Calling it good, we've had no feedback of anyone having issues for the last 4 hours.
Posted May 11, 2022 - 18:50 UTC
Monitoring
We think this is fixed, and is now in monitoring.
The DDoS detection was triggered which can happen occasionally because CDN activity is still going up overall. It looks like traffic bounces over a threshold for stricter rules now. The way DDoS detection works in Cloudflare is through a series of rules/heuristics which can be individually tuned ( which is what I looked at in https://github.com/CocoaPods/CocoaPods/issues/11355#issuecomment-1123465704 .)
What we were seeing was still similar CDN traffic patterns but some traffic occasionally getting through correctly. This was when we reached out to Cloudflare support. With their help we determined that it was likely that the migration of our rules from the central settings repo to the different CDN edges (e.g. for simplicity think the servers closest to users each time) were using the cached (older) settings.
This meant some regions didn't have the new rules saying ignore the rule about 'allow the custom user-agent' - the Cloudflare support folks clears up the cache and now its looking like everything is working fine.
I'm going to maybe give till the weekend till I go and turn back on some of the DDoS settings (but not the main culprit) - giving me now time to go do some [baking with my wife](https://mastodon.social/@orta/108243829603470408) on our day off :P
Posted May 11, 2022 - 15:18 UTC
Update
We've been in conversation with Cloudflare, still figuring it out, no hard ETA yet but things are happening
Posted May 11, 2022 - 14:35 UTC
Identified
Our DNS provider and front-end to the CDN Cloudflare has triggered a DDoS detection with normal CocoaPods traffic to the CDN. We have a raised support request with them and are figuring out how to bypass.