What just happened on archive.org today, as best we know:

Tens of thousands of requests per second for our public domain OCR files were launched from 64 virtual hosts on Amazon’s AWS services. (Even by web standards,10’s thousands of requests per second is a lot.)

This activity brought archive.org down for all users for about an hour.

We are thankful to our engineers who could scramble on a Sunday afternoon on a holiday weekend to work on this.

We got the service back up by blocking those IP addresses.

But, another 64 addresses started the same type of activity a couple of hours later.  

We figured out how to block this new set, but again, with about an hour of outage.

—- 

How this could have gone better for us:

Those wanting to use our materials in bulk should start slowly, and ramp up. 

Also, if you are starting a large project please contact us at [email protected], we are here to help.

If you find yourself blocked, please don’t just start again, reach out.

Again, please use the Internet Archive, but don’t bring us down in the process. Posted in News | 42 Replies