• Posted by Intent Media 18 Nov
  • 0 Comments

The Purginator: Dynamically throttled writes to DynamoDB

https://github.com/intentmedia/purginator

The purginator was designed to enable dynamically throttled write operations against a local or remote DynamoDB database. It also has the ability to generate test data in a local database to validate the correctness of a planned operation.

Our initial use case was to purge several hundred thousand records from a database that was in production use, without exceeding the provisioned write capacity. DynamoDB can be an incredible tool, and we have found some great uses for it. Ensuring that your DynamoDB usage doesn’t exceed your provisioned capacity can be a challenge, especially when multiple applications share the same database. In this case, the operation to purge the records needed to not impact normal application activity, since we didn’t want our production application’s database access to be throttled or interrupted.

The purginator was developed in Clojure, using open source libraries including Faraday and Commons Math. Today, we’re open sourcing it, and we welcome comments, questions, bug reports, and pull requests.

John Chapin is a Data Engineer at Intent Media, where he is building the next generation of predictive analysis tools alongside a world-class team. When he’s not hacking on a functional programming project, he can be found running along the Hudson River or planning his next trip abroad. John has a bachelor’s degree in computer science from James Madison University.

Post Comments 0