From 7 minutes to 7 seconds: speeding up a 1,000-item bulk import
A new API let clients submit up to 1,000 items (diamonds, jewelry, or gemstones) in a single request, and it was taking about 7 minutes to respond long enough to freeze the page. I switched it to insert records in batches, moved the per-record callback work to a background job (Sidekiq), and tightened the database queries (selecting only the fields it needed and adding indexes). That brought the response time down from ~7 minutes to ~7 seconds.
Situation
A new API let clients send up to 1,000 items (diamonds, jewelry, or gemstones) in one POST request. For each item it created a database record, and every record triggered extra callback work. On top of that, the query was pulling every column (SELECT *) instead of just the fields it needed. Together this made the API take about 7 minutes to respond, long enough to freeze the frontend.
Task
Make the API fast enough to handle 1,000 items in one request without freezing the page.
Action
I made four main changes: 1. inserted the records in batches instead of saving them one at a time 2. moved the per-record callback work into a background job using Sidekiq, so it no longer blocked the response 3. rewrote the query to select only the fields it actually needed instead of every column; and 4. added database indexes to speed up the lookups.
Result
The API's response time dropped from about 7 minutes to about 7 seconds ( roughly 60x faster ) and the frontend no longer froze when submitting large batches.