AI can help you get a service working quickly. Then an instance restarts while a request is still in flight. The caller retries, but the first attempt may already have written to the database. What happens when the request comes in again?
I’d want an answer before asking AI to build the service. It affects what state we save, which changes belong in the same transaction, and what an operator needs to see when something fails. Give AI the requirements that apply to your component, then use them to review the code and test what happens under load and during failures.
1. How much load do we need to handle?
Give AI the request rate on a normal day and during a burst, how much work can be in progress at once, and what latency and error rates you can live with. If the component accepts work to finish later, specify how fast it arrives and how soon it needs to be done. Those figures help you set concurrency and queue limits and decide how much capacity to test.
2. What infrastructure do we have available?
Tell AI how much CPU, memory, disk space, and network capacity the component can use. Include the connection budget for the database, how many replicas you can run, and which dependencies other services use too. A design that needs a new database or messaging platform may be hard to operate with what you have. Leave some capacity for production traffic and failures.
3. How much data must we process, and in how much time?
Start with how many items there are, how big they are, and when the last one needs to finish. To process 100 million records in 24 hours, you need an average of about 1,158 successful completions per second. Allow more capacity for retries, downtime, and items that take longer than others. Test the whole path against the deadline. A small benchmark tells you little about what happens when several replicas compete for the same resources.
4. What happens when the component restarts?
Look at what lives only in memory while an operation is in progress. Save whatever another instance would need to pick up unfinished work. Decide which operations can resume, which should run again, and which need to be reported as failed. If you run an operation again, check what happens when part of it already took effect.
5. What happens when an operation fails?
Decide what the caller sees and what happens to the work. The component might retry, mark the operation as failed, or keep it for someone to investigate. Record the failure and make it clear who owns the next step. If automatic recovery runs out of options, an operator should be able to find the failed operation and decide what to do with it.
6. Why could an operation fail?
A dropped connection may be worth retrying. Invalid input usually needs to be fixed first, and a missing record might be an expected result. Decide how to handle timeouts, HTTP errors, unexpected responses, bad configuration, and database failures. Pay particular attention to a timeout after an operation may already have taken effect. The caller may have missed the acknowledgment even though the operation succeeded.
7. How does the component affect the database?
Count the reads and writes needed for a typical operation, including coordination, status updates, and cleanup where they apply. Check how many connections it holds, how long transactions stay open, which locks it takes, and how much it writes. Set limits for when queries slow down or traffic picks up.
8. What happens when we scale?
Before you add replicas, decide which instance owns each operation, what state the instances share, and which operations can safely run at the same time. Multiply each replica’s limits by the number you can run. A connection limit of 50 per replica becomes 500 across ten replicas. Check that the database and other dependencies can handle the combined load, and that another instance can recover unfinished work when a replica goes away.
9. What happens when processing finishes?
Decide what counts as done and what the component should do next. For a service with a queue, an empty queue can still mean operations are in progress, retries are scheduled for later, or more records are on their way. If the service runs continuously, decide how it waits for new work.
10. What happens during a graceful shutdown?
Stop taking new work and decide how long to give operations that are already running. Close resources after the drain, and plan how to handle operations that need more time. Make sure the shutdown sequence fits within the time the platform gives the application before it terminates the process.
11. What happens when shutdown cannot be graceful?
A crash or host failure can stop the process before cleanup runs. Work out what has to be saved for recovery and which changes might already have taken effect. Try stopping an instance while an operation is in progress, especially between a call to another service and the local update that records its result.
12. What if an operation never finishes?
Give operations deadlines and use timeouts to stop waiting on individual calls. Decide how to find abandoned operations and, as far as the system allows, tell a slow operation from a failed instance. If another instance takes over, check what happens if the original instance resumes and tries to finish the same operation.
13. Should behavior change during peak hours?
When several services share capacity, traffic spikes in one can slow down the others. You might defer work that can wait until after peak hours or adjust concurrency and rate limits based on the load you see. If you use a schedule, specify the timezone. Give operators a way to slow or pause work when a dependency is struggling.
14. What happens when there is no work, or checking for work fails?
If the component polls for new work, decide how long to wait after an empty response or an error. Polling too often adds load while the system is idle. Waiting too long delays new work. Make sure the component can try again after a failed poll. If it relies on events, decide how it reconnects and discovers what it missed.
15. How do we investigate one failed operation?
Give each operation an ID you can follow through logs, traces, and stored state, including calls to other services. Give each retry attempt its own identity too. Record enough to explain what happened without exposing sensitive details the operator doesn’t need. An operator should be able to find failed operations without searching the logs on every instance.
16. How will the results be analyzed?
Find out what users or operators need to filter, compare, or report on before you settle on how to store the results. Add indexes for the queries you expect to run, then check the query plans with realistic data. Include the space those indexes take and the work of updating them in your estimates.
17. How much storage will the component require?
Estimate the space for inputs, pending work, results, indexes, and operational history. Decide how long to keep each kind of data and automate cleanup once it’s no longer needed for recovery, investigation, or analysis. Budget for the cleanup itself. Deleting rows may leave the space available for reuse without immediately returning it to the filesystem.
18. How can we minimize database round trips?
Look for queries that run once per record and related updates that use separate calls. Set-based SQL, bounded batches, or a database function can cut round trips. Use a transaction where related changes need to succeed or fail together, and avoid keeping it open while waiting for another service. Check what the change does to latency, lock times, and database load.
Make the answers testable
When I give AI a requirement, I include how I’d check that the code meets it. For shutdown, I’d write something like this.
When shutdown starts, stop taking new requests and give the ones already running a fixed amount of time to finish. In the test, stop the service just after it saves a change, before it sends the response. Send the same request to another instance and check that the change was applied only once.
I read the implementation and the tests before running them with realistic data. In that recovery test, I’d look for two things in particular. Does it stop the service at the right moment? Does it check what ended up in the database?
A restart is one case. I also want to see what finishes when a dependency slows down or more requests arrive at once, and how much extra load reaches other services. When cleanup runs, I check what it deletes and whether the data we need for recovery is still there.
Where I start
I start with the failure that would be hardest to undo. If a release changes what we write or when a caller can safely retry, I want those rules spelled out and tested before it goes live. For a different release, the first question might be how much traffic it can handle or what an operator can do when a dependency goes down.
Working through a similar challenge?
Tell me what you’re up against. I can help.