- Digital Engineering
Designing Systems That Scale: A Practical Field Guide
Designing Systems That Scale: A Practical Field Guide
Key takeaways
- If you cannot measure a requirement, you cannot design for it. "The site should be fast" is a wish, "95% of quote requests return in under 400 ms" is a requirement.
- Averages hide pain. A page that makes 20 API calls feels slow on roughly 1 in 5 loads when just 1% of calls are slow.
- Each extra nine of availability changes how you build, test and deploy, not just how much hardware you buy.
- Replication solves read load and redundancy. Sharding solves write and storage limits. Reach for replication first.
- Start simpler than you think. A clean layered application splits easily later, a premature split into twenty services does not merge back.
- Keep the customer-facing path short, and put every slow or fragile dependency behind a queue, a cache or a retry.
Most systems do not fail because of bad code. They fail because nobody decided early on how much load they must carry, how fast they must answer, or what happens when a server dies at 2 a.m.
That is what system design is about: making those decisions on purpose, before traffic makes them for you. This guide walks through the ideas we lean on most when we design large-scale platforms, from gathering requirements to picking a database to drawing the final architecture. It ends with a worked example from a domain we know well: buying an insurance policy online.
A story every Indian internet user knows
It is 9:59 a.m. You have the Tatkal booking page open, passenger details ready, UPI app unlocked. At 10:00 the button goes live, and so do lakhs of other people. Pages spin. Some users see an error and retry, which doubles the load. A few lucky ones get through, most get a timeout and then find the train sold out.
Nothing in that story is a coding bug. The site does exactly what it was programmed to do. What breaks is the design's assumptions: how many people arrive in the same second, what happens to a request while it waits, and whether a retry makes things better or worse.
Every large platform has its own 10 a.m. moment. For an e-commerce site it is a flash sale. For a streaming app it is the last over of a cricket final. For a life insurer it is the last week of March, when salaried customers rush to buy tax-saving policies before the financial year ends. Good system design is how you prepare for that moment before it arrives.
Start with drivers, not diagrams
Before anyone draws a box, three kinds of input should be on the table. Together they shape the architecture far more than any framework choice.
If you cannot measure it, you cannot design for it. "The site should be fast" is a wish. "95% of quote requests return in under 400 ms" is a requirement.
Real-world scenario: one calculator, two architectures
Imagine you are asked to build a premium calculator. The feature list is short: enter age, sum assured and term, get a premium. Now look at two versions of the same request.
| Driver | Version A: internal tool | Version B: public website |
|---|---|---|
| Users | A few hundred sales agents | Anyone on the internet |
| Peak traffic | Steady, office hours | Sudden spikes from ad campaigns and the March tax rush |
| Expected speed | A second or two is fine | Users leave if it feels slow, especially on mobile |
| Failure impact | Agents wait and retry | Lost sales, social-media complaints |
| Security | Behind the company VPN | Open to bots, scrapers and abuse |
| Sensible design | One app server and the main database | Stateless servers that scale out, a cache for rate tables, a CDN, rate limiting at the gateway |
The features are identical, yet Version A can ship in a week on a single server, while Version B needs a small distributed system. That difference comes entirely from the quality attributes and constraints. This is why experienced architects ask "how many, how fast, how often, and what if it breaks?" before they ask "which framework?".
The quality attributes that matter most at scale
You cannot maximise every quality at once, improving one usually costs another. These three come up in almost every large-system conversation.
Performance: watch the slow tail, not the average
Latency is the time a user waits for an answer, and it includes both the work done and the time a request sits in a queue. Throughput is how much work the system finishes per second.
Averages hide pain. If your average response is 120 ms but 1 request in 100 takes 3 seconds, a user who loads a page with 20 API calls will hit that slow path often. Track percentiles instead: p50 for the typical case, p95 and p99 for the tail. Also load-test until throughput stops rising, that knee in the curve is your real capacity, and response times climb sharply right after it.
The maths behind the slow tail
Here is why architects obsess over p99. Suppose only 1% of your API calls are slow. That sounds harmless, until you count how many calls one screen makes. A modern dashboard or product page often fires dozens of requests: profile, cart, offers, recommendations, notifications.
computed as 1 - 0.99^n, assuming 1% of calls are slow and calls are independent
A page is only as fast as its slowest call. So at 20 calls, nearly 1 in 5 page loads feels slow, and at 100 calls it is most of them. Your p99 for individual calls quietly becomes your users' typical experience.
Teams fight this in a few practical ways:
- Make fewer calls. An aggregation endpoint (often called a backend-for-frontend) returns everything one screen needs in a single response.
- Call in parallel, not in sequence. Ten parallel calls cost roughly the slowest one, ten sequential calls cost all of them added together.
- Set timeouts and fallbacks. If recommendations do not answer in 200 ms, render the page without them rather than making the user wait.
- Hunt the tail itself. Slow outliers usually come from a few causes: garbage-collection pauses, cold caches, lock contention, or one overloaded database shard.
Scalability: plan for growth in three directions
Availability: decide how many nines you can afford
Availability is the share of time users can actually use the service. 99.9% allows about 8.8 hours of downtime a year, 99.99% allows about 53 minutes. Each extra nine costs real money in redundancy and operations.
You get there with fault tolerance: run redundant copies of everything important, detect failures quickly with health checks, and recover automatically by pulling bad instances out of rotation or rolling back a bad deploy. Remember that people and deployments cause more outages than hardware does.
What each extra nine really buys
| Availability | Downtime per year | Downtime per month | What it usually takes |
|---|---|---|---|
| 99% | about 3.65 days | about 7.3 hours | A single server with backups |
| 99.9% | about 8.8 hours | about 44 minutes | Two or more instances behind a load balancer, a replicated database |
| 99.95% | about 4.4 hours | about 22 minutes | All of the above, plus automated failover and safe deploys |
| 99.99% | about 53 minutes | about 4.4 minutes | Multiple availability zones, no manual steps in recovery |
| 99.999% | about 5.3 minutes | about 26 seconds | Multiple regions, and a team that lives and breathes reliability |
Notice the last column. Each step is not a small upgrade, it changes how you build, test and deploy. Most business applications are well served by 99.9% to 99.95%. Promise 99.99% only when the business genuinely loses serious money every minute you are down.
The Friday evening deploy
A team pushes a new release at 6 p.m. on a Friday. All servers are updated at once. A configuration value is wrong, every instance starts failing, and the on-call engineer spends 40 minutes figuring out how to roll back.
The hardware was fine the whole time. The outage came from the release process. Three habits would have shrunk it to a blip:
- Roll out gradually. Update one instance or 5% of traffic first, watch error rates, then continue. This is often called a canary release.
- Make rollback one click. Keep the previous version ready to receive traffic, so recovery takes seconds, not a debugging session.
- Let health checks act. If an instance fails its checks, the load balancer should stop sending it traffic automatically, before a human even notices.
This is why mean time to recovery matters as much as mean time between failures. You cannot prevent every failure, but you can make each one short.
Turning goals into promises: SLA, SLO, SLI
Pick a few SLOs that users actually feel, and treat the gap below 100% as an error budget you are allowed to spend on releases and experiments.
The building blocks you will reach for again and again
Large systems are mostly assembled from a small set of well-understood parts. Knowing what each one is for, and where it hurts, is half of system design.
| Building block | What it does for you | Watch out for |
|---|---|---|
| Load balancer | Spreads requests across many servers and stops sending traffic to unhealthy ones | Sticky sessions that quietly break horizontal scaling |
| Global load balancing (GSLB) | Routes each user to the nearest healthy region, and fails over when a region goes down | DNS caching delays how fast failover takes effect |
| API gateway | One front door for clients: auth, rate limiting, routing, protocol translation | Business logic creeping in, it must stay thin and highly available |
| Message broker | Lets services hand off work asynchronously, so a slow or failed consumer does not block the sender | Duplicate and out-of-order messages, consumers must be idempotent |
| CDN | Serves static assets from servers close to the user | Stale content after a release, plan cache-busting and invalidation |
| Cache | Keeps hot data in memory (for example, Redis) to cut database load and latency | Invalidation and stampedes when a popular key expires |
For every box you add, ask what happens when it is slow, and what happens when it is down. If the answer is "the whole site stops", you need a second copy or a fallback.
Follow one request through the system
It helps to see these parts as a relay race. Here is what happens between a user tapping "Get quote" and the number appearing on screen.
the journey of one API request, 7 layers
The dashed line is the important trick: the service hands slow follow-up work to the broker and answers the user immediately. Every layer that can answer early, such as the CDN or the cache, protects every layer behind it.
Surviving a flash sale
An e-commerce site announces a big sale at 12:00 noon. Ten minutes before, traffic is normal. At 12:00:01 it is many times higher, and most of it wants the same 50 products. Here is how each building block earns its place that day.
- The CDN takes the first hit. Product pages, images and scripts are already cached at the edge, so the origin servers never see most page loads.
- The gateway keeps bots out. Rate limits per user and per IP stop scripts from hammering the "Buy" button a hundred times a second.
- A waiting room smooths the spike. Instead of letting everyone in and crashing, the site admits users in batches. A queue with an honest "you are number 4,312" beats an error page.
- Caches are warmed in advance. If the 50 sale products are loaded into Redis before noon, the database is not flooded the moment the cache is empty. When a popular key expires and hundreds of requests rebuild it at once, that is called a cache stampede, and it can take a database down.
- Stock is reserved atomically. Decrementing inventory must be a single atomic operation, or two buyers will get the last unit.
- Orders go through the broker. The order is saved and confirmed in milliseconds, while invoices, emails and warehouse updates are processed from the queue at a steady pace.
- Payments are idempotent. Users double-tap "Pay" when the screen is slow. An idempotency key on each payment attempt makes sure the second tap does nothing.
None of these ideas is exotic. What makes the difference is deciding them a week before the sale, not during it.
Choosing where your data lives
The database decision is the hardest one to reverse, so make it per use case rather than once for the whole system.
Relational or not?
Relational databases such as MySQL or PostgreSQL give you a fixed schema, rich queries and ACID transactions. Reach for them when correctness across several records matters: payments, policies, orders, anything an auditor will read.
Non-relational stores trade some of that for flexibility and scale:
Files, images and PDFs belong in object storage (S3 or similar), not in a database row. Use cheaper storage tiers for data that is rarely read.
Three ways to make any database go further
- Index the columns you filter and sort on. It is the cheapest performance win there is, at a small cost to writes.
- Replicate to get redundancy and to spread read traffic across copies.
- Shard (partition) when one machine can no longer hold the data or absorb the writes. Choose the shard key carefully, a bad one creates hot spots and painful cross-shard queries.
The CAP trade-off, in plain words
When the network between replicas breaks, which it eventually will, a distributed database must choose. It can refuse some requests to avoid serving stale data (consistency), or keep answering with possibly stale data (availability).
Neither is "right". A policy's payment status should favour consistency. A post's like count can happily favour availability. Decide per data type, and write the choice down.
Replication and sharding, side by side
These two techniques are often confused, but they solve different problems. Most large systems end up using both: each shard is itself replicated.
replication vs sharding, side by side
Reach for replication first when reads are the bottleneck or you need a standby. Reach for sharding only when a single primary can no longer keep up with writes or hold the data, because it adds real complexity.
The stale address
A customer updates her address and taps Save. The write goes to the primary. The next screen reads from a replica, which is a fraction of a second behind, and shows the old address. She assumes the update failed and does it again, then calls support.
This is replication lag, and it is normal. The fix is a pattern called read-your-own-writes: for a short time after a user changes something, send that user's reads to the primary, or return the saved value straight from the write response. Everyone else can keep reading from replicas.
The festival hot shard
An online store shards its orders table by city, which seems natural. Then the festive sale starts. Two metro cities generate most of the orders, so two shards run hot while the rest sit idle. Adding more shards does not help, because the load is not spread by city at all.
The team re-shards by a hash of the customer ID, which spreads orders evenly. The trade-off: a report like "all orders in one city" now has to ask every shard. They solve that by streaming orders into a separate analytics store built for such queries. The lesson is that the shard key should follow how the load is spread, not how the business thinks about the data.
Picking an architecture pattern
There is no best pattern, only the best fit for your team size, traffic and rate of change.
| Pattern | Fits best when | Main cost |
|---|---|---|
| Layered (multi-tier) | One team, a clear domain, moderate scale | The logic tier grows into a monolith that is slow to change |
| Microservices | Several teams need to ship independently, and parts scale differently | Network calls, distributed debugging and data consistency get harder |
| Event-driven | Many parts react to the same business fact, or you need an audit trail | Eventual consistency and harder end-to-end tracing |
Start simpler than you think. A well-structured layered application with clean module boundaries is easy to split later. A premature split into 20 services is hard to merge back.
If you do go to microservices, give each service one clear responsibility and its own data store. Two services sharing a table are really one service with extra latency.
Events decouple teams. When the policy service publishes "policy issued", the email, analytics and commission services can each react without the policy team knowing they exist. Two related ideas are worth knowing:
How real platforms usually evolve
Few successful systems were designed as microservices on day one. Most follow a path like this, moving to the next stage only when the current one starts to hurt.
a typical evolution path, 3 stages
Splitting a monolith without a big-bang rewrite
A portal started as one application with one database. Three years later, 25 developers work in the same codebase. A small change to the premium calculator waits two weeks for a release train, because the claims and policy modules ship in the same package.
The team does not rewrite everything. Instead, they apply a gradual approach often called the strangler pattern:
- Pick the part that changes most and scales differently. Here, the quote engine: it changes every few weeks and takes most of the traffic.
- Put a gateway in front of the monolith so traffic can be routed path by path without clients noticing.
- Build the new quote service beside the old code, with its own data store, and route a small share of quote traffic to it.
- Compare results, then move all quote traffic and delete the old module.
- Repeat for the next painful area, and stop when the remaining monolith is no longer a problem.
After the split, the quote team ships whenever it wants. The trade-off is new operational work: distributed tracing, service-to-service authentication and contract testing. That cost is worth paying only when release pain is real.
Worked example: an online insurance purchase platform
Let us apply all of this to a journey we know well: a customer gets a quote, fills in a proposal, pays, and receives a policy. The goal is to keep that path short and safe, and let everything else happen in the background.
The drivers
The architecture
online insurance purchase platform, target architecture
The customer only ever waits on four focused services behind one gateway, notifications, analytics and the legacy core system all catch up from events.
Why each choice was made
- Stateless, cached quote service. Quote traffic is read-heavy and bursty, so it scales out freely and keeps rating data in Redis.
- Idempotent payments. Payment gateways retry callbacks, so every payment carries an idempotency key and a second callback changes nothing.
- Relational storage where money and policies live. Payments and policies use ACID transactions and favour consistency over availability.
- Documents in object storage. KYC files go to S3-style storage with only references in the database, which keeps the database small and fast.
- The broker shields the customer from the legacy system. If the core system is slow or down, the event waits in the queue and is retried, while the customer already has a confirmation.
- A CDN in front of static assets. Pages and images load from servers close to the user, which takes load off the application entirely.
The purchase, step by step
Architecture diagrams show what exists. Sequence diagrams show what happens, in order. Here is the moment that matters most: the customer pays and gets a policy.
policy purchase, sequence of calls
The customer waits only for the steps on the left. The moment the policy service records the policy, it publishes an event and answers. Emails, analytics and the slow legacy sync all run afterwards from the broker.
When things go wrong
A design is only as good as its behaviour on a bad day. Walking through failures like these is the most valuable part of any design review.
| What fails | What the customer sees | What the system does |
|---|---|---|
| The payment provider charges the card but its callback never arrives | "Payment is being confirmed" for a few minutes | A reconciliation job asks the provider for the final status and completes the policy, the idempotency key prevents a second charge if the customer retries |
| The provider sends the same success callback twice | Nothing unusual | The payment service sees the same key, recognises a duplicate and ignores it |
| The legacy core system is down for two hours | Nothing, they already have a policy number | Events wait in the broker and are retried with growing delays, after too many failures they move to a dead-letter queue and alert the team |
| Quote traffic jumps during the March tax rush | Quotes stay fast | Quote instances scale out automatically, rate tables are served from cache, the gateway throttles abusive clients |
| A whole data-centre zone goes offline | A brief blip, possibly one retry | The load balancer stops routing to the failed zone, databases fail over to replicas in a healthy zone |
| A bug in the email service | Confirmation email arrives late | Events pile up safely in the queue, after the fix the service catches up and sends every email |
The customer-facing path is kept short and protected, and every slow or fragile dependency sits behind a queue, a cache or a retry. That one idea explains most of the architecture.
A checklist for your next design review
Good system design is less about clever diagrams and more about asking the right questions early. Before you sign off on a design, check that you can answer these:
- Is every key requirement written as a number we can test?
- Which SLOs will we track, and what is the error budget?
- What happens when each component is slow, and when it is down?
- Which data needs strong consistency, and which can be eventually consistent?
- Can each service scale out without sticky state?
- Are message consumers idempotent, and do retries have limits?
- Can we deploy, roll back and observe each part on its own?
If the team can answer all seven with confidence, the architecture is probably in good shape. If not, those gaps are where the next outage is hiding.
Frequently asked questions
Planning a platform that has to hold up at 10 a.m.
Auriga IT designs and builds large-scale platforms across ERP, digital products, cloud and data. If you are sizing a new system or reworking one that is starting to hurt, we can review the drivers with your team before a single box is drawn.
Talk to our architecture teamRelated content
Auriga: Leveling Up for Enterprise Growth!
Auriga’s journey began in 2010 crafting products for India’s [...]
