Welcome to my digital garden. 🌱

Tim Le
InsuranceCase study

Issuing a 30,000-member group policy for Insurance Company C

The platform could not issue or run a group health policy above roughly 11,000 members. I helped find every limit it hit at 30k and rebuild issuance as a background job that can roll back, so a bank client's 30,000 employees could be insured in one policy.

Company
Insurance Company C
Role
Software engineer
Period
Oct – Dec 2026
Built with
.NET
MongoDB
Vue
Typescript
GraphQL
Kubernetes
Issuing a 30,000-member group policy for Insurance Company C
30,000
members in one policy, up from about 11,000
1,430×
faster member upload check (31 min → 1.3 s)
28×
faster policy document data (226 s → 8.2 s)
< 1 s
longest pricing freeze, was 13 to 16 s
16 MB
every record now fits MongoDB's document limit
0
partial state left when issuance fails

Context

Insurance Company C runs a multi-tenant platform for group health insurance: brokers quote, underwriters approve, admins issue policies, and employees use a member portal to claim. A bank client wanted to insure about 30,000 employees and their families under one policy, issued on a fixed date in December.

The platform had been built for policies of a few hundred to a few thousand members. Nobody knew exactly where it would break at 30k, only that a 20k quotation had already failed in production.

Challenge

Before writing any fix, we measured the deployed code at 30k members, from member upload to member movements (adding, moving and removing members after issuance). Twelve blockers came out of it. The most serious ones were all the same shape: a record that grows with the number of members, until it passes MongoDB's 16 MB document limit.

Records that grow with members, sized at 30k
Before (MB)After (MB)MongoDB's 16 MB document limit
Member movement's transaction record
42 MB
moved members only
Billing transaction record1.6× smaller
17.6 MB, indented
10.8 MB, compact
Pricing kept on the policy5.1× smaller
10.8 MB
2.1 MB
Billing record's copy of every insured
~12 MB
first insured only
Data-warehouse copy of the pricing
11.2 MB
turned off

Measured with the production product, or estimated from the deployed code. Anything past the dashed line can't be saved.

The rest were time, not size:

  • Issuance ran in one browser tab. Over 30,000 calls in a row, about two hours of login lookups alone, an estimated 1 to 3 hours in total. Closing the tab left a half-issued policy, and nothing could undo a finished step.
  • Pricing froze the scripts service. One 30k pricing call took 13 to 16 seconds of uninterrupted work, and a pod busy for about 30 seconds fails its health check and gets restarted, taking other tenants' requests with it.
  • Underwriting was quadratic. One 30k call took 133 seconds, and each manual Approve re-ran it over the whole policy.
  • The pricing input was built from whole member records, with fields twice: about 1 GB at 26k, over the script engine's string limit, so the policy silently stored no pricing.
  • Member lists and reports were built in the browser from 237 MB of pricing, and the PDF of a 10,000-member list lost its headers on the last hundred pages.

Approach

1. Move issuance into a job that can roll back

The core change was replacing the browser-driven flow with one background job per policy, inside the policy service: a state machine driven by queue messages, one message per batch, with a ledger in MongoDB that records every change it makes.

The job does every reversible change first. Issuing the policy is the single point of no return: before it, a failure rolls back cleanly from the ledger; after it, steps retry until they succeed. When a rollback finishes, the admin who started issuance gets an email.

Issuance, before and after
BeforeOne browser tab · 12 steps · 2 loops of one call per member · 1 to 3 hours
Create loginsone lookup per member
Create individualsmatched by array position
Price, re-pricewhole policy, several times
Issue policy
Billingerrors shown as success
Documents and emailbuilt in the browser
AfterOne background job per policy · batches · a ledger of every change
Steps 1 to 6logins in bulk, members, codes, pricing once: all reversible
Step 7: issuethe only commit
Steps 8 to 11billing, documents, email: retried until done

A failure injected at any step before issuing leaves nothing behind; after it, the job finishes the work instead of undoing it.

The same job pattern then took over pricing, documents and member-movement approvals, on the existing infrastructure. We compared a workflow engine (Temporal) and decided against it: in-service jobs with a ledger covered every case without a new system to run.

2. Make every record fit

  • Store member ids only in the issue event, instead of every member's full record.
  • Keep slim pricing on the policy: each member's id and amount, plus totals per benefit type.
  • Write billing records compact instead of indented, and keep only the moved members' rows in a member movement's record.
  • Change the data-warehouse exports to read per-benefit totals first, so those slimmer records could ship without breaking finance reports.

3. Price each member once, without freezing anything

  • Rebuild the pricing input the way the product service builds it, with no duplicated fields.
  • Make the pricing script pause about every 50 ms, so the pod keeps answering health checks. Identical output, 0 to 2% slower, and the longest freeze fell from 13 to 16 seconds to under one.
  • Price each distinct member once and copy the result to identical members; a 30k quotation then takes about 2 seconds.
  • Run long pricing in a background job that every screen reuses, instead of each screen waiting on its own long request.

4. Fix the slow paths screen by screen

We then loaded real 10k and 20k test policies and fixed each screen that couldn't cope, measuring before and after every change.

Measured on 10k to 30k-member test policies
BeforeAfter
Validate a 30,000-row member upload1,431× faster
31 min
1.3 s
Add a member (duplicate and spouse checks), 20k30× faster
~6 min
8 to 12 s
Policy document data, 10k members28× faster
226 s
8.2 s
Open the Payment step, 20k23× faster
~3 min
~8 s
Member list request, 20k16× faster
100 to 180 s
9 to 11 s
Render a 2,500-page member list PDF4.3× faster
719 s
169 s
Save underwriting on a 30k policy33× faster
~20 s
0.6 s

A few of the fixes behind those numbers:

  • Member upload: a linear duplicate check instead of a quadratic one, and an import that resumes after a pod restart instead of losing the job.
  • Document data: read each member's parent once instead of checking every member against every primary, and build the table of benefits only when a query asks for it.
  • Member lists: render long lists in parts, merge them into one PDF and stamp "Page N of M", so a 10,000-member report is one file instead of 17.
  • Underwriting: save only members whose outcome changed, brotli-compressed (1.06 MB → about 70 KB per 1,000 members), and count the whole census once instead of per member.
  • Screens: count members instead of loading all of them just to show a number.

5. Ship safely to a shared platform

Every change shipped behind a flag for the insurer's Asia tenants only, so other tenants on the same services were untouched. The work ran as 20 stages and 70 tickets across backend, frontend, scripts and data-warehouse lanes, with two QA rounds, a 30k load test and UAT before the production release.

Results

  • The platform can issue and run a 30,000-member policy, where it used to stop at about 11,000.
  • Issuance runs as a background job with rollback: closing the browser no longer leaves a half-issued policy, and a failure before the commit leaves nothing behind.
  • Every record that grew with members now fits under 16 MB, with room to spare (the compact billing record fits up to about 44k members).
  • Pricing no longer restarts pods, and the slowest screens went from minutes to seconds.
  • A 20,000-member test policy issued end to end in 33 minutes during QA, and its members, documents and billing all checked out.

What I learned

  • Measure before you design. Half the blockers were invisible until we sized real records with the production product; estimates alone would have fixed the wrong things first.
  • Find the point of no return. Ordering a long process so that everything reversible happens before one commit made rollback simple instead of impossible.
  • Anything per member needs a second look at scale. A query, a log line or a copy that costs nothing at 500 members costs hours or megabytes at 30,000.
  • Prefer boring infrastructure. A state machine and a ledger on the services we already ran beat adding a new workflow engine weeks before a fixed launch date.