v7.5.2Stable
Scaling out: separate session and worker tiers
Follow up work from the 1 September window. The single application node is now a session tier and a worker tier that scale on their own, so a heavy campaign on one account no longer slows down every other account on the platform.
Highlights
- Session hosting and campaign workers now run on tiers that scale independently
- One account flooding the queue can no longer starve another account
- A large image campaign to 700 recipients went from 41 minutes to 9
- A session stuck below the ready state is detected and recycled without a manual restart
- Media processing moved off the request path
In this release6
- PerformanceSystemHigh impact
Session hosting and campaign workers split into separate tiers
Everything used to run on one node: the API, the Chromium sessions and the queue workers. A campaign burst competed for CPU with the very browsers that had to send it, which is the loop behind the problem on 1 September.
What we changed:
- Chromium sessions now live on a session tier that is sized by the number of linked accounts.
- Campaign, message and webhook workers live on a worker tier that autoscales on queue depth and CPU.
- The two tiers talk over Redis, so a worker can be added or removed without touching a live session.
- Each tier has its own health probe, so a bad worker is replaced without disturbing anyone's linked number.
- A deploy no longer restarts your session, because the API and the workers ship independently of the session tier.
2400 ms610 ms75% lowerp95 time from a job being picked up to the message leaving the node, at peak
app 7.5.2api 7.5.2sdk 1.4.2