Emergency maintenance: CPU relief and session recovery
A planned window on 1 September where we brought runaway Chromium CPU usage under control, stopped the RemoteAuth session cache from filling the heap, and fixed the sessions that were dropping back to the QR screen every half hour. No queued campaign was lost.
Highlights
- Sustained CPU on the sending nodes came down from 98 percent to about 31 percent
- The RemoteAuth session cache no longer holds the whole session archive in memory
- Numbers that were logging out every 30 to 60 minutes now stay linked
- Campaigns paused for the window resumed from the exact message they stopped on
- Delivery reports no longer mark a failed send as delivered
In this release5
- PerformanceSystemHigh impact
Chromium renderers no longer pin the CPU during send bursts
Every linked number runs its own Chromium page through whatsapp-web.js. After the library moved to the 1.31 line, each renderer kept a busy loop running even while the session was idle, which is the same behaviour reported upstream in issue #3619. With 40 or more sessions on one node the load average never dropped below the core count, and campaign workers started timing out waiting for a page that was too busy to answer.
What we changed:
- Sessions share a pooled browser instead of launching one Chromium per linked number.
- We start Chromium with a reduced flag set that turns off the renderer backgrounding timers, the GPU compositor and the Blink features we never use, which is what was driving the loop.
- Each page runs under a renderer process limit, so one noisy session cannot take the whole box with it.
- Idle sessions are parked after 10 minutes and rehydrated on the next send. That alone removed about two thirds of the live renderers on a normal day.