Prysm Falters in Fusaka Upgrade: Bug Triggers Resource Meltdown but Finality Holds

After the Fusaka upgrade on Dec 4, a nasty bug popped up in the Prysm consensus client. It caused loads of resource failures right after things went live.

Basically, when Prysm beacon nodes were trying to deal with certain attestations – mainly old or out-of-sync ones – they kept redoing these intensive historical states. This hogged all the CPU and memory. A lot of Prysm nodes ended up in a denial-of-service state, which stopped them from doing their validator jobs right.

Because of this, Ethereum's validator participation fell from over 95% to around 74–77%. About 40–42 epochs and roughly 18–19% of slots were missed. Prysm validators lost about 382 ETH in missed rewards (that’s over a million bucks!). Even with these issues, Ethereum still finalized transactions okay, mainly because most validators were using other consensus clients that didn't have this bug.

To fix things, Prysm operators quickly got some config fixes (like turning off last-epoch targeting with runtime flags) to stop the crazy redoes of historical states. Prysm developers then put out patched versions (like v7.0.1 and the upcoming v7.1.x). These versions changed how historical states and old attestations are handled, which got rid of the resource issue we saw during Fusaka. If you're running v7.0.0, it's a good idea to update to the latest fixed version. Also, use the suggested flags if you're stuck on older versions, keep an eye on how your system is running, and think about using other consensus clients like Lighthouse, Teku, or Nimbus to mix things up.