Yesterday, a Stack Overflow moderator election started, but I didn't get a notification in my inbox even though I have enough reputation to vote. I did get two other election messages at that same moment.
They're not in my global inbox either:
It was working two weeks ago when two other elections started, see this screenshot.
Ok, so the election notification system is a little old and wasn't idempotent. It was designed to just kick off a single background job that queued up all election inbox notifications for eligible voters. This would get queued at the moment in time we transitioned an election from one state to another. The problem is, if the background job was cancelled for any reason, nothing would or could automatically resume processing. And one thing that killed the background job is just deploying the application, which restarts all the pods. So we didn't end up sending out election notifications for most users on Stack Overflow for the current election.
We created a one time backfill route to check who was eligible and compare that to who had already gotten a notification. And then use the same background job to queue up the people who had not yet received an inbox notification. Shipped it to prod and sent out notifications - yay!
In so doing though, we accidentally queued the same job twice. Most people had not yet gotten a notification, so both background jobs started queuing notifications to the same group of people at the same time. Because the job kicks off in the background, it's quite pesky to try to turn it off. By the time we had noticed the issue, we weren't able to restart any pods manually (what caused the suspension in the first place).
Some users may see duplicate notifications like this
Tomorrow morning, we'll try to go through and see if we can remove duplicate inbox notifications for people. But by then some folks will have already logged in.
UPDATE to #2 - After discussing internally, we'd like to leave the duplicate notifications issue as is. It will impact ~500k users, of which ~5k have already seen it. There's no pre-existing route to remove that many notifications, so we'd have to ship a new one-time use script / job and possibly even risk me showing up here again for a Problem #3. The impact to users should be fairly low, and inbox notifications should get pushed out of view over time eventually.
If there are folks who feel strongly otherwise and think it's worth addressing, feel free to comment in support of the cleanup, but we have lots of other stuff to help deliver for the site and I'd love to focus more of our engineering time on that.
Hopefully, two notifications is better than none.
Sorry to folks for the extra inbox pings, and please go vote!
