Introducing a new proactive anti-spam measure

Accuracy data

The exact action will depend on how confident we are that the content is spam. Possibilities include:

• [...]
• Very High Confidence: Cast a binding spam flag resulting in deletion.

Our testing thus far has shown that very high similarity scores are a reliable indicator of spam, giving us confidence in taking decisive, automated actions.

Could you show us the actual data?

For a binding flag, I'd really like to know the data (e.g., the historical TP/FP rate), as a binding flag is much more significant than just a single non-binding spam flag.

Logging/transparency

Can the flags cast (binding/not-binding) be logged somewhere that humans can review? A chatroom, similar to CHQ,for example.

SmokeDetector has human review, and it pings the users who had their accounts cast autoflags if an FP is gotten on an autoflagged post.

Overlap with SmokeDetector autoflagging leading to unintentional immediate post deletion

There's a potential for conflict if this system and SmokeDetector both attempt to cast non-binding flags on the same post, but combined, the combination of the two deletes the post immediately. There are various reasons there's limits on the flags we cast, and implementing this as a separate system (instead of as a new SmokeDetector rule that integrates with SE's system) means that those limits are bypassed.

While SE can, of course, decide to ignore this, I'd suggest that someone carefully consider this possibility and its implications.

The site I moderate is the worst affected (in aggregate, but we don't get hit by the big waves) - and a good part of our spam mitigation strategy involves destruction of spammy accounts. We have good informal tooling, and I'm trying to best adjust our workflows to this new tooling.

One of the more useful tools in identifying and destroying current spammers is seeing which users got flagged and going through users with more than 3 (I'd assume y'all would know where it is, under) flags, and community reporting.

So - is there somewhere I can check if a post has a binding flag, is destruction still a good way to stop a spammer, and would accounts that have been flagged as such have greater weight in other spam prevention measures?

It's good that there's investment in tools that actually help communities, and that it's a network-wide investment rather than something focused on Stack Overflow.

However, I do have questions.

First, why only take advantage of recent posts deleted as spam? This is a text classification problem that could be done with supervised classification, and the network has a wealth of data - over 17 years of data on Stack Overflow alone. There are also continuous learning techniques to continue to retrain the models. There may be a data quality issue, but as a moderator on three sites, I'd be willing to volunteer some of my time to verify posts deleted as spam on my sites, and I suspect others would be willing to contribute time as well. Limiting to very recent spam feels like it would limit the usefulness of this tool.

From a technical perspective, are there any plans to write technical blog posts about this implementation? Since the platform is on Google Cloud now, I'd be interested to know if any of Google's AI/ML offerings are being used and how they are being used to solve this problem at Stack Exchange network scale.

For high-confidence posts, those would result in moderator review (unless there were enough other non-binding spam flags to handle the post immediately). For binding spam flags, there's probably something in a moderation dashboard somewhere to review them, but I'm not sure where it would be. Are there any plans to make automated actions more visible to moderators, if not high rep users, to be able to verify them? I suspect that a user may be able to cast a flag (or, if they have enough reputation, ask on meta), but this seems to be an unreasonable expectation.

Does the new tool have some name?

It is a bit awkward to talk about it as "the new tool" or "the new anti-spam measure" and other verbose descriptors, rather than a name. Compare it with "SpamRam".

I've previously voiced some concerns and offered suggestions about this project and - while some of them have been addressed (thanks!) - I think there are a few things that need more transparency for both posters and moderators.

For the posters, you say:

For legitimate users who believe their post was incorrectly blocked or deleted, the existing channels for appeal will be available.

A couple of things - you're not "blocking" any posts. Yes, that was considered but that seems to have been set aside (rightfully, in my mind). You're either deleting or flagging, the latter of which shouldn't lead to deletion for legitimate posts.

While it's true that users can go through the existing channels, it also requires that the user be aware of this issue. Flagging is easy to miss by many new users and it's unclear to me that users would consider flagging to be a way to get help in the case your post is deleted by the system. I would encourage y'all to create an OP-specific post notice that explains why the post was deleted and how the user can request review by a moderator. This could look something like:

Our system automations determined that this post resembles recent spam and was deleted. If this was in error, request that this be "looked at by a moderator" through the "flag" menu on the post and explain why it should be undeleted.

Additionally, are you putting any reputation-based safeguards in place? It seems uncommon for spam posts to come from users with more than 1 reputation, so I could imagine reducing false positives by downgrading a "very high confidence" deletion to only raise a single flag.

For moderators, a comment says:

Logging/transparency: As part of the release of this feature we're going to be pairing it with a dashboard that lists recently-deleted spam posts along with information about whether it was caught by this anti-spam measure, which we'll talk more about later. Initially we'll be limiting access of this dashboard to staff as we monitor its first efforts, but we'd like for moderators to have access to it as well.

I'm not sure I understand why mods shouldn't have access to this dashboard from the start (or why the dashboard wasn't mentioned in the question). I can understand limiting it to mods and then later expanding it to 10k users (and I specifically suggested this as a way to augment the 10k tools when discussed on the Mod Team) but I don't understand why you'd hamstring the mods in this way during the initial release.

Many tools pages have different actions based on user type - such as the blocklist page - so if you don't want to give mods actions you're considering building into the page, you can solve for that by special-casing the page by user type. Mods will need easier ways to review these deletions as they may otherwise go unnoticed by the mods. I don't think staff will be reviewing these lists across all 180 sites - please let the mods have access from the outset.

Hopefully, this tool will reduce spam wave duration and post volume significantly, which means it's more likely that mods and other engaged users will be unaware of what's going on, leading to reduced engagement in the oversight tools you create. One thing I think would be helpful is a way to ensure mods are aware that a spam wave occurred (or is occurring) on the site. This could be done as a new autoflag that is triggered when more than n posts are deleted in a period of time but you'd also want to allow it to scale to avoid constant flags that mods can't really do much about.

Let's say the system deletes 5 posts in the last few minutes, which would trigger the initial flag. Then to reduce the volume of flags, it could start counting and only raise additional flags hourly for each hour that met that minimum count of posts deleted. At the end of the first hour, maybe it reports "20 posts deleted as spam during this wave", and it continues every hour until the posts slow below the minimum trigger volume. If it was really fancy, once the wave has ended, it could raise a summary flag - "The spam wave has ended. Overall, 300 posts were deleted during this wave."

The flags can link to the dashboard so mods can review the posts if they wish and take any additional actions they deem necessary. I think it's reasonable to include all spam-deleted posts, not only ones deleted by the system.

I could also imagine this being implemented outside the flagging system, such as an RSS feed in designated chat rooms or something else like that but I'm not sure what the options might be. The issue with flags is that they can be dismissed by one mod, leaving the others in the dark - while that's not likely intentional, having something that's more informational and can't be dismissed for everyone by "handling" it would be better.

This sounds like a good idea to me. I note that you're looking to take some actions on edits, which in the past have been less protected. Great! I recommend you not cast an automatic spam flag against a post if the edit was done by a user who was not the post's author, including for Community Wiki posts. You can still log it somewhere, custom flag, etc.

Regarding Spam in SE Networks, we have some history with spam accounts on The Workplace. You may want to look into those efforts now that SE is taking more 'proactive' measures.

We have had Smoke Detector for a while now and it has been working well (kudos to the Charcoal team and @Makyen who is always there to support and help with Charcoal/Smoke stuff).

Some years ago we had a big spam user wave hitting us and I noticed a pattern: spam users were created, remained 'dormant' for a while, and eventually they activated and posted Spam, along with a Spamm-y user description and links (basically promoting a product/service).

At first I took upon the work of checking New Users created and go one by one and detect the ones that were obvious spammers and nuking them. However, eventually (I don't recall exactly when and with whom of the CM Team we talked to) the strategy changed: we were told to avoid nuking them and instead change their names to "SPAMMER", while also removing the spam content/links/profile pics.

This, as far as I understood, would help to still have those users around to study and learn about their nature and patterns. I still go hunting for new users now and then and give them the spammer treatment, but not as frequently as before as it is still very manual and time-consuming (and they keep coming regardless!).

My two cents on the things I think are evident signs of spammy users, and things that could be heuristics or criterion for Automated Spam detection:

• Most if not all of them include a Profile Picture of a Company or similar.
• Most if not all use their links for their company/product URL
• Most if not all use their location as the location of the company/product
• All of them include on their Profile Description such Spam content promoting the product/service

As a side note, I recently learned that SmokeDetector was/is having problems with SE servers limiting the access rates to chat pages (which hampers SD's job). I also recently learned that, apparently, SE already attempted to fix that (?) but now Cloudflare is again limiting and hampering SD's ability to do its job... maybe whitelist SD on Cloudflare or something? Disclaimer: my knowledge on this specific matter is not extensive so this paragraph might not be 100% accurate.

I think that supporting and helping existing efforts should be given a very high priority.

To conclude, I'm sharing some examples of the dormant users and the spammer treatment mentioned in this answer:

I literally went to Users > New Users > Creation Date and saw a clear example of a dormant Spam account (hasn't posted yet but their profile has Spam description and content). Here is a screenshot of that user's profile (I won't give the spam treatment to this one so you all can look at what I mean):

I browsed a bit back in time and here is an example of a user that was given the spam treatment along with a screenshot:

Hope that this helps in the process of fighting Spam in the SE network!

The payloads of these spam wave posts are mostly phishing scam contact numbers. Why do the spammers persist in posting even though the posts are usually deleted within ten minutes? The theory is that Stack Exchange is not actually the primary target, it's just a springboard. The scammers hope that their post gets scanned by a Web search engine crawler before it gets deleted on SE. That pollutes the ML data of the search engine, increasing the probability that the search engine will regurgitate the phishing scam number.

If that theory is correct then you could make Stack Exchange sites unattractive to those scammers by delaying the search engine crawlers, eg by increasing the crawl-delay parameter in robots.txt. Has this been considered?