I'm currently working in a dataset with censorship on profanity. Basically, fuck would be 4 heart emojis. Considering I'm trying to run a topic modelling w/ BERTopic, what kinda of preprocessing would be adequate to handle this situation?
Interesting question. There are a few directions you could take.
First, consider how much profanity is in your data and whether it actually impacts the topics.
From an NLP perspective, profanity can take different roles:
So treating everything the same way may not be ideal.
Here are some options:
It might not affect your topics much. In that case, leaving it as is can work.
You could reconstruct words using heuristics or a masked language model. This is noisy since multiple words can map to the same pattern.
Replace censored text with something like [PROFANITY] or [CENSORED].
You can also normalize all censored tokens to one form. Check that the token exists in the model vocabulary so it gets a meaningful embedding.
Another option is to use the model’s [MASK] token. It does not encode profanity, but the model understands it.
If you can estimate part of speech, use tokens like:
[PROFANITY_NOUN][PROFANITY_VERB]This keeps some structure.
Instead of changing text, map censored tokens to a general profanity vector. For example, an average of known profanity embeddings. This only helps if the model has seen such words.
Best choice depends on whether profanity matters for your topics. If it does, keep the signal. If not, simple normalization is enough.
Interesting problem.
