This repository was archived by the owner on Dec 10, 2025. It is now read-only.
This repository was archived by the owner on Dec 10, 2025. It is now read-only.
TFIDF content matching should check inter-scrape #97
Description
Description
Currently we remove duplicates everywhere but we only remove duplicates by description (TFIDF) between the masterlist and all scrape data.
We should allow masterlist to perform a content match to itself.
Steps to Reproduce
- scrape some jobs to .pkl
- copy-paste a row a few times, only changing the key_id
- run again with --no-scrape
Expected behavior
We should be running TFIDF inter-scrape data and inter-master csv
Actual behavior
Only duplicates in the ncoming dict are identified based on master CSV
Environment
- Build: 3.0.0