How to remove duplicate/repeated rows in csv with python?

Question

I am scraping web with python and getting data to .csv file that looks like this. If I append to the file, I might have some repeated/duplicate data. To avoid that what can i use? I am not sure about pandas - If i should open the file in pandas and then drop duplicates. I tried other methods of my own, but was unable to come up with a solution. I was thinking of using pandas as the last option

Date,Time,Status,School,GPA,GRE,GMAT,Round,Location,Post-MBA Career,via,on,Details,Note

2021-05-18,13:59:00,Accepted from Waitlist,Yale SOM,3.8,No data provided,740,Round 2 ,NYC,Non Profit / Social Impact,phone,2021-05-18,GPA: 3.8 GMAT: 740 Round: Round 2 | NYC,Interviewed and was waitlisted in R2. Just received the call this afternoon. Good luck everyone!

2021-05-18,13:51:00,Accepted from Waitlist,Yale SOM,3.8,323,No data provided,Round 2 ,Austin,Marketing,phone,2021-05-18,GPA: 3.8 GRE: 323 Round: Round 2 | Austin,Keep your head up! It all works out how it is supposed to.

Answer 1

If you want to do it with pandas

# 1. Read CSV
df = pd.read_csv("data.csv")

# 2(a). For complete row duplicate
pd.drop_duplicates(inplace=True)
             
# 2(b). For partials
pd.drop_duplicates(subset=['Date', 'Time', <other_fields>], inplace=True)

# 3. Save then
pd.to_csv("data.csv", index=False)

Answer 2

Maybe read through the lines one at a time, store them in a set (so there's no duplicates), and then write them back?

lines = set()
file = 'foo.txt'
with open (file) as fd:
    for line in fd:
        lines.add(line)
with open(file, 'w') as fd:
    fd.write(''.join(lines))