Dealing with Missing Data in Pandas DataFrame

Question

I'm currently working on a data science project utilizing Python and Pandas for data manipulation. My dataset, loaded from a CSV file, comprises various columns, but unfortunately, it has a lot of missing values. I'm seeking advice on the most effective approach to handle these gaps. Upon loading the dataset with pd.read_csv('data.csv'), I observed missing values scattered throughout the DataFrame. I'm unsure about the optimal strategy to address this issue—whether to remove rows with missing values using dropna() or to replace missing values with a specified value using fillna(). My dataset contains both numerical and categorical columns, and I'm concerned about the implications of removing or replacing missing values on the integrity of my subsequent analysis. Are there any established best practices or common strategies for dealing with missing data in Pandas that I should consider? Any insights or guidance would be greatly appreciated!

I've considered using methods like dropna() to remove rows with missing values or fillna() to replace missing values with a specified value. However, I'm not sure which approach is best for my dataset and whether there are other techniques I should consider.

Answer 1

You can use functions like simpleimputer to replace the nulls with mean or median in numeric columns and most frequent in categorical columns

from sklearn.impute import SimpleImputer
df_copy = df.copy()    
imputer = SimpleImputer(strategy='mean')
imputer2 = SimpleImputer(strategy= 'most frequent')
df = imputer.fit_transform(df.select_dtypes('number')
df = imputer2.fit_transform(df.select_dtypes('object')

Of course I could do this easier with a column transformer or a pipeline but this should be basic enough for you

df will be a numpy array, so you can convert it to a dataframe by

df_new =pd.DataFrame(df, columns = df_copy.columns)