Create multiple dataframes inside a for loop - pandas

Question

I am begginer in python.

I have three datasets that have a similar structure and I want to perform the same process for each one and save the results. My code run well but it only save the results for the last one base in bases. ¿How can I save the results in three different datasets?

My code looks like:

bases = [base1, base1_a, base1_b]
for base in bases:
    base = base.groupby(["Cliente_id", "Periodo_id"]).agg({"monto_2018": ["sum", "mean", "min", "max"]}).reset_index().round(1)
    base.columns = base.columns.get_level_values(1)
    base = base.set_axis(["Cliente_id", "Periodo_id", 'sum_v', 'mean_v', 'min_v', "max_v"], axis=1, inplace=False)

Answer 1

bases = [base1, base1_a, base1_b]
for base in bases:
    base = base.groupby(["Cliente_id", "Periodo_id"]).agg({"monto_2018": ["sum", "mean", "min", "max"]}).reset_index().round(1)
    base.columns = base.columns.get_level_values(1)
    base = base.set_axis(["Cliente_id", "Periodo_id", 'sum_v', 'mean_v', 'min_v', "max_v"], axis=1, inplace=False)

When you assign base = base.groupby() in the loop, you are reassigning the variable base to refer to a new dataframe which is the result of the groupby() action. This does NOT modify the original bases list.

To get a list of your new data frames, you should just create a new list:

bases = [base1, base1_a, base1_b]
results = [] # -----------------------------> create an empty list
for base in bases:
    base = base.groupby(["Cliente_id", "Periodo_id"]).agg({"monto_2018": ["sum", "mean", "min", "max"]}).reset_index().round(1)
    base.columns = base.columns.get_level_values(1)
    base = base.set_axis(["Cliente_id", "Periodo_id", 'sum_v', 'mean_v', 'min_v', "max_v"], axis=1, inplace=False)
    results.append(base) # ------------------> Add dataframe to the new list

I chose the name results here as a generic name. I strongly encourage you to use a different name that is more descriptive for what you are doing.

Answer 2

Avoid saving similarly structured data frames as separate objects in global environment. Right now it is three but can easily be 300! Instead, continue using a list (bases = [base1, base1_a, base1_b]) or even better dictionary as shown below then process it with list or dict comprehension using a user-defined method. Below uses named aggregation to avoid the column header manipulation.

# ASSIGN DATA FRAMES IN DICTIONARY 
# IDEALLY. YOU CREATE DICT DIRECTLY FROM SOURCE VIA read_* METHODS
df_dict = {"base1": base1, "base1_a": base1_a, "base1_b": base1_b}

def process_data(base):
    base = (
        base.groupby(["Cliente_id", "Periodo_id"], as_index=False)
            .agg(
                 sum_v=("monto_2018", "sum"),
                 mean_v=("monto_2018", "mean"),
                 min_v=("monto_2018", "min"),
                 max_v=("monto_2018", "max")
            )
            .round(1)
    )

    return base

# DICTIONARY COMPREHENSION TO ITERATE THROUGH ALL DATA FRAMES
new_df_dict = { k:process_data(v) for k,v in df_dict.items() }

Then call your dictionary elements as needed since you lose no functionality of data frame if saved in a container like list or dict:

new_df_dict["base1"].head()
new_df_dict["base1_a"].tail()
new_df_dict["base1_b"].describe()