Django Web Application - Monitor local folder and import csv files to database

Question

I have this Django web application project that needs to be hosted on a local network. In addition basic CRUD features, the scope requires to continuously monitor a local storage folder (C: or D: or E:) for csv files and import them to the database (Postgresql). I have already written the code for reading the csv files and importing them to the database and moving these csv files to another folder (after importing). What I don't know is where should I put this code and call the function (import_to_db), such that it runs continuously to scan the folder for new csv files? It cannot be a python command line interface. I am not fully conversant with Django REST Framework and not sure if it applies to this scope, since the csv files will be made available in a local folder. Any tips or references to examples/libraries would help.

Code to Import:

def get_files():
    csv_files = []
    for file in os.listdir(os.getcwd()):
        if file.endswith('.csv'):
            csv_files.append(file)
    return csv_files


def move_files_to_folder(csv_files, destination):
    try:
        os.mkdir("BackupFiles")
    except:
        print('BackupFiles Directory Already Exists')
    finally:
        for file in csv_files:
            shutil.move(file, destination)
    os.chdir(destination)
    return csv_files


def import_to_db():
    csv_files = get_files()
    engine = create_engine(
            url="postgresql://{0}:{1}@{2}:{3}/{4}".format(user, password, host, port, database))
    for file in csv_files:
        df = pd.read_csv(file, parse_dates=[1, 2], infer_datetime_format=True, encoding="utf-8")
        df.rename(columns={'ItemNumber': 'ItemNumber_id'}, inplace=True)
        df.to_sql('app_logdata', engine, if_exists='append', index=False)
        print('Imported Data Successfully')
    destination = os.getcwd() + os.sep + "BackupFiles"
    move_files_to_folder(csv_files, destination)
    print('CSV Files Moved To BackupFiles Folder', csv_files)
    # os.chdir('...')
    # print('App Folder = ', os.getcwd())


import_to_db()

chlorinelemon · Accepted Answer · 2022-10-29 03:49:33Z

0

After searching, I found out there are a few ways of achieving the objective. Using Cron, Scheduler, Rocketry etc.

answered Oct 29, 2022 at 3:49

chlorinelemon

239 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

chlorinelemon Over a year ago

Thanks. But, I was not able to narrow down on one particular tool since I am not sure about how to bundle the entire application and deploy it on the client's machine such that the application is protected. The objective is to deploy the web application in such a way that it can read csv files from a local directory on a continuous basis. In other words, how can I deploy the application on a local machine?

Collectives™ on Stack Overflow

Django Web Application - Monitor local folder and import csv files to database

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related