Importing a CSV file into a dataset, filter it by date, then into a postgresql database

Rather than reading the entire file into the dataset I would use .next() to read the file line by line, evaluate if the date is within your parameters, and if so append to a dataset. After the file is parsed, convert the dataset to PyDataSet and use a for loop to run system.db.runPrepUpdate for each record. Here are some good resources:

Exporting and Importing a CSV

Inserting from dataset to database

1 Like