Completed

Write Python script to import 1.8M rows of CSV data

I have the following table in a MySQL 2nd Gen 5.7 InnoDB :

CREATE TABLE [login to view URL] (

id INT(11) NOT NULL AUTO_INCREMENT,

name VARCHAR(256) DEFAULT NULL,

source VARCHAR(64) DEFAULT NULL,

price INT(11) NOT NULL,

year INT(11) NOT NULL,

make VARCHAR(64) NOT NULL DEFAULT '',

model VARCHAR(64) NOT NULL DEFAULT '',

mileage INT(11) NOT NULL,

transmission VARCHAR(64) DEFAULT NULL,

num_cylinders INT(11) DEFAULT NULL,

drive_type VARCHAR(64) DEFAULT NULL,

body_type VARCHAR(64) DEFAULT NULL,

fuel_type VARCHAR(64) DEFAULT 'Other',

title_type VARCHAR(64) DEFAULT NULL,

vin VARCHAR(64) DEFAULT NULL,

trim VARCHAR(64) DEFAULT NULL,

color VARCHAR(128) DEFAULT NULL,

location VARCHAR(256) DEFAULT NULL,

source_id VARCHAR(24) DEFAULT '',

url VARCHAR(512) DEFAULT '',

date_listed DATETIME DEFAULT NULL,

date_found DATETIME DEFAULT NULL,

date_updated DATETIME DEFAULT NULL,

num_doors INT(11) DEFAULT NULL,

date_analyzed DATETIME DEFAULT NULL,

seller_type VARCHAR(15) DEFAULT NULL,

details LONGTEXT DEFAULT NULL,

PRIMARY KEY (id)

)

I have a CSV file of 1.8 million rows that I need to have inserted into this database. I need a Python script to do this for me.

The CSV has the following columns:

name,

year,

make,

model,

trim,

mileage,

price,

transmission,

num_cylinders,

drive_type,

fuel_type,

body_type,

color,

location,

url,

sp_url,

date_posted,

date_found,

vin,

title_type,

source

The sp_url column from the CSV should not be imported.

All other columns from the CSV need to be imported.

The database columns num_doors, date_analyzed, seller_type, details, and source_id should all import NULL.

Some of the fields have double quotes " in them. These need to be removed.

Some of the fields are empty / blank. These need to be imported as NULL.

I could not attach the CSV file here because it is too large (700 MB).

Skills: Python, Software Architecture, Data Processing, MySQL, Data Extraction

See more: how to set low_memory=false in python, read csv python datetime, pandas dtype, pandas data types, pandas read_csv low_memory and dtype options, read_csv low_memory=false, how to set low_memory=false, python specify dtype option on import, read csv file using python script, script import csv file database, vbs script import csv sql server, script import csv mysql, python script csv tab, script import access csv files, perl script import csv mysql linux, script import map csv database php, need help write python script operate telit module, need write python script telit gc864quad module, python script import blogger, python script open read log file parse data

About the Employer:
( 1 review ) Riverton, United States

Project ID: #20699368

Awarded to:

gangabass

I can provide you a simple Python script that will import your CSV into MySQL database. All I need to start is a sample CSV from you.

$30 USD in 1 day
(450 Reviews)
7.3

15 freelancers are bidding on average $31 for this job

kipdev13

Nice to meet you. Very interested in your project. I have good talents and skills of python and i can import data from csv. We can share more details of work via chat. I will try to my best for work. Fast speed, Best q More

$50 USD in 1 day
(132 Reviews)
7.0
brianconey

Hello How are you I have checked your description I am a python expert and that is not a problem for me to edit python script with 1.8M rows data . It is the best proper for me to work on this project , so i am su More

$25 USD in 1 day
(90 Reviews)
6.5
AlexanderPGR

Hi, there! I have read your description carefully. I am Python expert and can do this task as you want. Please let me know your timeline and budget. Best regards.

$20 USD in 7 days
(15 Reviews)
4.7
pandaios

Hi, Nice to meet you. With 6+ years of experience in Web development, I developed a lot of big and small websites using HTML & PHP(Laravel, Codeigniter, Yii) & Python(Django, Flask) & JS(Angular, Node, React, Vue) & More

$20 USD in 7 days
(6 Reviews)
3.6
webdev1888

Hi,Sir! Nice to meet you! I read your description. I am a excellent Full-Stack Web developer and Python expert. I 'm able to complete your project perfectly. I hope you contact with me. Thank you!

$30 USD in 1 day
(10 Reviews)
3.7
ninestar801

Hi, I have gone through your requirements and I can do this task. The skills and resources needed for this project are in my genes. I can assure you for a complete professional work in given time [login to view URL] share comp More

$50 USD in 7 days
(6 Reviews)
3.2
uelionobre

Hi~ I have read your description carefully and checked your all rquirements fully like following: [login to view URL] sp_url column from the CSV should not be imported. [login to view URL] other columns from the CSV need to be imported. [login to view URL] da More

$50 USD in 1 day
(7 Reviews)
3.1
kozachenkodr

Hi. I have done many jobs with CSV -> Mysql using PYTHON. I have read your job carefully, so I am sure I will do it perfectly. Please ping me to discuss. Regards.

$30 USD in 7 days
(4 Reviews)
2.3
Hanzhequan

Hello,I have read yr job carefully. I am talented in Python&Excell. If u r excell is large,I need some data for test. If u hire me,I will do my best and u will receive great resutls. Thanks 4 reading.

$20 USD in 7 days
(2 Reviews)
1.5
prashant8967

i have experience in working with database and csv in python of 10mn rows, I can achieve thos task easily.

$40 USD in 3 days
(0 Reviews)
0.0
pintutrust143

Hi, I have a good typing speed and can perform your task with ease. I have gone through your details already and can perform your task on priority basis. please share details once.

$20 USD in 7 days
(0 Reviews)
0.0
kaajavi

Hi, I can process the file with a python3 script, with or without sql framework like pewee or sqlalchemy.

$35 USD in 2 days
(0 Reviews)
0.0
fireball111

Hi.. This is pretty straight forward. Probably the most time consuming thing will be ensuring that the db module (and other pip installs) are easy to install on your system (with what version of python?) prior to run More

$20 USD in 1 day
(0 Reviews)
0.0
Ceferty

Hi, I already made it. Do you need any kind of validation in the CSV fields? My other question is if you will send the CSV and do you expect just the script or the database to be imported? Let me know.

$30 USD in 7 days
(0 Reviews)
0.0