Process a Large Amount of Data Files Using Python

I have a large amount of data files that I’d like to have processed (and select information placed into an output CSV).

The purpose of the project will be:

1 – Develop a python script that will process the 180 input files (each input file is ~150mb and is in [login to view URL] format)

2 – Incorporate multi-threading to take full advantage of the high-power server that will be used for the processing (96 processors and 192gb of memory)

3 – Run the developed script on the files and generate the [login to view URL] (you will be given access to the server which may be accessed using RDP)

** Please see the attached write-up that explains the exact details in-depth.

** Please include a brief description in your PM along with your bid so I can tell that you reviewed the attached document read and fully reviewed the project description instead of using an auto-bidder to bid on the project. Just a sentence or two will do.

** This is a rush project. It is a requirement that the project needs to be completed within 24 hours. Hence the budget I have allocated.

Thank you!

Skills: Python, Data Processing, Data Entry, Software Architecture, Data Cleansing

See more: excel large data files, mysql load large data files, vbs file copying large amount files, extracting data webpages using python, process text files using vbscript, excel macro input large amount data, vbnet send large streams data using net, export excel using large amounts data, stx etx read rfid data using python, extracting data web pages using python, extract data html using python, large amount data, pylab loadtxt python load big amount data, using python modify output files, extract data bloomberg using python, web data mining using python, parse text files using python, analyse a data set using python, data comparison using python

About the Employer:
( 115 reviews ) New York, United States

Project ID: #21234666

Awarded to:


Hi, I have good experiences developing code in Python Programming language for scientific application and Machine learning projects more than 3 years . I read description files which have 5 parts. In first parts you n More

$230 USD in 1 day
(27 Reviews)

8 freelancers are bidding on average $226 for this job


Hello,sir. I have checked your project description. I have 7+years of experience software and data processing. I can implement your project with fully and completely. I can help you with my best. Thanks for your postin More

$250 USD in 7 days
(22 Reviews)

***** Scraping Expert ***** Hello. I have read your job description carefully. I have full experience in Web Scraping with PHP and Python. So I can do your project perfectly. I can start just now and I can deliver good More

$250 USD in 2 days
(7 Reviews)

Hi there, I am Python developer, having below given skills: Engineering professional with 10 years of experience in Software development. Mastering/Leading in the development of applications/tools using Python for 6 More

$250 USD in 7 days
(3 Reviews)

⭐⭐⭐⭐ i am python developer. ⭐⭐⭐⭐⭐⭐ I read through the job details extremely carefully and I am absolutely sure that I can do the project very well. * 6+ years experience in web and mobile app development. If you w More

$250 USD in 7 days
(6 Reviews)

Hello Sir, Nice to meet you. How are you doing sir? I read your project description. I think I am able to complete your project. Kindly message me for discuss your project. Thank your sir, Best Regard, Md Abu Shal More

$100 USD in 7 days
(2 Reviews)
$277 USD in 10 days
(0 Reviews)

The project sounds relatively straight forward. Taking info from files and populating a CSV into ordered, retrievable data. The program of which takes advantage of a high-power server by using multi-threading. I've w More

$200 USD in 3 days
(0 Reviews)