Process a Large Amount of Data Files Using Python

I have a large amount of data files that I’d like to have processed (and select information placed into an output CSV).

The purpose of the project will be:

1 – Develop a python script that will process the 180 input files (each input file is ~150mb and is in [login to view URL] format)

2 – Incorporate multi-threading to take full advantage of the high-power server that will be used for the processing (96 processors and 192gb of memory)

3 – Run the developed script on the files and generate the [login to view URL] (you will be given access to the server which may be accessed using RDP)

** Please see the attached write-up that explains the exact details in-depth.

** Please include a brief description in your PM along with your bid so I can tell that you reviewed the attached document read and fully reviewed the project description instead of using an auto-bidder to bid on the project. Just a sentence or two will do.

** This is a rush project. It is a requirement that the project needs to be completed within 24 hours. Hence the budget I have allocated.

Thank you!

Skills: Python, Data Processing, Data Entry, Software Architecture, Data Cleansing

See more: excel large data files, mysql load large data files, vbs file copying large amount files, extracting data webpages using python, process text files using vbscript, excel macro input large amount data, vbnet send large streams data using net, export excel using large amounts data, stx etx read rfid data using python, extracting data web pages using python, extract data html using python, large amount data, pylab loadtxt python load big amount data, using python modify output files, extract data bloomberg using python, web data mining using python, parse text files using python, analyse a data set using python, data comparison using python

About the Employer:
( 96 reviews ) New York, United States

Project ID: #21234666

Awarded to:


Hello! I understood your project. I see that you need some help for processing large amount data using python. Your document has 5 parts. I am on freelancer all day and I will implement all things that you want perfect More

$399 USD in 5 days
(23 Reviews)

Hi, I have good experiences developing code in Python Programming language for scientific application and Machine learning projects more than 3 years . I read description files which have 5 parts. In first parts you n More

$230 USD in 1 day
(15 Reviews)

13 freelancers are bidding on average $262 for this job


Hello,sir. I have checked your project description. I have 7+years of experience software and data processing. I can implement your project with fully and completely. I can help you with my best. Thanks for your postin More

$250 USD in 7 days
(20 Reviews)

Hi! I am very happy to see your project detail. I think it is just my project. :) As you can see in my profile, (my review: I have much experience with your job. please cont More

$250 USD in 2 days
(10 Reviews)

***** Scraping Expert ***** Hello. I have read your job description carefully. I have full experience in Web Scraping with PHP and Python. So I can do your project perfectly. I can start just now and I can deliver good More

$250 USD in 2 days
(4 Reviews)

Hi, Nice to meet you! I am very interesting your project and I am confident of I can help your job. I'm a professional Python expert with over 7 years of experience. Seems to be an interesting work and good opportunit More

$250 USD in 7 days
(6 Reviews)

Hi there, I am Python developer, having below given skills: Engineering professional with 10 years of experience in Software development. Mastering/Leading in the development of applications/tools using Python for 6 More

$250 USD in 7 days
(2 Reviews)

⭐⭐⭐⭐ i am python developer. ⭐⭐⭐⭐⭐⭐ I read through the job details extremely carefully and I am absolutely sure that I can do the project very well. * 6+ years experience in web and mobile app development. If you w More

$250 USD in 7 days
(3 Reviews)

Hello, How are you? I have read your description very carefully and it really attracts me. I am python expert with rich experience. I have done many projects with python before. I am sure I can satisfy your requirement More

$400 USD in 7 days
(1 Review)

Hello Sir, Nice to meet you. How are you doing sir? I read your project description. I think I am able to complete your project. Kindly message me for discuss your project. Thank your sir, Best Regard, Md Abu Shal More

$100 USD in 7 days
(0 Reviews)
$277 USD in 10 days
(0 Reviews)

Dear Sir I’ve read your job very carefully and understand what you want. I'm a Senior developer with 10+ years experience. I've special skill in Python. Any of your requirements are welcome. I can start your project More

$300 USD in 1 day
(0 Reviews)

The project sounds relatively straight forward. Taking info from files and populating a CSV into ordered, retrievable data. The program of which takes advantage of a high-power server by using multi-threading. I've w More

$200 USD in 3 days
(0 Reviews)