Help me to finish-tune up a java scraping project TODAY

Awarded Posted Jun 15, 2015 Paid on delivery
Awarded Paid on delivery

Hello i'm looking for a senior developer to finish a project today.

I need to add a gui to my project, some functionalities and improve speed between multithreading queries.

I can pay a maximum of 70 usd.

I'll hire the freelancer today. Thank you

This is the code to improve.

I use several apis and libraries.
This is a maven project.

What the project currently does?
______________________________________
Main goal is to obtain books data with contactable author. So are important both: book details and author contact details.

To do this i loop in several nodes in a xml - called config.xml - collecting the url of the bookstores. All the bookstores has the same layout, so the only change is the url itself.

Every url i collect start a thread, which try to download the information of n books, starting from a number specified in config.xml and finalizing in other number specified in config.xml

Because the way the bookstores is programmed, i first go to a form, search for the book' sku and then i go the search result and finaly i get the url of the book. Then i feed the url to an scraping api (import.io api) and i get the information of the book in a html table.

Then i look for if the book has biography of the author, if so, i feed an api wich do a semantic analizis -called opencalais- and after get the answer in xml, and i get the important data like city where the author lives, email, and every contact detail.

To get its phone and other relevant details, i feed another api -called intelius- which let me find all people named like the author living in specific city and i get phone numbers.

So i put all the book information and the possilbe contact details in a row in a table in a html file

Every succesful 50 records i start a new exportable file.

Each export is a html file with a table which has in each row the information of a book and the associated author details and posible phones.

The files exported includes a js file which let the user export the file to excel.

What is required?
_________________________________

Well, because i need to improve speed, I thought to start new threads (as set in config.xml) to increase the number of threads querying a url, to download more than one book at once, save the data in a sqlite db and when it goes to ask if has contactable data in the author biography, then start a new thread in opencalais and if it gets data to feed intelius (city where the autor lives indeed), then start a new thread in intelius too, and add the information collected to the book record in the database in a new table in author table with the possible contact details.

So the approach is increase the number of threads, and create a db (sqlite is the approach) to storage partial information, with book, and authors tables. I need help in design and program the new approach.

All this scrapping information has log just in console and modifies the xml file so, its the way the user see the progress, but it needs a gui.

Finally the gui must be able to export to excel the information too, so it needs some queries to the database and the exportability managment.

___

Other part is to get more information of new bookstores, increasing book information and narrowing the posibilites about the author contact details with the new bio information obtained ... well, may be it could be a second part of the project.


Java SQLite Web Scraping Windows Desktop XML

Project ID: #7865179

About the project

6 proposals Remote project Active Jun 15, 2015

6 freelancers are bidding on average $77 for this job

mantislin

Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi

$112 USD in 3 days
(111 Reviews)
6.7
narendragautam

Hi, I am interested. Please provide work details. i hope it will fairly estimated price. Please see my portfolio at http://www.freelancer.com/u/narendragautam.html Thank-you Narendra

$147 USD in 3 days
(7 Reviews)
4.3
Shankarone

Please let me know if you are using SQLite in your program . If you are not using SQLite , I will make sure your project is completed today . Message me and we can get started

$70 USD in 1 day
(12 Reviews)
3.8
bhoopendravw

i can start work with you today.i have good experience working in mutithreading environment. u can try me . i have 7 years of exp in java/j2ee technology. could you please throw some light on gui spec?? let me see More

$35 USD in 1 day
(2 Reviews)
3.1
lenin08

Hi There, I am Lenin and I have the 5 years of experience in Java. I am currently the working software professional in one of the leading IT services company. I went through the post and I am sure that I can do that More

$70 USD in 3 days
(1 Review)
2.9
fam85

Hello, I am a computer engineer; I glad to do your job. I am an expert in Information Security, Web Security, Network Security, also web programming (PHP, ASP) , Programming (JAVA, jsp, C#, ..), Mobile ( android ,J2ME) More

$77 USD in 1 day
(2 Reviews)
1.5
noam806

I can help you with you little project :) I can help you with you little project :) I can help you with you little project :) I can help you with you little project :)

$30 USD in 3 days
(0 Reviews)
0.0