Find Jobs
Hire Freelancers

Java scrapping improve expert

$30-250 USD

Closed
Posted over 3 years ago

$30-250 USD

Paid on delivery
I developed a Java program to scrap information from a website. The architecture of the solution involves: 1) using Java Selenium to send requests to the webpage via Chrome Webdriver to trigger authentication and authenticated requests; 2) routing the requests from Chrome (headless) to Java BrowserMobProxy to capture three HTTP headers (Authorization, X-CSRF-TOKEN, and Cookie) and one query string (without these, the server after some requests starts responding 512); and 3) use these 4 elements in HTTPs requests from Java directly to the webpage (i.e. without Selenium, Chrome, and BrowserMobProxy involved) to retrieve the desired information. This program does the basic functionality of extracting the information but has a few problems: It depends on an external non-Java component: Chrome WebDriver It depends on Java Selenium and Java BrowserMobProxy, two dependencies that I would like to remove It is not optimized (too much refresh and too long sleep periods) relatively to the limit upon which the Webpage (Cloudfare) starts responding 429 errors. Thus, the retrieval of the information is taking much more time than needed. Deliverables You will get the current program Java code and you will need to solve the problems above. To do so, you will need to: A. Find out how to authenticate and refresh the 3 headers and the query string without depending on Selenium, Chrome Webdriver, and BrowserMobProxy. As most of this data is likely generated in JavaScript, you will need knowledge about JavaScript and how to execute JavaScript from within Java or convert the JavaScript code to Java (preferable solution). B. You will need to identify the limit upon which the Webpage (behind Cloudfare) starts responding 429 errors. You will need to tune the refresh frequency of the headers and sleep periods to the limit identified. You will need to demonstrate the benefits of your changes by extracting the information currently extracted by the program and measuring how long it takes. Note: you will need to create your own login/password in the webpage. No additional requirements exist to register.
Project ID: 26951665

About the project

11 proposals
Remote project
Active 4 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
11 freelancers are bidding on average $177 USD for this job
User Avatar
Hi, Greetings! ✅checked your project details: Java scrapping improve expert ✅Completed Time: In project deadline We have worked on 650 + Projects. I have 6 + years of the experience in same kind of projects. If you are looking for a true Freelancer, I am the Right person for you. I am available almost 24-7 and am very responsive. I feel proud that I am a trusted Freelancer who pleases almost every single client. You can rest assure, your work will be delivered well in advance of others, with passion and accuracy. I guarantee you instant communication & responses when you need me. Why choose me? I think every client is the reason for my success. I only take projects which I am sure I can do quickly. My Portfolio Items: https://www.freelancer.com/u/schoudhary1553 I would really like to work with you on this project. If interested, Kindly contact me via chat for further details and discussion.. Thank you Sandeep Digital screencast
$240 USD in 5 days
5.0 (139 reviews)
7.3
7.3
User Avatar
I can replace existing scrape with python requests library with removing headache of jars and selenium and chrome driver. it will be very light weight. to resolve 429 errors we need to use proxy or delay. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation with selenium also. Know to deal with database like mongo, mysql, Elasticsearch. Also know to handle proxy and captcha while scrapping. Can scrape data from various websites . have done scraping of millions of products from Amazon eBay Walmart. have done scraping of social media websites like Facebook Instagram LinkedIn.
$167 USD in 2 days
4.9 (57 reviews)
6.2
6.2
User Avatar
Hello, I am very interested in your project”Java scrapping improve expert”. Webscrapping is my best skill. I have read the job description and I am interested in this job. I have 8 years experience in developing products using Software Architecture, .NET, C# Programming, Web Scraping and JavaScript. I have ever done so many projects using python selenium, beautiful soap and panda. I am sure I can offer good result and fast delivery for u in time. I have read your requirements and am ready to start working for you. Good luck from konstiantyn.
$140 USD in 7 days
4.7 (18 reviews)
4.9
4.9
User Avatar
Hi there, Let’s have a quick chat to discuss this project. I am expert in Python, PHP, JavaScript,Web Scraping,MYSQL.I do have expertise for this project. You can check my portfolio here:- https://www.freelancer.com/u/PoojaRautela417?page=portfolio&w=f&ngsw-bypass= Looking forward to hear from you soon. Regards Pooja Bohra
$250 USD in 3 days
4.9 (27 reviews)
4.9
4.9
User Avatar
Hi I am a very experienced statistician, data scientist and academic writer. I have completed several PhD level thesis projects involving advanced statistical analysis of data. I have worked with data from several companies and have done projects involving high level quantitative analysis and data interpretation skills to study the trends, time behaviour and compare the variables in the data. I can do advanced level analysis in SPSS, R, PYTHON, WEKA, TABLEAU, POWER BI, and EXCEL tools like machine learning, deep learning, AI, NLP, hypothesis testing, forecasting, T-test, ANOVA etc. Looking forward to discussion, Best Regards, Suyash
$250 USD in 7 days
3.8 (27 reviews)
5.7
5.7
User Avatar
Hii there , I am bidding on your project and I am good at this field I can do this for you within due time and honestly. I also have a few questions to discuss. Kindly contact me and we will discuss time and budget. Thanks
$140 USD in 7 days
4.9 (17 reviews)
4.3
4.3
User Avatar
Hello, I hpoe your family safe with Covid-19 I am a Java Full Stack Developer with hands-on experience working on Various websites, applications for more than 5+ years. I have an expert development team. All the resources are well experienced and ready to provide you quality solutions. Awaiting your valuable msg. Thanks, Best Regards Amit Rai
$140 USD in 7 days
5.0 (13 reviews)
3.0
3.0
User Avatar
Dear Employer, Thanks for posting the project . I have gone through your description " need to create your own login/password in the webpage. No additional requirements exist to register." and I believe I'm capable to work on this project . I have been working for the last 5 years as a full stack web developer and developed lots of website . I have the following skills below: WordPress and plugin development, PHP, Laravel, graphic design, e-commrce websites, bootstrap and landing page etc. Therefore, I think you will consider to send me a message for further discussion about your project . I would like you to come on chatbox so that we can discuss more in brief on that. Waiting for your positive response. I am interested in your requirement and can start work immediately. Kind Regards Nutan K
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi! I am happy to put my bid on your project. I have read your requirement carefully and I am confident in this project. I am a skillful and experienced web developer, I have a tons of experience with JAVA/Python/MY-SQL/. • HTML5 /CSS /JavaScript /jQuery I am confident to provide you best solution that completely satisfy your business needs in a short period of time.
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of ROMANIA
Băilești, Romania
5.0
1
Member since Mar 8, 2020

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.