Completed

Scrapy Web Crawling Help

Issue explanation:

Everything work just fine, except a crawl order, I add a priority method but didn`t work correctly.

Need to first write all author data, then all album and songs data and store to DB with this order.

I want to query items in a MySQL table by order from item in another one and make a better performance. On other words - need to rewrite some part of item pipeline (process_item method) to this work correctly.

Example:

First write all author items in Author table, and then order album items in Album table by authorId from Author table.

Thanks write other tables (what works) and go to another pages.

P.S. I didn`t know what missed, maybe Spider code need to be refactorized for different order.

Always is issue with a select query from another table and comparison like on example below.

(SELECT AuthorId FROM author WHERE AuthorName ='" + str([login to view URL]("albumAuthor")) + "')

Also I need to remove a duplicated items from a some attributes. Have a three item class for author, album and song parser. I was tried to make a another flow of spider and put all in one item class, but with no success. Order was a same. :(

Database structure: [login to view URL]

Github repository: [login to view URL]

Skills: Python, Web Scraping, MySQL, Data Processing, Scrapy

See more: lightspeed web store help, web crawling demo, broadlook web crawling, developing web crawling, web design help, free web design help amman, perl script web crawling, multilanguage web page help php, webscribble edit web jobs help, outsourced web crawling, web crawling script, custom web crawling service, web crawling projects freelancing, web crawling zend, web crawling grabbing, web crawling money, web crawling product pricing, web crawling amazoncom, joomla web site help, contract web crawling services

About the Employer:
( 3 reviews ) Belgrade, Serbia

Project ID: #20759838

Awarded to:

hunmin888

Hi, I am senior web developer and I am very interested in your details. I have read your details carefully and I think I can help you with my skills. I have rich experiences in Web Scraping using Python. If you hire me More

€15 EUR / hour
(53 Reviews)
6.5

8 freelancers are bidding on average €14/hour for this job

idragon712

Hello. I am very familiar with Web scrapping. As a web expert. I have strong skills and experiences in development for 7 years. I hope to work for you because my skill set is very suitable for your project. Will wait f More

€17 EUR / hour
(22 Reviews)
6.1
abhi68rajpura

Hi, I can rewrite parts of your script to make your pipeline work perfectly. I have done many projects in python , web scraping ,mysql Some of my projects include 1. Instagram Automation This tool does complete a More

€12 EUR / hour
(43 Reviews)
5.6
brightstar928

Hi I read your job description carefully and I can do your job perfectly. I have scraped many websites using python. I used to use BeautifulSoup or Selenium for web scraping. So I can know what you mean and I am ready More

€15 EUR / hour
(9 Reviews)
4.7
Darshit222

Dear sir, I am able to deliver you the high-end quality work according to your requirements. I will deliver to you the different creative ideas that will meet your satisfaction level. I believe in original and creative More

€15 EUR / hour
(10 Reviews)
4.1
ThinkStartPL

Hi there! I am 4+ years experienced developer as Python, Django, RoR & ReactJS. Please open the chat box for further discussion. Regards,

€13 EUR / hour
(2 Reviews)
3.7
achievers24

Hi, Thanks for sharing your requirements here. I’m an Experienced Freelancer with a demonstrated history of working in the internet industry. Skilled in Core PHP, MySQL, HTML, CSS, JQuery, JavaScript. I have worked m More

€12 EUR / hour
(1 Review)
0.0
henry9

Hi backend/data structures developer here. I've read your requirements and is interested in your project. bellow is an intro on my skills, but this is an automated message with only a generic price estimate. Message More

€15 EUR / hour
(0 Reviews)
0.0