Scrapy Web Crawling Help

Issue explanation:

Everything work just fine, except a crawl order, I add a priority method but didn`t work correctly.

Need to first write all author data, then all album and songs data and store to DB with this order.

I want to query items in a MySQL table by order from item in another one and make a better performance. On other words - need to rewrite some part of item pipeline (process_item method) to this work correctly.


First write all author items in Author table, and then order album items in Album table by authorId from Author table.

Thanks write other tables (what works) and go to another pages.

P.S. I didn`t know what missed, maybe Spider code need to be refactorized for different order.

Always is issue with a select query from another table and comparison like on example below.

(SELECT AuthorId FROM author WHERE AuthorName ='" + str([login to view URL]("albumAuthor")) + "')

Also I need to remove a duplicated items from a some attributes. Have a three item class for author, album and song parser. I was tried to make a another flow of spider and put all in one item class, but with no success. Order was a same. :(

Database structure: [login to view URL]

Github repository: [login to view URL]

Skills: Python, Web Scraping, MySQL, Data Processing, Scrapy

See more: lightspeed web store help, web crawling demo, broadlook web crawling, developing web crawling, web design help, free web design help amman, perl script web crawling, multilanguage web page help php, webscribble edit web jobs help, outsourced web crawling, web crawling script, custom web crawling service, web crawling projects freelancing, web crawling zend, web crawling grabbing, web crawling money, web crawling product pricing, web crawling amazoncom, joomla web site help, contract web crawling services

About the Employer:
( 0 reviews ) Belgrade, Serbia

Project ID: #20759838

Awarded to:


Hi, I am senior web developer and I am very interested in your details. I have read your details carefully and I think I can help you with my skills. I have rich experiences in Web Scraping using Python. If you hire me More

€15 EUR / hour
(48 Reviews)

13 freelancers are bidding on average €15/hour for this job


Hello. I am very familiar with Web scrapping. As a web expert. I have strong skills and experiences in development for 7 years. I hope to work for you because my skill set is very suitable for your project. Will wait f More

€17 EUR / hour
(14 Reviews)

Hi, I can rewrite parts of your script to make your pipeline work perfectly. I have done many projects in python , web scraping ,mysql Some of my projects include 1. Instagram Automation This tool does complete a More

€12 EUR / hour
(40 Reviews)

Hello, client. I am very interested in your scraping job. Web Scraping is my main skill. [login to view URL] These were my past jobs. Thanks. Silver bead.

€15 EUR / hour
(5 Reviews)

Hi I read your job description carefully and I can do your job perfectly. I have scraped many websites using python. I used to use BeautifulSoup or Selenium for web scraping. So I can know what you mean and I am ready More

€15 EUR / hour
(9 Reviews)

Dear sir, I am able to deliver you the high-end quality work according to your requirements. I will deliver to you the different creative ideas that will meet your satisfaction level. I believe in original and creative More

€15 EUR / hour
(10 Reviews)

Hi there! I am 4+ years experienced developer as Python, Django, RoR & ReactJS. Please open the chat box for further discussion. Regards,

€13 EUR / hour
(2 Reviews)

Hi. Please survey my reviews. I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years More

€15 EUR / hour
(4 Reviews)

Hello. I'm a full-stack and full-time developer. I'm an expert of scraping. I have scrapped many sites. Look at my portfolio. If you want hire me, I will do my best for your project. I hope many discussions. Thanks.

€15 EUR / hour
(1 Review)

Hello... My name is Toydemir Adbil, an experienced developer from Turkey with over 7 years of development experience. I have read your post very carefully and I have understood what you want. I am sure I can handle yo More

€15 EUR / hour
(0 Reviews)

Hi, nice to meet you. I am a Full Stack Developer with 4 years of experience working on web design and development. I have worked extensively around the web over these years and this involves accurate Web Scrapping in More

€15 EUR / hour
(0 Reviews)

Hi, Thanks for sharing your requirements here. I’m an Experienced Freelancer with a demonstrated history of working in the internet industry. Skilled in Core PHP, MySQL, HTML, CSS, JQuery, JavaScript. I have worked m More

€12 EUR / hour
(0 Reviews)

Hi backend/data structures developer here. I've read your requirements and is interested in your project. bellow is an intro on my skills, but this is an automated message with only a generic price estimate. Message More

€15 EUR / hour
(0 Reviews)