Crawl Domain Find Expred Domains

We are looking for a crawler to crawl every page of a website looking for external links pointing to expired domains.

User should definde a list of sites to crawl via text file. Crawler should work logically crawling all pages of a site and not be sitemap dependent. Only unique external domains should be logged to prevent duplicate domain availability lookups.

User should also be able to define a list of urls to ignore checking for availability; eg. [login to view URL] etc. these domains should be user defined in a blacklist text file.

Results should be given in a csv file listing linking domain and available domain.

Skills: Python, Web Scraping

See more: find keywords rank domain, find records domain siteexperts exchangecom, find domain value, expired domain finder software, how to find expired domain names with traffic, expired domain search engine, how to find expired domains with scrapebox, expired domains api, best expired domain finder, expired domains with traffic, expired domain marketplace, find buyer domains, enom api find available domains, need script find expired domains name, domain extractor extracts domains search engine, domain spy closed domains, crawl domain excel, crawl web find urls, crawl website find emails, find domains owned using domain

About the Employer:
( 5 reviews ) 保定市, China

Project ID: #19760507

5 freelancers are bidding on average $65 for this job


Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields do More

$200 USD in 5 days
(112 Reviews)

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO More

$35 USD in 3 days
(55 Reviews)

hi sir, i'm a python developer. i read the project explanation, actually i've programmed kind of a site crawler before, there should a broken-link finder there shouldn't be problem, can do it for you. thanks

$30 USD in 3 days
(3 Reviews)

Hello, I made a project like this. It is in my github account. You can use it. But It is not doing exactly what do you want. So I can edit it and create a version for you. You might want to look to my github account: More

$40 USD in 1 day
(0 Reviews)

Hey, I think i may be able to help you out on this project since i have worked on a similar project at work. Im new to freelancer, though i have years of experience in software development in python.

$20 USD in 1 day
(0 Reviews)