Develop a custom crawler for my search engine.
A crawler that can crawl all the articles title, url, description from all domains specified and store it in database
should be able to update the database every week without having to effect my search engine.
should not crawl where ever [login to view URL] is present.