ShopLocal Web Crawlers / Spiders for Nishant

The attached file summarizes the crawler requirements for the Best Buy crawler. Other crawlers will be included in their respective folders. Your bid should be a fixed price bid and not time-dependent ("T&E" bid). However, please review the contract for time constraints and other requirements. We are interested in hiring PHP developers/team on a long-term contract basis who are knowledgeable about the design and development of web crawlers/spiders using PHP. We currently have crawlers developed for two retail sites—-one with local store information and the other with online store information—-and are interested in significantly expanding the number of sites crawled (see listings below). The crawlers extract various information (product, store, and pricing) from the sites. Database used is a MySQL database (see attachment for db schema). You are bidding on the development of one (1) website crawler for BestBuy.com. If we like the final script, we will contract to have additional scripts created for a set agreed-upon price per script. Please put your bid in for the initial script and mention the price for any additional scripts. Final selection will be made on the bid for this initial script as well as the price of each incremental script. Note: Also note that existing crawler scripts (there are 2 available) will be provided to the winner of the bid for this project to expedite the development process. Please consider this in your bids. All site requirements are enumerated in the attached document, entitled "060525_SiteReqDoc_BestBuy_v1.4.doc". Additionally, the contract If you wish to bid on the project, please follow the steps below: 1. Complete the NDA located at: [login to view URL];tstart=0 2. Read the [login to view URL] crawler requirements in the attachments at [login to view URL];externalID=11 3. Complete the MS Word Service Agreement by revising the underlined text at [login to view URL];categoryID=15. All documents must be signed in all required appropriate signatories within each document and may be emailed as attachments in their entirety or faxed to (413) 622-8080. If you had any questions, you can post them at [login to view URL] Note that you must be registered in order to post documents; however, you can view documents without registering. The preliminary listing of sites to be crawled are as follows (the final site may or may not be selected from the following listing): Local Shopping Sites: 1. [login to view URL] (crawler already developed for this site) 2. [login to view URL] 3. [login to view URL] 4. [login to view URL] Online Shopping Sites: 1. [Trademark masked].com 2. [login to view URL] 3. [login to view URL] 4. [login to view URL] 5. [login to view URL] 6. [login to view URL] 7. [login to view URL] 8. [login to view URL] 9. [login to view URL] 10. [login to view URL] ** Note: Furthermore, the Scripts should be flexible enough (extensible) to crawl all subcategories (e.g. Electronics, Computers, Housewares, Toys, etc.) available on the site. Brick-and-Mortar Companies which have an Online Presence: 1. [login to view URL] 2. [login to view URL] (crawler already developed for this site) 3. CompUSA 4. [login to view URL] 5. [login to view URL] 6. [login to view URL] 7. B&[login to view URL] 8. J&R 9. PC Richards ** Note: For local stores and pricing of products above, you may need to specify a zip code. The script should be able to take a text file of zip codes and run the search accordingly. The script can run in a batch manner with zip-code text file as a feed. NOTES: 1. For the local online sites, the Script should read zip codes in from a text file and search within a certain radius from that zip code. The text file will contain zip codes delimited by a separator. 2. The Script should crawl all subcategories under the primary categories on each website. This hierarchy should be retained in the database. The script should be intelligent enough to search down each hierarchy of items categorized under the parent category as specified by each individual site. Hierarchical format will be specified in the tables. 3. Data will need to be captured in the respective product tables, including the category, product, and brand tables without duplication of records. See each table for more information. 4. The script should enable another PHP page to call it On-Demand (i.e., initiated when required) and the data parsed into the appropriate tables, without duplication of records. 5. Contractor will agree to and sign the Service Agreement contract. 6. Contractual rate is negotiated for each script/website. Each payment will be contingent upon meeting milestones throughout the project (i.e., delivering working code). The final payment will be made at the end of the entire project dependent on meeting all requirements and conclusion of testing (see Service Agreement for more detail). 7. Contractor must participate in bi-weekly meetings and provide daily status updates. Once a project starts, you must provide daily updates and must be available on a ad-hoc basis for conference calls via Skype and GotoMeeting (already paid for). 8. Contractor will agree to use Mantis to track bugs and features. Account(s) will be created for the user. 9. Scripts must be periodically backed up, uploaded and tested on our servers. Final acceptance of a script is contingent on testing and approval by management. 10. Contractor will provide project timeline, noting milestones of when various scripts and web crawlers will be delivered (unless specified in the Service Agreement). If the contractor anticipates that it will miss the deadline, management must be notified or risk breach of contract. Any delays in the project timeline must be agreed by both parties. 11. Contractor will work closely with PeerShopper™ team members in testing and integrating scripts into existing code. 12. Contractor will provided evidence of previous work using PHP, preferably for developing web crawlers. 13. You must include 1 month of assistance and upgrades on a weekly basis.