Hi.
We are working on a new web crawling platform and what you need can be easily handled. Let me know and we can discuss more, here are some details:
OK, let me describe just a little bit what is the crawling platform about. At the moment if we want to make a crawler we go to freelancer website or using some other crawling platform and usually a lot of computer power resources and eventually, money is used. Now, to reduce the costs of crawling web pages(including dynamically generated pages, like JavaScript) we want to use a network of personal computers connected together and processing web pages. We thought about a solution where anyone can connect his or her personal computer to this network and get paid for the automatic rendering web pages. So, we have 2 players on this platform: webmaster(or anyone which want to create a spider to get data) and clients(which will connect to the network and process web pages). So, clients can make money by processing web pages. We liked this idea and made some tests. Already wrote a simple application which clients can install and run on their computers in the background. This application is really smart in a lot of ways: can render JavaScript pages, can filter out unused data like CSS, Adds, Images. This smart filtering makes the web rendering really fast. In the end, we want to make a marketplace for web spiders. Soon we will have a landing page and more info about this platform.
Thx and let me know if interested to work on this.