Closed

create example scripts using celery and rabbidmq to run different tasks on different remote nodes

I'm creating a distributed system. But I'm not aware of Celery/RabbitMQ setups.

My plan is to create 5 different workers with different tasks (example):

- Worker1 - send icmp request

- Worker2 - scrap a url content

- Worker3 - execute a whois on a domain

- Worker4 - get a list of N urls from "somewhere" (could be the same rabbitmq?) and fetch the title tag.

- Worker5 - fetch all ips related with those 300 urls.

When the celery client send a task request through the broker:

- I want to scrap this url: a.b.c

The worker2 should respond and execute the task, and return the result back to the celery client. All the other workers should never receive this request, as those don't have the proper function/task.

When the celery client send a task request through the broker:

- I want to fetch the title tag from a list of 230 or 1000 websites

The Worker4 should respond, fetch the list of websites, execute the task and return the result back to the celery client.

Each worker runs on a different linux box on the internet.

I will also need the ability of have a pool of worker1 boxes. Let's say 5 linux boxes with the same worker1 software, and the first to respond will perform the task (Round-robin).

The celery client must be a Flask page with:

- a table with 10 domains and 5 urls

- When the user click on one of those domains, and hit the button 'run whois', the celery task must be triggered

- When the user select 3 of those urls and select the option "fetch title", the celery task must be triggered

The response must also be presented via flask webapp (can be a different page)

Skills: Python, DevOps, Data Scraping, Scripting

See more: create flyers trifolds using adobe photoshop, create floor plan using autocad, create online admission using php javascript html, create dyanamic website using php, create website voting using java mysql tomcat, web accounting example scripts, web banking example scripts php mysql, create contact form using formailpl, create simple banner using css, create registration form using mysql, create report php using pdf, create image swapping using flash, create xml file using mysql, vlc example scripts, create acknowledgement banner using scripts, example jboss create web service using annotation, example php code create order report using fpdf, create ear file using maven example, how to run selenium ide scripts using java eclipse, provide example scripts for automating the ui testing of a video player like youtube using selenium

About the Employer:
( 0 reviews ) Brazil

Project ID: #28332447

16 freelancers are bidding on average $544 for this job

(15 Reviews)
5.6
benliao

Hello I have read your job description carefully. Also, I have rich experience in python and web scraping So I am interested in and I am confident in your project. I always provide perfect results to my clients as you More

$500 USD in 8 days
(6 Reviews)
4.2
DRG1990

Hi, I summarizes all descriptions and provides clear and concise explanations. I am an expert of Scraping with Python experience. I have done several similar projects in the backend development of numerous internet s More

$250 USD in 5 days
(5 Reviews)
4.1
(8 Reviews)
4.2
braincenter

Hello, Hope this message finds you well, I checked your details and I believe that my experience is what you are looking 4. I have been working on similar projects for the past eight years, and I have the essential sk More

$500 USD in 7 days
(10 Reviews)
4.3
kholmogorovadev

Hi. I am a Flask expert I know Celery/RabbitMQ I can help you as well I'll provide you good result Please contact me best regards Aleksandra

$500 USD in 7 days
(3 Reviews)
2.9
sharmashivangi00

Hello, Hope you are doing well..!! I am a Full Stack Developer with over 4 years of experience in Python backend frameworks like Django and Flask along with the frontend frameworks like React.js and Angular. I have b More

$500 USD in 7 days
(3 Reviews)
3.3
rostikhr111

Hi, How are you? I am very interested in your project. I have rich experience in web development using Flask and Django. Especially I have rich experience in Schedule, Asynchronous using Celery, AsyncIO. I am sure th More

$500 USD in 7 days
(1 Review)
2.7
nikolai198061

Hi, client. I am a senior Python developer who has rich experiences. Especially, I have professional experiences in Web Scraping. I've successfully done a lot of projects to scrape data like email, URL, ads, contact, p More

$500 USD in 7 days
(3 Reviews)
2.8
AndreiiWebGats

Hello, Nice to meet you. I am very pleased to present this proposal for your consideration as a fullstack developer. I have deep knowledge in real-time data processing and experience in RabbitMQ, Redis, socket.io. One More

$500 USD in 7 days
(2 Reviews)
2.9
demydkasilov

Hi, sir I hope you are doing well and staying safely without COVID-19 effect. I have carfully checked your requirement. I am a web scraping expert and have good experience with scraping using python, node.js and php fo More

$500 USD in 7 days
(2 Reviews)
2.4
Rustamdev

Hello. I have rich experiences in python scraping. So, if you hire me, I will do my best for you and for your project. Thanks. Rustam.

$250 USD in 7 days
(1 Review)
1.8
stepinnsolution

Hello man. I am a full stack developer and have around about 12+ years of experience in the development field. I am the CEO of Stepinnsolution. Stepin Solution to award leading the digital world. I want to tell you We More

$1500 USD in 7 days
(1 Review)
0.6
iwtechnocrats41

HI, Hope you are well! I have gone through your requirement properly and our company's main policy is to create and maintain long-term relationships rather than earning profit. This would be a win-win situation for More

$500 USD in 30 days
(1 Review)
0.0
idevenster

Can I have a go at your project? I have been doing Python, Java, Node Js for the past 10 years. I think I can help you out. I don't need money as such for now

$500 USD in 3 days
(0 Reviews)
0.0
shanarana001

Hello successful buyer I have go through your request and it is fully understand and i can do it perfectly and the work will be done professionally and perfectly because am an expert in this field Kindly contact me now More

$250 USD in 7 days
(0 Reviews)
0.0