Setup Airflow + sample DAG + simple Flask Server on Google Kubernetes Engine cluster

Setup Airflow on Google Kubernetes Engine cluster:

You need to have Airflow deployement experience and Google Kubernetes Engine experience

- Airflow 1.10.2 or newer

- Mysql DB 5.7

- Executor: KubernetesExecutor

- 1 cluster

- 1 node (to be confirmed)

I want to use pure Airflow and open source libraries. I want to have the option to move the whole project to another provider, so I prefer not to use GKEPodOperator and similar Google specific things unless it is the only option.

Part A

Prepare necessary configuration files (yaml, docker-compose, helm,..., airflow config file) + full instruction to deploy airflow on GKE (assume 2 options: cluster exists or new cluster, so instructions should include cluster creating and setup on GKE) with Mysql DB (docker file will be provided by me) and KubernetesExecutor. Need to have persistence, logging, Ingress controller/LoadBalancer, port exposure inside and to outside. I should be able to access the airflow webserver to manually run dags

- 1 pod will contain the airflow webserver and Airflow scheduler and the mysql database (with persistence, Stateful Set) and mongoDB container (docker file will be provided by me ) or use 2 pods (one for airflow and one for Mysql and MongoDB)

- 1 pod for every task instance. Workers are created in containers dynamiccaly and disappear when the task/dag is finished

- DAG sync mode: PersistentVolume

- 3 PersistentVolume: 1 for logs, 1 for Dags and plugins, 1 for databases (mysql and mongoDB) or 1 persistent volume with 3 sum folders (one for each)

- Ability to trigger DAG runs with Airflow REST API

- Ability to send variables (in json/dict format) and data (in pandas Dataframe format) when calling a Dag or between tasks inside a dag

- ability to access the Airflow Webserver thought the internet (authentication required)

Part B

Create a sample dag with 3 tasks (1 python operator, 1 bash operator and 1 KubernetesPodOperator) to demo the process (should use XCOM and a sample code in plugin to demo how to import external python scripts into a dag)

Create instructions and code to test the Dag (trigger it) using Airflow REST API and from some python code running in a container deployed on GKE (Will be used in Part C)

Create a new DB inside Mysql ("myUsers) to hold users data and user management: need 3 types of users: basic, premium and admin. create one user for each type. sql schema to be provided by me.

Part C

Create and deploy (statefulSet, port exposure) a sample Flask server "[login to view URL]" in a docker container with routes:

- "login" to verify user against DB myUsers

- "addUser" and "deleteUser" available only for admin

- "saveFile" to store a file in "toProcess/[username]" folder under the PersistentVolume (using FTP)

- "trigger_dag" (variable "dagName") that will trigger a specific dag either by using Airflow Rest API or using python/bash scripts (both should be supported, the one to use will be a setting/variable inside "[login to view URL]" )

No HTML files to be returned from this Flask server, only data (in json format) and return code.

Create a simple python code ([login to view URL]) that uses python requests to test all functions of airflowFlaskApp. It can be run from anywhere (my pc for example). It should test saveFile with a csv or excel file.


Other details and info required will be discussed as needed

All code should be documented (functions should have comments explain all variables and return values, and main part of the code).

Python 3.6+ should be used

All python code should have [login to view URL] using pipreqs

Instructions should include how to update code without stopping the server (on GKE)

All access to the internet (outside the cluster) should be secure: need to create necessary secure connections and instructions to setup (certificates needed)

other skills required: Airflow, Flask, Docker, Kubernetes, Google Kubernetes Engine, MYSQL

Skills: Python, Docker, Kubernetes

See more: simple linux server, set simple linux server tasks, simple echo server, simple web server javame, setup live streaming video windows 2003 server, simple echo server java application, simple http server post, simple client server socket application, sample share point portal server 2003 home page, simple chat server, setup simple exchange server outlook, smartfox server google app engine, tcp server google app engine, google kubernetes engine pricing, flask on google compute engine, google kubernetes engine master, google kubernetes engine iap, google kubernetes engine, google kubernetes engine tutorial

About the Employer:
( 4 reviews ) Beirut, Lebanon

Project ID: #22365358

7 freelancers are bidding on average $146 for this job


Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl More

$250 USD in 7 days
(30 Reviews)

I am a devops expert and I can install and configure airflow in gke, test it with sample program. Looking forward to work for you. Thank you.

$150 USD in 10 days
(3 Reviews)

Hi, I am a Certified Kubernetes Administrator and with my vast experience I can help you in setting up Airflow and other sample applications on a k8s cluster. I've 4+ years of experience working on Kubernetes, Python A More

$88 USD in 7 days
(6 Reviews)

Hi there Harry, I am a experts developer, having 8+ years experience developed 640+ websites/systems. Not only that, we’re also UI designers (FULL STACK). My best skills: PHP, Wordpress, PHP Codeigniter, PHP Laravel, More

$140 USD in 7 days
(0 Reviews)

Hello, I'm Kelvin, I can provide you perfect work as you looking for . we can discuss more in chat . I will be waiting for your response . I have very great experience about your project , if you want discuss more ple More

$140 USD in 7 days
(0 Reviews)

I am a devops engineer with experience in Python and cloud environments. I would be happy to help you build out your cloud environment

$111 USD in 7 days
(0 Reviews)

Please click on the "CHAT" button so that I can ask a few questions concerning your project. I will provide an exceptional quality project on time, leaving you fully satisfied that you got your money’s worth!

$140 USD in 5 days
(1 Review)