Steps to set up airflow

10/8/2017 - 6:39 AM

1. Setting up virtualenv 
virtualenv env

2. activate it using:
source ./env/bin/activate

3. pip install airflow # will install airflow version 1.8 to use latest one use: pip install apache-airflow

4. Add "export AIRFLOW_HOME=~/<directory>" to your bashrc/zshrc
echo "export AIRFLOW_HOME=/home/mahendra/pyconf/app_airflow" >> .zshrc

5. initialize the database
airflow initdb

6. start the web server
airflow webserver -p 8080

7. Start the scheduler
airflow scheduler

8. Add the dag 

Dag link: https://gist.githubusercontent.com/userimack/1c5b8ae12506d03025669328ee21b303/raw/886d5165d45ea5e91f203775e06f27fb4175517f/download_data.py

Download script Link: https://gist.githubusercontent.com/userimack/6fe752ecba8df9374e32d44e3c7ca1b8/raw/f36849eb90569cfa30691e4c0bad33b8031fe9bf/download.sh

Enjoy :)

Cacher is the code snippet organizer for pro developers

We empower you and your team to get more done, faster

Steps to set up airflow