aditi5
2/24/2020 - 10:02 AM

Workflows concepts

apache_airflow.md
content_copyfile_download
  • Rendered
  • Source
  • Benefits of airflow:
    • DAGs
      • Dags are a way to setup workflows, they can setup a sequence of operations that can be individually retried on failure and restarted where the operation failed. Dags provide a nice abstraction to a series of operations.
    • Programmatic Workflow Management:
      • Airflow provide a way to setup programmatic workflows, Tasks for instances can be generated on fly within a dag. While Sub-DAGs and XCom, allows to create complex dynamic workflows.
  • command
  • initdb
  • 1 - meta database of airflow initialization. All meta data related to airflow and also related to
  • your data pipelines
commands
content_copyfile_download
(.sandbox) airflow@4cd0ceac9940:~/airflow$ history
    1  python -m venv .sandbox
    2  source .sandbox/bin/activate
    3  wget https://raw.githubusercontent.com/apache/airflow/master/requirements/requirements-python3.8.txt
    4  wget https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.8.txt -O requirements-python3.8.txt
    5  ls
    6  pip install "apache-airflow[crypto,celery,postgres,kubernetes,docker]"==1.10.12 --constraint ./requirements-python3.8.txt
    7  ls
    8  airflow initdb
    9  ls
   10  cd airflow/
   11  ls
   12  head airflow.cfg 
   13  airflow scheduler &
   14  airflow webserver &
   15  history



- Docker image build from docker file :
- docker build -t airflow-basic .
- sudo docker run --rm -d -p 8080:8080 airflow-basic
  -- rm : remove docker container as soon as container is stopped 
    -d: run container as deamon 
    - p expose 8080 port to 8080 on local


Enter a docker container with container ID:
sudo docker exec -it add52d26edfd /bin/bash