本文所涉及完整配置文件:https://github.com/KingFalse/veronica/blob/main/airflow2.0/docker-compose.yml
2020年12月17日airflow2.0发布,一直没有时间去更新,今天记录一下安装流程.
首先是docker-compose.yml
文件,与1.x版本没多大变化,我遇到的主要是两个命令有变化:airflow upgradedb -> airflow db upgrade
跟 airflow create_user -> airflow users create
version: "3.7" # 完整文件: https://github.com/KingFalse/veronica/blob/main/airflow2.0/docker-compose.yml # 启动一个airflow实例并创建用户:admin 密码:admin123 x-airflow-env: &airflow-env AIRFLOW__CORE__DEFAULT_TIMEZONE: system #解决UTC时区问题 AIRFLOW__CORE__EXECUTOR: LocalExecutor #加载示例任务 AIRFLOW__CORE__LOAD_EXAMPLES: 'True' AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: 'False' AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgres+psycopg2://airflow:www.kagura.me@postgres:5432/airflow AIRFLOW__CORE__FERNET_KEY: 81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs= AIRFLOW_CONN_METADATA_DB: postgres+psycopg2://airflow:www.kagura.me@postgres:5432/airflow AIRFLOW_VAR__METADATA_DB_SCHEMA: airflow AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC: 10 AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 60 # dags目录自动刷新时间,60秒 AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: system #解决UTC时区问题 AIRFLOW__WEBSERVER__RBAC: 'True' AIRFLOW__WEBSERVER__AUTHENTICATE: 'True' AIRFLOW__WEBSERVER__AUTH_BACKEND: airflow.contrib.auth.backends.password_auth #启用密码认证登陆 AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'True' x-airflow-vol: &airflow-volumes - /etc/localtime:/etc/localtime - ./dags:/opt/airflow/dags services: postgres: image: postgres:12 container_name: airflow_postgres_1 restart: always environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=www.kagura.me - POSTGRES_DB=airflow volumes: - ./pgdata:/var/lib/postgresql/data/ ports: - "5433:5432" scheduler: image: apache/airflow container_name: airflow_scheduler_1 restart: always depends_on: - postgres - webserver environment: <<: *airflow-env ports: - "8793:8793" volumes: *airflow-volumes command: scheduler healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3 webserver: image: apache/airflow container_name: airflow_webserver_1 hostname: webserver restart: always depends_on: - postgres environment: <<: *airflow-env volumes: *airflow-volumes ports: - "8080:8080" entrypoint: bash -c " airflow db upgrade && airflow users create -r Admin -u admin -e admin@example.com -f False -l King -p admin123 && pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/ && airflow webserver " healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 32
现在你已经安装好了一个开箱可用的airflow.打开浏览器:http://127.0.0.1:8080/
应该能看见airflow登陆页面. 用户名:admin
密码:admin123
,直接登陆即可使用
设置容器pip镜像
docker exec -u root airflow_webserver_1 pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/
docker exec -u root airflow_scheduler_1 pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/
安装其他的pip包
docker exec -u root airflow_webserver_1 pip3 install pymysql
docker exec -u root airflow_scheduler_1 pip3 install pymysql
设置容器apt镜像
docker exec -u root airflow_webserver_1 sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list
给容器安装apt包
docker exec -u root airflow_webserver_1 apt install -y git
安装airflow插件(在线代码编辑器)
docker exec -u root airflow_webserver_1 pip3 install airflow-code-editor
未经允许不得转载:鹞之神乐 » docker-compose安装airflow 2.0纪要