We have built a CLI to get you up and running with Airflow as quickly as possible. This guide details how to create, setup and run your Airflow project.
Guide requirements are
- astro-cli installed
- an Airflow DAG you wish to run
Creating Your Project
Before running your first DAG you will need to initialize your Astronomer EE Airflow project.
Creating A Project Directory
Create a project directory in your desired root directory.
Change into the root development directory on your machine. This is often the directory where you have chose to house various development repositories on your computer. If you don’t have one yet, we suggest creating a “dev” directory in your home path
Create a project directory. This directory name will become the name of your Astronomer EE Project. The best project names will indicate the department (or if you’re lean like us, the organization) and the purpose of the project. Using this example, Astronomer may then name their project
Change into the newly created project directory
Now that you are in your project directory, you will need to initialize the project. This command will scaffold out the needed files and default configuration for your project.
astro airflow init
You will now see the following files and folders
- .astro/ - .dockerignore - Dockerfile - dags/ - include/ - packages.txt - plugins/ - requirements.txt
Importing Your Project
You have initialized your Astronomer Airflow project and now you can begin to build your project from scratch or import and existing one.
Directed acyclic graphs (DAG) are the configuration for your workflows and a core component of an Airflow Project. If you are migrating an existing Airflow project you likely have several DAG files you wish to import. You can place all DAG files into the
dags/ directory. They will be imported to your Docker image when you deploy or test locally. This directory gets added to your Docker image in the
The Airflow Plugin system can be used to make managing workflow logic easier or even expand the functionality of Airflow itself. If you have any plugin requirements or would like to bring in a plugin from airflow-plugins, they can be put into the
plugins/ project directory. As in the case of the
dags/ directory, this directory will get added to your docker image in the
In order to keep our images lightweight, we ship with only the Python standard lib. When you find yourself needing modules not included in the standard lib, you can modify
requirements.txt. Your requirements file can be used to add additional Python requirements in the same way a standard Python requirements file works.
Running Your Project Locally
We have built tooling to make development and testing of your Airflow projects as simple as possible. From your root project directory you can start your Airflow project.
astro airflow start
This command will inject any dependencies discussed above into the Docker image and then run that Docker image locally. After a successful build you will see hyperlinks to your local Airflow resources.