Usage of Astronomer Enterprise Edition requires an Astronomer Enterprise Subscription. Start a free 14-day trial.

Objective

We have built a CLI to get you up and running with Airflow as quickly as possible. This guide details how to create, setup and run your Airflow project.

Requirements

Guide requirements are

Creating Your Project

Before running your first DAG you will need to initialize your Astronomer EE Airflow project.

Creating A Project Directory

Create a project directory in your desired root directory.

Change into the root development directory on your machine. This is often the directory where you have chose to house various development repositories on your computer. If you don’t have one yet, we suggest creating a “dev” directory in your home path ~/dev/.

cd [EXAMPLE_ROOT_DEV_DIR]

Create a project directory. This directory name will become the name of your Astronomer EE Project. The best project names will indicate the department (or if you’re lean like us, the organization) and the purpose of the project. Using this example, Astronomer may then name their project astronomer-airflow.

mkdir [EXAMPLE_PROJECT_NAME]

Change into the newly created project directory

cd [EXAMPLE_PROJECT_NAME]

Project Initialization

Now that you are in your project directory, you will need to initialize the project. This command will scaffold out the needed files and default configuration for your project.

astro airflow init

You will now see the following files and folders

- .astro/
- .dockerignore
- Dockerfile
- dags/
- include/
- packages.txt
- plugins/
- requirements.txt

Importing Your Project

You have initialized your Astronomer Airflow project and now you can begin to build your project from scratch or import and existing one.

DAGs

Directed acyclic graphs (DAG) are the configuration for your workflows and a core component of an Airflow Project. If you are migrating an existing Airflow project you likely have several DAG files you wish to import. You can place all DAG files into the dags/ directory. They will be imported to your Docker image when you deploy or test locally. This directory gets added to your Docker image in the $AIRFLOW_HOME directory.

Plugins

The Airflow Plugin system can be used to make managing workflow logic easier or even expand the functionality of Airflow itself. If you have any plugin requirements or would like to bring in a plugin from airflow-plugins, they can be put into the plugins/ project directory. As in the case of the dags/ directory, this directory will get added to your docker image in the $AIRFLOW_HOME directory.

Requirements

In order to keep our images lightweight, we ship with only the Python standard lib. When you find yourself needing modules not included in the standard lib, you can modify requirements.txt. Your requirements file can be used to add additional Python requirements in the same way a standard Python requirements file works.

Running Your Project Locally

We have built tooling to make development and testing of your Airflow projects as simple as possible. From your root project directory you can start your Airflow project.

astro airflow start

This command will inject any dependencies discussed above into the Docker image and then run that Docker image locally. After a successful build you will see hyperlinks to your local Airflow resources.