Usage of Astronomer Enterprise Edition requires an Astronomer Enterprise Subscription. Start a free 14-day trial.

Overview

Astronomer Enterprise Edition is an enterprise-grade data engineering platform that allows any developer to jump to the task of creating and managing data pipelines very quickly. The platform includes a Heroku-like CLI capability to deploy pipelines, as well as a New Relic-like interface to monitor and troubleshoot running pipelines.

The platform leverages leading open source tools including Kubernetes, Airflow, Prometheus, and Grafana — and also includes a library of open-source Airflow connectors that we have been building and curating for the past year.

Architecture

We’ve build out pre-configured Docker containers w/ Celery and metrics/monitoring. The Astronomer Airflow module consists of seven components, and you must bring your own Postgres and Redis database.

Airflow Module

This kit is fully open-sourced (Apache 2.0) and you can experiment with it at https://open.astronomer.io/.

DAG Deployment

Astronomer Enterprise makes it easy to deploy these containers to Kubernetes - but more importantly, to give Airflow developers a CLI to deploy DAGs through a private Docker registry that interacts with the Kubernetes API.

Components that glue this all together include Phoenix and Commander.

Airflow Deployment

Remember to run astro airflow init after creating a new project directory.

Any Python packages can be added to requirements.txt and all OS level packages can be added to packages.txt in the project directory.

Additional RUN commands can be added to the Dockerfile. Environment varaibles can also be added to ENV.

Astronomer CLI

The Astronomer CLI is under very active development to support Airflow-related commands.

Airflow CLI

We’ll also make it easy to use the Airflow CLI remotely (i.e. run commands from your local terminal that execute in the cloud Airflow).

Other community contributions

We’re building out a library of Airflow Plugins and we’re also doing an Airflow podcast.

Building the Documentation

Documentation is built on Jekyll and hosted on Google Cloud Storage.

Build the docs site locally:

cd docs
bundle install

Run it:

bundle exec jekyll serve