Astronomer Enterprise Edition is an enterprise-grade data engineering platform that allows any developer to jump to the task of creating and managing data pipelines very quickly. The platform includes a Heroku-like CLI capability to deploy pipelines, as well as a New Relic-like interface to monitor and troubleshoot running pipelines.
The platform leverages leading open source tools including Kubernetes, Airflow, Prometheus, and Grafana — and also includes a library of open-source Airflow connectors that we have been building and curating for the past year.
We’ve build out pre-configured Docker containers w/ Celery and metrics/monitoring. The Astronomer Airflow module consists of seven components, and you must bring your own Postgres and Redis database.
This kit is fully open-sourced (Apache 2.0) and you can experiment with it at https://open.astronomer.io/.
Astronomer Enterprise makes it easy to deploy these containers to Kubernetes - but more importantly, to give Airflow developers a CLI to deploy DAGs through a private Docker registry that interacts with the Kubernetes API.
Remember to run
astro airflow init after creating a new project directory.
Any Python packages can be added to
requirements.txt and all OS level packages can be added to
packages.txt in the project directory.
We’ll also make it easy to use the Airflow CLI remotely (i.e. run commands from your local terminal that execute in the cloud Airflow).
Other community contributions
Building the Documentation
Documentation is built on Jekyll and hosted on Google Cloud Storage.
Build the docs site locally:
cd docs bundle install
bundle exec jekyll serve