Usage of Astronomer Enterprise Edition requires an Astronomer Enterprise Subscription. Start a free 14-day trial.

Preparing

Before installing Astronomer Enterprise’s Airflow module, you’ll need to bring Postgres and Redis that will serve as back-ends to the platform. You can bring these databases with you or run them in Kubernetes via helm package(s). This guide will assume that you want to run via the helm packages, which can be useful for local development at least.

Redis Deploy

Redis is an in-memory data structure store that we will be using as a message broker for Celery.

  1. run helm install stable/redis
  2. Capture the resource URL from successful deployment out
    • The output contains a header titled “NOTES:”
    • In a new text doc save this URL
  3. Follow the output instructions to capture your REDIS_PASSWORD
    • echo $REDIS_PASSWORD
    • Save this in a text doc with the Redis URL
  4. Verify by running helm ls to see the status of your deployment

PostgreSQL Deploy

PostgreSQL is an open-source relational DB that will serve as the back-end to Apache Airflow, Celery and Grafana. There is currently a minor bug in the stable helm chart, so for this install we will be using the Astronomer fork.

  1. git clone https://github.com/astronomerio/charts.git
  2. cd charts/stable
  3. run helm install postgresql
  4. Capture the resource URL from successful deployment output
    • The output contains a header titled “NOTES:”
    • In a new text doc save this URL
  5. Follow the output instructions to capture your PGPASSWORD
    • echo $PGPASSWORD
    • Save this in a text doc with the PostgreSQL URL
  6. Verify by running helm ls to see the status of your deployment

Astronomer Deployment

Use the password that is output by the previous installs to construct connection strings needed to deploy the platform.

Note: If deploying a database in a separate Kubernetes namespace, you’ll need to use fully qualified URLs for the PostgreSQL and/or Redis host. For example, intentional-marmot-postgresql would become intentional-marmot-postgresql.default.svc.cluster.local where default is the database’s Kubernetes namespace.

Verification

  1. Navigate to your Kubernetes Dashboard
  2. Change your namespace on the left-side panel from default to astronomer-system
  3. Verify that all your pods are all-green