Overview
Explore Apache Airflow, an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have platform as a support. Gain an in-depth learning session to understand what Airflow is, and how it works and deep dive into the advanced concepts - such as learning how to create plugins and make real dynamic pipelines.
Key Takeaways
At the end of this programme, you will be able to:
- Create plugins to add functionalities to Apache Airflow
- Use Docker with Airflow and different executors
- Master core functionalities such as DAGs, Operators, Tasks, Workflows and etc
- Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs
- Identify the difference between Sequential, Local and Celery Executors, understand how do they work and how you can use them
- Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc
- Install and configure Apache Airflow
- Think, answer and implement solutions using Airflow to real data processing problems
Who Should Attend
Please refer to the job roles section.
Prerequisites
Some prior programming or scripting experience. Python experience will help but not a must.
Programme Structure
This programme will cover the following topics:
- Getting Started with Airflow
- The important views of the Airflow UI
- Coding your first data pipeline with Airflow
- The new way of scheduling
- Databases and Executors
- Implementing advanced concepts in airflow
- Creating Airflow Plugins with Elasticsearch and PostgreSQL
Refer to the programme page for further details on the content provider(s).
NOTE
This self-paced e-learning course is accessible by all Public Service Officers via this link.
This programme is complimentary.
Learners are required to register via the LEARN Portal
Step-by-Step Guide
1. Login to LEARN Portal
2. If using laptop:
- Type module title, e.g. The Complete Hands-On, in the search bar.
- Click on title.
- Click on "Start Learning" to proceed.
3. If using mobile:
- Tap on the “Discover” tab.
- Type module title, e.g. The Complete Hands-On, in the search bar.
- Tap on title.
- Tap on "Start Learning" to proceed.
Note that the process for accessing CSC LEARN modules is generic.
