Airflow Orchestration And Ingestion Engineer
Job Description
Workflow Migration:
- Analyze and convert Oozie workflows into Airflow DAGs using Python-based orchestration.
- Design and implement reusable, modular, and optimized Airflow pipelines for data ingestion, transformation, and orchestration.
- Maintain a one-to-one mapping between legacy workflows and Airflow DAGs, ensuring no data loss or business interruption.
Responsibilities Duties:
Cloud Data Migration:
- Collaborate with data engineering teams to migrate Cloudera Hadoop workloads to Databricks on AWS.
- Leverage Airflow for scheduling and orchestrating data workflows on AWS-based services (e.g., S3, EMR, Glue, Redshift).
Key Skills:
Pipeline Optimization:
- Optimize data ingestion pipelines to achieve high throughput and low latency on AWS cloud infrastructure.
- Integrate Airflow workflows with Databricks for data transformations and analytics.
Experiance Qualifications:
Error Handling and Monitoring:
- Implement robust error-handling mechanisms, task retries, and alerting within Airflow workflows.
- Set up monitoring dashboards using tools like CloudWatch, Prometheus, or Airflow’s built-in features.
Benefits:
Training, health, insurance, commuting support, lunch service etc.