Cloud Data Engineer
Job Description
Design and Development:
- Design, develop, and maintain scalable ETL pipelines using cloud-native tools (AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, etc.).
- Architect and implement data lakes and data warehouses on cloud platforms (AWS, Azure, GCP).
- Develop and optimize data ingestion, transformation, and loading processes using Databricks, Snowflake, Redshift, BigQuery and Azure Synapse.
- Implement ETL processes using tools like Informatica, SAP Data Intelligence, and others.
- Develop and optimize data processing jobs using Spark Scala.
Responsibilities Duties:
Data Integration and Management:
- Integrate various data sources, including relational databases, APIs, unstructured data, and ERP systems into the data lake.
- Ensure data quality and integrity through rigorous testing and validation.
- Perform data extraction from SAP or ERP systems when necessary.
Key Skills:
Performance Optimization:
- Monitor and optimize the performance of data pipelines and ETL processes.
- Implement best practices for data management, including data governance, security, and compliance.
Experiance Qualifications:
Collaboration and Communication:
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Collaborate with cross-functional teams to design and implement data solutions that meet business needs.
Documentation and Maintenance:
- Document technical solutions, processes, and workflows.
- Maintain and troubleshoot existing ETL pipelines and data integrations.
Benefits:
Training, health, insurance, commuting support, lunch service etc.