Job Summary:

We are looking for a skilled Data Engineer to join our team and help build and maintain our data infrastructure. The ideal candidate will be responsible for designing, implementing, and managing our data processing systems and pipelines. You will work closely with data scientists, analysts, and other teams to ensure efficient and reliable data flow throughout the organization.

Key Responsibilities:

1. Design, develop, and maintain scalable data pipelines for batch and real-time processing

2. Implement ETL processes to extract data from various sources and load it into data warehouses or data lakes

3. Optimize data storage and retrieval processes for improved performance

4. Collaborate with data scientists and analysts to understand their data requirements and provide appropriate solutions

5. Ensure data quality, consistency, and reliability across all data systems

6. Develop and maintain data models and schemas

7. Implement data security measures and access controls

8. Troubleshoot data-related issues and optimize system performance

9. Stay up-to-date with emerging technologies and industry trends in data engineering

10. Document data architectures, pipelines, and processes

Requirements:

1. Bachelor's degree in Computer Science, Engineering, or a related fields 2. 2-4 years of experience in data engineering or similar roles 3. Strong programming skills in Python, Java, or Scala 4. Proficiency in SQL and experience with relational databases (e.g., Databrick, PostgreSQL, MySQL) 5. Familiarity with cloud platforms (AWS, Azure, or Airflow) and their data services 6. Knowledge of data warehousing concepts and ETL best practices 7. Experience with version control systems (e.g., Git) 8. Understanding of data cleansing, data modeling and database design principles 9. Solid problem-solving skills and attention to detail 10. Good communication skills and ability to work with technical and non-technical team members.

Preferred Qualifications:

1. Experience with Azure data platform (ADF, Databrick) 2. Familiarity with data visualization tools (e.g., Tableau, Power BI) 3. Knowledge of stream processing technologies (e.g., Kafka, API, Google Big Query, MongoDB, SFTP sources) 4. Experience with containerization technologies (e.g., Docker)

5. Experience to deal with large data and optimization skill in development

6. Understanding of machine learning concepts and data science workflows