Key Responsibilities

• Design, develop, and maintain robust and scalable data pipelines using tools such as Apache Airflow, PySpark, and cloud-native services (e.g., Azure Data Factory, Microsoft Fabric Pipelines).

• Manage data ingestion from APIs, files, and databases into data lakes or data warehouses (e.g., Microsoft Fabric Lakehouse, Iceberg, DWS).

• Ensure seamless data integration across on-premise, cloud, and hybrid environments.

• Implement data validation, standardization, and transformation to ensure high data quality.

• Apply data encryption, masking, and compliance controls to maintain security and privacy standards.

• Collaborate with Data Scientists to deploy ML models and integrate predictive insights into production pipelines (e.g., using Azure Machine Learning or Fabric Notebooks).

• Support AI-powered automation and data insight generation through tools like Microsoft Co-pilot Studio or LLM-powered interfaces (chat-to-data).

• Assist in building lightweight AI chatbots or agents that leverage existing datasets to enhance business efficiency.

Qualifications & Skills

• 3–5+ years of experience in Data Engineering or AI Engineering roles.

• Proficiency in Python, SQL, and big data frameworks (Apache Airflow, Spark, PySpark).

• Experience with cloud platforms: Azure, Huawei Cloud, or AWS.

• Familiar with Microsoft Fabric services: OneLake, Lakehouse, Notebooks, Pipelines, and Real-Time Analytics.

• Hands-on with Microsoft Co-pilot Studio to design chatbots, agents, or LLM-based solutions.

• Experience in ML model deployment using Azure ML, ModelArts, or similar platforms.
• Understanding of vector databases (e.g., Qdrant), LLM orchestration (e.g., LangChain), and prompt engineering is a plus.