Meaningful work,
meaningful life

Job Description:

Position Overview

We are seeking an experienced Azure Data Engineer with 6–8 years of hands-on experience in designing, developing, and optimizing large-scale data platforms on Microsoft Azure. The ideal candidate will possess deep expertise in Azure Databricks, Azure Data Factory, Delta Lake, and modern data engineering practices. This role involves building scalable data pipelines, enabling advanced analytics, and ensuring efficient data availability for business and AI-driven use cases.

Key Responsibilities

Data Engineering & Pipeline Development

· Design, build, and maintain end-to-end data pipelines using Azure Databricks (PySpark/Scala), ADF, and Delta Lake.

· Implement ETL/ELT workflows with robust logging, monitoring, orchestration, and error-handling capabilities.

· Develop high-performance data ingestion frameworks for structured, semi-structured, and unstructured data.

Data Modeling & Architecture

· Design and maintain data models, including Bronze–Silver–Gold architectures on the Lakehouse.

· Contribute to data lake and data warehouse architecture leveraging Azure Synapse, Delta Lake, and Databricks Unity Catalog.

· Optimize data storage and compute costs using best practices in partitioning, caching, and workload management.

Databricks Expertise

· Write scalable and optimized PySpark/Scala code for transformations and batch/streaming workloads.

· Implement Delta Lake features (ACID transactions, time travel, optimization, Z-order, VACUUM).

· Work with MLflow, Databricks job clusters, workflows, and CI/CD integration.

Azure Platform & Integration

· Integrate data pipelines with Azure Data Factory, Azure Functions, Event Hub, Service Bus, and other Azure services.

· Manage environments through Infrastructure as Code (IaC) using Terraform, ARM, or Bicep.

· Configure and maintain data security and governance using Azure AD, Key Vault, Purview, and role-based access.

Performance Optimization & Quality

· Ensure high-performance processing and troubleshoot issues related to cluster performance, cost, and scalability.

· Implement data quality frameworks, unit testing, and observability through tools like DQ Frameworks, Databricks expectations, or Great Expectations.

Collaboration & Stakeholder Interaction

· Work closely with data architects, analysts, and business stakeholders to understand requirements and deliver data solutions.

· Participate in Agile ceremonies, sprint planning, and documentation.

· Provide thought leadership by recommending modern tools, best practices, and optimization strategies.

Required Skills & Qualifications

Technical Skills

· 6–8 years of experience in data engineering, with at least 3+ years in Azure Databricks.

· Proficiency in PySpark/Scala, SQL, Delta Lake, and Databricks workflows.

· Strong experience with Azure Data Factory for orchestration and pipeline management.

· Hands-on experience with Azure Data Lake Storage Gen2, Azure Synapse (SQL Pools or Serverless), Event Hub, and Key Vault.

· Solid understanding of data warehousing, distributed computing, and Lakehouse principles.

· Experience building and maintaining CI/CD pipelines using Azure DevOps or GitHub Actions.

· Knowledge of cloud security, encryption, access management, and compliance standards.