Lead Data Engineer

Job Description

We are seeking a highly skilled Senior Data Engineer with 9+ years of 100% hands-on experience building and maintaining enterprise-grade data pipelines. This is a pure Individual Contributor role focused on writing production-quality code, developing scalable ETL/ELT solutions using PySpark and AWS, and orchestrating workflows with Airflow. If you thrive on solving complex technical problems and shipping robust, well-tested code, this role is for you.
Key Responsibilities
•    Develop and maintain robust, scalable ETL/ELT pipelines using PySpark on AWS EMR
•    Build data ingestion and transformation workflows from diverse sources (S3, EMR, RDS, Kafka, APIs) into AWS-based data lakes and warehouses
•    Write clean, modular, testable Python code following best practices and coding standards
•    Implement comprehensive unit tests using pytest/unittest with mocking, fixtures, and high code coverage
•    Design and build production-grade Airflow DAGs for workflow orchestration, scheduling, and monitoring
•    Optimize Spark jobs for performance, memory efficiency, and cost reduction
•    Implement CI/CD pipelines for automated testing and deployment using Jenkins, GitHub Actions, or AWS CodePipeline
•    Troubleshoot and debug complex data pipeline issues in production environments
•    Collaborate with Data Scientists, Analysts, and Platform Engineers to deliver data solutions
•    Ensure data quality, security, and compliance standards are met
Required Skills & Qualifications
•    9+ years of hands-on data engineering experience (no management responsibilities required)
•    Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
•    Expert-level Python programming – OOP, design patterns, clean code practices
•    Advanced PySpark/Spark skills – partitioning strategies, shuffle optimization, memory tuning, broadcast joins
•    Strong unit testing expertise using pytest/unittest – mocking, parametrized tests, fixtures, TDD mindset
•    Hands-on Airflow experience – DAG design, custom operators, sensors, XComs, debugging failed tasks
•    Deep AWS experience: S3, EMR, Glue, Redshift, Lambda, Step Functions, IAM, CloudWatch
•    Solid understanding of data lake and warehouse architectures (medallion architecture, Delta Lake)
•    Strong SQL skills – complex queries, window functions, query optimization
•    Proficiency with Git, code reviews, and collaborative development workflows
•    Experience with CI/CD pipelines and automated testing frameworks

Nice to Have (Preferred)
•    Familiarity with Docker for containerized data workloads
•    Exposure to streaming data (Kafka, Spark Streaming)
•    Knowledge of data quality frameworks
•    Background in financial services or regulated industries
•    Understanding of data security and privacy practices (GDPR)

Mandatory Competencies