Test Automation Data - Senior Engineer

Job Description

Design and develop automation scripts for:
API testing (REST/JSON)
ETL pipeline validation
Data transformation checks in Databricks
Build and maintain reusable automation components using Python, PySpark 
Automate regression suites covering DBT model outputs, schema validations, and upstream/downstream data layers.
Implement automated data validation across:
Databricks Delta Lake
SQL Server sources
AWS S3 raw layers

Data Engineering Test Automation
•    Automate validation of DBT transformations (tests, snapshots, seed data checks).
•    Build SQL-based and script-based automation for: 
o    Data reconciliation
o    Aggregation validation
o    Schema evolution testing
o    Data freshness checks
•    Use Databricks APIs or automation tools to validate notebook runs and workflows.
API & Integration Automation
•    Develop API automation scripts for: 
o    Data ingestion
o    Data consumption
o    Metadata services
•    Use tools like Postman/Newman or Python requests for automated Web/API testing.
CI/CD Integration
•    Integrate automation suites with: 
o    AWS CodePipeline / GitHub Actions / Jenkins / GitLab CI
•    Configure pipelines to run tests automatically for every code push, DBT model change, or Databricks workflow update.
Agile Collaboration
•    Participate actively in Agile ceremonies—stand-ups, sprint planning, grooming, retros.
•    Work closely with Data Engineers, DBT Developers, Cloud Engineers, and Product Owners.
•    Provide automation insights, effort estimates, and feasibility judgments.
Defect Management
•    Log defects with clear data evidence in Jira.
•    Collaborate with teams to identify root cause (pipeline logic, DBT model,  AWS service failure, etc.).
•    Maintain traceability between requirements → test cases → automated scripts.
Quality Improvement & Standards
•    Enhance test coverage and reliability by contributing to: 
o    Automation strategy
o    Data testing best practices
o    Test data generation utilities
o    Error handling and logging improvements
•    Advocate for quality-first development in a data engineering environment.

Technical Skills:
•    Strong experience in API and Automation Testing.
•    Hands-on experience automating data validations for: 
o    Snowflake
o    Databricks (SQL/PySpark)
o    SQL Server
•    Understanding of DBT (models, tests, documentation, lineage).
•    Strong SQL skills (joins, CTEs, window functions, reconciliations).
•    Experience with Python automation frameworks (pytest, unittest), or Java (TestNG, JUnit).
•    Exposure to AWS data services: 
o    S3, Glue, Lambda, Step Functions, Athena, EMR (optional)
•    CI/CD exposure (GitHub Actions, Jenkins, AWS CodePipeline).
•    Experience with test management tools (Zephyr, Jira).

Mandatory Competencies