GCP Datalake - Senior Engineer

Job Description

1. Data Engineering:

Cloud Platform Expertise: Strong knowledge of GCP (Google Cloud Platform) services, including Composer, Data Fusion, BigQuery, Cloud Storage, and Dataflow.

Data Pipeline Development: Experience in building and maintaining data pipelines using technologies like Apache Airflow, Python, SQL, and potentially other ETL tools.

Data Modeling & Schema Design: Ability to design and implement efficient data models and schemas for data ingestion, storage, and processing.

Data Quality & Validation: Expertise in ensuring data quality, validating data integrity, and implementing data quality checks within the pipeline.

Troubleshooting & Debugging: Proficiency in identifying and resolving issues within the pipeline, including performance bottlenecks, data inconsistencies, and error handling.

CI/CD & Automation: Experience with continuous integration and continuous delivery (CI/CD) pipelines for automating data pipeline deployments and updates.

2. Data Integration & Connectivity:

API Integration: Expertise in integrating with various APIs (e.g., Salesforce, Workday, Peoplesoft, Siplast, etc.) and understanding API security, authentication, and data formats.

Data Transformation & Manipulation: Skill in transforming data using different methods (e.g., data cleaning, aggregation, filtering, etc.) and applying appropriate data transformation techniques.

Database Technologies: Experience with relational databases

3. Application & Platform Support:

Troubleshooting & Problem Solving: Ability to identify and resolve issues related to application performance, data integration, and platform stability.

Monitoring & Alerting: Experience in setting up monitoring systems to track pipeline performance, data quality, and potential failures.

Incident Management: Skill in handling and managing incidents, communicating updates to stakeholders, and following incident resolution procedures.

Documentation & Communication: Clear communication skills and ability to document procedures, best practices, and technical documentation for the team and stakeholders.

4. Domain Expertise:

Business Understanding: A deep understanding of the business processes and data requirements related to the specific data pipelines.

Data Understanding: Knowledge of the data structures, formats, and relationships within the various source systems.

Technical Collaboration: Ability to effectively collaborate with other teams (e.g., business analysts, data analysts, application developers) to understand requirements and troubleshoot issues.

Prior experience in SAP, Salesforce etc esp finance domain

5. Additional Skills:

Version Control (Git): Understanding and experience with version control systems like Git for managing code and pipeline changes.

Cloud Security: Knowledge of cloud security best practices and implementing security measures for data pipelines and cloud infrastructure.

Mandatory Competencies