Data Engineer - Lead

Job Description

Role:
Primary Skill(s): Lead Data Engineer
▪  Optimize data pipelines for performance and efficiency. 
▪  Work with databricks clusters and configuration management tools.
▪  Use appropriate tools in the cloud data lake development and deployment. Developing/implementing cloud infrastructure to support current and future business needs.
▪  Provide technical expertise and ownership in the diagnosis and resolution of issues.
▪  Ensure all cloud solutions exhibit a higher level of cost efficiency, performance, security, scalability, and reliability.
▪  Manage cloud data lake development and deployment on AWS /Databricks.
▪  Manage and create workspaces, configure cloud resources, view usage data, and manage account identities, settings, and subscriptions in Databricks
 
Roles and Responsibility
▪  Technical Leadership: Demonstrate leadership, and ability to guide business and technology teams in adoption of best practices and standards
▪  Design & Development: Design, develop, and maintain robust, scalable, and high-performance data estate
▪  Architecture: Architect and design robust data solutions that meet business requirements & include scalability, performance, and security.
▪  Quality: Ensure the quality of deliverables through rigorous reviews, and adherence to standards.
▪  Agile Methodologies: Actively participate in agile processes, including planning, stand-ups, retrospectives, and backlog refinement.
▪  Collaboration: Work closely with system architects, data engineers, data scientists, data analysts, cloud engineers and other business stakeholders to determine optimal solution & architecture that is future-proof too.
▪  Innovation: Stay updated with the latest industry trends and technologies, and drive continuous improvement initiatives within the development team.
▪  Documentation: Create and maintain technical documentation, including design documents, and architectural user guides.
 
Skills & Qualification
▪  Bachelor’s degree in information technology or a related field. Equivalent experience may be considered.
▪  Overall experience of 8-12 years in Data Engineering 
▪  Ability to understand and utilize execution plans effectively.
▪  Excellent communication and client interfacing skills, both verbal and written.
▪  Excellent problem-solving and analytical skills.
▪  Ability to work effectively in a fast-paced, agile, and high intensity - dynamic environment.
 

 

Mandatory Competencies