Databricks - Lead

Job Description

Key Responsibilities
Platform Administration & Configuration

• Manage and maintain the AWS Databricks platform, including cluster configuration, workspace setup, and compute resources.
• Implement and optimize Databricks configurations based on business and security requirements.
• Work closely with architects to understand system requirements and make necessary configuration adjustments.
• Monitor performance, availability, and cost of Databricks resources, optimizing where necessary.

User Access Management & Security

• Develop and manage user access controls, ensuring least privilege principles.
• Define and enforce Identity and Access Management (IAM) policies in AWS and Databricks.
• Work with security teams to audit user access, permissions, and data security policies.
• Manage integration with Azure AD, Okta, or other authentication providers for Single Sign-On (SSO).

User Persona Development

• Define and manage user personas for different types of Databricks users, including data engineers, data scientists, analysts, and administrators.
• Assign appropriate permissions and roles based on business needs and compliance policies.
• Develop governance frameworks to ensure proper access provisioning and de-provisioning.

Collaboration & Support

• Work closely with architects, data engineers, and security teams to ensure best practices are followed.
• Support troubleshooting, debugging, and issue resolution related to Databricks performance, security, and configuration.
• Document platform configurations, best practices, and user guidelines.
• Provide training and support to Databricks users across the organization.

Monitoring & Compliance

• Set up monitoring and alerting mechanisms for performance, security, and cost management.
• Ensure compliance with industry regulations (e.g., GDPR, HIPAA) and internal security policies.
• Perform regular audits and reviews to maintain platform security and compliance.

Required Skills & Qualifications

• 5-7 years of experience as a Databricks Administrator or similar role.
• Strong experience with AWS services (IAM, S3, EC2, Lambda, Glue, etc.).
• Expertise in Databricks administration, workspace management, and security configurations.
• Hands-on experience with AD groups, user access management, RBAC, and IAM policies.
• Experience in developing and managing user personas within enterprise environments.
• Strong understanding of network security, authentication, and data governance.
• Proficiency in Python, SQL, and Spark for troubleshooting and automation.
• Familiarity with Terraform, CloudFormation, or Infrastructure as Code (IaC) is a plus.
• Knowledge of CI/CD pipelines and DevOps best practices is desirable.
• Excellent communication and documentation skills.

Preferred Certifications

• AWS Certified Solutions Architect – Associate / Professional
• Databricks Certified Data Engineer / Administrator
• Certified Information Systems Security Professional (CISSP) – Nice to have

Mandatory Competencies