Job Description

Onsite at client location in Mountain View, CA (4 days per week)

Long term contract with possibility of conversion (~3 years)

Average hourly rate of $80/hour depending on experience and interview performance

Next Venture's financial services consulting client is seeking a Databricks Administrator with subject matter expertise in AWS services and products for a long term contract position out of their end client's office in Mountain View, CA.

The end client is just beginning their Databricks journey and is seeking a seasoned administrator to leadin setting up, managing, and securing the Databricks platform within an AWS environment, ensuring it runs smoothly for the entire team.

Responsibilities:

Key Responsibilities:

Databricks & Cloud Platform Administration:

• Deploy, configure, and maintain Databricks workspaces on AWS, ensuring high availability and security.
• Manage cluster configurations, autoscaling, and compute resources for optimal performance.
• Administer Unity Catalog , enforcing governance policies for structured and unstructured data assets.

Security, Compliance & Data Governance:

• Implement Unity Catalog for centralized access control , enforcing RBAC (Role-Based Access Control), attribute-based policies, and lineage tracking .
• Define fine-grained permissions for tables, views, and files stored in AWS S3 .
• Ensure compliance with banking industry regulations (e.g., SOX, PCI-DSS, GDPR ) through data masking, encryption, and audit logging .
• Manage IAM roles and authentication mechanisms between AWS and Databricks.

Performance Optimization & Monitoring:

• Optimize Spark job performance and cluster tuning using best practices.
• Monitor Databricks workspace health with CloudWatch, Datadog, or Splunk .
• Automate cost control measures through idle cluster termination, autoscaling policies, and query optimizations .

Integration & Automation:

• Integrate Databricks with AWS services (S3, Glue, Redshift, Kinesis, IAM) and external databases .
• Automate workspace provisioning and Unity Catalog policies using Terraform or Infrastructure-as-Code (IaC) .
• Implement CI/CD pipelines for Databricks notebooks, jobs, and configurations.

Collaboration & Support:

• Work closely with data engineers, cloud architects, security teams, and compliance officers to align Databricks with banking security policies.
• Troubleshoot platform issues, user access problems, and data pipeline failures .
• Develop best practices and documentation for Databricks administration and Unity Catalog implementation.

Qualifications & Experience:

• 5+ years of experience administering Databricks on AWS in an enterprise environment.
• Hands-on experience with Unity Catalog , including fine-grained permissions, lineage tracking, and governance policies .
• Strong knowledge of AWS services (S3, Glue, Redshift, IAM, Lambda, Kinesis).
• Expertise in Spark cluster management, Databricks Jobs, and SQL analytics .
• Experience in cloud security frameworks and data compliance for regulated industries.
• Familiarity with Terraform, CloudFormation, or Infrastructure-as-Code for automation.
• Understanding of networking & VPC configurations for Databricks on AWS.

Preferred Skills:

• Experience with Databricks Delta Lake, MLflow, and AI/ML workloads .
• Proficiency in Python, Bash, or PowerShell for scripting and automation.
• Knowledge of Airflow, Kafka, or other data orchestration tools .
• Databricks Certified Associate/Professional Administrator certification is a plus.

Job Tags

Similar Jobs