NewFree base camp this Saturday: Agentic AI for Enterprise. Limited seats.

← All jobs

LTM

AWS Databricks

On-site · Hyderabad8–12 yrsfulltime

Role: AWS Databricks Developer Location: Pan India Years of Experience: 8–12 Years Shift: 11:30 AM to 9:00 PM Notice Period: Immediate Joiners Primary Skills Databricks Development: Hands-on development experience using Databricks for large-scale data processing and analytics. Data Engineering: Strong experience in building and maintaining ETL/ELT pipelines. Processing Frameworks: Strong knowledge of Apache Spark, PySpark, Spark SQL, Python, Scala, and Power BI. Delta Lake: Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance optimization. Data Pipelines: Experience in developing batch and near real-time data ingestion pipelines. Data Modelling: Experience in implementing analytical and dimensional data models, including Star and Snowflake schemas. Cloud Storage: Experience with AWS cloud-based storage systems. Development Best Practices: Adherence to coding standards, documentation, and data engineering best practices. Collaboration: Ability to work closely with architects, BI teams, and data science teams to deliver end-to-end solutions. Secondary Skills Exposure to ML/AI workloads on Databricks, including feature engineering. Familiarity with MLflow or similar model tracking tools. Experience with Databricks CI/CD and Git-based version control. Understanding of data governance, security, and compliance concepts. Familiarity with Agile/Scrum delivery methodologies. Domain experience in Insurance.

The role

What you'll do

  • Develop, maintain, and optimize large-scale data processing and ETL/ELT pipelines using Databricks and Apache Spark.
  • Design and implement batch and near real-time data ingestion pipelines on AWS
  • Implement and optimize Delta Lake solutions, including ACID transactions, schema evolution, partitioning, and performance tuning
  • Develop solutions using PySpark, Spark SQL, Python, and Scala
  • Design and implement dimensional and analytical data models, including Star and Snowflake schemas
  • Work with AWS cloud storage services for scalable data ingestion and processing

You bring

Requirements

  • 8–12 years of overall IT experience with strong hands-on experience in Databricks and Data Engineering.
  • Strong expertise in Apache Spark and PySpark
  • Good hands-on experience with Spark SQL, Python, and Scala
  • Strong experience in developing and maintaining ETL/ELT data pipelines
  • Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance optimization
  • Experience in batch and near real-time data ingestion

Skills

AWS Databricks DevelopmentData EngineeringApache Spark / PySparkSpark SQLPythonDatabricks ML/AI WorkloadsFeature EngineeringMLflow