LTM
AWS Databricks
Role: AWS Databricks Developer Location: Pan India Years of Experience: 8–12 Years Shift: 11:30 AM to 9:00 PM Notice Period: Immediate Joiners Primary Skills Databricks Development: Hands-on development experience using Databricks for large-scale data processing and analytics. Data Engineering: Strong experience in building and maintaining ETL/ELT pipelines. Processing Frameworks: Strong knowledge of Apache Spark, PySpark, Spark SQL, Python, Scala, and Power BI. Delta Lake: Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance optimization. Data Pipelines: Experience in developing batch and near real-time data ingestion pipelines. Data Modelling: Experience in implementing analytical and dimensional data models, including Star and Snowflake schemas. Cloud Storage: Experience with AWS cloud-based storage systems. Development Best Practices: Adherence to coding standards, documentation, and data engineering best practices. Collaboration: Ability to work closely with architects, BI teams, and data science teams to deliver end-to-end solutions. Secondary Skills Exposure to ML/AI workloads on Databricks, including feature engineering. Familiarity with MLflow or similar model tracking tools. Experience with Databricks CI/CD and Git-based version control. Understanding of data governance, security, and compliance concepts. Familiarity with Agile/Scrum delivery methodologies. Domain experience in Insurance.
The role
What you'll do
- Develop, maintain, and optimize large-scale data processing and ETL/ELT pipelines using Databricks and Apache Spark.
- Design and implement batch and near real-time data ingestion pipelines on AWS
- Implement and optimize Delta Lake solutions, including ACID transactions, schema evolution, partitioning, and performance tuning
- Develop solutions using PySpark, Spark SQL, Python, and Scala
- Design and implement dimensional and analytical data models, including Star and Snowflake schemas
- Work with AWS cloud storage services for scalable data ingestion and processing
You bring
Requirements
- 8–12 years of overall IT experience with strong hands-on experience in Databricks and Data Engineering.
- Strong expertise in Apache Spark and PySpark
- Good hands-on experience with Spark SQL, Python, and Scala
- Strong experience in developing and maintaining ETL/ELT data pipelines
- Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance optimization
- Experience in batch and near real-time data ingestion
Skills