Azure Databricks Data Engineer – Apache Spark & Azure Data Factory

About the job

Job Summary Synechron is seeking a Senior Data Engineer to build and optimize cloud-based data platforms using Azure and Databricks. The role will deliver reliable, secure, and cost-efficient data pipelines that support enterprise business objectives.

Software Requirements Required:

- Azure, Azure Databricks, Azure Data Factory, and Azure Data Lake

- Azure Key Vault, service principals, and Azure CLI

- Apache Spark, PySpark, Python, T-SQL, and Linux/Shell scripting

- Databricks Jobs, Workflows, Azure Pipelines, and CI/CD

- Git and cloud-based data engineering tools Preferred:

- Agile delivery experience

- PL/SQL

- Data-quality, monitoring, and infrastructure automation tools

Overall Responsibilities

- Design, develop, and maintain ETL/ELT pipelines using Spark and Databricks.

- Build batch and streaming data solutions.

- Develop Azure Data Factory pipelines and automate deployments.

- Optimize Databricks clusters, caching, queries, and cloud resource usage.

- Secure data and credentials using Key Vault and service principals.

- Monitor pipelines, resolve issues, and improve reliability.

- Collaborate with engineering teams and stakeholders.

- Document solutions and support continuous technical improvement.

Technical Skills (By Category)

- Programming: Essential—Python, PySpark, T-SQL, Shell scripting. Preferred—PL/SQL.

- Data Management: Essential—Data lakes, ETL/ELT, SQL, data modeling, partitioning, and optimization. Preferred—Data governance and lineage.

- Cloud: Essential—Azure, Databricks, Data Factory, Data Lake, Key Vault, Azure CLI, and service principals. Preferred—Infrastructure as code and cloud cost management.

- Frameworks: Essential—Apache Spark, Spark SQL, PySpark, Databricks Jobs, and Workflows. Preferred—Delta-based storage and streaming frameworks.

- Tools and Methods: Essential—Git, CI/CD, Azure Pipelines, testing, monitoring, and Agile practices.

- Security: Essential—Secure credential management, role-based access, and least-privilege principles.

Experience Requirements

- 10+ years in data engineering or Azure cloud technologies.

- 3+ years optimizing Databricks workloads.

- Experience with cluster sizing, caching, query optimization, Spark, ADF, Python, SQL, and Linux.

- Enterprise data platform or regulated-industry experience is preferred.

- Equivalent practical experience, certifications, or substantial project work may be considered.

Day-to-Day Activities

- Develop PySpark, Python, SQL, and shell-based solutions.

- Build and monitor ADF pipelines and Databricks Workflows.

- Tune Spark jobs and manage cluster configurations.

- Investigate failures and resolve production issues.

- Participate in Agile meetings, design reviews, and code reviews.

- Collaborate with engineers, stakeholders, and platform teams.

- Produce pipeline documentation, test evidence, and support materials.

- Make technical decisions within approved architecture and security standards.

Qualifications

- Bachelor’s or master’s degree in Computer Science, Information Technology, Mathematics, Data Engineering, or a related field.

- Equivalent professional experience may be considered.

- Azure or Databricks certification is preferred.

- Completion of Synechron security, compliance, and project training is required.

- Ongoing learning in Azure, Databricks, Spark, security, and cloud efficiency is expected.

Professional Competencies

- Analytical problem-solving and root-cause analysis.

- Effective teamwork and technical leadership.

- Clear communication with technical and business stakeholders.

- Ability to manage priorities, deadlines, and dependencies.

- Adaptability and willingness to learn new technologies.

- Focus on secure, maintainable, reliable, and cost-efficient solutions. S​ YNECHRON’S DIVERSITY & INCLUSION STATEMENT

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law .

Candidate Application Notice