The Role: Staff Data Engineer II
We are currently seeking an accomplished Staff Data Engineer II to join our dynamic team. The ideal candidate will possess a wealth of experience in data engineering, with a focus on designing and implementing complex pipelines, cloud-based data migration projects, Spark pipeline development, medallion architecture, data modeling, on-premise to cloud Spark code optimization, data cataloging, and quality implementation using Unity Catalog, as well as expertise in Delta Live Table (DLT) pipeline implementations. Candidates with knowledge of AWS services will be given preference.
Responsibilities:
Data Engineering Excellence:
Design, develop, and implement sophisticated data engineering pipelines to address diverse business needs.
Working knowledge of data formats like JSON, parquet, csv, and orc etc.
Implementation knowledge of batch and streaming data ingestion.
Cloud Data Migration Leadership:
Lead data migration projects on cloud platforms, collaborating closely with clients to understand their on-premise ETL design and pipeline.
Excel in Spark pipeline development, ensuring scalable and efficient data processing.
Medallion Architecture Expertise:
Apply medallion architecture principles to design robust and scalable data systems.
Should able to design the Bronze, Silver and Gold tables.
On-Premise to Cloud Spark Code Optimization:
Optimize Spark code for seamless migration from on-premise to cloud environments, ensuring optimal performance.
Data Modeling Prowess:
Develop and maintain comprehensive data models, contributing to effective data governance.
Data Cataloging and Quality Implementation:
Implement robust data cataloging strategies, maintaining high-quality data through effective data quality frameworks using Unity Catalog.
Delta Live Table (DLT) Leadership:
Lead the design and implementation of Delta Live Table (DLT) pipelines, ensuring secure and tamper-resistant data transactions.
Knowledge of the delta tables, streaming tables, managed and external tables.
Customer Collaboration:
Work closely with customers to understand their on-premise ETL design and pipeline, providing optimal solutions for migration to the cloud.
Take a lead role in migration components, providing best practices for high-level design, low-level design, and data governance.
Requirements
Basic Qualifications:
Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects.
Bachelor's or higher degree in Computer Science, Data Engineering, or a related field.
Preferred Qualifications:
Proficiency in Spark, SQL, Python, and other relevant data processing technologies.
Expertise in on-premise to cloud Spark code optimization and medallion architecture.
Deep of Databricks and related components like Delta Live Table (DLT) pipeline implementations.
Familiarity with AWS services (additional cloud platform experience is a plus).
Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams.
Benefits
What We Offer
An opportunity to be part of an agile, highly proficient and experienced AI/ML team
An opportunity to work on challenging data science and machine learning problems with customers and seeing your work deployed in action
A fast-paced software development environment that uses the latest open-source tools across the development stack
Benefits
We provide a competitive salary and benefits package, a vibrant work environment, and numerous opportunities for professional growth. You'll have the opportunity to work with a team of industry experts on exciting projects that transform businesses and create significant value. Join us to revolutionize the way companies leverage technology for digital transformation.
OnebyZero is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.