As a Senior Data Engineer, you will work to define the data ontology for all of Confidential (Registered Users Only), establish standard methodologies for data operations and lifecycle management, design and build analytics tooling and frameworks, and influence event instrumentation. Additionally, this role is highly multi-functional, requiring close collaboration with Data Science, Experimentation, and Machine Learning teams to understand customer requirements and analytics applications, as well as with Data Infrastructure and Storage teams to develop integrated solutions.
Join us and be a part of a dynamic team driving innovation and growth at Confidential (Registered Users Only). This role will report to our Engineering Manager on the DSE team.
You Will:
- Partner with Data Science, Product, and Engineering to collect requirements to define the data ontology for all of Confidential (Registered Users Only)
- Lead and mentor a growing team of Data Engineers to support Confidential (Registered Users Only)'s ever-evolving data needs
- Design, build, and maintain efficient and reliable batch and streaming data pipelines to populate core data sets
- Develop scalable frameworks and tooling to automate analytics workflows and streamline users interactions with data products
- Establish and promote standard methodologies for data operations and lifecycle management
You Have:
- 8+ years of professional experience working building scalable ETL pipelines on industry standard ETL orchestration tools (Airflow, Dagster, Luigi, Google Cloud Composer, etc.) with deep expertise in SQL, PySpark, or scala.
- 3+ years leading data engineering development directly with business or data science partners
- Built, scaled, and maintained Multi-Terabyte data sets and have an expansive toolbox for debugging and unblocking large scale analytics challenges (skew mitigation, sampling strategies, accumulation patterns, data sketches, etc.)
- Experience with at least one major cloud's suite of offerings (AWS, GCP, Azure).
- Developed or enhanced ETL orchestrations tools or frameworks
- Worked within standard GitOps workflow (branch and merge, PRs, CI / CD systems)