Oops! This job is no longer active.

Please contact the hiring team for pending updates, if any

Data Engineer

Pune

5 - 8 years
Last active
29
68% Matching
Apache Spark
Vertica
ETL
Scala
Tableau
Data Modeling
Data Warehouse
Data Quality
Informatica
Apache Hive
Applied
Active
29
Apache Spark
Vertica
ETL
Scala
Tableau
Data Modeling
Data Warehouse
Data Quality
Informatica
Apache Hive

  About opportunity

Responsibilities

  • Design, develop and maintain resilient, secure and scalable data pipelines that handle data collected from 800M+ monthly active users.
  • Evaluate new technologies, build prototypes, formulate deployment & scaling plans for improvements in data infrastructure and engineering.
  • Ensure that data pipelines are following all the necessary security protocols through periodic reviews.
  • Lead the development of data reporting for customers & internal stakeholders.
  • Ownership of the data-pipeline quality, instrumentation and logging.
  • Mentor and work with the other team members.
  • Keep calm and learn everyday.

Requirements

  • 5+ years of experience with Hadoop ecosystem.
  • Experience of running data-warehouse technologies like Hive, HBase, Hadoop at scale.
  • Experience of using Spark to build data-pipelines.
  • Experience in the design and implementation of distributed systems at scale. Must have exposure to distributed systems concepts like leader election, consensus, clocks.
  • Deep knowledge of JVM including GC tuning and Posix compliant Operating Systems (we develop on Mac OS X and deploy on GNU/Linux).
  • Experience of writing Unit, Functional & Regression tests. Knowledge of generative testing is preferred.
  • Excellent verbal and written communication skills.
  • Handy with the shell and automation tools.
  • Bachelor’s Degree in Computer Science (or equivalent)

Nice to have

  • Experience of stream processing frameworks like Flink / Storm.
  • Knowledge of BI tools that can be used for internal reporting.
  • Knowledge of Functional programming.
  • Experience of developing production systems in Clojure.
Report an error

Was this job relevant for you?

Data Engineer

Helpshift   •   Pune