Summit Africa RecruitmentSummit Africa Recruitment
Job title: Big Data Engineer
Employment type:Full Time
Experience:4 to 7 years
Job published:21 May 2020
Job reference no:2286172474

Job Description

We’re looking for experienced Big Data Engineers with experience in building data warehouses and / or transactional data models within the Big Data environment.

That means that Spark / PySpark experience, as well as experience with the Hadoop EcoSystem, which leads on to any Python, Scala or Java coding experience. We need someone who can build data pipelines (ELT / ETL) within the Big Data environment.

Some additional skills and experience required would be:

  • In depth architectural knowledge of Spark and Hadoop

  • Expert in building ETL pipelines using Spark (Pyspark)

  • Experience using Spark with HDFS,

  • Experienced writing data pipelines using functional programming (Python, Java, Scala)

  • Advanced ANSI SQL experience

  • Firm understanding of Big Data and traditional data processing, understanding the differences in depth to make informed design decisions

  • Firm understanding of data Modelling OLAP vs. OLTP vs. Hybrid models

  • Firm understanding of dimensional modelling i.e. Kimball

Skills that would be beneficial:

  • Experience using Spark with Cloud Storage and Hiveserver2 LLAP (HWC)

  • Proven track record of developing DAGs using Apache Airflow

  • Exposure to Ranger and Atlas