Brillio

Senior Lead Data Engineer

Brillio

full-time

Posted on:

Location Type: Hybrid

Location: San Francisco • 🇺🇸 United States

Visit company website
✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowApacheHadoopHDFSMapReducePythonScalaShell ScriptingSparkSQLUnixYarn

About the role

  • Senior Lead Data Engineer

Requirements

  • - Key Responsibilities:
  • - Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
  • - Manage and optimize the infrastructure ensuring reliability and performance
  • - Design and implement data models and schemas for enterprise data warehouse and data marts
  • - Write and maintain scripts in Python for various applications
  • - Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
  • - Work closely with analysts and product managers to understand data requirements and deliver high-quality data
  • - Provide technical guidance and mentorship to other team members fostering best practices in data engineering
  • - Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
  • **
  • Required Skills:
  • - High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
  • - Good knowledge in Python for data manipulation and scripting
  • - Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
  • - Hands-on experience with workflow management tools like Apache Airflow
  • - Familiarity with Unix shell scripting for automation and system management
  • **

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills
HadoopSpark SQLPythonSQLApache AirflowScalaUnix shell scriptingdata modelingdata warehousingELT pipelines
Soft skills
technical guidancementorshipcollaborationproblem-solvingcommunication