3Core Systems, Inc

Senior Data Engineer

3Core Systems, Inc

contract

Posted on:

Location Type: Hybrid

Location: Plantation • Florida • United States

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

CloudPySparkPythonScalaSparkSQLTerraformUnity

About the role

  • Administer, optimize, and scale our Databricks Lakehouse environment, ensuring high performance, cost efficiency, and operational excellence.
  • Develop, maintain, and enhance our data platform infrastructure and security configurations using Terraform.
  • Manage and enforce Unity Catalog for data governance, access control, and metadata management.
  • Implement and manage CI/CD pipelines for data pipelines, dbt projects, and infrastructure deployments using GitHub Actions.
  • Automate operational tasks, monitoring, and alerting for the data platform.
  • Design and implement data ingestion patterns into Databricks using Delta Lake, optimizing for large-scale data processing and storage.
  • Develop, optimize, and troubleshoot complex Spark jobs (PySpark/Scala) for data processing and transformation within Databricks.
  • Manage and extend data ingestion pipelines using Airbyte (or similar modern tools like Fivetran, Stitch).
  • Collaborate with Analytics Engineers to translate business requirements into efficient and scalable data models using dbt (Data Build Tool).

Requirements

  • Bachelor's degree in Computer Science, Data Engineering, or a related technical field required.
  • 5+ years of progressive experience as a Data Engineer, with a strong focus on cloud-based data platforms.
  • Extensive experience with Spark (PySpark/Scala), Delta Lake, Unity Catalog, Databricks SQL, and platform administration.
  • Proven experience with dbt for data modeling, transformation, and testing.
  • Strong proficiency with Terraform for defining, provisioning, and managing cloud infrastructure and Databricks resources as code.
  • Expertise in Git and GitHub Actions for version control and implementing robust CI/CD pipelines.
  • Proficiency in SQL and at least one programming language (Python strongly preferred, Scala is a plus).
Benefits
  • Professional development opportunities
  • Remote work options

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills
DatabricksTerraformUnity CatalogCI/CD pipelinesdbtDelta LakeSparkPySparkScalaSQL