Principal Data Engineer
insightsoftware
CloudOracle
Job Level
Mid-LevelSenior
Tech Stack
AirflowCloudDockerKubernetesPythonSQL
About the role
- Remote data engineer for Sweed, a remote cannabis retail software startup.
- Migrating reporting infrastructure to a new data platform (Airbyte, dbt, Dagster, ClickHouse, Cube).
- Set up data export pipelines for clients and data streams for ML workflows.
- Establish pipeline monitoring and alerting.
- Ingest pipelines and integrate data sources with Airbyte or Fivetran.
- Create and optimize data transformation models using dbt, macros, and write data quality tests.
- English: Upper-Intermediate+; communicate with English-speaking team; read/write docs in English.
- Team size 140; remote-first; Europe to Bali; flexible hours; USD salary; remote work.
- Project is 4 years on market; daily feature launches; focus on customer trust and delivery processes.
Requirements
- Experience in Python for data processing tasks.
- Understanding of batch and real-time data processing.
- Proficiency with Git, CI/CD systems, and working with cloud platforms.
- Expertise in one or more pipeline orchestration frameworks (e.g., Airflow, Dagster, Kestra, mage.ai). Manage data pipelines, automate schedules, and handle error processing.
- Strong proficiency in SQL for complex queries and data manipulation.
- Experience in design and optimizing queries and data storage structures in ClickHouse, with experience handling large datasets.
- Understanding of C# is a plus.
- Experience with Docker, Kubernetes (k8s), and Helm charts is highly desirable.
- Upper-Intermediate English or higher.
