Data Engineer

Senior Data Engineer

• Design, build, and maintain scalable data platforms and services
• Implement data pipeline solutions in Python and cloud platforms like AWS
• Optimize data delivery and reduce cloud costs through automation
• Monitor Data Quality, Observability, Governance, and Lineage
• Collaborate with internal teams and client stakeholders
• Write unit/integration tests and document design/implementation

Lead Data Engineer

• Design, build, and optimize ETL/ELT workflows using Databricks, SQL, Python/pyspark & Alteryx
• Work on cloud platforms (Azure, AWS) to build and manage data lakes, data warehouses, and scalable data architectures
• Use Databricks for big data processing, analytics, and real-time data processing
• Collaborate with cross-functional teams to deliver impactful data solutions
• Leverage CI/CD pipelines to streamline development, testing, and deployment of data engineering workflows

Data Engineer II

• Lead and oversee project delivery, ensuring timely completion with high quality.
• Collaborate with clients and offshore leads to gather requirements and define high-level designs.
• Direct development and unit testing activities while maintaining strong alignment with project goals.
• Communicate effectively with stakeholders, providing regular updates on task and project statuses.
• Identify, track, and address project risks or issues promptly.
• Work closely with management to ensure seamless execution of tasks.
• Provide technical guidance to the team, outlining design, implementation, and delivery strategies.

Lead Data Engineer – AWS

• Designs and develops data architecture for new and existing applications and data sources
• Establishes Data Quality, Data Governance and Master Data Management Best Practices to enable the business to maintain clean and accurate data
• Designs ETL frameworks and features to allow for robust and scalable data pipelines
• Incorporates new pipelines into the existing data model, augmenting as needed
• Manages and maintains several production systems
• Engages with various internal cross-functional departments to strategically design, develop and implement data pipelines while understanding the underlying data
• Increases team productivity by developing, identifying, and implementing better tools and processes
• Exemplifies good documentation, coding, and testing best practices
• Develops standard operating procedures for the use of artificial intelligence and data engineering principles within clinical trials
• Prototypes new ideas/technologies to create proof of concept and demos
• Provides mentorship for other data engineers
• Assists in executing responsibilities of Data Engineers including the following:
• o Maintains and develops ETL pipelines
• o Maintains an Enterprise Data Warehouse by updating and translation logic and support warehouse servers.
• o Follows Data Quality, Data Governance and Master Data Management Best Practices
• o Delivers high quality software design documentation
• o Prototypes new ideas/technologies to create proof of concept and demos
• o Contributes to the development of standard operating procedures
• o Performs other related duties incidental to the work described herein
• Adherence to all essential systems and processes that are required at SDC to maintain compliance to business and regulatory requirements
• Act as a resource for other team members for debugging, code reviews and other software development lifecycle activities.
• Contract Research Organization experience and familiarity with its operations
• The above statements describe the general nature and level of work being performed by individuals assigned to this classification. This document is not intended to be an exhaustive list of all responsibilities and duties required of personnel so classified.

• Develop and maintain ETL/ELT pipelines on Databricks
• Work with PySpark & SQL for data transformation
• Ingest, clean, and validate structured/unstructured data
• Support data pipeline optimization & debugging
• Collaborate with teams for data needs & requirements

Databricks Data Engineer

• Lead script-level repointing across 593 downstream scripts (PySpark, BTEQ, Shell, SQL, IICS, SSIS) and 154 Databricks MLOps scripts from Teradata/SQL Server/Minio to Snowflake.
• Execute SQL syntax conversion for Teradata BTEQ scripts to Snowflake-compatible SQL, leveraging utility frameworks provided by the SI Partner (Infosys).
• Drive Airflow variable and connection updates across all migrated workflows, ensuring pipelines point to Snowflake endpoints post-cutover.
• Manage the script modification and change management process, coordinating production-to-parallel sync within the required 1-business-day SLA.
• Own the deployment and sign-off process for each migration wave, maintaining runbooks and go/no-go checklists per domain group.
• Manage data copy requests for Minio-related assets and coordinate initial load and incremental sync activities with the SI Partner.
• Lead and coordinate a team of 6–8 Data Engineer I and Data Engineer II resources (onshore and offshore) across parallel migration workstreams.
• Assign scripts per complexity band (S/M/L) to team members, track daily progress against the LOE estimates, and escalate blockers promptly.
• Review code changes, validate SQL conversion quality, and enforce engineering standards before submission to EBI team for review.
• Mentor junior engineers on Snowflake-specific constructs (virtual warehouses, stages, Snowpark, streams & tasks) and Teradata-to-Snowflake syntax differences.
• Conduct daily stand-ups and provide weekly migration status reports to the Program Manager and client stakeholders.
• Own the parallel run and reconciliation cycle — execute data validation against Teradata source and Snowflake target, targeting sign-off within 2–3 reconciliation iterations per workflow.
• Drive automation of unit testing and validation frameworks in collaboration with the SI Partner, tracking outcomes in the centralized validation dashboard.
• Triage and root-cause data mismatches, performance gaps, and pipeline failures during the parallel run phase; coordinate resolution with SI Partner support as needed.
• Ensure UAT sign-off from the EBI Data team on each domain group before scheduling production cutover.
• Conduct performance and load testing on migrated Snowflake workloads; incorporate SI Partner guidance on query optimization and warehouse sizing.
• Serve as the EXL technical point of contact with the EBI client team for the repointing workstream — attending daily syncs, steering calls, and design reviews.
• Coordinate with Infosys (SI Partner) on alignment of major SQL conversion utilities prior to team execution, ensuring conversion frameworks are available at program kick-off.
• Translate migration risks and data reconciliation issues into clear business-impact language for non-technical stakeholders.
• Maintain shared documentation: migration decision log, runbooks, issue trackers, and lessons learned repository.

Senior Data Engineer – Snowflake Migration Lead

• Work in collaborative environment including pairing and mobbing with other cross-functional engineers
• Work on a small agile team to deliver working, tested software
• Work effectively with fellow data engineers, product owners, data champions and other technical experts
• Demonstrate technical knowledge/leadership skills and advocate for technical excellence
• Develop exceptional Analytics data products using streaming, batch ingestion patterns in the Google Cloud Platform with solid Datawarehouse principles
• Be the Subject Matter Expert in Data Engineering and GCP tool technologies

• Coding transformation in Pyspark Notebooks
• Using Azure Data Factory (ADF) for data orchestration.
• PG-SQL/PL-SQL to TSQL Conversion for migration of legacy code.
• Data validation and comparison
• Performance Tuning (Synapse pipelines & Dedicated Pools)
• Execution & Unit testing of ADF/Synapse pipelines
• Release management using Git & Azure Devops
• Creation of HLD & TDD artifacts
• Working with cross-functional teams to effect seamless component delivery
• Status Reporting (Daily / Weekly)
• Working in Agile PODS and experience of Agile methodology

Azure Data Engineer

• D365 CID administration & support
• Support and extend an existing CID setup onboard new sources, tune unification rules, manage refresh schedules and environment configuration.
• Triage and resolve CID issues raised by the customer’s business and IT teams: failed refreshes, broken segments, profile-count discrepancies, export failures, permission issues.
• Manage Support tickets with Microsoft relating to CID issues.
• Operate and support upstream data pipelines feeding CID.
• Validate data on date and data fitness for CID (schema, keys, types, nulls, duplicates) — and fix or escalate when it isn’t.
• Build and maintain monitoring on pipeline runs, refresh status, row counts, and SLAs.
• Perform root-cause analysis end-to-end — source → lake → pipeline → CID → consumer — and produce clean write-ups and permanent fixes, not just patches.
• Run regular service reviews with the customer’s data and CRM stakeholders, log changes through formal change control, and keep documentation current.

D365 Customer Insight Data Engineer

• Implement automated data quality checks, validation rules, and anomaly detection at each stage of the data pipeline to prevent corrupt, incomplete, or biased data from reaching AI models.
• Define and enforce data quality SLAs (e.g., completeness, timeliness, consistency, accuracy thresholds)
• Build monitoring and alerting systems to detect pipeline failures, data drift, and schema changes that could degrade AI model performance.
• Conduct root cause analysis on data quality incidents and implement preventative controls.
• Implement and maintain end-to-end data pipelines, including processes for structured, semi-structured, unstructured datasets.
• Work with AI Deployment Engineers to optimize data pipelines for AI model training and data stores for AI model retention.
• Ensure data availability and reliability across operational AI systems.
• Implement data retention, data lineage, and other data governance standards across various data pipelines and products.
• Collaborate with both technical and non-technical teams for seamless data integration with automation tools.
• Implement both batch and real-time/streaming data pipelines to support diverse AI and/or automation use cases.

AI Data Engineer

• Design, build, and maintain scalable data pipelines for ingestion, transformation, and delivery using Fivetran and API Ingestion with AWS S3
• Develop and optimize complex SQL queries and Python scripts to support ETL/ELT workflows
• Manage and maintain data workflows within Databricks, ensuring performance, reliability, and data quality
• Collaborate on data modeling, schema design, and governance best practices
• Build and maintain Databricks dashboards and reports that surface actionable insights to business stakeholders
• Partner with analysts and business teams to translate reporting needs into scalable data solutions
• Communicate clearly and proactively with stakeholders across business units, setting realistic expectations on timelines and deliverables
• Prioritize workload effectively across multiple concurrent projects and shifting business needs
• Document data processes, pipeline logic, and architectural decisions to enable team-wide transparency
• Participate in sprint planning and contribute to Agile development cycles
• Identify opportunities to leverage AI, machine learning, or cloud-native tooling to enhance data products

Data Engineer III

• Partner with technical and non-technical stakeholders to translate business requirements into scalable, reliable, and maintainable data engineering solutions.
• Design, develop, and maintain data integration solutions using Microsoft SQL Server, Azure Data Factory, Azure Databricks, Microsoft Fabric, and related Azure services.
• Build, manage, and optimize ETL/ELT pipelines using Azure Data Factory and Microsoft Fabric Pipelines, including pipeline orchestration, parameterization, scheduling, monitoring, and error handling.
• Develop and maintain data transformation logic using Azure Databricks notebooks and Microsoft Fabric notebooks, with a focus on performance, scalability, reusability, and operational reliability.
• Design and implement data ingestion, transformation, validation, and delivery processes across structured, semi-structured, and external data sources.
• Develop and optimize data integration processes utilizing Azure Data Factory, Azure Databricks, Azure Storage Accounts, Azure Data Lake Storage, Azure Key Vault, Microsoft Fabric Lakehouses, Warehouses, Pipelines, and Notebooks.
• Build and maintain SQL-based data models, stored procedures, views, automation scripts, and database objects to support operational and analytical workloads.
• Support the modernization and migration of legacy data integration processes from SSIS and on-premises platforms to cloud-based Azure and Microsoft Fabric architectures.
• Design and implement reusable pipeline patterns, data quality checks, validation controls, logging, alerting, and performance monitoring processes.
• Support new client onboarding efforts by designing and implementing reliable data ingestion, transformation, reconciliation, and validation workflows.
• Troubleshoot and resolve data pipeline, data quality, performance, and integration issues in collaboration with teams across Intelligence & Analytics.
• Design, develop, and maintain datasets, semantic models, and data structures that support reporting, analytics, and visualization solutions across Power BI, Excel, Microsoft Fabric, and other business intelligence platforms.
• Contribute to data engineering standards, documentation, best practices, and reusable frameworks that improve scalability, maintainability, and operational efficiency.
• Research, evaluate, and recommend emerging technologies and platform capabilities that improve data architecture, automation, performance, and reliability.

• Design, evolve, and maintain a canonical insurance data model (CIDM).
• Define and enforce standards for data capture, persistence, normalization, and extensibility.
• Architect and govern effective dating, versioning, and historical tracking strategies.
• Model complex insurance relationships including 1-1, 1-many, and many-many.
• Define end-to-end technical architecture for integration with downstream systems.
• Design and optimize relational schemas in SQL Server and PostgreSQL.
• Partner with developers to build and refine .NET-based services.
• Define and maintain REST API contracts for data interchange.
• Establish and maintain data governance frameworks including data contracts.

Data Architect, Duck Creek/Guidewire

• Serve as the primary technical lead and escalation point for enterprise data engineering initiatives.
• Bridge business requirements, architectural standards, and engineering implementation.
• Partner with business analysts, architects, BI teams, DevOps, and data engineers to support successful solution delivery.
• Interpret and clarify technical implementation requirements for data engineering teams.
• Guide implementation decisions across Databricks pipelines, transformations, and data models.
• Review engineering implementations for consistency, scalability, maintainability, and alignment to standards.
• Support troubleshooting and root cause analysis for data quality issues, failed pipelines, performance concerns, and production defects.
• Act as L1/L2 support lead for enterprise data platform operational issues.
• Perform lineage and downstream impact analysis for data model and pipeline changes.
• Guide implementation of reusable engineering patterns, medallion architecture, and gold-layer datasets.
• Coordinate defect triage, release support, deployment validation, and production stabilization activities.
• Support adoption of engineering standards, CI/CD processes, governance controls, and operational best practices.
• Mentor and guide data engineers on technical implementation approaches and enterprise standards.
• Drive consistency across engineering teams, platforms, and data products.
• Document technical patterns, implementation standards, operational procedures, and support processes.

Data Engineering Technical Lead

• Design and implement scalable data lake/lakehouse architectures on AWS using S3 and Iceberg.
• Develop and optimize ETL/ELT pipelines using PySpark and orchestrate workflows with Apache Airflow.
• Integrate and manage Snowflake as the enterprise data warehouse and business layer for analytics.
• Ensure data quality, governance, security, and performance across the data platform.
• Lead technical design discussions, define best practices, and drive data engineering standards.
• Mentor and guide a team of data engineers, fostering a collaborative and high-performance culture.
• Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders.

Lead AWS Data Engineer

• Design, build, and maintain enterprise ETL/ELT pipelines in Microsoft Fabric using Dataflow Gen2, Data Factory pipelines, and other Fabric-native tools
• Collaborate closely with other Data Engineers and Developers in India to deliver scalable, well-orchestrated data solutions that support a global business
• Ingest data from new and existing source systems into Microsoft Fabric using a combination of out-of-the-box connectors, APIs, and custom integration approaches
• Transform and standardize disparate datasets into report-ready tables optimized for Power BI and downstream analytics use cases
• Engineer and maintain dimensional models, including star schemas with fact and dimension tables, to support enterprise reporting and analytical scalability
• Collaborate with analysts and business stakeholders to understand data requirements and translate them into robust warehouse and semantic-ready data structures
• Create and monitor jobs, alerts, and data quality checks to identify missing, incomplete, or inaccurate data and perform root cause analysis to resolve issues
• Optimize data transformations, query performance, and warehouse design to improve reliability, efficiency, and cost management
• Prepare and maintain data architecture documentation, pipeline documentation, system diagrams, and ERDs to clearly communicate lineage and transformation logic
• Assist in building roadmaps for new data sources, warehouse enhancements, and process automation opportunities within the Fabric ecosystem
• Identify opportunities to improve reporting enablement, automation, and analytical capabilities across the business
• Other duties as assigned, including ad hoc project work

• Design and optimize data schemas for LLM consumption
• Build robust data pipelines using Python and C#/.NET
• Implement and maintain Vector Databases to support Retrieval-Augmented Generation
• Work with Gravitee API Gateway to enforce data masking and access control
• Manage the OpenAPI and MCP metadata for AI agents

• Define and evolve the enterprise MDM architecture strategy in alignment with business priorities, enterprise architecture standards, and digital transformation goals.
• Design scalable data architecture patterns for managing core and master data across domains such as customer, product, vendor, reference, organizational, and other critical enterprise entities.
• Translate business architecture, capabilities, and processes into data architecture, domain models, and end-to-end enterprise data flows.
• Develop conceptual, logical, and physical data models to support MDM and broader enterprise data initiatives.
• Establish architecture principles, standards, and best practices for: Master data, Reference data, Metadata, Data quality, Data lifecycle management.
• Collaborate with business teams, product owners, enterprise architects, application teams, data engineering teams, and governance partners on global strategic initiatives.
• Partner with stakeholders to define critical data entities, business rules, matching and merging logic, survivorship rules, stewardship processes, and consumption patterns.
• Architect and document end-to-end data flows across upstream and downstream systems, ensuring consistency, traceability, reuse, and quality.
• Design integration patterns for: Master data synchronization, Event-based distribution, API-driven access, Batch pipelines, Real-time data exchange.
• Ensure alignment between MDM solutions and enterprise integration, analytics platforms, operational systems, and modern cloud data platforms.
• Embed data governance principles into architecture designs, including ownership, stewardship, lineage, controls, classification, and compliance requirements.
• Define and support frameworks for data quality, including profiling, standardization, deduplication, enrichment, monitoring, and issue remediation.
• Drive adoption of metadata management, lineage, and catalog capabilities to improve transparency and trust in enterprise data assets.
• Evaluate business and technical requirements and recommend target-state architecture solutions across cloud and enterprise platforms.
• Contribute to architecture roadmaps, reference architectures, capability assessments, and investment planning.
• Review solution designs and guide implementation teams to ensure adherence to architectural standards and long-term maintainability.
• Support strategic programs from discovery through implementation and operational transition.
• Identify opportunities to simplify the data landscape, reduce duplication, improve interoperability, and increase data reuse.
• Stay current on evolving technologies, architecture patterns, and MDM/data management trends, while mentoring team members and contributing to the broader architecture community of practice.

Data Architect – Master Data Management

• Responsible for acquiring, curating, and publishing data both on prem and in the cloud for analytical or operational uses for basic to moderate scenarios.
• Ensures the data is in a ready-to-use form that creates a single version of the truth across all data consumers, including business/technology users, reporting and visualization specialists and data scientists with coaching and support.
• Utilizes skills to translate business analytic requests/requirements into design, development, testing, deployment, and production maintenance tasks.
• Works with various technologies from big data, relational and non-relational databases, cloud environments, different programming languages and various reporting tools and is familiar with a few but requires training for some
• Consults on data projects of intermediate complexity, with limited coaching and guidance, to understand design requirement and existing business processes to lead in the design, development, and implementation of data products.
• Practices code management and integration with current architectural and data governance guidelines with reduced oversight required of leadership.
• Produces data building blocks, data models, and data flows for varying client requests such as dimensional data, standard and ad hoc reporting, data feeds, dashboard reporting, and data science research and exploration with limited assistance and coaching.
• Creates business user access methods to structured and unstructured data. Utilizes techniques such as mapping data to a common data model, natural language processing, transforming data as necessary to satisfy business rules, AI, statistical computations, and validation of data content.
• Translates business data stories into a technical story breakdown structure and work estimates for a schedule or planned agile sprint.
• Projects are of increasing complexity but require less coaching and assistance from leadership. Develops and maintains moderately complex scalable data pipelines for both streaming and batch requirements and builds out new API integrations to support increased demands of data volume and complexity.
• Preps/cleanses data to optimize for downstream reporting via Farmers standard visualization or AI/ML tools with coaching and feedback.
• Applies sound knowledge of data governance concepts with limited experience in application. Executes on mid size projects and large tasks within timelines.
• Develops data project plans and executes with some supervision.
• Performs other duties as assigned

See all jobs on JobTailor

Senior Data Engineer

Lead Data Engineer

Data Engineer II

Lead Data Engineer – AWS

Senior Data Engineer

Databricks Data Engineer

Senior Data Engineer – Snowflake Migration Lead

Data Engineer

Azure Data Engineer

D365 Customer Insight Data Engineer

AI Data Engineer

Data Engineer III

Data Engineer

Data Architect, Duck Creek/Guidewire

Data Engineering Technical Lead

Lead AWS Data Engineer

Data Engineer

Senior Data Engineer

Data Architect – Master Data Management

Data Engineer