Data Engineer

• Design, implement, and maintain scalable data pipelines using ELK Stack (Elasticsearch, Logstash, Kibana) and Beats for monitoring and analytics.
• Develop data processing workflows to handle real-time and batch data ingestion, transformation and visualization.
• Implement techniques like grok patterns, regular expressions, and plugins to handle complex log formats and structures.
• Configure and optimize Elasticsearch clusters for efficient indexing, searching, and performance tuning.
• Collaborate with business users to understand their data integration & visualization needs and translate them into technical solutions.
• Create dynamic and interactive dashboards in Kibana for data visualization and insights that can enable to detect the root cause of the issue.
• Leverage open-source tools such as Beats and Python to integrate and process data from multiple sources.
• Collaborate with cross-functional teams to implement ITSM solutions integrating ELK with tools like ServiceNow and other ITSM platforms.
• Anomaly detection using Elastic ML and create alerts using Watcher functionality.
• Extract data by Python programming using API.
• Build and deploy solutions in containerized environments using Kubernetes.
• Monitor Elasticsearch clusters for health, performance, and resource utilization.
• Automate routine tasks and data workflows using scripting languages such as Python or shell scripting.
• Provide technical expertise in troubleshooting, debugging, and resolving complex data and system issues.
• Create and maintain technical documentation, including system diagrams, deployment procedures, and troubleshooting guides.

ELK Data Engineer

Data Engineer

Data Engineer – SME

Data Engineer – MTS

Data Engineering Manager, Big Data