Be the first to apply!

Data Engineer

Orysys Limited   Colombo • Full-time

We are seeking a highly skilled and experienced Data Engineer to lead the architecture, design, and deployment of our enterprise data infrastructure. You will bridge the gap between data engineering, backend services, and machine
learning operations (MLOps).

Key Responsibilities:

  • Pipeline Architecture: Architect and build highly scalable, fault-tolerant ETL/ELT pipelines from scratch.
  • Big Data Ecosystems: Design and manage open-source big data environments using Hadoop, Apache Spark, Kafka, and NiFi for high-throughput, real-time, and batch data processing.
  • Backend & API Integration: Lead backend development for data serving using Python FastAPI and Spring Boot. Secure APIs utilizing OAuth2 protocols.
  • Orchestration & Transformation: Utilize Apache Airflow for complex workflow orchestration and dbt for robust data transformation.
  • DevOps & Infrastructure: knowledge in Docker and Kubernetes. Leverage deep knowledge of Linux commands and network configurations for seamless deployments.
  • MLOps & Advanced Data: support MLOps practices for model deployment and monitoring. Handle complex time-series data storage and analysis.

Requirements and Qualifications:

  • Experience: Minimum 3 years of hands-on experience in Data Engineering.
  • Programming: Strong proficiency in Python. Solid experience with backend frameworks like FastAPI and Spring Boot.
  • Big Data & Streaming: experience with Hadoop environments, Apache Spark, Kafka, and NiFi.
  • ETL/Orchestration: Deep expertise in ETL pipeline creation, productionizing data workflows, Airflow, and dbt.
  • Databases: Advanced SQL skills with deep knowledge of DB2 SQL and PostgreSQL. Capable of handling massive time-series datasets.
  • Infrastructure & Security: Proficiency with Linux CLI, Docker, Kubernetes, network configuration, and OAuth2 implementation.
  • MLOps: Proven knowledge and hands-on experience with MLOps principles and productionizing machine learning workflows
  • Exposure to Google Cloud Platform (GCP) services and architecture. (or any other vendor)