itpro.lk

applicant

• Design, build, and maintain scalable pipelines using PySpark on AWS. • Experience of Databricks on AWS is a must have • Work with AWS services such as S3, Glue, EMR, and Redshift for data storage, transformation, and querying. • Hands-on experience with Redshift, including performance tuning and data modelling. Strong SQL skills with experience in querying and optimizing large datasets. • Manage and monitor data workflows using orchestration tools like Apache Airflow • Knowledge of CI/CD workflows for data engineering projects. • Utilize Git for version control, ensuring proper collaboration and tracking of code changes. • Establish and follow best practices for repository management, branching, and code reviews. • Good to have DBT Exposure -Contribute to DBT transformations and assist in setting up data modeling workflows. • Working experience on Agile and Scrum methodologies