Senior Data Engineer

Job Description

Veracity Digital creates software solutions for startups to mature organizations to bring their ideas into life! Our main focuses is on delivering Al-driven digital transformation solutions. We help businesses with integrating artificial intelligence, machine learning, application development, data analytics, and cloud services, to help businesses innovate and operate more efficiently.

Personal Profile

• 4+ years of experience as a Data Engineer, building production-grade, large-scale data platforms.
• Proficiency in designing, developing, and optimizing ETL pipelines with Apache Spark for batch and streaming data workloads.
• Hands-on expertise scheduling and monitoring data wokflows in Apache Airflow, including Astronomer, with strong proficiency in authoring DAGS to automate ingestion, cleaning, pseudonymization, transformation and loading processes.
• Advanced Python and SQL skills for developing, tuning, and optimizing large-scale data workflows.
• Demonstrated experience archtecting and managing cloud-based data warehouses with Amazon Redshift, Snowflake, Trino (Presto), and DBT.
• Proven ablity to develop CI/CD pipelines, containerize deliverables using Docker, and orchestrate deployments in Kubernetes.
• Sedity utilizing Trino (Presto) for federated querying across Redshift, S3, and other sources for real-time, unified analytics.
• Strong experience building BI solutions with Power BI, Metabase,
• Apache Superset, and AWS QuickSight.
• Experience with data migration tools such as Airbyte for ensuring data quality and integrity.
• Exceptional problem-solving, communication, and collaboration skills, with a track record of mentoring peers and driving complex, cross-tearnnines.