Senior Data Engineer

Headquartered in Abu Dhabi, United Arab Emirates, specializes in developing AI-powered supply chain solutions that integrate fragmented SAP, Ariba, and unstructured data into actionable intelligence. The company focuses on architecting ultra-secure data lakehouses to deliver real-time analytics, secure high-performance pipelines, and enable advanced generative AI applications, supporting critical enterprise workflows and procurement automation in defense and technical sectors. Job Summary We are seeking a Senior Data Engineer to architect and develop the core data infrastructure that will underpin our AI-driven transformation. In this pivotal role, you will move beyond traditional ETL processes to design and deploy high-security Data Lakehouse environments, establishing a single, authoritative source of truth for our AI systems. You will lead the technical implementation of 'Workstream 1: Data & Platform Foundations,' a critical initiative for our flagship projects. Your responsibilities will include integrating complex enterprise systems such as SAP S/4HANA and Ariba, as well as processing unstructured data like technical drawings and regulatory documents. By collaborating across teams, you will build high-performance pipelines and systems that power Intelligent Supply Chain forecasting and generative AI tools. Operating within a structured 'Sprint Zero' framework, you will ensure robust data lineage, security, and compliance with defense-grade standards. This role demands expertise in both structured and unstructured data engineering, with a focus on creating scalable, secure, and high-performance data architectures. Your work will directly enable AI agents to optimize procurement processes and support engineers in developing next-generation systems, making you instrumental in transforming raw data into strategic advantage for national defense capabilities. Key Responsibilities - Design and deploy defense-grade Data Lakehouse architectures to serve as the Single Source of Truth for AI-driven Supply Chain Intelligence, ensuring high-performance pipelines and systems for Intelligent Supply Chain forecasting and generative AI tools. - Lead the technical execution of Workstream 1: Data & Platform Foundations, mapping rigid enterprise systems (e.g., SAP S/4HANA, Ariba) and collaborating with cross-functional teams to integrate and process complex unstructured data (technical drawings, regulatory text) for AI applications. - Architect and deploy ingestion pipelines to extract high-volume transactional data from ERP systems like SAP S/4HANA, Ariba, and PLM, ensuring near real-time availability for forecasting models and AI agents. - Build connectors for external market intelligence feeds (e.g., S&P Global, Orbis, EcoVadis) to enrich internal procurement data with macroeconomic and geopolitical signals for enhanced decision-making. - Design and implement a standardized procurement data model and taxonomy across multiple entities, harmonizing fragmented datasets into a cohesive analytics layer. - Engineer pipelines to ingest, process, and transform unstructured technical data (PDF tender documents, CAD metadata, historical CONOPS) into vector-ready formats for Retrieval-Augmented Generation (RAG) applications. - Manage and optimize Vector Databases (e.g., Weaviate) to store embeddings of archival proposals and engineering snippets, ensuring high-speed retrieval for AI drafting assistants and generative tools. - Establish data lineage and traceability protocols to link requirements to physical components, supporting Model-Based Systems Engineering (MBSE) and the Digital Thread implementation. - Implement Role-Based Access Control (RBAC), audit logging, and data redaction policies to ensure compliance with export controls and strict on-premise security requirements. - Deploy automated data quality frameworks to validate Bill of Materials (BOM) completeness and cost data accuracy before ingestion into AI models. - Optimize data pipelines for on-premise GPU clusters and air-gapped environments, ensuring efficiency and performance within existing infrastructure constraints. - Operate within a structured Sprint Zero environment to ensure data lineage, security, and governance meet defense-grade standards. Qualifications And Experience - 5+ years of experience in Data Engineering, with at least 2 years focused on building pipelines for Machine Learning or Generative AI applications in an enterprise setting. - Expert proficiency in Python, SQL, and modern data engineering frameworks including Apache Spark, Kafka, and Airflow. - Strong experience extracting data from complex ERP environments, specifically SAP S/4HANA and SAP Ariba, with familiarity in SAP BTP as a plus. - Deep understanding of Data Lakehouse architectures (Databricks/Delta Lake), Relational Databases (PostgreSQL), and Vector Databases (Weaviate/Milvus). - Experience building pipelines for RAG solutions, Conversational AI agents, and classical ML models using tools such as dbt, dagster, or prefect. - Proficiency with containerization (Docker, Kubernetes) and CI/CD pipelines for deploying data workflows in secure environments. - Experience in Supply Chain, Manufacturing, or Defense sectors, with the ability to understand 'Bill of Materials' (BOM) structures and procurement lifecycles. - Ability to navigate the governance challenges between agile data work and rigid systems engineering requirements, ensuring data deliverables meet formal Stage Gate reviews. - Proven ability to collaborate with Data Scientists and Backend Engineers to define data schemas that support predictive modeling and AI agents.