Job Description:

• Design and implement robust data pipelines for loading (ETL) processes, extracting data from various sources and loading it into a data lake to ensure consistent, reliable data availability and workflow automation by using tools like Apache Airflow, Spark, or custom scripts.
• Evaluate organizational business strategies to advise on and design optimal data models, ensuring the data architecture supports business objectives by leveraging data modeling software and best practices in database design.
• Collaborate on data requirements   to understand and gather data requirements, designing efficient data models and schemas that facilitate effective data usage while ensuring adherence to data modeling standards and principles by working closely with cross-functional teams.
• Apply Python-based libraries such as Pandas, PySpark, or Dask for big data management tasks to manipulate and process large datasets effectively by optimizing data workflows and processing times.
• Manage data storage, processing, and automated functions on cloud platforms to leverage cloud technologies for scalable and efficient data handling and hosting solutions through cloud platforms, particularly Google Cloud Platform (GCP).
• Develop and maintain web scraping methods to acquire new and relevant data sources, enriching the data ecosystem and supporting broader data-driven insights by implementing web scraping techniques.
• Create and implement APIs to facilitate seamless data integration across various systems and platforms, ensuring interoperability and real-time data availability           by planning and designing API, using appropriate frameworks and tools, cloud service providers.
• Construct and scale data products and services  to meet the analytical and operational needs of the enterprise, enhancing data accessibility and utility across business function through encompasses planning, development, deployment, and continuous improvement.
• Monitor data performance and quality to ensure data integrity and accuracy throughout its lifecycle to support reliable decision-making by regularly monitoring, evaluating and implementing measures.
• Update knowledge on industry trends and emerging technologies to improve data engineering practices and introduce innovative solutions that keep the organization at the technological forefront through workshops, training sessions, professional communities, industry conferences, peers networking.

Skills:

• Good command of English                  
• Strong mathematical & statistical skills                      
• Strong verbal and written communication skills to clearly explain complex technical details and data-driven insights to non-technical stakeholders                        
• Excellent problem-solving skills to troubleshoot and optimize data pipelines                  
• Skills in developing APIs for data integration and for facilitating communication between different software systems                       
• Strong analytical and problem-solving skills to navigate challenges in data processing and integration                               
• Skills in automating manual processes and scripting with Python to enhance data workflows                     
• Generate insightful data visualizations and reports to support business decision-making (experience with tools like Tableau or Power BI is a plus)                            

Education:

Bachelor’s degree in computer science, engineering, or a related field                 

Knowledge:

• Proven experience (2 years+) as a Data Engineer or similar role in a complex data environment, preferably within the Food & Beverage industry                     
• Hands-on experience with cloud computing platforms, particularly Google Cloud Platform (GCP), for large-scale data storage, processing, automation & running ML environments               
• Proficiency in Python programming and experience with Python-based big data management libraries (e.g., Pandas, NumPy, Spark)                             
• Strong understanding of database concepts and experience with SQL and NoSQL databases                     
• Deep understanding of ETL (Extract, Transform, Load) processes, tools, and best practices to manage data pipelines efficiently                       
• Awareness of data security principles, data governance standards, and regulations (like GDPR, HIPAA) relevant to data management and processing                       
• Familiarity with web scraping techniques and tools for acquiring data from diverse online sources             
• Experience developing and consuming APIs for data integration purposes