Overview:
We are seeking a highly skilled and experienced Senior Data Engineer to join our team. The ideal candidate will possess a strong background in data architecture, management, and integration, particularly within the energy commodities or financial services sectors. This role will involve designing robust data solutions, ensuring data quality, and implementing ETL processes to support our business objectives.
Key Responsibilities:
* Data Architecture Development: Design, implement, and maintain scalable and reliable data architecture frameworks that support the organization's strategic goals and objectives.
* ETL/ELT Design and Implementation: Architect and develop ETL and ELT processes utilizing SnapLogic and other integration tools to extract, transform, and load data from various sources into centralized data stores.
* Data Modeling: Collaborate with business stakeholders to create comprehensive data models, optimizing logical and physical data design and ensuring alignment with business requirements.
* Data Management and Governance: Establish and enforce data management best practices, including data governance frameworks to ensure data accuracy, consistency, and availability across the organization.
* Quality Assurance: Conduct comprehensive data quality assessments and audits, identifying and resolving data issues to maintain high standards of data integrity.
* Data Integration: Facilitate seamless data integration across various systems and platforms by developing and maintaining data ingestion pipelines that support real-time and batch processing.
* Collaboration with Teams: Work closely with data scientists, business analysts, and software engineers to understand data needs and provide suitable data solutions and architecture for analytics and reporting.
* Cloud Database Management: Manage relational and non-relational databases using Snowflake and Oracle, including performance tuning, optimization, and effective indexing strategies.
* Data Analysis and Visualization: Utilize Python and business intelligence tools (Power BI, Tableau) to analyze complex datasets and create meaningful visualizations and reports that facilitate data-driven decision-making.
* Machine Learning Implementation: Collaborate with machine learning practitioners to implement machine learning solutions, ensuring the proper infrastructure and data pipelines are in place to support model development and deployment.
* Training and Knowledge Transfer: Mentor and train junior staff, providing knowledge transfer on data architecture and engineering best practices, tools, and technologies.
* Vendor Management: Evaluate, select, and manage relationships with vendors and technology partners to integrate new data tools and solutions, staying abreast of vendor trends that impact data strategies.
* Research and Innovation: Stay current with emerging trends, technologies, and best practices in data architecture, big data analytics, and AI/ML, recommending innovative solutions that enhance the organization's data capabilities.
Relevant Skills and Qualifications:
* Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field; Master's degree preferred.
* Proven experience in data architecture, data engineering, or a related field.
* Proficiency in SQL, T-SQL, and data modeling techniques.
* Experience with ETL/ELT tools, specifically SnapLogic.
* Familiarity with cloud data warehousing solutions, particularly Snowflake or Oracle.
* Strong programming skills in Python, with experience in data libraries such as Pandas and NumPy.
* Familiarity with modern data integration and data pipeline architectures.
* Knowledge of machine learning and artificial intelligence concepts and tools.
* Experience using BI tools (Power BI, Tableau) for data visualization and reporting.
* Excellent analytical skills with a focus on data quality and management.
* Strong communication and interpersonal skills, with experience in vendor management.
