Sr AWS Data Engineer

Job Description

Our client is looking for an experienced Data Engineer to join a newly formed team and work on an exciting new strategic project to create a common data platform for the organization.

You will work closely with Product Owners, Architecture, Data Science, QA and Release Management to develop data products directly impacting the business through increased operational efficiency and generated revenue. The position requires you to collaborate on requirements, produce appropriate solution designs, generate development tasks, and work with the team to implement the solution.

To apply for this role, in addition to having a proven technical background you must also have experience and track-record of working on enterprise-level applications. You will be joining a large team of diverse skills and experience and the job requires a motivated individual who is experienced in taking responsibility and can independently deliver results.



The role:

* Ingest from multiple sources and develop data processing pipelines;
* Automate test and deployment of your infrastructure;
* Work closely with business analysts, data engineers and product owners to gain deeper understanding of business processes and data requirements;
* Define and manage meta-data standards, data mappings and the data dictionary;
* Define data quality rules, validation checks and automated cleansing methods for data pipelines;
* Use data engineering and DevOps techniques on large datasets to gain business insight;
* Implement and test business logic, transformations and calculations to support data pipelines;

Qualifications

You Have:

* Good experience with Python and Spark;
* Expertise with AWS;
* Proficiency with data lake related technologies like Apache Hudi, Apache Iceberg or Databricks Delta Lake;
* Experience using DevOps and automation techniques (CI/CD, infrastructure-as-code, monitoring tools);
* Proficiency in test automation (TDD, unit tests, Deequ will be considered an asset);
* 3+ years experience with data-related process controls, data mapping, and ETL processes and tools;
* 3+ years experience working with relational databases;
* Well-developed analytical skills, quantitative, and problem-solving skills;