Digitek Resourcing Ltd T/A DigiTech Resourcing
Big Data Engineer
·Financial Market Sector
·£65-75k PA
·London offices / Hybrid working
Intro context:
Are you looking to work in an increasingly competitive market? With data being essential to improving business decisions in providing customer service. Through strategic data, we want to establish a strong data platform that meets the data provisioning needs of consumers.
through a cloud first approach, we’re able to provide on demand performance and reduce time to market applying solid methodologies and automation. Reporting to the Senior Data Engineering Lead. The role requires interaction with all areas of the Business and close collaboration with other IT departments.
KEY RESPONSIBILITIES & ACCOUNTABILITIES
·Translate raw data into usable datasets to support business requirements.
·Deliver robust, high-performing and scalable solutions for loading, transforming and processing company data.
·Develop adhering to the current development best practices whilst contributing to the continual improvement of these practices.
·Maintain formal data and metadata structures (lineage) to support integrity and searchability of information via a data dictionary.
·Provide handover documentation for delivered solutions.
·Working collaboratively with:
·the data engineer lead to understanding policy and strategies
·our data architect to define and build data marts
·our data platform team to ensure high level of performance and provide support for delivered solutions
·Provide consultation to Project Managers, Business Analysts and Managers around current processes, best practices and ideas to solve business challenges
In addition, this will:
·Reduce Costs and Grow our business – Proactively identify and act on opportunities to reduce waste, improve efficiency, increase productivity and grow our business.
·Play my part in embedding the Blueprint Behaviours and put the customer at the centre of everything we do – Actively ensure the behaviours we have committed to as an organisation are demonstrated, reinforced and recognised in my own area of influence
·Risk Management – Identify, assess, document, manage, escalate and report the risks associated with my role and business area, ensuring an appropriate control environment
Foundational skills:
·Deliver high-performing and scalable data pipelines for loading, transforming and processing structured and semi structured data using near real time and batch processing connecting to multiple data sources and populating multiple targets on-prem and cloud.
·Working collaboratively with:
oOur Principal and Lead data engineer to understanding policy and strategies
oour Data Architect to define and build data marts
oour Data Platform team to ensure high level of performance and provide support for delivered solutions. Note: Support for delivered solutions will fall in remit of RUN team but support skills are expected to continue Level 1 to Level 3 for a period of time.
·Provide handover documentation for delivered solutions and consultation to Project Managers, Business Analysts and Managers around current processes and ideas to solve business challenges
Development focused skills:
·Ensure adherence to current development best practices and active contribution to the evolution of setting standards and improve development methodologies
·Maintain formal data and metadata structures (lineage) to support integrity and searchability of information via a data dictionary
·Create re-usable metadata driven data pipelines which can be leveraged for future development.
·Strong knowledge and 3+ years development experience of Big Data technologies such as Nifi, Azure Data Factory, Hive, Spark, Kafka, HBase
·5+ years’ Experience of end-to-end implementation of Data Engineering and Analytics over a Cloud infrastructure, preferably with Azure. Technically versatile with Azure offerings including Databricks or Spark, Azure Data Factory, Blob storage, Azure Data Lake
·Experience developing data pipelines using agile & DevOps.
·Strong troubleshooting, performance tuning, Security awareness
APPLICATION OF KNOWLEDGE, PROBLEM SOLVING & INNOVATION
·Continue research and learning of data engineering tool and capabilities to improve the time to market and maintainability of the data solutions.
·Challenge status quo and explore alternatives.
·Understand and apply regulatory and compliance considerations to all data processing solutions, GDPR and PCI.
·Contribute to Agile ALM including refinement, planning, release and retrospective sessions
·Promote and emphasize a test strong framework and automation to support CI and CD
INTERACTION/INTERPERSONAL SKILLS
·Sharing knowledge and experience with Data Engineers.
·Liaise with Data Integration Engineers, Data Engineers and Data architect to improve solution design.
·Communicate changes through Change Management processes
·Escalate issues, delays and concerns to the Data Engineer Lead and relevant Business stakeholders
Occasional travel to other sites to engage with business analysts in understanding their requirements
working conditions
·37.5 hours per week. Based in London with occasional attendance in other locations as and when required
·Owing to present Covid restrictions this role will need the candidate working remotely from home interim. Necessary guidance, equipment and accesses will be provided
Next steps – how to apply?
If this is something you wish to be considered for? Or require more information about the role? Please Apply now.