Data Engineer - Boston, United States - Gardner Resources Consulting

Description

Our healthcare client is committed to leveraging innovative data engineering solutions, particularly the Medallion architecture, to drive operational excellence and deliver transformative insights in patient care.

• Minimum of 3 years' experience with Databricks, PySpark, SQL, Spark clusters, and Jupyter Notebooks.
• Expertise in building data lakes using the Medallion architecture and working with delta tables in the delta file format.
• Familiarity with CI/CD pipelines and Agile methodologies, ensuring efficient and collaborative development practices.
• Strong understanding of ETL processes, data modeling, and data warehousing principles.
• Experience with data visualization tools like Power BI is a plus.
• Knowledge of cybersecurity data, particularly vulnerability scan data, is preferred.
• Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

• Develop, test, and maintain largescale data processing systems using Databricks, PySpark, SQL, Spark clusters, delta tables, and the innovative Medallion architecture.
• Collaborate closely with cybersecurity analysts to gather data requirements and deliver effective solutions aligned with Medallion architecture principles.
• Ensure data quality and implement robust data governance standards, leveraging the scalability and efficiency offered by the Medallion architecture.
• Design and implement ETL processes, including data cleansing, transformation, and integration, optimizing performance within the delta file format framework.
• Build and manage data lakes based on Medallion architecture principles, ensuring scalability, reliability, and adherence to best practices.
• Monitor and optimize data pipelines, integrating CI/CD practices to streamline development and deployment processes.
• Collaborate with crossfunctional team members to implement data analytics projects, utilizing Jupyter Notebooks and other tools to harness the power of the Medallion architecture.
• Embrace Agile methodologies throughout the development lifecycle to promote iterative and collaborative development practices, enhancing the effectiveness of Medallionbased solutions.