Jobs
>
Remote

    Web Scraping Data Engineering Manager - Remote, United States - Comply

    Comply
    Comply Remote, United States

    3 weeks ago

    Default job background
    Full time
    Description

    As the Data Engineering Manager for web scraping at COMPLY, you will play a pivotal role in the design and implementation of advanced web scraping tools and services. You will lead the development of workflows that enhance our data extraction capabilities, ensuring high accuracy and quality through diligent monitoring and testing. Under your guidance, our team will efficiently scale operations to meet increasing data demands. Your leadership will extend to mentoring in-house staff and managing offshore contractors, ensuring all activities align with COMPLY's strategic goals.

    Reporting directly to the Head of Data & Analytics, you will spearhead efforts to refactor our web scraping frameworks, significantly improving the speed, accuracy, and durability of our data processing. Your insights and deep technical expertise will be key in sustaining COMPLY's competitive advantage and establishing new benchmarks in data excellence.

    Job Responsibilities:

    • Robust System Design: Develop and refine advanced web scraping tools and services to efficiently extract and ingest data from diverse web sources.
    • Proactive Monitoring: Implement comprehensive monitoring systems to track the performance and health of web scraping operations, enabling early detection of issues before they affect data quality.
    • Quality Assurance: Conduct rigorous testing partnering with QA to ensure the accuracy and reliability of data collected through web scraping. Implement quality control measures to maintain high standards.
    • Leadership: Lead a team of geographically distributed data engineers, providing mentorship and direction to enhance skills and foster a collaborative, innovative work environment.
    • Issue Resolution: Quickly identify and resolve technical issues and discrepancies in the data collection process, ensuring minimal disruption to operations.
    • Continuous Improvement and Innovation: Continuously evaluate and improve scraping processes and methodologies to increase efficiency and adapt to changing technological landscapes.
    • Strategic Planning: Collaborate with cross-functional teams to align web scraping activities with overall business objectives and strategic goals.
    • Project Execution: Ensure timely execution of project deliverables by overseeing project schedules and resource allocation.
    • Continuous Learning: Proactively stay current with advancements and trends in web scraping technologies to keep our practices in line with top industry standards.

    Qualifications:

    • Education: Bachelor's or higher in Computer science, engineering, analytics, or information technology or similar.
    • Experience: 10+ years in data engineering with 2+ years focused on web scraping. 3+ years managing a team.
    • Technical Expertise: High Proficiency in Python a must Experience with Celery, RabbitMQ, and Django to manage existing web scraping processes. Understanding of Elasticsearch indices, complex queries, and search optimization
    • Web Scraping Framework: Deep knowledge in Scrapy, Beautiful Soup, Selenium or similar framework
    • Leadership: 3+ Experience in senior or lead roles, demonstrating growth and increased responsibility in data handling.
    • Web Technologies: Thorough understanding of HTML, CSS, and JavaScript to navigate complex websites.

    Nice to Have:

    • Industry Knowledge: Experience in financial services, political contributions or other relevant industries
    • Certifications: AWS (preferred), Azure, or GCP. Data specific certification a plus.
    • Other Technologies: Kubernetes, Redis, Jenkins, Postgres, ETL, dbt, NoSQL, SQL

    COMPLY is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, disability, sex, sexual orientation, gender identity, or national origin. Nothing in this job posting should be construed as an offer or guarantee of employment.

  • WPRO TALENTS

    Engineering Manager

    2 weeks ago

    WPRO TALENTS Remote, United States

    This is a remote position. · Our client is a wallet for everything onchain. It's the safest & easiest way to explore the web3. · Our client is looking for an experienced and driven Engineering Manager to help us grow and solidify our engineering team. You will focus on people an ...

  • Zerion

    Engineering Manager

    1 week ago

    Zerion Remote, United States Full time

    Zerion is a wallet for everything onchain. It's the safest & easiest way to explore web3. · We're looking for an experienced and driven Engineering Manager to help us grow and solidify our engineering team. You will focus on people and the engineering process so that we can conti ...

  • Secureframe

    Engineering Manager

    2 weeks ago

    Secureframe Remote, United States Full time

    The Engineering Manager role at Secureframe is a key position spearheading the development and refinement of the workforce product set, focusing initially on debugging and then on rebuilding policies and user access frameworks in collaboration with the Product Manager. Joining Se ...

  • Cortex

    Engineering Manager

    1 week ago

    Cortex Remote, United States Full time

    About Cortex · Cortex is the internal developer portal that cuts noise for developers with paved paths to production. Companies like Netflix, Dropbox, and Adobe use Cortex to catalog, score, and assign action to improve service quality and velocity, so devs can get back to work t ...

  • Bolt

    Engineering Manager

    1 week ago

    Bolt Remote, United States Full time

    · Bolt is on a mission to democratize commerce. We relentlessly prioritize our retailers—putting their brands front and center while enabling frictionless shopping at any touchpoint in the customer journey. At the center of it all is our rapidly growing universal shopper network ...

  • Dutchie

    Engineering Manager

    3 days ago

    Dutchie Remote, United States Full time

    · About Dutchie · Founded in 2017, Dutchie is a comprehensive technology platform powering dispensary operations, while providing consumers with safe and easy access to cannabis. Dutchie aims to further support the positive societal change the cannabis industry brings to the wor ...

  • Donnelley Financial Solutions

    Engineering Manager

    1 week ago

    Donnelley Financial Solutions Remote, United States Full time

    Donnelley Financial Solutions (DFIN) is a leader in risk and compliance solutions, providing insightful technology, industry expertise and data insights to clients across the globe. We're here to help you make smarter decisions with insightful technology, industry expertise and d ...

  • Smartsheet

    Manager, Engineering

    3 weeks ago

    Smartsheet Remote, United States Full time

    Smartsheet, a leader in collaborative project management and productivity software, is seeking a Manager, Engineering to spearhead one of our Content Automation teams in Australia. This role is pivotal in our mission of integrating and transforming content between structured and ...

  • O(1) Labs

    Engineering manager

    3 weeks ago

    O(1) Labs Remote, United States Full time

    We are looking for talented and motivated engineers to join the Engineering team at O(1) Labs. This is a chance to join a small, collaborative team and have a ton of independence while working on fascinating cross-disciplinary problems in computing and building on the first crypt ...

  • 7shifts

    Engineering Manager

    6 days ago

    7shifts Remote, United States Full time

    · 7shifts is an all-in-one team management software platform designed for the restaurant industry. We're used by 1 in 10 restaurant workers and over 50,000 restaurants across North America. Our mission is to simplify Team Management and improve performance for restaurants, with ...

  • DFIN

    Engineering Manager

    2 weeks ago

    DFIN Remote, United States Full time

    Donnelley Financial Solutions (DFIN) is a leader in risk and compliance solutions, providing insightful technology, industry expertise and data insights to clients across the globe. We're here to help you make smarter decisions with insightful technology, industry expertise and d ...

  • Cloudflare

    Engineering Manager

    3 weeks ago

    Cloudflare Remote, United States Full time

    About Us · At Cloudflare, we are on a mission to help build a better Internet. Today the company runs one of the world's largest networks that powers millions of websites and other Internet properties for customers ranging from individual bloggers to SMBs to Fortune 500 companies ...

  • Rentable

    Engineering Manager

    2 weeks ago

    Rentable Remote, United States Full time

    We're Changing the Rentals Industry · We're a profitable, growth-stage company building industry-leading martech and data SaaS products for the rentals industry. While originally known for building and operating one of the U.S.'s largest rental marketplaces - Rentable, we recentl ...

  • BuyChain

    Engineering Manager

    1 week ago

    BuyChain Remote, United States Full time

    Lead the growing global team of developers and engineers · Provide Additional Dev Support · Support CTO in weekly sprint planning and management · Support developers with issues and confirmation on tasks · Support developers in the code or architecture as needed or bandwidth all ...

  • Paytient

    Engineering Manager

    1 day ago

    Paytient Remote, United States Full time

    · About Paytient: · We're on a mission to help people better access and afford care. · Every day, millions of people, and their loved ones, need to see a doctor. For most of us, that moment is an uncertain one - we're unsure of what's wrong, who to go to, how long it'll take to ...

  • E2 Optics

    Program Engineer Manager

    3 weeks ago

    E2 Optics Remote, United States Full time

    E2 Optics is an award-winning, Woman-Owned technology integrator, headquartered in Denver, Colorado. We help clients deploy remarkably efficient solutions that are modern, turnkey, and minimize operational costs. We design low voltage solutions that seamlessly integrate audiovisu ...

  • Ciena

    Engineering Project Manager

    2 weeks ago

    Ciena Remote, United States Full time

    Ciena is committed to our people-first philosophy. Our teams enjoy a culture focused on prioritizing a personalized and flexible work environment that empowers an individual's passions, growth, wellbeing and belonging. We're a technology company that leads with our humanity—drivi ...

  • ServiceNow

    Manager, Software Engineering Management

    1 week ago

    ServiceNow Remote, United States Full time

    Company Description · At ServiceNow, our technology makes the world work for everyone, and our people make it possible. We move fast because the world can't wait, and we innovate in ways no one else can for our customers and communities. By joining ServiceNow, you are part of an ...

  • TransUnion

    Software Engineering Managers

    1 week ago

    TransUnion Remote, United States Full time

    We'd Love to See: · Job Description · Software Engineering Managers for Various and unanticipated worksites throughout the U.S. (HQ: Chicago, IL). Manage a software engineering team to lead client focused product development. Responsible for managing software design, architectur ...

  • Hypixel Studios

    Platform Engineering Manager

    3 weeks ago

    Hypixel Studios Remote, United States Full time

    We're looking for a Platform Engineering Manager to join our team at Hypixel Studios, which collaborates remotely from around the world. Our members range from industry newcomers to experts with 25+ years of experience. Team members come from a diverse set of backgrounds, but sha ...