Optimise scraper system

Only for registered members United States

1 month ago

Default job background
I have a complex scraper system in python, scraping several sites 24/7, and many many URLs on these several sites. · It runs across multiple virtual machines. It is not set up efficiently to scale, and is very heavy on CPU usage, hence having to run it across a number of virtual ...
Lorem ipsum dolor sit amet
, consectetur adipiscing elit. Nullam tempor vestibulum ex, eget consequat quam pellentesque vel. Etiam congue sed elit nec elementum. Morbi diam metus, rutrum id eleifend ac, porta in lectus. Sed scelerisque a augue et ornare.

Donec lacinia nisi nec odio ultricies imperdiet.
Morbi a dolor dignissim, tristique enim et, semper lacus. Morbi laoreet sollicitudin justo eget eleifend. Donec felis augue, accumsan in dapibus a, mattis sed ligula.

Vestibulum at aliquet erat. Curabitur rhoncus urna vitae quam suscipit
, at pulvinar turpis lacinia. Mauris magna sem, dignissim finibus fermentum ac, placerat at ex. Pellentesque aliquet, lorem pulvinar mollis ornare, orci turpis fermentum urna, non ullamcorper ligula enim a ante. Duis dolor est, consectetur ut sapien lacinia, tempor condimentum purus.
Get full access

Access all high-level positions and get the job of your dreams.



Similar jobs

  • Work in company Remote job

    Build Automated Nationwide Contractor Lead Scraper + Daily Email Delivery System

    Only for registered members

    +Job summary · I need an experienced developer to build a stable, daily automated scraping system. · +Responsibilities · Deduplicate data ...

    Freelance

    2 weeks ago

  • Work in company Remote job

    Python Developer Needed for Web Scraper Integration

    Only for registered members

    We are seeking a skilled Python developer to create a web scraper that will seamlessly integrate with our existing backend system. The ideal candidate should have experience in developing scrapers that can efficiently extract and process data. Strong familiarity with Python libra ...

    $10 - $15 (USD) per hour

    5 days ago

  • Work in company Remote job

    Web Scraping Expert for News Aggregation

    Only for registered members

    We are seeking an experienced web scraping specialist to develop a scraper that collects news articles from a predefined list of websites across multiple categories and stores them on a WordPress website. · ...

    1 month ago

  • Work in company Remote job

    Senior Airflow and Playwright Selenium Scraper Engineer

    Only for registered members

    We're looking for a senior engineer to help us with our scraping architecture. · turn this system into something that runs with absolute consistency, ...

    $20 - $45 (USD) per hour

    1 month ago

  • Work in company Remote job

    Robust Web Scraper and Cron Job Deployment

    Only for registered members

    We are seeking a skilled freelancer to build a robust web scraper and deploy a cron job for it on our system. This is a part-time project with a short duration, perfect for someone looking to demonstrate their skills in data scraping and automation. · ...

    $80 - $0 (USD) budget

    1 week ago

  • Work in company

    Scraper Operator

    Only for registered members

    The Scraper Operator is responsible for safely and efficiently operating scraper equipment to move, · load,hual,spreed,andgradeearthmaterialsinsupportoflargescaleearthmovingandsitedevelopmentprojects. · Thisrolerequiresexactequipmentcontrol,strongawarenessofgradeandhaulconditions ...

    Cheyenne Full time

    1 month ago

  • Work in company Remote job

    Python Web Scraper Setup

    Only for registered members

    We are looking for a skilled freelancer to assist with the setup and configuration of an existing Python web scraper. The code has already been written, but we need an expert to help us ensure it's correctly configured to scrape data from our specified sources. · This role requir ...

    1 week ago

  • Work in company

    Senior Engineering Analyst, Search Antiscraper

    Only for registered members

    We're reimagining what it means to search for information – any way and anywhere. · In joining the Search team, you'll have an opportunity to make an impact on billions of people globally. · Investigate and analyze patterns of abuse on Google Search. · Analyze datasets to identif ...

    Mountain View $174,000 - $258,000 (USD)

    2 weeks ago

  • Work in company Remote job

    Web Scraper Development for Leaderboards

    Only for registered members

    We are looking for a serious and capable person who understands real-world web scraping challenges, can reason through inconsistent, dynamic site behavior, implements practical, scalable solutions, focuses on reliability, performance and correctness. · ...

    3 weeks ago

  • Work in company Remote job

    Apify + Python Web Scraping Expert for Automated Tender Monitoring

    Only for registered members

    I'm building an automated monitoring system that scans government offices, school districts, and public organizations' noticeboards and procurement portals to identify tender-related documents and early-stage procurement opportunities. · The goal is to detect potential leads (ten ...

    $10 (USD) per hour

    1 week ago

  • Work in company Remote job

    Experienced Web Scraping Specialist for Ticket Marketplaces

    Only for registered members

    We're looking for a highly experienced web scraping professional to support data collection from primary and secondary ticket marketplaces and closely related event-access platforms. · ...

    1 month ago

  • Work in company Remote job

    Data Science

    Only for registered members

    We are looking for a strong Machine Learning Engineer / Automation Engineer who will build intelligent systems for discovering strategic auxiliary segments — additional flight segments and routes that dramatically reduce the price of premium tickets while remaining 100% compliant ...

    1 month ago

  • Work in company Remote job

    Complex Data Scraping

    Only for registered members

    We're looking for a Senior Engineer to own our property data pipeline. · Design and operate distributed scraping systems processing 300K+ records daily with 99.5%+ reliability. · Architect parallel processing pipelines with concurrent workers, intelligent rate limiting, and horiz ...

    $30 - $70 (USD) per hour

    1 month ago

  • Work in company Remote job

    Build N8N server on AWS and automate tasks including scraping and email

    Only for registered members

    Project Overview · We currently use + Apify to scrape product data. While functional, the process is too slow for large datasets. · We need to move to a faster, scalable architecture built in N8N, capable of efficiently scraping: · -5,000 – 20,000 SKUs per batch · -Large dataset ...

    1 week ago

  • Work in company Remote job

    Apify + Python Web Scraping Expert for Tender Monitoring System

    Only for registered members

    I'm building an automated monitoring system that scans government offices, school districts, and public organizations' noticeboards and procurement portals to identify tender-related documents and early-stage procurement opportunities. · The goal is to detect potential leads (ten ...

    $10 - $18 (USD) per hour

    1 week ago

  • Work in company Remote job

    Full-Time Lead Scraper

    Only for registered members

    We're looking for a full-time lead scraper who can build and manage high-intent lead generation systems for multiple outbound campaigns. · ...

    $250 - $0 (USD) budget Full time

    2 weeks ago

  • Work in company Remote job

    Recruitment Operations

    Only for registered members

    You will start by fixing a messy CRM structure, move into automating our entire market mapping process, · as well as our reporting system and finally internalize our lead generation to replace a high-cost external · agency. · Phase 1: CRM Infrastructure & Data Architecture · The ...

    $8 - $25 (USD) per hour

    1 week ago

  • Work in company Remote job

    Web Scraper Developer

    Only for registered members

    We are looking for a Web Scraper Developer to build a system that automatically collects airport lounge data from official sources and outputs it in a structured format. The goal is to create a reusable, config-driven scraper that can be extended to many airports. · I want you to ...

    1 month ago

  • Work in company Remote job

    Full Stack Dev Needed: Facebook Marketplace Scraper

    Only for registered members

    A web based SaaS where users enter search criteria for Facebook Marketplace. The system monitors these searches 24/7 and sends an instant SMS notification when a new listing is found. · Must monitor Facebook Marketplace listings. · ...

    $15 - $35 (USD) per hour

    1 month ago

  • Work in company Remote job

    Automated deal-scraping engine

    Only for registered members

    Automated deal-scraping engine, scrape multiple sources, deduplicate listings and store data in a partitioned PostgreSQL database. · Technical Stack · • Language: (v20+) Non-blocking I/O · • Scraping: Playwright with playwright-stealth. · • Database: PostgreSQL (v16+) with pgvec ...

    $150 - $0 (USD) budget

    4 days ago