Senior Data Engineer - San Francisco, United States - Django REST framework

Description

People Data Labs is hiring a
Remote Senior Data Engineer

About Us
At People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy.

We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education data consumed through our suite of APIs.

PDL is an innovative, fast-growing, global team backed by world-class investors, including Craft Ventures, Flex Capital, and Founders Fund.

We scour the world for people hungry to improve, curious about how things work, and willing to challenge the status quo to build something new and better.

Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks

Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets.

Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production.
Devising solutions to largely-undefined data engineering and data science problems.
Work with stakeholders in Engineering and Product to assist with data-related technical issues and support their infrastructure needs
Technical Requirements
5-7+ years industry experience with clear examples of strategic technical problem solving and implementation
Experience withPython Expertise with Apache Spark (Java, Scala, and/or Python-based)
Experience with SQL
Experience building scalable data processing systems (e.g., cleaning, transformation) from the ground up.
Experience using developer-oriented data pipeline and workflow orchestration (e.g., Airflow (preferred), dbt, dagster or similar)
Knowledge of modern data design and storage patterns (e.g., incremental updating, partitioning and segmentation, rebuilds and backfills)
Experience working in Databricks (including delta live tables, data lakehouse patterns, etc.)
Experience with cloud computing services (AWS (preferred), GCP, Azure or similar)
Experience with data warehousing (e.g., Databricks, Snowflake, Redshift, BigQuery, or similar)
Understanding of modern data storage formats and tools (e.g., parquet, ORC, Avro, Delta Lake)
Professional Requirements
Must thrive in a fast paced environment and be able to work independently
Can work effectively remotely (able to be proactive about managing blockers, proactive on reaching out and asking questions, and participating in team activities)
Strong written communication skills on Slack/Chat and in documents
You are experienced in writing data design docs (pipeline design, dataflow, schema design)
You can scope and breakdown projects, communicate and collaborate progress and blockers effectively with your manager, team, and stakeholders

Nice To Haves:
Degree in a quantitative discipline such as computer science, mathematics, statistics, or engineering
Experience working with entity data (entity resolution / record linkage)
Experience working with data acquisition / data integration
Expertise with Python and the Python data stack (e.g., numpy, pandas)
Experience with streaming platforms (e.g., Kafka)
Experience evaluating data quality and maintaining consistently high data standards across new feature releases (e.g., consistency, accuracy, validity, completeness)
Stock
Unlimited paid time off
Health, fitness, and office stipends
The permanent ability to work wherever and however you want
No C2C, 1099, or Contract-to-Hire. Recruiters need not apply.

People Data Labs does not discriminate on the basis of race, sex, color, religion, age, national origin, marital status, disability, veteran status, genetic information, sexual orientation, gender identity or any other reason prohibited by law in provision of employment opportunities and benefits.

Salary and compensation

No salary data published by company so we estimated salary based on similar jobs related to Design, Python, Education, Cloud, Senior and Engineer jobs that are similar:
$65,000 — $105,000/year
Distributed team
Paid time off
4 day workweek
Company retreats
Mental wellness budget
Equity compensation
No whiteboard interview
No politics at work
We hire old (and young)
Location

Please reference you found the job on Remote OK, this helps us get more companies to post here, thanks

When applying for jobs, you should NEVER have to pay to apply. You should also NEVER have to pay to buy equipment which they then pay you back for later. Also never pay for trainings you have to do.

Those are scams NEVER PAY FOR ANYTHING Posts that link to pages with "how to work online" are also scams.

Don't use them or pay for them. Also always verify you're actually talking to the company in the job post and not an imposter.

A good idea is to check the domain name for the site/email and see if it's the actual company's main domain name.

Scams in remote work are rampant, be careful Read more to avoid scams .

When clicking on the button to apply above, you will leave Remote OK and go to the job application page for that company outside this site.

Remote OK accepts no liability or responsibility as a consequence of any reliance upon information on there (external sites) or here.

#J-18808-Ljbffr