Talent AgentClassic Search

23 batch data engineer jobs at 20 companies in San Francisco, CA

1mo
Save
Mark Applied
Hide
Data Engineer
San Francisco or New York City
HybridFull Time
Beast Industries
Beast Industries: Privately held creator-led holding producing digital entertainment, snacks, software, and consumer brands for global audiences.
3+ YOE3+ years building and operating high-volume production data pipelines, with streaming and batch experience, event instrumentation, schema design, data quality, event-driven architectures, and major cloud data stacks.
AWS, GCP, Databricks, Snowflake
Job PostingView all
1w
Save
Mark Applied
Hide
Senior Data Engineer (Data Platform)
El Segundo or Austin or Washington or San Francisco
$80k-$210k/yr OnsiteFull Time
Gallatin AI
Gallatin AI: Defense technology providing AI-powered logistics decision-support software to military and allied defense organizations.
8+ YOE8+ years data engineering experience; production experience with open table formats (Iceberg/Delta/Hudi), distributed query engines (Trino/Presto/Spark SQL), streaming/batch pipelines (Kafka/NiFi/Debezium/Spark), Python, SQL, Kubernetes, Docker, AWS data services; ability to obtain security clearance.
Apache Iceberg, Delta, Hudi, Trino, Presto, Spark SQL, Kafka, NiFi, Debezium, Spark, Python, SQL, Airflow, Dagster, Argo Workflows, TypeScript, Vue, React, Kubernetes, Docker, AWS S3, AWS Glue, AWS Aurora, Azure, Nessie, Polaris, OpenLineage, DataHub, lakeFS, Palantir Foundry
Job PostingView all
3mo
Save
Mark Applied
Hide
Senior Data Engineer
San Francisco, California, United States
$180k-$220k/yr HybridFull Time
Metriport
Metriport: Open-source healthcare data infrastructure providing APIs that help digital health organizations access and exchange patient data.
6+ YOE6+ years data engineering experience building and scaling pipelines; experience across ingestion, storage, processing, warehousing; production coding in TypeScript/Python; mentoring experience; located in or willing to relocate to San Francisco Bay Area.
Spark, Parquet, Iceberg, Delta, S3, Snowflake, BigQuery, Redshift, TypeScript, Python, Kafka, Kinesis, dbt, Airflow, Dagster, FHIR, HIE, IHE, EHR/EMR, NPI, TEFCA, ADT, HL7, HEDIS, RAF, SNOMED, LOINC, ICD-10, Node.js, AWS, ECS, Lambda, SQS, SNS, Batch, CDK, PostgreSQL, Aurora, DynamoDB, Athena, SageMaker
Job Posting
1mo
Save
Mark Applied
Hide
Data Engineer, Meta Superintelligence Labs (Mobile Client)
Menlo Park, California, United States
$177k-$247k/yr OnsiteFull Time
Meta
MetaNASDAQ: META: Builds technologies that help people connect, find communities, and grow businesses.
7+ YOEBachelor's degree in Computer Science or related field; 7+ years data experience; 7+ years with SQL, ETL, data modeling, and programming; experience with batch pipelines, distributed SQL, and mobile instrumentation.
SQL, ETL, Python, C++, C#, Scala
Job PostingView all
3mo
Save
Mark Applied
Hide
Software Engineer - X Data
Palo Alto, California, United States
$125k-$400k/yr OnsiteFull Time
xAI
xAI: Artificial intelligence research and development.
3+ YOE3+ years software engineering experience; expertise in Python,Rust,Scala,Go or Java; experience with data pipelines, realtime and batch processing, and distributed systems.
Python, Rust, Scala, Go, Java, BigQuery, Trino, Clickhouse, Flink, Kafka, Spark, Scalding, SQL
Job PostingView all
3w
Save
Mark Applied
Hide
Senior Data Platform Engineer
Boston or San Francisco or Vancouver
$125k-$205k/yr OnsiteFull Time
Later
Later: Private influencer marketing and social media management serving brands, agencies, and creators.
6+ YOE6+ years in data, backend, or platform engineering; strong Python, SQL, dbt; GCP and BigQuery experience; infrastructure as code, CI/CD, orchestration; batch and streaming architectures; security and data governance knowledge.
GCP, BigQuery, AWS, dbt, Fivetran, Kubernetes, Python, SQL
Job PostingView all
2mo
Save
Mark Applied
Hide
AI Context & Data Infrastructure Engineer
San Francisco, California, United States
$250k-$300k/yr OnsiteFull Time
Town
Town: Privately held applied-AI software building assistants that help people create and use software at work.
Significant hands-on experience with lexical and semantic search, large-scale realtime and batch data infrastructure, pipelines, indexing, storage, and retrieval tradeoffs; ranking, knowledge graphs, or LLM retrieval are bonuses.
BM25, ANN, LLM
Job PostingView all
3mo
Save
Mark Applied
Hide
ML Infra Engineer (Data Systems)
San Francisco, California, United States
OnsiteFull Time
Physical Intelligence
Physical Intelligence: AI robotics developing foundation models and learning algorithms for robots and physically actuated devices.
Strong software engineering fundamentals with experience building distributed systems, large-scale data pipelines, object storage and batch/streaming systems; ownership mindset and performance focus.
ClickHouse, Ray, Flink, Spark
Job PostingView all
3mo
Save
Mark Applied
Hide
Member of Technical Staff - Data Platform
New York City or San Francisco or London
OnsiteFull Time
Reflection AI
Reflection AI: Private AI research lab building open foundation models and agentic software for enterprise, government, and regulated-industry users.
Experienced data engineer with production-grade pipeline experience for large-scale batch and streaming workloads; strong debugging, cost/performance optimization, and data quality/governance skills.
Spark, Flink, Beam, Airflow, Dagster, Kafka, PubSub, Parquet, Iceberg, Delta Lake, BigQuery, Snowflake, Great Expectations
Job PostingView all
3mo
Save
Mark Applied
Hide
Member of Technical Staff — Data Ingestion & Quality
San Francisco, California, United States
OnsiteFull Time
Causal Labs
Causal Labs: Private AI research building physics foundation models for weather prediction and control for businesses and governments.
Experience building large-scale data pipelines and QA systems, familiarity with streaming and batch ingestion, vendor collaboration, and strong problem-solving and domain learning skills.
Apache Spark, Ray, Beam
Job PostingView all
6h
Save
Mark Applied
Hide
Senior Bioinformatics & Genomics Data System Engineer
San Carlos, California, United States
$126k-$157k/yr OnsiteFull Time
Natera
NateraNASDAQ: NTRA: Global leader in cell-free DNA testing and diagnostics.
6+ YOEBachelor's in CS, software engineering, bioinformatics or related; 6+ years industry experience (3+ with Master's, 2+ with Doctorate); 3+ years in regulated environment (CLIA/FDA); 3+ years in software engineering or DevOps; Python, AWS, RDBMS, Linux, Git.
Python, AWS, Batch, ECS, S3, EC2, HealthOmics, PostgreSQL, MySQL, Linux, Git, GitHub, GitLab
Job PostingView all
1mo
Save
Mark Applied
Hide
Software Engineer, Data Platform (Foundations Lead)
San Francisco or New York City
$281k-$334k/yr HybridFull Time
Notion
Notion: Collaborative workspace for notes, projects, and AI tools.
Deep data-platform expertise in large-scale batch and streaming systems; leadership of data platform or infrastructure teams; team-building and talent development; hands-on engineering judgment; platform-product thinking; organizational influence.
Kafka, Debezium, S3, Iceberg, Spark, EMR, Athena
Job PostingView all
2mo
Save
Mark Applied
Hide
State Estimation Engineer - Data Collection Systems
San Jose, California, United States
$150k-$300k/yr OnsiteFull Time
Figure
Figure: Develops general-purpose humanoid robots.
4+ YOERequires 4+ years building multi-sensor fusion and state estimation systems, expertise in real-time filtering and batch optimization, advanced kinematics and optimization knowledge, and high-performance C++ and Python skills.
C++, Python, GTSAM, Ceres, Machine Learning (ML)
Job PostingView all
2w
Save
Mark Applied
Hide
Senior Back End Engineer, Data
Los Angeles or San Francisco
$170k-$225k/yr OnsiteFull Time
Altruist
Altruist: Private fintech wealth platform and self-clearing custodian providing integrated tools for independent financial advisors.
4+ YOE4+ years software engineering experience; proficiency in Python or Java; experience with databases, data warehouses, ETL pipelines, batch processing (Spark), AWS, Kubernetes; AI-assisted development tools; B.A./B.S. in Computer Science or equivalent.
Python, Java, Spark, AWS, Kubernetes, Cursor, Claude Code, Snowflake, Airflow, dbt
Job PostingView all
1w
Save
Mark Applied
Hide
Sr. Software Engineer - Big Data
San Francisco, California, United States
$130k-$197k/yr HybridFull Time
LiveRamp
LiveRampNYSE: RAMP: Public data collaboration platform helping brands, publishers, retailers, and media platforms unify, activate, and measure data.
6+ YOE6+ years software development with big data; 3+ years Spark/Hadoop batch processing; 3+ years Java and SQL; 2+ years cloud platforms (GCP, AWS, Azure); experience with Docker, Terraform, Kubernetes.
Spark, Hadoop, Java, SQL, GCP, AWS, Azure, Docker, Terraform, Kubernetes, BigTable, BigQuery, Dataproc, PubSub, Kafka, Hive, Apache Beam, Helm, Gradle
Job PostingView all
2mo
Save
Mark Applied
Hide
Engineering Manager, Data Platform
San Jose, California, United States
$209k-$438k/yr OnsiteFull Time
TikTok USDS Joint Venture LLC
TikTok USDS Joint Venture LLC: Ensuring U.S. data security and content integrity for TikTok.
5+ YOE3+ MgmtBachelor's or master's degree or equivalent experience; 5+ years building enterprise data platforms, including 3+ years leading engineering teams. Requires batch and streaming systems, SLA, incident response, architecture, and communication expertise.
Spark, Flink, Hadoop, Kafka, Druid, ClickHouse
Job Posting
3mo
Save
Mark Applied
Hide
Member of Technical Staff, Data Engineering
Palo Alto or San Francisco
$150k-$300k/yr OnsiteFull Time
Parallel Web Systems
Parallel Web Systems: AI infrastructure providing web search, research, extraction, and monitoring APIs for AI agents and enterprise workflows.
Deep experience in distributed data processing, data modeling, system reliability, batch and streaming pipelines, and building data quality, lineage, and observability systems.
Job PostingView all
1w
Save
Mark Applied
Hide
Senior Software Engineer, ML Evaluation & Introspection Infrastructure
Mountain View, California, United States
$162k-$260k/yr HybridFull Time
Aurora Innovation, Inc.
Aurora Innovation, Inc.NASDAQ: AUR: Self-driving technology developing hardware, software, and data services for autonomous trucking and passenger mobility.
5+ YOE5+ years professional software engineering experience building backend, data or ML infrastructure; strong Python and SQL; experience with large-scale batch/data pipelines on cloud infrastructure; experience building measurement/evaluation systems; distributed compute and container orchestration (e.g., Kubernetes, AWS Batch).
Python, SQL, Kubernetes, AWS Batch
Job PostingView all
3w
Save
Mark Applied
Hide
Senior Software Engineer, Cloud Infrastructure
San Carlos, California, United States
$173k-$225k/yr HybridFull Time
Beacon AI
Beacon AI: Private aviation technology building AI pilot-assistance systems for commercial and defense pilots.
4+ YOEU.S. Person status required; 4+ years AWS serverless/container experience; LLM systems production experience; AWS depth (VPC, IAM, KMS, CloudWatch, S3, ECS/EKS, Lambda, Step Functions, Bedrock, SageMaker); Python data engineering; security mindset.
AWS, AWS CDK, Terraform, GitHub Actions, CodeBuild, CodePipeline, VPC, PrivateLink, IAM, KMS, Secrets Manager, CloudWatch, OpenTelemetry, AWS Bedrock, SageMaker, ECS, EKS, Lambda, Batch, LangChain, OpenSearch Serverless, Aurora PostgreSQL, pgvector, Pinecone, S3, DynamoDB, Step Functions, Airflow, Glue, Lake Formation, Athena, EventBridge, SQS, Redis, Triton Inference Server, TensorRT-LLM
Job Posting
3mo
Save
Mark Applied
Hide
Senior Machine Learning Engineer - ESPN
Glendale or San Francisco or New York City
$142k-$199k/yr OnsiteFull Time
ESPN
ESPN: Disney-controlled American sports media serving fans through television networks, digital platforms, streaming, radio, and live events.
5+ YOE5+ years building production data or ML systems; distributed systems, streaming and batch technologies, Python, cloud-native environments, observability, and a related bachelor's or master's degree.
Kafka, Kinesis, Spark, Flink, Python, Java, Scala, Go, C++, AWS, Kubernetes
Job PostingView all

Explore Jobs

Expand Your Job Search