Backend developer · Lead / principal
Staff Software Engineer (Agentic Search) - Crawler
- Amsterdam area
- On-site
- Permanent job
- Salary not listed
- Sponsors work permits
- Posted 16-10-2026
- English is enough
Technology asked for
- Go
- C++
- Linux
••• is building an agent-native search platform that gives AI agents low-latency, observable, programmatic search APIs grounded in fresh real-world data. This Staff Engineer role sits in the content acquisition team, which owns the crawling and ingestion infrastructure feeding the search index. Main tasks: design, build and operate web-scale distributed crawlers that handle billions of URLs; create ingestion workflows for crawlers, structured feeds and partner sources; develop crawl scheduling, prioritisation, recrawl and freshness strategies; build URL discovery, deduplication and content extraction; keep the systems reliable under high throughput; define metrics for coverage, freshness and quality; track bandwidth and infrastructure cost; and work with indexing and ML teams. Must-haves: 8+ years building backend or distributed systems, strong Go or C++, experience with large-scale systems (10k+ RPS, billions of URLs), solid knowledge of HTTP, DNS and TLS, crawling and scraping, and experience running and debugging production distributed systems. Streaming pipelines, Kafka-like messaging and Spark or Flink are a plus. The process includes coding interviews.