We are looking for a Senior Software Engineer to work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption
In this role, you will focus on building systems that discover, fetch, and continuously refresh content from the open web and other large-scale data sources. You will design distributed crawling, scheduling, and ingestion infrastructure capable of operating at internet scale while balancing coverage, freshness, resource efficiency, and reliability. You will work on systems that process billions of URLs, manage high-throughput data flows, and ensure that high-quality content is consistently available to downstream indexing and retrieval systems
Design, implement, and operate web-scale crawling systems for acquiring content from the internet
Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
Ensure reliable operation of crawling infrastructure under high-throughput conditions
Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
Monitor resource usage, bandwidth consumption, and infrastructure cost
Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
Enable safe experimentation with crawling strategies and content acquisition policies