web crawler

A researcher says OpenAI's automated agents scanned a United Nations statistics website roughly 16,500 times, raising fresh questions about the load and consent issues created by AI web crawlers. The claim highlights growing friction between AI companies collecting web data at scale and the sites whose infrastructure absorbs the traffic.

Automated AI agents flooded and hammered a United Nations website, according to a tech news brief. The incident highlights growing strain on public web infrastructure as AI-driven crawlers and bots increasingly hit sites at scale. The UN has not issued a detailed public statement on the disruption, and reports so far offer limited detail on which site or service was affected or for how long.

Security researcher Rowan Howard-Jones reports that OpenAI's automated agents hit the UN Conference on Trade and Development's statistics site more than 16,000 times between April and June. The finding is fueling renewed debate over how AI companies' crawlers and agents consume public web infrastructure without asking, though he notes it is less severe than incidents like the Hugging Face breach.

A new animation showing a spider moving through a web as a metaphor for how a web crawler indexes pages online is circulating widely. The visual breaks down a technical process into an easy-to-grasp analogy, drawing attention from both tech audiences and general viewers who are sharing and discussing it.

Cloudflare argues that the internet now serves a second audience beyond human readers: automated systems such as AI crawlers and agents that consume web content at scale. The piece discusses what this shift means for publishers and the open web, as machine traffic grows and content is increasingly read by software rather than people.