Crawling

A program (a crawler) roams the web and scoops up pages. Google Search pioneered the practice, and AI companies now use it to gather training data.

It has become ground zero for copyright disputes in the AI era — publishers, writers, and artists have filed lawsuits claiming their articles, images, and writing were scraped without permission to train AI. Sites can post a "do not scrape" sign called robots.txt, but it carries little enforcement power, so crawlers that ignore it regularly make headlines. At the same time, a growing trend is settling the issue through licensing deals (paying for content use).