Reddit is making a decisive move against AI firms — or at least pushing for them to pay their dues.
This week, Reddit announced revisions to its Robots Exclusion Protocol, commonly known as robots.txt. While this may seem like a technical adjustment, it’s part of a broader clash between AI companies eager to use content for training their language models and content owners who hold the rights.
The robots.txt file is crucial for websites to specify how they can be crawled, such as allowing Google to index their pages for search results.
However, the value exchange with AI is less straightforward. For websites relying on traffic and engagement, there’s little incentive to let AI companies freely access their content without reciprocating with traffic or, worse, plagiarizing their material.
By updating its robots.txt file and maintaining measures like rate limiting and blocking unidentified bots and crawlers, Reddit appears committed to curbing practices that have drawn criticism, particularly from companies like Perplexity AI.
Watch our coverage for more insights and share your thoughts in the comments below!
