Here’s something that doesn’t get nearly enough attention in conversations about the AI boom: a huge amount of computing power going into AI web crawling is being spent on things AI systems don’t need – navigation menus, JavaScript, advertising scripts, layout code. All of it gets sent to the crawler, processed by the model and turned into tokens, at cost, even though the only thing that actually matters is the readable text.
A free, open-source WordPress plugin called WordPress Markdown for Agents, from The Chancery Lane Project, is going after exactly that waste. The plugin serves clean Markdown versions of web pages to AI crawlers instead of full HTML, stripping out scripts, navigation and formatting that machines don’t need.
Early testing shows page size reductions of around 80% and token reductions of up to 90% when AI systems access pages via Markdown rather than standard webpage rendering. The plugin also generates an llms.txt index file and exposes Markdown versions through standard discovery methods for AI agents.
Why This Is A Bigger Deal Than It Sounds
The electricity figure cited at launch is dramatic: under the article’s modelling assumptions, broad adoption of a tool like this across WordPress sites could produce energy savings equivalent to powering the entire United States for 24 hours.
That figure depends heavily on adoption rate, traffic volume and modelling choices, so it should be read as an illustration of scale rather than a verified measurement. But the directional point holds: even the smallest improvements in crawl efficiency compound dramatically when multiplied across the billions of AI web requests happening daily, the combined effect on computing resources and power consumption is massive.
WordPress powers more than 40% of the web. If a significant proportion of those sites served clean Markdown to AI crawlers rather than full HTML pages, the reduction in tokens processed would cut compute and energy costs significantly.
The cost of AI computing infrastructure is the industry’s most widely debated constraint. A free plugin that chips away at that from the content delivery side is an unusual and underreported angle on the problem.
If You Build With Web Data, Pay Attention
For entrepreneurs creating AI tools that rely on internet data – research tools, market intelligence platforms, content aggregators, RAG pipelines, agent frameworks that browse the web – there’s a clear and immediate effect. If your scraper grabs the whole HTML page, you’re paying for a lot of useless fluff. A plugin like this, adopted at the source, reduces that cost without any change on your end. The more sites that implement it, the cheaper web-dependent AI products become to run.
The plugin’s llms.txt index file deserves a separate mention. This is a standardised format that tells AI agents what content is available on a site and how to access it cleanly – essentially a sitemap built for AI crawlers rather than search engines. As more AI agents browse the web autonomously, having an llms.txt in place means those agents can find and read your content without processing full HTML pages. If your product is content-heavy, setting this up today is a no-brainer. It requires minimal effort right now, but compounds quietly as AI web browsing becomes standard
There’s also a broader principle that applies beyond this specific tool. The AI efficiency debate has primarily centred on model design: creating lighter, faster and more budget-friendly systems. This is a reminder that significant efficiency gains are also available at the data layer, before the model ever sees anything. Cleaner inputs mean fewer tokens, fewer tokens mean lower costs and lower costs mean more accessible AI products for the founders who can’t afford to run at hyperscaler margins.
The plugin is free, open source and available now. For anyone running a WordPress site that AI agents read – or building products that crawl the web – it’s one of those rare things: a zero-cost infrastructure improvement with a real potential upside.
