If for some reason an agentic software forgets to check the llms.txt then catch them with an AWS WAF Challenge which checks to see if the pages being loaded actually get loaded into a browser DOM.

If you want to step it up a notch then add in a Captcha every now and then.

If they fail, either redirect them to your llms.txt or markdown version website. If they continue to try to browse the human HTML website then get strict with them and hit them with a block.