21 points demegire 3 hours ago 4 comments
AMerrit 3 hours ago | parent
Interesting that in some cases they are using the AI Agent crawler names to hide themselves. With the explosion in bot traffic it gets much easier to hide yourself in the crowd.
flockonus 1 hour ago | parent
I wonder if Cloudflare blocks it through their default protection?
I'm ok with having agents / LLMs reading my site, but not happy about scanners.
nerdralph 28 minutes ago | parent
Some useful information, though I'd like more details on headers sent like Sec-Fetch.
For passive blocking, TLSv1.3 filters out some bad bots and scrapers.
Most bad bots follow 302 redirect, but don't parse html and act on meta refresh.
Most bad bots don't store cookies, but I've seen a couple that do.