Yet another Hacker News

Stay discoverable in search while disallowing AI training

35 points djfergus 1 hour ago 15 comments

gchamonlive 1 hour ago | parent

Cloudflare, enabling the problem and the solution since, how long has it been?

arm32 1 hour ago | parent

(checks watch)

17 years

Fun lava lamp story, though

gleezard 1 hour ago | parent

A bit too late honestly (?). With so many people who have shifted over to reading AI summaries as a primary search response, those with AI-enabled sites will win by attrition.

There is no going back from this. And the internet is a relatively new phenomenon. Recklessly, blindly applying ads to pages in hopes of generating revenue is a very silly thing to do. Technology with ad blockers and now AI summaries has taken that away. New business models, perhaps actually decent ones are required.

Death of ads everywhere? Good fucking riddance.

Posted from LibreWolf.

tracerbulletx 1 hour ago | parent

All this attitude does is tear down the only viable income source for independent publishers and demonizes them for trying to make money, while everyone let's huge corporations off the hook for it because "well that's just what they do"

gleezard 28 minutes ago | parent

Advertising in the way it’s done is demonic in and of itself. I don’t care - find a better business model.

arm32 1 hour ago | parent

What does this new setting actually do? Does it block their IP ranges too, I hope? As if Meta, for instance, is actually going to respect Accountable, via themselves or their partners, quite frankly is eyebrow raising at best.

zergrush 1 hour ago | parent

what weirds me out is the analytics

theres no way my index.html page with nothing is getting 10000 hits a day

wtf?

itake 1 hour ago | parent

7 per minute.

I use my high school’s website to test Internet connectivity bc the domain is short and they don’t do a TLS redirect (making it easy to detect WiFi portals).

JoshTriplett 1 hour ago | parent

neverssl.com

AnonC 1 hour ago | parent

> Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search.

> We also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable. These organizations separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search.

I find it difficult to trust that either Meta or OpenAI would use their separate search and training crawlers only for the respective purposes. Their pinky promises have no value, IMO. Both companies are premised on deceptive behaviors.

skybrian 1 hour ago | parent

I didn't know websites could opt out of providing data to Google's AI training. Looks Google added support for this via 'Google-Extended' in robots.txt back in 2023:

https://blog.google/innovation-and-ai/products/an-update-on-...

mskalski 1 hour ago | parent

I wonder if protocols like Web Bot Auth [1] will see wider adoption. At least as a supported mechanism for those bots which identify themselves. The rest probably still have to be treated with Anubis. In my free time I've recently been experimenting with a Web Bot Auth implementation as an Envoy dynamic module [2] to have a way to define some additional policies for the traffic from bots.

[1] https://datatracker.ietf.org/doc/draft-ietf-webbotauth-https... [2] https://github.com/michalskalski/envoy-web-bot-auth

fraywing 50 minutes ago | parent

I'm playing around with it for my MCP hiring protocol ojcp[1] and it seems to work very well for signing attestations at the header level.

[1] https://github.com/ojcp-org/ojcp

mskalski 26 minutes ago | parent

Thanks for sharing, it is interesting use case

kinduff 59 minutes ago | parent

I maintain a cloud IP ranges database, and I'm going to test this out.

I have my doubts, though. A formal title like "Accountable" (capitalized) sounds deliberate, but I can't help imagining the renewal email:

"Hey, want to renew your Accountable™ license? Just pinky promise again that you use your IPs for what you say you do."