YHK - yet another hacker news clone

Web Search API

142 points tosh 3 hours ago 81 comments

binarymax 2 hours ago | parent

Why not use those providers directly? Does Cloudflare need to be in the middle of everything?

unified101 2 hours ago | parent

Ease of integration and billing. Failover. Higher trust.

not_math 2 hours ago | parent

To add to this, some organizations just prefer using one provider for their cloud service. So if they build on Azure/Google Cloud/AWS, then everything needs to be on there. Cloudflare probably wants to offer the same here, where everything can be built on Cloudflare.

binarymax 2 hours ago | parent

Not sure where the trust claim lands, but the first two are now exceedingly trivial with code agents. A little more work perhaps, but not hard at all. I’ve done this myself (not with those providers) with several search platforms.

bayesianbot 2 hours ago | parent

CloudFlare AI Gateway was quite convenient for me - I wanted to give my zeroclaw instance a limited budget to services like Image generation/Replicate/Fal.ai, which would mean for each service I'd have to run my own proxy that stores the keys and cuts off the real calls if we go over the limits. Easy to do but still extra thing to build and maintain.

Instead I put my API keys to cloudflare, set limits, and gave the agent the CloudFlare token, and in minutes it could contact tens of services.

edit: not to mention instead of loading balance to each service I could just keep balance on cloudflare that covers them all

478336632929 2 hours ago | parent

Why would anyone trust Cloudflare?

wongarsu 1 hour ago | parent

Same way people trust Microsoft: "We already use them, and using them for this additional service exposes no data they wouldn't already have access to from all the other services we buy from them"

ForHackernews 1 hour ago | parent

I trust Cloudflare more than I trust some other players in the arena. They have a decent track record of being neutral infrastructure provider. They seem technically strong, deploying Rust widely and caring about performance in a way that most firms do not. They're likely covertly funded by intelligence services so they don't have economic incentives to enshitify their offerings or deliberately screw me over.

Put it this way: I'd rather Cloudflare owns the Internet than Google, Meta, Amazon or Alibaba.

OutOfHere 29 minutes ago | parent

> or deliberately screw me over.

Cloudflare is however known to deliberately screw over some of their clients.

> I'd rather Cloudflare owns the Internet

I'd rather no one does, certainly not a firm that feeds into the NSA.

ForHackernews 20 minutes ago | parent

Citation? The only case I'm aware of is when they removed DDoS protection for the Daily Stormer[0], and tbqh that made me feel more positively toward them. Yes, it was an impulsive, emotionally-motivated choice but that was relatable to me.

[0] https://blog.cloudflare.com/why-we-terminated-daily-stormer/

goalieca 1 hour ago | parent

Curious what other people’s experience is with cloudflare billing. When you go through an AE, everything seems made up anyways.

rithdmc 2 hours ago | parent

It would be difficult for them to provide intelligence to the US without being in the middle of everything.

ForHackernews 2 hours ago | parent

Cloudflare are setting themselves up as the arbiter who will decide which requests are a) human, b) authorized AI bots, c) illicit/banned bots.

Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.

timpera 2 hours ago | parent

Cloudflare seems to be very excited to eventually get a 30% cut on pay-to-crawl.

As for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.

someonebaggy 1 hour ago | parent

Cloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.

viraptor 1 hour ago | parent

They give you the tools. Your options are:

- let more traffic in, eat the compute cost

- block larger cohorts of traffic, affect many real users

- babysit the rules to get them just right, lose time doing that

In my experience the residential proxies exist but are not that common and many aren't trying as hard as they could. It's really a war of which side wants to spend more attention on the problem.

tucnak 2 hours ago | parent

National security, bro.

patwolf 2 hours ago | parent

I've been using the web search in OpenRouter, which is similar in that it's a wrapper around other search engine providers. It's really convenient to be able to experiment with new models and new search engines without having to go through corporate hoops to subscribe to a new service.

hobofan 1 hour ago | parent

I think Cloudflare is (for companies already using it) approaching the status of trusted main cloud supplier (which usually would be AWS, GCP, Azure) via which the majority of cloud costs are billed (so you don't have to go through a fresh procurement process).

ryandvm 1 hour ago | parent

I don't know what you mean by "trusted", but how many times do folks have to go through the same loop?

   - Company has great initial product
   - Company gets popular
   - Shareholders demand infinite growth
   - Company becomes rent-seeker
   - GOTO 10
I'm with OP - a company that wants to insert itself in the middle of everybody's business is not being altruistic, they're playing the long game.

viraptor 1 hour ago | parent

> I don't know what you mean by "trusted",

It means "hi spending approver, I'm going to add $100 to our CF account" instead of "hi accounting+management+security, please initiate the process of evaluating new third party vendor Foo for use in my project, I hope we can get it approved and integrated into SSO sometime next month".

moralestapia 36 minutes ago | parent

You tell us, you're the president of bonsai.io.

Why would I use bonsai? Why not use ElasticSearch directly?

tom1337 2 hours ago | parent

I wonder if the three search engines get access to cloudflare protected sites without any captcha or bot interventions

astonex 2 hours ago | parent

Most likely not. Their Crawling service for example does not bypass the cloudflare protections either.

weird-eye-issue 2 hours ago | parent

You are conflating a couple of different things here

There actually is such a thing as verified bots on Cloudflare that gets through most blocks (and these services are likely are part of that), but ultimately it just depends on how the website owner has things set up in Cloudflare

dbbk 1 hour ago | parent

I hate to be rude but once again I am begging people to actually read the post. It is mentioned in the second paragraph that they are all verified bots.

weird-eye-issue 45 minutes ago | parent

Actually it doesn't say they are verified bots but it does say that they follow the verified bots standards. But yes I think we can assume they are verified bots which I already said in my comment so I'm not really sure what the purpose of your reply is?

viraptor 1 hour ago | parent

> There actually is such a thing as verified bots on Cloudflare that gets through most blocks

Verified bot is just a label. What you do with that information is entirely up to you as the operator. It doesn't say anything anything about the service and doesn't provide any guarantees about the traffic.

weird-eye-issue 47 minutes ago | parent

No it's not just a label because it's a category that gets used in the firewall settings. A lot of sites allow only verified bots and block non-verified ones. So yes as I already mentioned website owners can of course block verified bots but it's much much less common than blocking non-verified ones because most site owners don't want to accidentally deindex their site from Google...

timpera 2 hours ago | parent

Interesting to see that they didn't include the Brave Search API, which is really great and imo a better experience than Exa.

The absence of the Perplexity Search API is to be expected though, knowing how much these two companies despise each other.

AznHisoka 2 hours ago | parent

Why do they despise each other?

timpera 45 minutes ago | parent

Cloudflare has accused Perplexity of stealth crawling. Cloudflare's CEO has been pretty harsh in its comments: https://twitter.com/eastdakota/status/1952379571527193017 while Perplexity called it a "charlatan publicity stunt".

Oras 2 hours ago | parent

Weird choice by CloudFlare, would been great if they have shared why it was created.

I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.

I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?

Herz 2 hours ago | parent

How does Cloudflare manage to hit the HN front page almost daily? Don't get me wrong, they build cool stuff, but the frequency is wild.

dewey 2 hours ago | parent

Because it's their release week, so there's multiple new products every day. The overlap of people using HN and Cloudflare is pretty large, so not that surprising.

AznHisoka 2 hours ago | parent

A related Q: why are their products so popular? I get why CDN/DDOS protection is but what about everything else? I have never ever found a use for stuff like Workers. (Sincerely asking, not dismissing them as useless)

someonebaggy 1 hour ago | parent

Workers is their version of Lambda

winstonp 1 hour ago | parent

Their free tier for stuff like Workers and D1 is quite generous.

sebzim4500 1 hour ago | parent

Pages is a very convenient way of deploying static websites/SPAs with a generous free tier. You just need to find the tiny links in their dashboard to avoid accidentally using workers instead (which is supposed to supersede it but is clearly worse for this usecase).

viraptor 1 hour ago | parent

Workers have a nice and easy deployment model (when it's not broken) compared to AWS lambda, so I get why people are tempted. It's one simple file compared to 4 separate pieces of infra. But yes, please, use anything else that doesn't pay for the CloudFlare protection racket. For example there's https://bunny.net/edge-scripting/

sva_ 12 minutes ago | parent

I resisted using them for a long time but it is really so convenient

Running your stuff, even private stuff, through a tunnel is great so that you don't have to expose your VPS' IPv4.

thepoet 2 hours ago | parent

I suspect also to do with internal Slack etc. where employees vote on launch posts (a lot of them on HN since long). Not a scam or accusing anyone but this probably propels a lot.

iphonecorridor 2 hours ago | parent

For those developers out there, the best is still Gemini Flash Lite 2.5 believe it or not. It gives you 1000 google searches per day for free. Compare to Flash Lite 3.x which is 5k PER MONTH and then a few pennies PER SEARCH. Nuts. Didn’t realize search was so expensive.

Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”

It’s really really good for low cost search!

rvz 2 hours ago | parent

> Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”

Don't give them (G) ideas.

iphonecorridor 2 hours ago | parent

The idea i do want to give them… guys, differentiate your Gemini models with free to low cost search. It’s what your known for! Lean into it.

twoodfin 38 minutes ago | parent

Not joking: They need to figure out how to sell ads for agents first.

Google Search only works as a business because human eyeballs see (and brains choose to click on) ads at rates that justify advertisers’ (massive in aggregate) dollars.

iancuandrei 1 hour ago | parent

"This model is being retired on October 20th, 2026"

LatticeAnimal 1 hour ago | parent

Google AI's deprecation page [0] says that there is "No shutdown date announced" for gemini-2.5-flash-lite.

(Was your comment a joke? Or did google announce this through different channels?)

0: https://ai.google.dev/gemini-api/docs/deprecations

apwheele 1 hour ago | parent

So I wish I could use Google for https://veruscite.com/, but the number of Google searches are a hard cap on the account! So yes that is fine for agentic coding, but for an app that relies on web-search is not sufficient.

I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).

stavros 23 minutes ago | parent

Can someone clarify how this works? Why does an LLM give me a search API? The search APIs I've always used were just "post query, get JSON".

karmakaze 14 minutes ago | parent

Can Gemini Flash Lite 2.5 be made to return raw search results. Some 'search' providers I looked at returned summaries, or vector relevance matches (of presumably a smaller page set).

anon373839 2 hours ago | parent

> All three support Zero Data Retention for requests made through Cloudflare

But does CloudFlare itself commit to zero data retention? If not, this isn’t too meaningful.

freakynit 2 hours ago | parent

Tried one query on ceramic.ai (the default provider for cloudflare web search api): "qwen-3.8 flash next and rtx 5090 best inference setup" ... 0 results ... same query on google and ddg both yield proper results.

Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.

And I had thought that I finally had found a cheaper search alternative.

scosman 2 hours ago | parent

I made a zero ads SERP using one of these "AI first" search providers: https://github.com/scosman/froogle (live version https://froogle.fyi). In this case Keenable.ai. Generally the same pattern: it's not usable.

freakynit 2 hours ago | parent

You know the craziest part? This time I searched for their own website address: "ceramic.ai" .. results came... none pointing to the website or any page on it.

Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.

vscarpenter 2 hours ago | parent

I guess I'm not understanding the value here - to compete with Google and the likes, the scale, cost and complexity would be huge. Appreciate new entrants in an existing field but not seeing this one.

owebmaster 2 hours ago | parent

Codex and Claude Code need to do countless web searches, I'd guess they have a partnership with Google. Open models don't have this partnership so the search API needs to come from somewhere.

ramesh31 1 hour ago | parent

>"to compete with Google and the likes, the scale, cost and complexity would be huge."

CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".

qznc 2 hours ago | parent

My coding agent uses the hister cli, i.e. a local index. That often requires me to seed it manually as a downside. The upside is that it caches website contents via browser plugin, which is a nice workaround for bot blocking.

Thanks asciimoo for https://github.com/asciimoo/hister

okokwhatever 1 hour ago | parent

Who thought 5 years ago that searching online would have a cost...

breakingcups 1 hour ago | parent

It always did, but who pays it is shifting.

sreekanth850 1 hour ago | parent

Create bot detection and bot protection, then sell crawlers. Is this the peak of hypocrisy?

doginasuit 11 minutes ago | parent

Let's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.

ronfriedhaber 1 hour ago | parent

Interesting to see how this can be compared with Exa, Alas, Cloudflare really is shipping many great orthogonal products recently.

breakingcups 1 hour ago | parent

Seems like this uses Exa, as well as two other providers.

fnordsensei 1 hour ago | parent

I've been quite satisfied with Kagi[1]'s API.

1: https://kagi.com/api/docs/openapi

ctolsen 59 minutes ago | parent

Me too, but it’s kind of expensive. Would be nice if they included some API usage in their subscription.

jamesponddotco 6 minutes ago | parent

Same here, I built their search into my Home Assistant MCP toolbox, and couldn’t be happier.

corentin88 1 hour ago | parent

Funny how search was a graveyard for startups for almost two decades. And since ChatGPT releases (or so) it’s a trending place again.

8bite 40 minutes ago | parent

The pricing is so different between these:

ceramic.ai - $0.25 per 1,000 requests

Exa - $7.00 per 1,000 requests

Linkup - $5.00 per 1,000 requests

Does anyone have insights on the quality differences? Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.

EDIT: formatting

stefs 24 minutes ago | parent

i looked up what alternatives there'd be and openrouter also makes its search API available. it's exa too, but $4 per 1000 results.

nreece 23 minutes ago | parent

I recently tried a bunch of web search APIs. Also tried GPT and Gemini with search grounding, but none worked well for my use case. You can see some good comparisons and benchmarks at https://mattcollins.net/web-search-apis-for-llms and https://openbenchmarks.com

OutOfHere 32 minutes ago | parent

This has got to be a serious antitrust violation. First Cloudflare bans all the other bots, then it allows its liaised bots.

jasonjmcghee 22 minutes ago | parent

> All three support Zero Data Retention for requests made through Cloudflare

And then on the providers page:

    Property              Value
    provider              exa
    Zero Data Retention   No

jeromechoo 9 minutes ago | parent

Very difficult to promise ZDR if you’re scraping Google and Bing.

karmakaze 17 minutes ago | parent

I was just looking into these as DeepSeek Harness w/ Qwen3.8-27B relies heavily on search. I was going to go with Serper.dev[0] $1 per 1000 (or lower in quantity).

The providers[1] behind this Web Search API have very different rates:

    Ceramic.ai: $0.25 per 1,000 requests
    Linkup:     $5.00 per 1,000 requests
    Exa:        $7.00 per 1,000 requests
[0] https://serper.dev/

[1] https://developers.cloudflare.com/web-search/providers/

0fflineuser 7 minutes ago | parent

I am pretty sure exa specifically say it trains on your data in it's privacy policy, so how can it be ZDR ?