YHK - yet another hacker news clone

An Update on Wayback Machine Access

102 points ChrisArchitect 1 hour ago 49 comments

Onavo 42 minutes ago | parent

Why not just offer a paid endpoint for the crawlers? It's not like the demand is going to go away anytime soon.

It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs.

I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.

croes 40 minutes ago | parent

It’s one thing to archive other companies content, it’s another to sell the access to it

Onavo 39 minutes ago | parent

That's for the lawyers to sort out, they have a lot of flexibility as a US nonprofit. The case law isn't that clear cut for this.

simonw 37 minutes ago | parent

Internet Archive was almost destroyed by a copyright lawsuit from book publishers within the last few years. I expect they aren't excited to take on any additional risk of similar lawsuits right now.

xp84 22 minutes ago | parent

major [citation needed] on that. There are very limited exceptions to the massive power of copyright -- and they're mainly granted to libraries in the form of narrow waivers. And just the cost of fighting the most powerful copyright holders can bankrupt you -- especially if you're a relatively modestly-funded nonprofit.

celsoazevedo 15 minutes ago | parent

They need access to sites to archive them. It's already hard to do it as it is, imagine if they start selling access to content. They'd be shooting themselves on the foot, independently of what the law says.

faefox 26 minutes ago | parent

Yeah, who does the Internet Archive think it is, (insert literally any AI company here)?

bonoboTP 12 minutes ago | parent

Which AI company is selling access to reliable verbatim copies of websites? I don't mean "it may regurgitate a paragraph", but as a reliable service where you can repeatably get website content snapshots to a reliability level that makes such a use case viable?

Using the information for training purposes is not the same thing. Not legally the same and otherwise.

drdexebtjl 35 minutes ago | parent

Sites would just block the Internet Archive crawler as well.

xp84 24 minutes ago | parent

My guess? Because even with a paid endpoint, the type of unscrupulous yahoo that is DDOSing IA today would probably still abuse the free endpoints because they can. The revenue that might come from a paid endpoint could help to scale up, but with how slow IA usually seems, I suspect there is an upper limit to how much traffic they can serve without a LOT more revenue.

This is a major "this is why we can't have nice things" situation in my opinion. IA is one of the most valuable gems of the Internet. The only thing that even comes close to preserving our shared history. The damage being caused (both by the effective DDOSing and by the knock-on impact that abuse has in encouraging publishers to remove their content from the archive) is incredibly serious.

imglorp 4 minutes ago | parent

Micropayments would solve so many Internet problems. It's not too late to adopt.

Content creators could charge by page instead of depending on malware/ad/surveillance revenue. Spam is cut if there's a charge per mail. Scraping abuse goes away, along with a bunch of DDOS garbage.

The impact is a few cents per page or mail, negligible for a human. But if you're consuming a trillion pages per day, you'd reconsider.

KPGv2 3 minutes ago | parent

> Why not just offer a paid endpoint for the crawlers?

Because then you're definitely violating US copyright law. There are four prongs of fair use analysis, and one of them is the "nature of the use." In this case, you'd be turning into a commercial use.

simonw 42 minutes ago | parent

> Here’s what’s going on. The Internet Archive’s Wayback Machine has been hit by waves of high-volume automated traffic, and we’ve put protections in place to keep the service running.

I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior.

In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've ålso already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.

packetslave 40 minutes ago | parent

This is absolutely something that's happening. There are even paid scraper API's that offer "Wayback Machine fallback" as a feature.

bsimpson 19 minutes ago | parent

It's an open secret that you can often circumvent paywalls by searching Wayback.

gambiting 16 minutes ago | parent

Every single paid article linked on HN has the way back machine link as the very first comment.

ValentineC 15 minutes ago | parent

The links are usually to Archive.today (aka archive.ph and a bunch of other domains), not Wayback Machine (which is run by Internet Archive).

unkeen 10 minutes ago | parent

*APIs

luckylion 25 minutes ago | parent

What sites would they be targeting? Generic "just give me anything"? Whenever I check regular sites on IA, the coverage is spotty -- they'll have the homepage and a few important pages, but it quickly fizzles out.

Very understandable, you can't store all 15000 pages of any random website and update them etc etc, but that makes them pretty useless for indirect scraping because you usually don't want a tiny taste, you want everything.

toomuchtodo 24 minutes ago | parent

It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.com), or behind Cloudflare for aggressive bot and scraping protection. Hard to defend against abuse of a public resource when its intent is public access with as little restriction as possible.

https://en.wikipedia.org/wiki/Tragedy_of_the_commons

(no affiliation)

ronsor 15 minutes ago | parent

Reddit has no excuses for the anonymous old.reddit.com removal; they're simply greedy.

On the other hand, the Internet Archive is a non-profit offering a free public resource.

toomuchtodo 12 minutes ago | parent

Examples provided as technical examples, strong feelings are out of scope for this thread.

bradly 1 minute ago | parent

Just yesterday from my one of my sessions with Sol:

> Hacker News and the Rails forum are blocking the text fetcher, so I'm using the browser workflow to inspect the pages directly

CqtGLRGcukpy 38 minutes ago | parent

> We’re getting better at telling abusive bots apart from the people who depend on the Wayback Machine every day. If you think you were blocked in error, email info@archive.org with your operating system, browser, and IP address, and we’ll look into it.

tech234a 27 minutes ago | parent

I wonder if they'll end up behind Anubis at some point. I'm surprised it hasn't happened already.

stickfigure 7 minutes ago | parent

Plenty of threads on HN about this, Anubis does not work.

BeetleB 27 minutes ago | parent

Wow, but I wonder if there's more to it.

I've not been able to access web.archive.org from my work computer - I always get the 429 error.

But I then pull out my phone and can access it just fine. All along I was assuming my company was blocking it. Still weird that it happens every time from my work PC and never from my home one.

dotmanish 24 minutes ago | parent

Could be due to some scrapers from either your work ISP block, or the larger block which lends IPs to multiple workplaces.

flexagoon 23 minutes ago | parent

I assume that's because the IP range of your company network overlaps with a range used by some scrapers, and if it doesn't happen on your phone even in the corporate network, then IA probably checks some extra signals like the user agent in addition to the IP

timpera 25 minutes ago | parent

I really appreciate the Archive team's efforts to make the Wayback Machine more responsive, and have donated a few times to support them.

Unfortunately, the restrictions have been way too strict for the last few months: from my residential IP, simply moving the mouse on the calendar for a specific URL is enough to get stuck on 429 error messages for a while; and from corporate ISPs (for example, on airport WiFi), you often can't access the WM at all. I hope they'll find a way to relax those.

UltraSane 19 minutes ago | parent

Why not put it in S3 with downloader pays?

Kayvanian 10 minutes ago | parent

As a public resource the hope is for Wayback to be free to access. I imagine putting up a paywall would be their last resort.

lousken 14 minutes ago | parent

AI companies should pay billions to wayback machine for access

KPGv2 5 minutes ago | parent

I think that'd raise serious copyright concerns, if the Wayback machine started selling other people's intellectual property.

xyst 9 minutes ago | parent

> abusive bots

Are the abusive bots in the room with us?

ignoramous 3 minutes ago | parent

https://archive.vn/WWENg