Yet another Hacker News

GitHub Incident with Actions, Pages, and Webhooks

105 points damontal 1 hour ago 40 comments

uneekname 1 hour ago | parent

My Actions are stuck waiting :-/

jefb 1 hour ago | parent

same here

dc3k 1 hour ago | parent

Another day another GHA incident

rvz 1 hour ago | parent

Last time these intermittent incidents happened was days ago and for this exact same incident was 11 days ago. [0]

After everything announced in GitHub Universe and with that to improve the reliability of GitHub overall, yet we have another incident afterwards.

It doesn't matter what GitHub does, only you can decide for yourself to self-host rather than going all in on GitHub and it continues to go down.

Predicted this 6 years ago and self hosting has aged perfectly well. [1]

[0] https://news.ycombinator.com/item?id=50154406

[1] https://news.ycombinator.com/item?id=22868406

kriztw 1 hour ago | parent

Yep. Ironically this downtime is currently delaying our ongoing migration to self-hosted forgejo...

tempest_ 1 hour ago | parent

We use a self hosted gitlab-ce instance which has pretty high up time running on a 10 year old server (and its backup)

Takes a few minutes a month to update and babysit. We only basically use the CI and the repository functionality though so gitlab has been "feature complete" for us for a while. Probably I we could swap to something like gitea or forgeo but the gitlab CI seems like a nice balance between powerful when it needs to be and simple otherwise for our use case so there is nothing driving us to swap.

pksunkara 1 hour ago | parent

I think all they have to do is stop giving free CI to everyone and this reliability issue will be solved immediately.

Disclaimer: I am building a code forge myself at https://juju.bi

willio58 1 hour ago | parent

I've heard this claimed dozens of times. I wonder if anyone at Github can confirm this theory and maybe give insight into why they haven't just done this?

tmpz22 1 hour ago | parent

Im guessing the lock-in is strong enough that they can eat the reputational hit while getting meaningful training data by subsidizing the free users.

Github has a decade of positive reputation to deal out, I don't see Grok or OpenAI jumping in and competing with that.

StilesCrisis 35 minutes ago | parent

I wanted to set up a Cloudflare static page and all the tooling is completely designed around Github. If you want any other service, you have to roll it yourself.

cls59 13 minutes ago | parent

The official Cloudflare CLI can ship static site content with one command: "wrangler pages deploy"

Easy enough to run from another CI or automated system.

dannyw 15 minutes ago | parent

I suspect the 1/4 (?) ownership by Microsoft plays a factor into OpenAI.

Otherwise I reckon they’d have cloned GitHub already; possibly with a “Mirror to CodexHub? Free usage reset”.

oleg_antonyan 1 hour ago | parent

free CI on github is so ridiculously overpowered it's hard to believe they still provide it

MrDunham 55 minutes ago | parent

I think it's brilliant. I'm now paying $30 per month for Github actions as a duo shop... it's freemium working the way it should. Give them the first hit and all.

And yes, I need to optimize (and am), but they're already getting hundreds from little me. I even started going towards the CI doing browser walks, code tests, AI rework, etc. It saved me from having another dev machine.

Great stuff, powerful as all heck, and once you see the power it was—for me—impossible NOT to think of 6 other uses and blow through the free CI tier.

NorthSouthNorth 18 minutes ago | parent

The thing is, once you start paying for it, then a service like blacksmith is so attractive and easy to replace gh, that it seems a no brainer. We swapped like 30-40 gh actions from gh runners to Blacksmith runners and cut our ci times by about 4x on almost all of our workflows. And all you do is auth their gh app and update an image tag.

Especially if you can take advantage of the high vcpu runners, then you can get some really fast pipelines going (not affiliated w/ them, just an enjoyer).

m4rtink 1 hour ago | parent

Isn't that why most people are on GitHub in the first place ? Which then, Windows style eventually turns into people wanting to use GitHub also for company work, where the real money comes from for GitHub.

Without free CI in GitHub, you might as well say:

"they can just shut the whole thing down and this reliability issue will be solved immediately"

jjice 53 minutes ago | parent

Maybe it is, but I know that lots of us used GitHub way before Actions launched. I think even if Actions wasn't free, GitHub would still have a massive advantage because it is the most well known place to host code, especially by less technical users (which is where I see a lot of finger pointing going with the increase in GitHub usage).

I would not be surprised if a serious amount of this usage is not user-intended and instead the LLM just inserts some sort of CI job off the bat and people roll with it.

sergiotapia 42 minutes ago | parent

Not even that heavy handed. They just need to _severely_ rate limit the top 0.1% of free users that are raping their hardware. I bet they are whales in usage terms and the root cause of many heartache.

nerdypepper 1 hour ago | parent

like clockwork! right now on the frontpage:

The federated code forge (https://news.ycombinator.com/item?id=50305573)

talon8635 1 hour ago | parent

No way I don’t believe it

peri-cl 1 hour ago | parent

The shape of this HN duplicate detector actually works as a GitHub uptime chart:

https://news.ycombinator.com/from?site=githubstatus.com

It's driven by aggregated user complaints, so it's pretty high-quality. Comment counts are a proxy for "how bad was the outage" + "how long was the outage thread topical for".

edit to add: Here's a scatterplot of the same,

https://files.catbox.moe/hhfh4l.png

hridyaagrawal 59 minutes ago | parent

lol

xiaoyu2006 26 minutes ago | parent

Really smart insight lmao

SoftTalker 1 hour ago | parent

Nice graphic on that page of a bunch of cartoon workers tending to what looks like a Rube Goldberg contraption. It's all just lighthearted fun and games.

montroser 1 hour ago | parent

It bugs me how there can be an acknowledged live incident happening today with Actions, and yet down the page where it shows last 30d uptime for actions, today's little rectangle is a chipper green.

hoppp 1 hour ago | parent

It's hard coded probably.

deathanatos 57 minutes ago | parent

If you hover over the green bar for today, the tooltip knows there's an incident.

whalesalad 50 minutes ago | parent

if day.endswith('y'): return GREEN

dozerly 34 minutes ago | parent

A completely different, non GH controlled status page had to be made to accurately report status for them.

Nedomas 1 hour ago | parent

Self-hosted GitHub Actions on AWS spot are still working.

After many outages we had to build our own free runner https://github.com/supercorp-ai/superci

howunfortunate 24 minutes ago | parent

This works as long as the schedulers don't go down (which they have, repeatedly in past incidents)

m4rtink 1 hour ago | parent

I'm sure there is a good meme to use for such a frequently occurring problem, but it would be too over used by this point.

enjoyyourlife 1 hour ago | parent

Giving Github more attention

senderista 58 minutes ago | parent

Lesson for every infra provider: physically separate your free tier infra from your paid tier infra.

the_real_cher 38 minutes ago | parent

It's getting to be hilarious at this point, honestly.

TavsiE9s 36 minutes ago | parent

No way, is it a day ending in "y" again?!

time0ut 23 minutes ago | parent

I wonder if I can hook failed git push into betting on github's status on polymarket.

CodingJeebus 19 minutes ago | parent

It's pretty easy to see where that ends. The headline writes itself: "Github engineers arrested for betting on downtime on prediction markets and intentionally causing outages"

huijzer 23 minutes ago | parent

Ever since I switched to Forgejo (could do Gitea or other self-hosted too), my outages have gone down to basically zero. It’s only down when I do maintenance but since I’m not pushing code at the same time it doesn’t matter

Pxtl 21 minutes ago | parent

I was confused why the github file browser was falling over.