YHK - yet another hacker news clone

GitHub Incident with Git Operations, Pull Requests and Actions

140 points gagan2020 1 hour ago 106 comments

geooff_ 1 hour ago | parent

Actions being down is one thing - I can't even land a PR.

vitosartori 1 hour ago | parent

PR? I can't even push haha

laurent123456 1 hour ago | parent

I can't even create issues (error 500)

gagan2020 1 hour ago | parent

Same here. I wanted to push something on Github and its was giving 500. This is most severe issue in recent times.

balanceiskey15 1 hour ago | parent

GitHub added new webhook IP ranges this morning, curious if that's related.

gagan2020 1 hour ago | parent

That might be the cause but let's see.

There 4 updates contains webhook mention in 2 of the updates.

"Webhooks is experiencing degraded performance"

iercan 1 hour ago | parent

first time seeing "git push" on github throwing an "Internal Server Error"

cowlevel 1 hour ago | parent

gitea, forgejo, gitlab CE, and ssh still working fine

hermannj314 48 minutes ago | parent

Exactly. I'm running forgejo in a container, pushing backups to a private S3 storage, and sent the forgejo team a small liberapay payment.

Maybe 30 minutes to get it all setup and then I moved on with my life.

user43928 45 minutes ago | parent

I never noticed downtime with public GitLab. It's also generally better than GitHub and Actions.

A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.

cowlevel 37 minutes ago | parent

Of course. Azure is the reason GitHub is down so often, too.

mort96 6 minutes ago | parent

I have to second this, GitLab CI is pretty good. I'll never be on board with infrastructure people's obsession with putting code in yaml but outside of that, the model makes much more sense to me.

A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.

It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".

alexellisuk 1 hour ago | parent

I thought it was odd that we'd not had one of these for a few weeks.

A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.

cowlevel 38 minutes ago | parent

There was one yesterday.

malfist 35 minutes ago | parent

Not had one for a few weeks? There was an outage on Monday (as usual)

caaqil 1 hour ago | parent

Who even runs GitHub at this point?

dam_jackalopes 1 hour ago | parent

I just work here man.

Kudos 1 hour ago | parent

Copilot.

herpdyderp 1 hour ago | parent

According to themselves, 225 million people.

https://github.com/about

john_strinlai 11 minutes ago | parent

thats how many people use it, not employees.

esafak 1 hour ago | parent

Who's the commanding officer here, soldier?

freakynit 1 hour ago | parent

Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.

embedding-shape 50 minutes ago | parent

I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?

It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.

enraged_camel 16 minutes ago | parent

>> Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?

This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.

embedding-shape 14 minutes ago | parent

Sure, and backups should be replicated over 3 mediums, and you shouldn't deploy bugs into production, and...

Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?

pixelesque 1 hour ago | parent

One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"

aarestad 1 hour ago | parent

99% of the time, this is a server returning a generic 500 HTML response to an HTTP call that was expecting a JSON response.

aleksandrm 1 hour ago | parent

"Rewrite GitHub in Rust. Make no mistakes."

jeremyjh 1 hour ago | parent

codefloe.com is still up

freakynit 1 hour ago | parent

So, "git push" is down ... how are github engineers gonna push the fix now?

whalesalad 1 hour ago | parent

sudo git push

pkthunder 59 minutes ago | parent

Make no mistakes.

tripleee 52 minutes ago | parent

in vim over ssh of course.. agents can use vim right?

dodslaser 41 minutes ago | parent

Yeah, but they keep trying to use Emacs commands...

C-x M-c M-unfuck-git

marcosdumay 16 minutes ago | parent

You'd use tramp... What is way better than the vim's solution.

zrail 13 minutes ago | parent

Real talk I have a M-x unfuck-this-buffer. All it does is toggle the Unicode input mode that I sometimes accidentally trigger but have never figured out how.

bearjaws 51 minutes ago | parent

rsync and sftp, like god intended

redanddead 44 minutes ago | parent

Like the olden days

ChuckMcM 36 minutes ago | parent

Nah in the olden days we'd load a tape with patches and the code would read a patch off tape, apply it, then read the next one. lol.

Sohcahtoa82 16 minutes ago | parent

I prefer Dropbox.

PackMan97 15 minutes ago | parent

Might I ask why you aren't working directly on the prod server? God doesn't use staging or dev, he just codes directly on prod.

sdcfgy 4 minutes ago | parent

God uses UUCP

xydone 34 minutes ago | parent

git push --force. No time for the lease either

snihalani 12 minutes ago | parent

s3/blob storage

oslem 1 hour ago | parent

Color me shocked

nstott 49 minutes ago | parent

This is so frustrating. I wish it were easier to migrate away from this tool

Brian_K_White 1 hour ago | parent

holy crap push is a new low

acclay 59 minutes ago | parent

I'm tired, boss.

rapind 57 minutes ago | parent

Based on the comments in this thread, you'd think that Reddit was down.

icantevenhold 23 minutes ago | parent

Complaining that HN is becoming Reddit is imo even worse than posting Reddit-level comments - those are at least sometimes funny

iambenm 21 minutes ago | parent

"Please don't post comments saying that HN is turning into Reddit." [0]

[0] https://news.ycombinator.com/newsguidelines.html

schrodinger 17 minutes ago | parent

This isn't that imo... do you think it violated the spirit of that rule?

embedding-shape 16 minutes ago | parent

What do you think parent is trying to say, except that this HN comment section looks like reddit? I'm trying my hardest to apply charitable reading, but even this has limits.

gchamonlive 5 minutes ago | parent

[delayed]

microsoftedging 4 minutes ago | parent

Perhaps about the quality of the comments. In my opinion it's not really that bad but reddit-style comments are often cheap shots that are slightly off topic, some of which I see present here

Rapzid 3 minutes ago | parent

That it looks like reddit is down so a bunch of people used to posting low effort slop over there ended up here posting low effort slop?

This thread isn't "HN" and that comment wasn't saying it was turning into Reddit..

But let's call a spade a spade. I got worried my account would be flagged due to the amount of down voting I was doing.

Drupon 2 minutes ago | parent

Always thought this rule is hilarious because the fact that such comparisons are so overwhelmingly common that they needed to address it with this bitchy little rule (adding one link to an example per word is a classic tell that an internet moderator is spending their highly compensated moderation time Extremely Mad lol).

However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.

DaSHacka 18 minutes ago | parent

I suspect it'll only become more common as Reddit enshittifies further and further. The platform is basically unusable for anyone, thus its suitability as a 'containment zone' for the redditor phenotype goes down with it.

I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.

ofjcihen 10 minutes ago | parent

“People here have opinions that I don’t like.”

sdcfgy 5 minutes ago | parent

Nah. I can't see anyone blaming it on Trump, Israel, datacentre water having a memory, their parents, their neighbour, spirits or spez.

rvz 57 minutes ago | parent

The last time GitHub went down was...yesterday.

GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.

It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.

time0ut 57 minutes ago | parent

Well it is a Wednesday.

I noticed because SSO was down. They don't even report on that as far as I can tell.

coliveira 56 minutes ago | parent

That's what you get when the bulk of open source is now depending on Microsoft.

knowknow 55 minutes ago | parent

Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?

TavsiE9s 54 minutes ago | parent

I mean... they got acquired by Microsoft.

cowlevel 40 minutes ago | parent

Almost 10 years ago

embedding-shape 49 minutes ago | parent

They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.

And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.

I wouldn't hold my breath here.

mjr00 46 minutes ago | parent

> has something changed to cause this many outages?

AI coding. Which can be interpreted one of two ways:

* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or

* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.

cowlevel 40 minutes ago | parent

It also coincides with GitHub moving to exclusively Azure infrastructure. Azure is a terrible shit show for reasons that have nothing to do with AI.

nosefrog 25 minutes ago | parent

Haha I hope this is true, it fits my experience on Azure. 0 machines available. If someone from GitHub wants to chime in.

franktankbank 22 minutes ago | parent

Azure sucks balls. Its support has no clue whats going on.

o_m 43 minutes ago | parent

Give them some slack. Their core product is AI, not version control

ainar-g 41 minutes ago | parent

> Was github always this shoddy or has something changed to cause this many outages?

The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/

MadrasTh0rn 52 minutes ago | parent

Love it

gwbas1c 52 minutes ago | parent

What's frustrating is that:

1: The status site says that pull requests are working,

2: A PR I'm working on right now is missing commits,

3: The contact support page errors out.

atsjie 9 minutes ago | parent

Contact support page is probably just a waste of time even if it did work.

bossyTeacher 51 minutes ago | parent

This is becoming way more frequent than a monthly event. Does anyone know why GitHub keeps having incidents?

christophilus 49 minutes ago | parent

It’s mentioned on most threads, but their traffic and usage is way up due to AI / agentic programming.

cowlevel 38 minutes ago | parent

Because it's vibecoded and because it's running on Azure

tripleee 49 minutes ago | parent

Three days in a row now right?

HenkieTenkie 49 minutes ago | parent

Another case of tactical downtime.

sleepybrett 48 minutes ago | parent

i see they started letting the github engineers ship code again after that shitshow last month.

user- 43 minutes ago | parent

I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.

And while I was typing this comment it just blipped again. Atleast this time I know why.

I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.

DaSHacka 16 minutes ago | parent

> I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.

Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.

zrail 10 minutes ago | parent

There aren't any restrictions anymore on who you can invite to your team. I added a passkey admin account and added other users via email invites. The account owner is still my GitHub account but I pretty much don't use it day to day.

honeycrispy 38 minutes ago | parent

Funny, yesterday git operations had an unreported outage too. I couldn't pull changes for ~10 minutes.

odo1242 32 minutes ago | parent

To be fair to GitHub, outages are pretty short (~40 minutes), and they rarely impact critical parts of the service, nor do they affect one’s ability to work (git offline)

slowin 26 minutes ago | parent

Not being able to push is pretty critical.

lenerdenator 25 minutes ago | parent

I'm not able to get code pushed, meaning I can't address PR feedback, meaning I can't get things approved, meaning code isn't getting merged to main and thus a ticket isn't getting closed. Sounds pretty critical to me.

OrderlyTiamat 22 minutes ago | parent

40 minutes is an eternity when you can't push to a gitops repo with a PRD issue.

mschuster91 21 minutes ago | parent

... and that is (yet another example) why pipeline driven gitops is a really darn bad idea.

Rapzid 10 minutes ago | parent

If I had a super critical gitops pipeline that couldn't deal with the odd provider outage I'd be using my cloud providers repo service.

Same as with container registries; prod shouldn't be pulling from Dockerhub.

m2ger 22 minutes ago | parent

git push ending up with Internal Server error is similar to save file ending up with disk full error. yes, it can work after 40 minutes, and one can go to lunch and retry later, but here in Europe the troubles started at the end of a (normal) workday.

mococa 17 minutes ago | parent

In Mars, right?

mort96 10 minutes ago | parent

What... what parts of GitHub would you consider "critical parts of the service" if not git operations, pull requests and CI?

acedTrex 30 minutes ago | parent

Im so grateful that coding is solved

esafak 29 minutes ago | parent

This is such an opportunity for competitors. I hope they don't squander it.

lenerdenator 6 minutes ago | parent

The drive to win a market by accumulating customer market share is no longer the main drive behind business ventures in our economy. It has been replaced with the drive to increase equities market valuation. This, at times, contradicts accumulating or retaining customer market share.

Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.

They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.

hdgvhicv 23 minutes ago | parent

Must be a Y in the day

udave 23 minutes ago | parent

Hope it survive the next few years of codification of all of humanities unfinished cheap ideas.

sdcfgy 4 minutes ago | parent

This is horribly accurate.

mococa 18 minutes ago | parent

Again?

projectescape 16 minutes ago | parent

GH reliability has been a regular question mark lately, and many time issues are not reported publicly too.

In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos

throw03172019 11 minutes ago | parent

Just another day.

zer0x4d 4 minutes ago | parent

My self hosted Gitlab still at 99.9% uptime since the last time github/gitlab were down.