YHK - yet another hacker news clone

Moving our warehouse off DuckDB

84 points emschwartz 1 hour ago 42 comments

thisisauserid 1 hour ago | parent

Is Quack, the DuckDB client-server protocol, something that could have helped them with the scaling problems they encountered?

0c3ca83 1 hour ago | parent

Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?

It feels like a lot of churn and not a lot of measurements or engineering.

coolgoose 1 hour ago | parent

I don't get the blog post either.

huqedato 59 minutes ago | parent

This sounds more like: shoot first ask later. (OMG!)

lokar 59 minutes ago | parent

To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.

I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).

gonzalohm 38 minutes ago | parent

How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case

dominotw 33 minutes ago | parent

> It feels like a lot of churn and not a lot of measurements or engineering.

They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.

avree 1 hour ago | parent

Haven't seen Posthog's website in a while. Man, it's hideous.

dm03514 1 hour ago | parent

I use it as a free web analytics, the UX is so bad and convoluted, it's like a bad joke.

LawrenceKerr 1 hour ago | parent

I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.

But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.

mattmcal 1 hour ago | parent

I actually love this design but I nevertheless agree it's atrocious.

SlackingOff123 1 hour ago | parent

And what's with that bs cookie banner? If the only cookies are functional/essential, they don't need to show a cookie banner. If it's for analytics, then I should be able to reject it. Otherwise, it's just a dick-over.

stickfigure 47 minutes ago | parent

I actually like the menus. They work great on mobile too.

We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.

pnathan 30 minutes ago | parent

really? I think its one of the best websites going these years.

bitpush 1 hour ago | parent

> Five months ago, we rebuilt our warehouse [1] on DuckDB instead of ClickHouse.

Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)

https://posthog.com/blog/why-we-rebuilt-our-data-warehouse

wheelerwj 1 hour ago | parent

Engineers like to experiment and we get the benefit of learning from their experiences. Not sure why you’re upset.

weego 1 hour ago | parent

I mean, mostly it just comes off as egregious self-indulgence to try one thing and then jump to "we built the solution ourselves!" when the problem is not novel at all. These things are just tiresome.

ambicapter 25 minutes ago | parent

Maybe it's the "me first" mindset you're bringing to engineering?

crorella 55 minutes ago | parent

The intro to that post is so bad: "PostHog's original data warehouse had a single goal: delay the moment a company needs to hire their first data engineer."

It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.

ecshafer 42 minutes ago | parent

Ironically a good data engineer should probably let them avoid needing 3 different data warehouse systems within a year.

Aurornis 1 hour ago | parent

> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue

They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?

The article then ends before saying if the new setup solved their problems.

It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.

I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.

sgt 1 hour ago | parent

What's wrong with ClickHouse again?

Aurornis 1 hour ago | parent

I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.

vindex10 52 minutes ago | parent

Russian ties?

https://news.ycombinator.com/item?id=33082224

spooneybarger 47 minutes ago | parent

Unless something has changed in the last year, a lot of teams have serious problems with stability.

throwaw12 19 minutes ago | parent

I am curious to learn more about those issues, anyone have shared their experience publicly? would love to read

renegade-otter 58 minutes ago | parent

It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.

sreekanth850 1 hour ago | parent

after 3 months, how we switched back to forked clickhouse.

tmach32 1 hour ago | parent

Interestingly, it seems that a data lake is the hot new thing to build in house. Hell, we did it too. If you don't mind, I wrote up exactly what these new data lakes tend to look like: https://telemetrymachine.com/blog/the-new-state-of-the-art-i...

Thaxll 1 hour ago | parent

Translation: frontier models are so good that we can rewrite our own warehouse infra.

uutangohotel 1 hour ago | parent

I would have appreciated more technical details with methodologies and explicit failures observed. Oh and a human voice would be nice for a change.

> Customers ran more complex queries than we expected

Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?

> Predicting CPU and memory for a specific query was even harder.

What attempts / methodologies were used to try this? What was hard? What failed?

> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork

This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)

> Missing reconnects and managing TTLs and idle connections at scale is hard

Wonder if they tried pgdog?

crorella 57 minutes ago | parent

Feels off that only 5 months ago the company just migrated to DuckDB only to now move to something else.

Who did the growth estimation of the data? This looks like an example of bad planning.

whinvik 48 minutes ago | parent

The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.

But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.

npalli 47 minutes ago | parent

This idiotic churn used to be restricted to front end devs and backend folks were spared. Thanks for Agents now every part of the stack is going to be in constant churn thanks to resume builders. At no point in the article did you get a sense they have a handle on what their problem is and why a certain technology and architecture would be a better fit. Just try it out, churn out, try it out.

mgfist 46 minutes ago | parent

HogLake is quite the name ..

mvkel 45 minutes ago | parent

This reminds me so much of when we would bring a new marketing leader and they would plant their flag by nuking all of the existing marketing automation processes to bring in theirs, with no net impact on the business. Bike shedding.

1a527dd5 43 minutes ago | parent

I'm surprised at this. That is a lot of engineering effort without revealing why they chose they own format and chose to query it via Trino.

bhouston 42 minutes ago | parent

6 months from now: We are moving our warehouse off of Trino + HogLake to newer technology XXX.

This is ridiculous. Why can you not foresee such issues? Do you not do testing with mock data at the size you need? Migrating production between stacks so often means you are not getting new features in. This is pure churn.

hasyimibhar 41 minutes ago | parent

I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].

[1] https://clickhouse.com/blog/jump-trading-uses-clickhouse-wit...

fhn 30 minutes ago | parent

This post is pointless. Tells us how it went, show performance numbers, show benefits, resource usage, ease of use, etc.

time0ut 28 minutes ago | parent

The git repo [0] has a lot more of the technical details.

[0] https://github.com/PostHog/hoglake

kwillets 12 minutes ago | parent

Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.

I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.

I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.

But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.