Swanky new site, deprecating poobrains and a new webframework coming whenever

Finally, after procrastinating for months and then overcompensating by obsessing over details, our swanky new site is finally online.

This site was developed after we deprecated poobrains, the custom webframework running our previous site.

Dropping poobrains after pouring so much blood and sweat into it was a hard decision to make, but probably the right call, for what we believe are hysterical raisins.

Historically speaking, the development of Flask and poobrains was deeply intertwined. We've been working on what would become poobrains since at least 2011 and the project and its predecessors were all based on Flask since day one.

Especially in the beginning, it was still pretty early days for Flask and when something we needed wasn't possible due to limitations of Flask, it was often resolved by just dropping into the IRC channel, chatting a bit with mitsuhiko (Flask's project founder) and either being pleasantly surprised by the next version of Flask covering our needs more elegantly than our proposed improvements would have or – and this is what caused problems for us later on – being pointed to some obscure Flask internal that allowed us to do what we needed.

Due to this, the use of Flask internals that were never part of the official API accumulated. And for a long time, this wasn't much of a problem – with new Flask versions rarely introducing changes that broke compatibility with poobrains' code.

In hindsight, we can only surmise that mitsuhiko probably had some vague recollection of someone depending on this internal or that one and this is what kept things stable for us.

But when mitsuhiko dropped maintainership, bitrot started accumulating fast enough that it became impossible to realistically maintain poobrains as side-project for a single person.

And – to be absolutely clear – this is not the fault of Flask's new maintainership, who are doing a splendid job keeping one of our favorite software projects alive and thriving.

The changes that kept breaking poobrains weren't part of the public API and we can't fault the maintainership for making them. There never was any sort of agreement or communication to preserve the state of those internals. We could have kept proper track of which internals we used where and for what, but we didn't.

It's just a matter of unlucky circumstances that poobrains had to die.

Of course, a predecessor for poobrains was an immediate concern.

After rolling things around in our brainpan for a while, we decided that the best way to minimize the risk of the next iteration of our framework falling victim to changes in upstream was to simply minimize the upstream as much as possible.

And thus, ooze was born.

Based on raw WSGI, an extremely opinionated view of what the web should be like and delusions of grandeur, ooze aims to be everything that poobrains was and more.

The general aim to have a webframework that actually respects privacy and is specifically geared towards DIY data journalism with an integrated suite of tools for data-analysis and -visualization hasn't changed.

What has changed, is pretty much the entire underlying structure.

Foregoing any underlying framework or library for the web parts should be a huge help in minimizing breaking upstream changes accumulating over time, which in turn enables us to put energy into actual progress instead of fighting an uphill battle against the 'rot.

It also means taking a hiatus from the project doesn't incur additional technical debt anymore, which enables us to demote ooze to a "whenever"-project.

The simple truth is that between a boatload of projects, infrastructure, a job, chronic depression and trying to build up a life after homelessness, there simply aren't enough spoons left to regularly put large amounts of time into this project.

This might or might not change, it'll be done whenever.

That said, ooze does already have some interesting components.

Firstly, a custom templating engine inspired by CL-WHO which uses some simple AST modifications to have templates that directly translate to callable python functions which can be cached and return a traversable and modifiable tree of objects corresponding to the resulting HTML DOM.

Secondly, an SQL generator with an ORM-like interface that does just string manipulation and covers pretty much all of PostgreSQL's syntax. This coincides with a decision to specifically and only support PostgreSQL as database backend. For one, it's yet another cut dependency which will make the database interface more stable (at least within ooze itself), for another, it will be easy to extend this to add support for PostGIS features when we finally get to port over geodata features from poobrains.

Actual interaction with the database happens through some custom code inheriting from relevant classes of the SQL generator and extending their interface to allow easily accessing the database – and while we dropped peewee as a dependency, the interface of both the SQL generator and the database layer are still strongly inspired by it.

We're also strongly considering adding… *cough*

*deep inhale*

javascript.

Specifically, by adding partial page updates with server-side object rendering through a websocket. Of course, if this happens, it will be vanilla ECMAScript (not to be confused with VanillaJS, an irksome project name if ever there was) instead of depending on some shit framework.

The thought behind this is that, especially for more complex form applications like the analysis and visualization tooling, full page requests can quickly become intensive both in terms of computation and bandwidth, so partial page updates could reduce UI latency, unnecessarily spent compute cycles and therefore also the softwares' CO² footprint.

Besides all that, routing (including Subsites akin to Flask's Blueprints) also works already and there's even the humble beginnings of a form system, but alas, ooze is far from ready for prime-time yet.

Sessions are still missing, as are users, access control, any of the analysis and visualization tools and a myriad of other things.

Well – Flask of course!

If we wait until ooze is ready to run an actual, usable website, then we'd have to postpone the relaunch until 2027 or something…

and the previous, poobrains-based, site has already been suffering from bitrot for years. We've been fixing the reader-facing bits for hours on end everytime we update Flask for years now, but some of the administrative parts are in a woeful state of disrepair and this is entirely too much effort for no gain.

So we ported over some of the basic concepts and features of poobrains, particularly renderable objects and a slimmed down version of the form system, slapped some 2FA with TLS client certs and SHA3-512 based password storage on top, wrote an entirely new markdown integration using markdown-it-py with some custom extensions, ported and much better integrated scored links, did a metric fuckton of templating and styling, some asset design and probably half a million more things which we forget and that's how we arrived at this site.

Now for the reasoning, poobrains deprecation and ooze's unfinished state are certainly a major part of it, but this was also done so the code for this site is only for this site and doesn't have to adhere to the quality standards we have for a framework.

Our hope is that this will allow us to play things a bit more dirty and iterate faster through what we actually want in terms of features and tools for ooze while keeping required maintenance efforts minimal.

For now tho, we'll probably take it a bit easy on the web stuff and rather put our energy elsewhere. There are some posts we want to write on a few projects in various states of existence, ranging from physical manufacturing stuff to random gamedev prototyping, those projects themselves and we've been slacking on bouldering, too… 🤔