Grumpy Archive

Just leaving this here for reference as I'm currently moving my old stuff to Codeberg from GitHub and other places.

About

This has been my homegrown, self-hosted news reader for many years. For legal reasons (see below) I've never made it accessible to the public and eventually stopped running it for myself as well, when exploding server costs made it unfeasible (storing and searching hundreds of millions of articles needs a bit of decent hardware).

I started this mostly because news sites like Google News had (and still have) terrible recommendation algorithms and don't seem to take users' reading preferences into account at all, so I came up with a (very naive) Bayes algorithm implemented in pure SQL that somehow immediately worked much better than anything else I've seen on news sites so far. From there a bunch of other random features were added.

Features

• discover RSS/Atom feeds from websites and fetch/import them periodically
• initially show random news for the user to rate as good/bad/ignore
• update the user's preferences profile in the background (minion worker)
• over time recommend articles that the user will likely be interested in
• when opening an article, it fetches the entire article content using a homegrown "readability" algorithm, in some cases even circumventing paywalls, so you can read it right there in Grumpy Archive and don't have to visit an ad-riddled news site (note: this is the main reason why I don't recommend running this as a public service, news publishers wouldn't be overly fond of it and are known to sue sites over stuff like this)
• options to export full articles as markdown or send to a pastebin (telegra.ph) for sharing
• remove duplicates: big/breaking news are covered by lots of different sources, so after reading/rating an article, other articles that Grumpy Archive considers to be about the same event won't be suggested for a certain amount of time
• allow the user to rank/prioritize sources
• assign a clickbait score to articles and use it to prioritize more professional coverage of news (uses a series of common clickbait patterns and is only implemented for German language news, adding support for other languages shouldn't be a huge problem though)
• full-text search over all articles with a bunch of search filters for the proper news nerd (e.g. "I think I've read something about this some time ago" → search for articles similar to article X but older than one month, and much more)
• save named searches
• create RSS feeds from saved searches
• keep a per-user blacklist to permanently hide articles matching a list of negative search queries

Over time a bunch of more obscure features/tools were added, such as a language learning mode for articles in foreign languages (allowed you to click on words to get a translation/definition and save them to a dictionary for later vocabulary training), several Reddit bots and probably a lot that I forgot about.

Most of the heavy lifting is done by PostgreSQL and the tsearch2 full-text search engine in particular, not only for the search itself but the stemmed tokens in tsvector columns are also used for the Bayes recommendation algorithm (see lib/Grumpy/Archive/Account.pm starting at line 92 as well as lib/Grumpy/Archive/Task/Dictionary.pm).

Installation

TBD

To be quite honest this requires a couple of 3rd party PostgreSQL extensions that haven't been maintained for a while and can be a hassle to install with current Postgres versions, and I don't have the time right now to either remove the dependency for those extensions or document how to make them work. If anyone's actually interested in running this, let me know.

A Few Notes

• Since this never was meant to be public I maintained the source in a private repository hosted on my own server and also kept configuration files including credentials in there. For that reason this (public) is a fresh import without Git history.
• Please note that in lib/XML there's a full copy of the Perl module XML::Feed used for feed parsing and discovery. I don't quite remember why I did this, most likely I had to make some changes specific to this project that wouldn't be useful for anyone else so I didn't contribute them back to mainstream. Copyright of course remains with the original author Dave Cross and the original license under which XML::Feed was published applies. Just thought I should mention this here for transparency.
• The directory public/fonts is supposed to contain a couple of fonts, IBMPlexSans-Regular.ttf in particular, which I won't add to the repository for copyright reasons.