snugrss

Generate RSS feeds from websites, filter existing RSS feeds and enrich them with scraped article contents.

Installation

cargo install --git https://codeberg.org/jkramer/snugrss.git

Make sure ~/.cargo/bin is in your $PATH.

Configuration

snugrss uses $XDG_CONFIG_HOME/snugrss/feeds.toml for its configuration by default, alternatively you can specify a it using -c /somewhere/else.toml.

Here is a sample config.

output = "/tmp"

[[generate_feed]]
title = "Neuerscheinungen auf kriminetz.de"
file = "kriminetz.rss"
url = "https://www.kriminetz.de/neuerscheinungen"
item = "div.article-content"
heading = "h2.medium"
link = "h2 a"
content = "div.field-type-text-with-summary"
media = "img"

[[generate_feed]]
url = "https://www.punchfork.com/recipes/instant-pot-dinner"
title = "New Recipes"
file = "recipes.rss"
item = "div.story-card"
link = "a"
heading = "h1"
rebase = "body"
content = "div.closeup-container"

[[process_feed]]
file = "tagesschau_innenpolitik.rss"
url = "https://www.tagesschau.de/inland/innenpolitik/index~rss2.xml"
readability = true

[[process_feed]]
file = "some_feed.rss"
url = "https://site.com/rss.xml"
resolve_redirects = true
exclude = [
    { field = "url", regex = "reddit\.com" },
    { field = "title", regex = "sport|weather" },
    { field = "description", regex = "some|boring|stuff" },
]
include = [
    { field = "title", regex = "exciting.*stuff" }
]

output: directory where feeds will be saved

[[generate_feed]] and [[process_feed]] start different feed section, one for generating new feeds by parsing websites using CSS selectors, the other for processing an existing RSS feed, filtering its items and extending it with hopefully the full article contents by scraping the pages the item links point to.

[[process_feed]] only requires two settings: file which is the file name the feed will be saved as (in the directory defined in output), and url which is the URL of the RSS feed to process. exclude allows you to remove items that match a regex, while include allows you to exclude all items that do NOT match the regex. Filters can be applied to item URL, title or description/summary. See example above. readability = true activates fetching of item urls and extracting article texts for the item description (default is false). resolve_redirects follows redirects of the original URL and replace the item link with the final location. This is done before filters are applied, so if you have any URL filters, they will check against the final URL after resolving redirects.

[[generate_feed]] requires file as well plus a bunch of extra settings for generating the feed:

• title: title of RSS feed
• url: URL of the website to scrape
• item: CSS selector for the base node that represents one item
• heading: CSS selector for the item title, relative to the item node
• link: CSS selector for the item link, relative to the item node, needs to have href attribute that contains the item page URL
• content: optional CSS selector for the item content, relative to the item node
• media: optional CSS selector for the item image, relative to the item node, needs to have a src attribute that contains the image URL
• rebase: optional CSS selector for new base node on the item's page - if set, each item's link will be fetched and the node found using this selector on the loaded page will be used as new base for all other per-item selectors (heading, content, media), which are expected to be relative to the rebase node

Running

RUST_LOG=snugrss snugrss [-c ...] [-o ...]

Use -c to provide an alternative configuration (default is $XDG_CONFIG_HOME/snugrss/feeds.toml). -o allows you to override the output directory as defined by output in the configuration. Setting RUST_LOG=snugrss gets you a bunch of debug output, by default only errors are printed.

Caching

Pages that are fetched for scraping (when scrape = true is set or for all items in process_feed feeds) are cached in $XDG_CACHE_HOME/snugrss. The base URL (defined in url generate_feed blocks) is never cached.