anynews/republisher
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
republisher/README.md

2.2 KiB
Raw Permalink Blame History

republisher-redux

nix develop
uv sync --all-groups
cat > repub.toml <<'EOF'
out_dir = "out"

[[feeds]]
name = "Guardian Project Podcast"
slug = "gp-pod"
url = "https://guardianproject.info/podcast/podcast.xml"

[[feeds]]
name = "NASA Breaking News"
slug = "nasa"
url = "https://www.nasa.gov/rss/dyn/breaking_news.rss"
EOF
uv run repub --config repub.toml

out_dir may be relative or absolute. Relative paths are resolved against the directory containing the config file. Each feed now needs a user-provided slug, which is used for output paths and filenames. Optional Scrapy runtime overrides can be set in the same file:

[scrapy.settings]
LOG_LEVEL = "DEBUG"
DOWNLOAD_TIMEOUT = 30

Additional feed definitions can also be imported from one or more TOML files, including a pygea-generated manifest.toml:

feed_config_files = ["/absolute/path/to/pygea/feed/manifest.toml"]

Imported files only need [[feeds]] entries with name, slug, and url.

See demo/README.md for a self-contained example config.

TODO

  • Offlines RSS feed xml
  • Downloads media and enclosures
  • Rewrites media urls
  • Image normalization (JPG, RGB)
  • Audio transcoding
  • Video transcoding
  • Image compression - Do we want this?
  • Download and rewrite media embedded in content/CDATA fields
  • Config file to drive the program
  • Daemonize the program
  • Operationalize with metrics and error reporting

License

republisher-redux, a tool to mirror RSS/ATOM feeds completely offline

Copyright (C) 2024-2026 Abel Luck

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.