2024-04-18 15:27:00 +02:00
|
|
|
# republisher-redux
|
|
|
|
|
|
|
|
|
|
``` shell
|
2026-03-29 12:59:08 +02:00
|
|
|
nix develop
|
|
|
|
|
uv sync --all-groups
|
2026-03-29 13:52:23 +02:00
|
|
|
cat > repub.toml <<'EOF'
|
|
|
|
|
out_dir = "out"
|
|
|
|
|
|
|
|
|
|
[[feeds]]
|
2026-03-29 14:44:45 +02:00
|
|
|
name = "Guardian Project Podcast"
|
|
|
|
|
slug = "gp-pod"
|
2026-03-29 13:52:23 +02:00
|
|
|
url = "https://guardianproject.info/podcast/podcast.xml"
|
|
|
|
|
|
|
|
|
|
[[feeds]]
|
2026-03-29 14:44:45 +02:00
|
|
|
name = "NASA Breaking News"
|
|
|
|
|
slug = "nasa"
|
2026-03-29 13:52:23 +02:00
|
|
|
url = "https://www.nasa.gov/rss/dyn/breaking_news.rss"
|
|
|
|
|
EOF
|
|
|
|
|
uv run repub --config repub.toml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`out_dir` may be relative or absolute. Relative paths are resolved against the
|
2026-03-29 14:44:45 +02:00
|
|
|
directory containing the config file. Each feed now needs a user-provided
|
|
|
|
|
`slug`, which is used for output paths and filenames. Optional Scrapy runtime
|
|
|
|
|
overrides can be set in the same file:
|
2026-03-29 13:52:23 +02:00
|
|
|
|
|
|
|
|
```toml
|
|
|
|
|
[scrapy.settings]
|
|
|
|
|
LOG_LEVEL = "DEBUG"
|
|
|
|
|
DOWNLOAD_TIMEOUT = 30
|
2024-04-18 15:27:00 +02:00
|
|
|
```
|
|
|
|
|
|
2026-03-29 14:44:45 +02:00
|
|
|
Additional feed definitions can also be imported from one or more TOML files,
|
|
|
|
|
including a `pygea`-generated `manifest.toml`:
|
|
|
|
|
|
|
|
|
|
```toml
|
|
|
|
|
feed_config_files = ["/absolute/path/to/pygea/feed/manifest.toml"]
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Imported files only need `[[feeds]]` entries with `name`, `slug`, and `url`.
|
|
|
|
|
|
2026-03-29 13:52:23 +02:00
|
|
|
See [`demo/README.md`](/home/abel/src/guardianproject/anynews/republisher-redux/demo/README.md) for a self-contained example config.
|
|
|
|
|
|
2024-04-18 15:43:03 +02:00
|
|
|
## TODO
|
|
|
|
|
|
|
|
|
|
- [x] Offlines RSS feed xml
|
|
|
|
|
- [x] Downloads media and enclosures
|
|
|
|
|
- [x] Rewrites media urls
|
2024-04-18 17:28:09 +02:00
|
|
|
- [x] Image normalization (JPG, RGB)
|
2024-04-19 13:22:49 +02:00
|
|
|
- [x] Audio transcoding
|
|
|
|
|
- [x] Video transcoding
|
2024-04-19 15:53:03 +02:00
|
|
|
- [ ] Image compression - Do we want this?
|
|
|
|
|
- [x] Download and rewrite media embedded in content/CDATA fields
|
2026-03-29 14:10:52 +02:00
|
|
|
- [x] Config file to drive the program
|
2024-04-18 15:43:03 +02:00
|
|
|
- [ ] Daemonize the program
|
|
|
|
|
- [ ] Operationalize with metrics and error reporting
|
2024-04-18 15:27:00 +02:00
|
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
republisher-redux, a tool to mirror RSS/ATOM feeds completely offline
|
|
|
|
|
|
2026-03-29 14:10:52 +02:00
|
|
|
Copyright (C) 2024-2026 Abel Luck
|
2024-04-18 15:27:00 +02:00
|
|
|
|
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
|
|
|
it under the terms of the GNU Affero General Public License as
|
|
|
|
|
published by the Free Software Foundation, either version 3 of the
|
|
|
|
|
License, or (at your option) any later version.
|
|
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
GNU Affero General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU Affero General Public License
|
|
|
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
