2024-04-18 15:27:00 +02:00
|
|
|
# republisher-redux
|
|
|
|
|
|
|
|
|
|
``` shell
|
2026-03-29 12:59:08 +02:00
|
|
|
nix develop
|
|
|
|
|
uv sync --all-groups
|
2026-03-29 13:52:23 +02:00
|
|
|
cat > repub.toml <<'EOF'
|
|
|
|
|
out_dir = "out"
|
|
|
|
|
|
|
|
|
|
[[feeds]]
|
|
|
|
|
name = "gp-pod"
|
|
|
|
|
url = "https://guardianproject.info/podcast/podcast.xml"
|
|
|
|
|
|
|
|
|
|
[[feeds]]
|
|
|
|
|
name = "nasa"
|
|
|
|
|
url = "https://www.nasa.gov/rss/dyn/breaking_news.rss"
|
|
|
|
|
EOF
|
|
|
|
|
uv run repub --config repub.toml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`out_dir` may be relative or absolute. Relative paths are resolved against the
|
|
|
|
|
directory containing the config file. Optional Scrapy runtime overrides can be
|
|
|
|
|
set in the same file:
|
|
|
|
|
|
|
|
|
|
```toml
|
|
|
|
|
[scrapy.settings]
|
|
|
|
|
LOG_LEVEL = "DEBUG"
|
|
|
|
|
DOWNLOAD_TIMEOUT = 30
|
2024-04-18 15:27:00 +02:00
|
|
|
```
|
|
|
|
|
|
2026-03-29 13:52:23 +02:00
|
|
|
See [`demo/README.md`](/home/abel/src/guardianproject/anynews/republisher-redux/demo/README.md) for a self-contained example config.
|
|
|
|
|
|
2024-04-18 15:43:03 +02:00
|
|
|
## TODO
|
|
|
|
|
|
|
|
|
|
- [x] Offlines RSS feed xml
|
|
|
|
|
- [x] Downloads media and enclosures
|
|
|
|
|
- [x] Rewrites media urls
|
2024-04-18 17:28:09 +02:00
|
|
|
- [x] Image normalization (JPG, RGB)
|
2024-04-19 13:22:49 +02:00
|
|
|
- [x] Audio transcoding
|
|
|
|
|
- [x] Video transcoding
|
2024-04-19 15:53:03 +02:00
|
|
|
- [ ] Image compression - Do we want this?
|
|
|
|
|
- [x] Download and rewrite media embedded in content/CDATA fields
|
2024-04-18 15:43:03 +02:00
|
|
|
- [ ] Config file to drive the program
|
|
|
|
|
- [ ] Daemonize the program
|
|
|
|
|
- [ ] Operationalize with metrics and error reporting
|
2024-04-18 15:27:00 +02:00
|
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
republisher-redux, a tool to mirror RSS/ATOM feeds completely offline
|
|
|
|
|
|
|
|
|
|
Copyright (C) 2024 Abel Luck
|
|
|
|
|
|
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
|
|
|
it under the terms of the GNU Affero General Public License as
|
|
|
|
|
published by the Free Software Foundation, either version 3 of the
|
|
|
|
|
License, or (at your option) any later version.
|
|
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
GNU Affero General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU Affero General Public License
|
|
|
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
