republisher/PLAN.md

Plan

1. Task Record

The work spans two local repositories:

• /home/abel/src/gitlab.com/guardianproject-ops/pygea
• /home/abel/src/guardianproject/anynews/republisher-redux

Requested outcome:

1. Refactor pygea so it no longer hardcodes feed inputs in pygea/main.py.
2. Make pygea accept a TOML config file in the same general style as republisher-redux instead of pygea.ini
3. Replace tuple-based feed definitions such as ("Titulares", True, None) with proper keyed data shaped like: {"name": "Titulares", "only_newest": True, "content_type": None}.
4. Add a required user-provided slug field alongside each feed name in pygea. like: {"name": "Titulares", "only_newest": True, "content_type": None, "titulares"}.
5. Stop using the hash-based subdirectory name in pygea; use the configured slug instead.
6. Create a demo/ directory in pygea with an example config, similar to republisher-redux.
7. Change pygea output from manifest.json to manifest.toml.
8. Make pygea write manifest.toml in [[feeds]] format that republisher-redux can consume directly.
9. Each generated manifest feed entry must include:
   • name
   • slug
   • url
10. The manifest url must be an absolute file:// URI pointing to that feed's rss.xml.
11. Extend republisher-redux so its runtime config can load additional feed definitions from a separate TOML file, specifically the pygea-generated manifest file.
12. Keep current republisher-redux features intact while adding the extra feed-config source.
13. Update docs in both repos so the new workflow is discoverable.
14. Add or update tests in both repos.
15. Verify both projects are working.
16. Stage the resulting changes.
17. Draft a commit message, but do not commit.

Operational context and nuance to preserve:

• The intended deployment is two systemd services on the same machine, one for pygea and one for republisher-redux.
• The user will handle the systemd units; this task is only about application/config/docs/test changes.
• The purpose of slug is operational clarity and stable filesystem paths, especially for wiring pygea output into republisher-redux.
• slug must be user-supplied, not auto-generated.
• name may remain human-facing, including strings that are awkward for filesystem paths.
• republisher-redux should be able to merge feeds declared directly in its own config with feeds loaded from the external TOML manifest.
• Final validation should include formatter and flake checks, and work should be staged but not committed.

2. Execution Plan

1. Finish refactoring pygea runtime configuration:

   • Introduce a TOML config loader and validation.
   • Replace import-time config reads and hardcoded feed tuples.
   • Make feed definitions explicit objects with name, slug, only_newest, and content_type.

2. Finish refactoring pygea output behavior:

   • Write feed output under slug-based directories instead of hash-based directories.
   • Emit manifest.toml in [[feeds]] format with absolute file:// URLs.
   • Add demo/ examples and update docs.

3. Add pygea tests and packaging/check updates:

   • Cover config parsing, manifest generation, and slug-based output behavior.
   • Update pyproject.toml, flake.nix, and related files as needed so tests are part of normal validation.

4. Update republisher-redux config handling:

   • Extend feed definitions to include slug.
   • Use slug for path/log/output naming while preserving name as the user-facing label.
   • Add a config option for loading additional feed definitions from one or more external TOML files.
   • Merge direct feeds and imported feeds with duplicate detection.

5. Update republisher-redux tests and docs:

   • Cover slug-aware feed config loading and external TOML feed imports.
   • Document how to consume a pygea manifest.

6. Validate both repos:

   • Run formatting where required.
   • Run repo tests.
   • Run nix flake check in both repos.

7. Finalize without committing:

   • Review diffs.
   • Stage the intended files only.
   • Draft a commit message for user review.