docs: some reorganisation of documentation

.gitignore

@@ -163,3 +163,10 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
+
+# Testing Onion keys
+/scripts/self_signed_onion_service/
+/scripts/future_onion_service/
+/scripts/expired_onion_service/
+/scripts/wrong_name_onion_service/
+/scripts/new_onion.rest

docs/conf.py

@@ -19,6 +19,8 @@ author = 'Bypass Censorship'
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinxcontrib.mermaid',
+    'sphinxcontrib.openapi'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -35,7 +37,7 @@ exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+html_theme = 'furo'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

docs/index.rst

@@ -4,23 +4,23 @@ Documentation Home
 .. image:: /_static/home.png
    :width: 800
 
-.. toctree::
-   :maxdepth: 2
-   :caption: Publisher Guide
-
-   publisher/api.rst
-
 .. toctree::
    :maxdepth: 2
    :caption: User Guide
 
    user/index.rst
-   user/proxies.rst
+   user/mirrors.rst
    user/eotk.rst
    user/bridges.rst
    user/lists.rst
    user/automation.rst
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Publisher Guide
+
+   publisher/api.rst
+
 .. toctree::
    :maxdepth: 2
    :caption: Admin Guide

docs/publisher/api.rst

@@ -1,149 +1,6 @@
-=====================
+=============
-Bypass Censorship API
+Publisher API
-=====================
+=============
 
-Introduction
+.. openapi:: ../../schemas/publisher.yaml
-------------
+   :examples:
-
-The Bypass Censorship API allows publishers to interact with the Bypass Censorship platform and to obtain short links,
-snapshot links, and mirror links for their content.
-
-.. note::
-
-   This documentation is generic and applies to any deployment of the Bypass Censorship platform.
-   For the base URL and authentication details specific to your use, contact your vendor.
-
-The API supports the following endpoints:
-
-.. list-table::
-   :header-rows: 1
-
-   * - Endpoint
-     - Method
-     - Description
-   * - /me
-     - GET
-     - Retrieve information about the authenticated user's pool.
-   * - /link
-     - GET
-     - Generate a short link or a direct mirror link.
-   * - /<hash>
-     - GET
-     - Redirect to the live mirror associated with the given hash.
-
-Authentication
---------------
-
-The API endpoints are protected by authentication. The authentication method depends on the endpoint:
-
-- /me endpoint: The API key is provided in the request arguments using the `key` parameter. If no API key is provided, the "public" resource pool is used.
-- /link endpoint: The API key is provided in the request arguments using the `key` parameter. If no API key is provided, the "public" resource pool is used.
-
-Endpoints
----------
-
-/me
-^^^
-
-Retrieve information about the authenticated user's pool.
-
-- Method: GET
-- URL: /me
-
-Query Parameters:
-
-- `key` (optional): The API key of the pool. If not provided, the public API key is used.
-
-Response:
-
-- Success: Returns the information about the pool in JSON format.
-- Failure: Returns a 403 response if the API key is invalid.
-
-/link
-^^^^^
-
-Generate a circumvention link for a specific resource.
-This may create a new link or return an already existing link if it has been previously created.
-
-- Method: GET
-- URL: /link
-
-Query Parameters:
-
-- `url` (required): The URL for which a circumvention link needs to be generated.
-- `type` (optional): The type of link to generate. Valid values are "live", "short", or "snapshot". If not provided, "direct" is used.
-- `key` (optional): The API key of the pool. If not provided, the public API key is used.
-- `expiry` (optional): The duration that a snapshot will remain published for. Valid values are "long" and "short". If not provided, "short" is used.
-- `footer_text` (optional): Defines the footer text to be used when publishing a snapshot.
-- `footer_link` (optional): Defines the footer link to be used when publishing a snapshot.
-
-When `footer_text` is not specified, no footer link will be added to a snapshot.
-When `footer_link` is not specified, but footer text is specified, a link will be added to a live mirror of the
-snapshotted article.
-
-When `type` is specified as "snapshot" and a private API key is specified, a request will be made to the resource to
-check the status code returned. If the status code is not 200 and a snapshot had previously been published then that
-snapshot will be deleted and future attempts to access it will fail. This mechanism may be used by publishers to
-perform self-service take downs of unwanted snapshots.
-
-Response:
-
-- Success: Returns the generated link in JSON format.
-- Failure: Returns a 403 response if the resource is not on a supported domain.
-- Failure: Returns a 404 response if no live mirror exists for the resource.
-- Failure: Returns a 500 response if it was not possible to generate a snapshot for a resource.
-
-/<hash>
-^^^^^^^
-
-Redirect to the live mirror associated with the given hash.
-
-- Method: GET
-- URL: /<hash>
-
-Path Parameters:
-
-- `hash` (required): The hash associated with the short link.
-
-Query Parameters:
-
-- `force` (optional): Always redirect to a mirror and ignore GeoIP information.
-- `force_ip` (optional): Override the client IP address and base the GeoIP lookup on this IP address instead.
-
-Response:
-
-- Success: Redirects the client to the original URL.
-- Failure: Returns a 404 response if the hash is invalid or not found.
-
-Example Usage
--------------
-
-.. code-block:: bash
-
-   $ curl -X GET /me?key=<api_key>
-   $ curl -X GET /link?url=<url>&type=<type>&key=<api_key>
-   $ curl -X GET /<hash>
-
-Note: Replace `<api_key>`, `<url>`, `<type>`, and `<hash_>` with actual values.
-
-Authentication and Permissions
--------------------------------
-
-The API endpoints require valid API keys for authentication. The API key can be obtained from the pool configuration.
-
-API keys have different permissions:
-
-- Public API Key: Can be used to access live mirror links, snapshots, and resolve short links.
-- Private API Key: Can be used to access live mirror links, snapshots, and create new short links.
-
-Response Format
----------------
-
-The API endpoints return JSON responses.
-Successful responses contain the requested data, while failure responses provide relevant error messages.
-
-Error Handling
---------------
-
-In case of errors, the API endpoints return appropriate HTTP status codes and error messages.
-Clients should handle these errors gracefully and take appropriate actions.

docs/user/index.rst

@@ -11,7 +11,7 @@ to counter the measures put in place by the censor.
 Collateral Freedom
 ------------------
 
-*Used by:* :doc:`Web Proxies <proxies>`, :doc:`Tor Bridges <bridges>`
+*Used by:* :doc:`Web Mirrors <mirrors>`, :doc:`Tor Bridges <bridges>`
 
 "Collateral freedom" is an anti-censorship strategy that attempts to make it economically prohibitive for censors to
 block an Internet resource.

docs/user/mirrors.rst

@@ -1,36 +1,23 @@
-Web Proxies
+Web Mirrors
 ===========
 
 Background
 ----------
 
-A web proxy can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable.
+A web mirror can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable.
 Web proxies work by forwarding requests to the original website, and providing a different URL to access that content.
 By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website's URL.
 
-Web proxies also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content.
+Web mirrors also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content.
 This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the
 deployment of new URLs, and for the :doc:`distribution lists <lists>` to allow applications and end-users to discover
 new URLs.
 
 Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge.
 
-Simple vs. Smart Proxies
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-We currently distinguish between simple and smart proxies.
-This distinction will become redundant in the future as we aim to allow for all providers to offer the smart proxy functionality.
-Smart proxies differ from simple proxies in that they perform active rewriting of the responses from the original website.
-This modifies links to resources to ensure that they are always loaded via the mirror system rather than from the original—blocked—website.
-
-Smart proxies are only supported on AWS right now.
-
 Architecture
 ------------
 
-.. image:: /_static/proxies/architecture.png
-   :width: 800
-
 Before configuring any web origins to create mirrors, you must first configure an origin group.
 The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration.
 The shared configuration will include which pools will require mirrors for the origins.
@@ -38,9 +25,8 @@ The shared configuration will include which pools will require mirrors for the o
 For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed.
 This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors.
 Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality.
-No smart proxy instance will be created for providers where it is not necessary.
 
-Web origins represent the original website that has been blocked.
+Web origins represent the source website that has been blocked.
 
 Web Origins
 -----------
@@ -52,7 +38,7 @@ CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure
 efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the
 quality of service for all the websites using the CDN.
 
-These rate limits will be sized according to the expected rate of requests from an average user, however the proxy
+These rate limits will be sized according to the expected rate of requests from an average user, however the mirror
 system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN.
 When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a
 denial of service (DoS) attack and prevent access to the website.
@@ -92,85 +78,10 @@ Auto-Rotate
 Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
 redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
 
-Smart Proxy
-"""""""""""
-
-.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
-
-Select this field if the mirror requires a smart proxy to function properly. This is an optional field and is disabled
-by default. Enabling this will limit the platforms that the mirror can be deployed to as not all platforms support
-smart proxies at this time.
-
 Asset Domain
 """"""""""""
 
-.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
-
 Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field
 and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart
 proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror
 hosted by a smart proxy.
-
-Static Origins
---------------
-
-Static origins can be used to host static content at a cloud provider, and then have mirrors generated for that content
-rather than proxying to an external web server.
-
-Once created a static origin will provide you with details on where and how to upload the web content.
-
-Static origins can also be extended with additional resources:
-
-* `Keanu Convene <https://letsconvene.im/>`_ - A matrix-based chat platform that supports sharing of any kind. Rooms are
-  private and encrypted by default. No software is required to join a room other than a web browser, and participants
-  can be invited via a link.
-* `Clean Insights <https://cleaninsights.org/>`_ - A private measurement platform. It is focused on assisting in
-  answering key questions about app usage patterns, and not on enabling invasive surveillance of all user habits.
-
-These resources can be added/removed after the creation of the static origin without having to change the static storage
-location or access credentials.
-
-New Static Origin
-^^^^^^^^^^^^^^^^^
-
-To create a new static origin, click "Create new static origin" at the top of the list. This will present
-you with the new static origin form:
-
-.. image:: /_static/static/new.png
-   :width: 800
-
-Description
-"""""""""""
-
-Enter a brief description of the static website that you are creating in this field. This is also a required field.
-
-Group
-"""""
-
-Select the group that you want the origin to belong to from the drop-down menu in this field. This is a required field.
-
-Auto-Rotate
-"""""""""""
-
-Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
-redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
-
-Matrix Homeserver
-"""""""""""""""""
-
-Select the Matrix homeserver from the drop-down box to enable Keanu Convene on mirrors of this static origin.
-
-Keanu Convene Path
-""""""""""""""""""
-
-Enter the subdirectory to present the Keanu Convene application at on the mirror. This defaults to "talk".
-
-Enable Clean Insights
-"""""""""""""""""""""
-
-When enabled, a Clean Insights Measurement Proxy endpoint is deployed on the mirror to allow for submission of results
-from any of the supported Clean Insights SDKs.
-
-Web Proxies
------------
-

requirements-docs.txt

@@ -0,0 +1,4 @@
+furo
+sphinx
+sphinxcontrib-mermaid
+sphinxcontrib-openapi

schemas/publisher.yaml

@@ -0,0 +1,132 @@
+openapi: 3.0.3
+info:
+  title: Bypass Censorship API
+  description: |
+    The Bypass Censorship API allows publishers to interact with the Bypass Censorship platform and to obtain short links,
+    snapshot links, and mirror links for their content.
+    
+    **Note**: This documentation is generic and applies to any deployment of the Bypass Censorship platform.
+    For the base URL and authentication details specific to your use, contact your vendor.
+  version: 1.0.0
+servers:
+  - url: https://example.com
+    description: Example server
+paths:
+  /me:
+    get:
+      summary: Retrieve information about the authenticated user's pool
+      parameters:
+        - name: key
+          in: query
+          description: The API key of the pool. If not provided, the public API key is used.
+          required: false
+          schema:
+            type: string
+      responses:
+        '200':
+          description: Success
+          content:
+            application/json:
+              schema:
+                type: object
+                description: Information about the user's pool
+        '403':
+          description: Invalid API key
+  /link:
+    get:
+      summary: Generate a circumvention link for a specific resource
+      parameters:
+        - name: url
+          in: query
+          description: The URL for which a circumvention link needs to be generated.
+          required: true
+          schema:
+            type: string
+        - name: type
+          in: query
+          description: |
+            The type of link to generate. Valid values are "live", "short", or "snapshot".
+            If not provided, "direct" is used.
+          required: false
+          schema:
+            type: string
+            enum: [live, short, snapshot, direct]
+        - name: key
+          in: query
+          description: The API key of the pool. If not provided, the public API key is used.
+          required: false
+          schema:
+            type: string
+        - name: expiry
+          in: query
+          description: |
+            The duration that a snapshot will remain published for. Valid values are "long" and "short".
+            If not provided, "short" is used.
+          required: false
+          schema:
+            type: string
+            enum: [long, short]
+        - name: footer_text
+          in: query
+          description: Defines the footer text to be used when publishing a snapshot.
+          required: false
+          schema:
+            type: string
+        - name: footer_link
+          in: query
+          description: Defines the footer link to be used when publishing a snapshot.
+          required: false
+          schema:
+            type: string
+      responses:
+        '200':
+          description: Success
+          content:
+            application/json:
+              schema:
+                type: object
+                description: Generated circumvention link
+        '403':
+          description: Resource not on a supported domain
+        '404':
+          description: No live mirror exists for the resource
+        '500':
+          description: Could not generate a snapshot for the resource
+  /{hash}:
+    get:
+      summary: Redirect to the live mirror associated with the given hash
+      parameters:
+        - name: hash
+          in: path
+          description: The hash associated with the short link
+          required: true
+          schema:
+            type: string
+        - name: force
+          in: query
+          description: Always redirect to a mirror and ignore GeoIP information.
+          required: false
+          schema:
+            type: boolean
+        - name: force_ip
+          in: query
+          description: Override the client IP address and base the GeoIP lookup on this IP address instead.
+          required: false
+          schema:
+            type: string
+      responses:
+        '302':
+          description: Redirects the client to the original URL
+          headers:
+            Location:
+              description: The URL of the live mirror
+              schema:
+                type: string
+        '404':
+          description: The hash is invalid or not found
+components:
+  securitySchemes:
+    apiKeyAuth:
+      type: apiKey
+      in: query
+      name: key