docs: improving docs for proxies

docs/user/proxies.rst

@@ -1,29 +1,124 @@
 Web Proxies
 ===========
 
-Web proxies provide alternate URLs to access censored resources.
-These can be accessed through a normal web browser.
-They use frequently changing URLs to evade censorship.
-Some functionality on more complex websites may not work fully.
-The URLs will have a limited lifetime as, once discovered, they can be blocked by censors.
+Background
+----------
 
+A web proxy can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable.
+Web proxies work by forwarding requests to the original website, and providing a different URL to access that content.
+By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website's URL.
+
+Web proxies also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content.
 This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the
 deployment of new URLs, and for the :doc:`distribution lists <lists>` to allow applications and end-users to discover
 new URLs.
 
+Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge.
+
+Simple vs. Smart Proxies
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+We currently distinguish between simple and smart proxies.
+This distinction will become redundant in the future as we aim to allow for all providers to offer the smart proxy functionality.
+Smart proxies differ from simple proxies in that they perform active rewriting of the responses from the original website.
+This modifies links to resources to ensure that they are always loaded via the mirror system rather than from the original—blocked—website.
+
+Smart proxies are only supported on AWS right now.
+
+Architecture
+------------
+
+.. image:: /_static/proxies/architecture.png
+   :width: 800
+
+Before configuring any web origins to create mirrors, you must first configure an origin group.
+The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration.
+The shared configuration will include which pools will require mirrors for the origins.
+
+For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed.
+This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors.
+Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality.
+No smart proxy instance will be created for providers where it is not necessary.
+
+Web origins represent the original website that has been blocked.
+
 Web Origins
 -----------
 
+Rate Limiting
+^^^^^^^^^^^^^
+
+CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure that the network resources are
+efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the
+quality of service for all the websites using the CDN.
+
+These rate limits will be sized according to the expected rate of requests from an average user, however the proxy
+system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN.
+When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a
+denial of service (DoS) attack and prevent access to the website.
+
+Deploying mirrors for websites hosted on CDNs will require co-operation from the CDN provider. If you find that mirrors
+are producing many "Rate Limited Exceeded" or "Access Denied" errors then you may be suffering from this problem. Ask
+your administrator to configure the `Bypass-Rate-Limit-Token` header in the portal and at the CDN to disable the rate
+limiting for requests originating via the mirrors.
+
+New Web Origin
+^^^^^^^^^^^^^^
+
+To create a new web origin, click "Create new origin" at the top of the list. This will present
+you with the new origin form:
+
+.. image:: /_static/origin/new.png
+   :width: 800
+
+Domain Name
+"""""""""""
+
+Enter the domain name that you want to create a mirror for in this field. This is a required field.
+
+Description
+"""""""""""
+
+Enter a brief description of the website that you are creating a mirror for in this field. This is also a required field.
+
+Group
+"""""
+
+Select the group that you want the mirror to belong to from the drop-down menu in this field. This is a required field.
+
+Auto-Rotate
+"""""""""""
+
+Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
+redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
+
+Smart Proxy
+"""""""""""
+
+.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
+
+Select this field if the mirror requires a smart proxy to function properly. This is an optional field and is disabled
+by default. Enabling this will limit the platforms that the mirror can be deployed to as not all platforms support
+smart proxies at this time.
+
+Asset Domain
+""""""""""""
+
+.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
+
+Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field
+and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart
+proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror
+hosted by a smart proxy.
+
 Static Origins
 --------------
 
-These have not yet been implemented.
+Static origins can be used to host static content at a cloud provider, and then have mirrors generated for that content
+rather than proxying to an external web server.
 
-Simple Proxies
---------------
+Once created a static origin will provide you with details on where to upload the web content.
 
-Smart Proxies
--------------
+Web Proxies
+-----------
 
-Where a simple proxy leads to a broken user experience, it may be necessary to use a smart proxy to mitigate the
-brokenness.