170 lines
7.5 KiB
ReStructuredText
170 lines
7.5 KiB
ReStructuredText
Web Proxies
|
|
===========
|
|
|
|
Background
|
|
----------
|
|
|
|
A web proxy can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable.
|
|
Web proxies work by forwarding requests to the original website, and providing a different URL to access that content.
|
|
By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website's URL.
|
|
|
|
Web proxies also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content.
|
|
This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the
|
|
deployment of new URLs, and for the :doc:`distribution lists <lists>` to allow applications and end-users to discover
|
|
new URLs.
|
|
|
|
Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge.
|
|
|
|
Simple vs. Smart Proxies
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
We currently distinguish between simple and smart proxies.
|
|
This distinction will become redundant in the future as we aim to allow for all providers to offer the smart proxy functionality.
|
|
Smart proxies differ from simple proxies in that they perform active rewriting of the responses from the original website.
|
|
This modifies links to resources to ensure that they are always loaded via the mirror system rather than from the original—blocked—website.
|
|
|
|
Smart proxies are only supported on AWS right now.
|
|
|
|
Architecture
|
|
------------
|
|
|
|
.. image:: /_static/proxies/architecture.png
|
|
:width: 800
|
|
|
|
Before configuring any web origins to create mirrors, you must first configure an origin group.
|
|
The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration.
|
|
The shared configuration will include which pools will require mirrors for the origins.
|
|
|
|
For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed.
|
|
This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors.
|
|
Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality.
|
|
No smart proxy instance will be created for providers where it is not necessary.
|
|
|
|
Web origins represent the original website that has been blocked.
|
|
|
|
Web Origins
|
|
-----------
|
|
|
|
Rate Limiting
|
|
^^^^^^^^^^^^^
|
|
|
|
CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure that the network resources are
|
|
efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the
|
|
quality of service for all the websites using the CDN.
|
|
|
|
These rate limits will be sized according to the expected rate of requests from an average user, however the proxy
|
|
system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN.
|
|
When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a
|
|
denial of service (DoS) attack and prevent access to the website.
|
|
|
|
Deploying mirrors for websites hosted on CDNs will require co-operation from the CDN provider. If you find that mirrors
|
|
are producing many "Rate Limited Exceeded" or "Access Denied" errors then you may be suffering from this problem. Ask
|
|
your administrator to configure the `Bypass-Rate-Limit-Token` header in the portal and at the CDN to disable the rate
|
|
limiting for requests originating via the mirrors.
|
|
|
|
New Web Origin
|
|
^^^^^^^^^^^^^^
|
|
|
|
To create a new web origin, click "Create new origin" at the top of the list. This will present
|
|
you with the new origin form:
|
|
|
|
.. image:: /_static/origin/new.png
|
|
:width: 800
|
|
|
|
Domain Name
|
|
"""""""""""
|
|
|
|
Enter the domain name that you want to create a mirror for in this field. This is a required field.
|
|
|
|
Description
|
|
"""""""""""
|
|
|
|
Enter a brief description of the website that you are creating a mirror for in this field. This is also a required field.
|
|
|
|
Group
|
|
"""""
|
|
|
|
Select the group that you want the mirror to belong to from the drop-down menu in this field. This is a required field.
|
|
|
|
Auto-Rotate
|
|
"""""""""""
|
|
|
|
Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
|
|
redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
|
|
|
|
Smart Proxy
|
|
"""""""""""
|
|
|
|
.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
|
|
|
|
Select this field if the mirror requires a smart proxy to function properly. This is an optional field and is disabled
|
|
by default. Enabling this will limit the platforms that the mirror can be deployed to as not all platforms support
|
|
smart proxies at this time.
|
|
|
|
Asset Domain
|
|
""""""""""""
|
|
|
|
.. note:: This is a transitional option that will be removed once all platforms support smart proxies.
|
|
|
|
Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field
|
|
and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart
|
|
proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror
|
|
hosted by a smart proxy.
|
|
|
|
Static Origins
|
|
--------------
|
|
|
|
Static origins can be used to host static content at a cloud provider, and then have mirrors generated for that content
|
|
rather than proxying to an external web server.
|
|
|
|
Once created a static origin will provide you with details on where and how to upload the web content.
|
|
|
|
Static origins can also be extended with additional resources:
|
|
|
|
* `Keanu Convene <https://letsconvene.im/>`_ - A matrix-based chat platform that supports sharing of any kind. Rooms are
|
|
private and encrypted by default. No software is required to join a room other than a web browser, and participants
|
|
can be invited via a link.
|
|
* `Clean Insights <https://cleaninsights.org/>`_ - A private measurement platform. It is focused on assisting in
|
|
answering key questions about app usage patterns, and not on enabling invasive surveillance of all user habits.
|
|
|
|
These resources can be added/removed after the creation of the static origin without having to change the static storage
|
|
location or access credentials.
|
|
|
|
New Static Origin
|
|
^^^^^^^^^^^^^^^^^
|
|
|
|
Description
|
|
"""""""""""
|
|
|
|
Enter a brief description of the static website that you are creating in this field. This is also a required field.
|
|
|
|
Group
|
|
"""""
|
|
|
|
Select the group that you want the origin to belong to from the drop-down menu in this field. This is a required field.
|
|
|
|
Auto-Rotate
|
|
"""""""""""
|
|
|
|
Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
|
|
redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
|
|
|
|
Matrix Homeserver
|
|
"""""""""""""""""
|
|
|
|
Select the Matrix homeserver from the drop-down box to enable Keanu Convene on mirrors of this static origin.
|
|
|
|
Keanu Convene Path
|
|
""""""""""""""""""
|
|
|
|
Enter the subdirectory to present the Keanu Convene application at on the mirror. This defaults to "talk".
|
|
|
|
Enable Clean Insights
|
|
"""""""""""""""""""""
|
|
|
|
When enabled, a Clean Insights Measurement Proxy endpoint is deployed on the mirror to allow for submission of results
|
|
from any of the supported Clean Insights SDKs.
|
|
|
|
Web Proxies
|
|
-----------
|
|
|