2024-12-02 00:00:28 +00:00
Web Mirrors
2022-05-26 15:59:12 +01:00
===========
2023-03-30 13:09:33 +01:00
Background
----------
2022-05-26 15:59:12 +01:00
2024-12-02 00:00:28 +00:00
A web mirror can help users by providing alternate URLs to access censored resources, allowing them to bypass technical censorship and access information that may be otherwise unavailable.
2023-03-30 13:09:33 +01:00
Web proxies work by forwarding requests to the original website, and providing a different URL to access that content.
By accessing the web proxy URL, users can access the same content that would be blocked by censors if accessed through the original website's URL.
2024-12-02 00:00:28 +00:00
Web mirrors also use frequently changing URLs to evade censorship, making it more difficult for censors to block access to the content.
2022-05-26 15:59:12 +01:00
This assumption of a limited lifetime is built-in to the system, allowing for automated block detection to trigger the
deployment of new URLs, and for the :doc: `distribution lists <lists>` to allow applications and end-users to discover
new URLs.
2023-03-30 13:09:33 +01:00
Additionally, web proxies can be accessed via a normal web browser, making them easily accessible to users without requiring any special software or technical knowledge.
Architecture
------------
Before configuring any web origins to create mirrors, you must first configure an origin group.
The group is used to organise larger numbers of web origins so that mirrors can be created for them with a shared configuration.
The shared configuration will include which pools will require mirrors for the origins.
For each configured origin group, a smart proxy instance will be created at each provider where proxies are deployed.
This instance runs the active proxy that will rewrite URLs in responses from the origin webserver to ensure that all resources are loaded via the mirrors.
Some providers, e.g. Fastly, do not require an instance as they already provide the means for URL re-writing in their standard functionality.
2024-12-02 00:00:28 +00:00
Web origins represent the source website that has been blocked.
2023-03-30 13:09:33 +01:00
2022-05-26 15:59:12 +01:00
Web Origins
-----------
2023-03-30 13:09:33 +01:00
Rate Limiting
^^^^^^^^^^^^^
CDNs (Content Delivery Networks) can impose rate limiting on websites to ensure that the network resources are
efficiently utilized, to protect the websites from DDoS (Distributed Denial of Service) attacks and to maintain the
quality of service for all the websites using the CDN.
2024-12-02 00:00:28 +00:00
These rate limits will be sized according to the expected rate of requests from an average user, however the mirror
2023-03-30 13:09:33 +01:00
system is a bottleneck that aggregates requests from multiple users and passes these on to the original CDN.
When a single system is used to send a large number of requests to a CDN like this, the CDN may interpret this as a
denial of service (DoS) attack and prevent access to the website.
Deploying mirrors for websites hosted on CDNs will require co-operation from the CDN provider. If you find that mirrors
are producing many "Rate Limited Exceeded" or "Access Denied" errors then you may be suffering from this problem. Ask
your administrator to configure the `Bypass-Rate-Limit-Token`
limiting for requests originating via the mirrors.
New Web Origin
^^^^^^^^^^^^^^
To create a new web origin, click "Create new origin" at the top of the list. This will present
you with the new origin form:
.. image :: /_static/origin/new.png
:width: 800
Domain Name
"""""""""""
Enter the domain name that you want to create a mirror for in this field. This is a required field.
Description
"""""""""""
Enter a brief description of the website that you are creating a mirror for in this field. This is also a required field.
Group
"""""
Select the group that you want the mirror to belong to from the drop-down menu in this field. This is a required field.
Auto-Rotate
"""""""""""
Select this field if you want to enable auto-rotation for the mirror. This means that the mirror will automatically
redeploy with a new domain name if it is detected to be blocked. This field is optional and is enabled by default.
Asset Domain
""""""""""""
Select this field if you want to use mirrors of this domain to host assets for other domains. This is an optional field
and is disabled by default. All of the asset domains within a group will be available for all mirrors using smart
proxies. The smart proxy will rewrite URLs for the asset domains that are included in the source code of the mirror
hosted by a smart proxy.
