Skip to main content

Manage CDN resource search indexing

Last update:

Search indexing is a process where search bots scan, analyze, and save information about website pages. The better a search bot understands the structure and content of a website, the higher it may rank the site in search results. Higher rankings attract more traffic.

When a CDN is used, a separate website address is created, which provides access to the same content. If a search bot sees both the main address and the CDN version of the site, it may consider them duplicates and lower the site's search engine rankings; therefore, by default, we hide the CDN version of the site from indexing.

If you want search bots to index content served via CDN, you can open the CDN version of the site for indexing. You can always hide the CDN version of the site from indexing.

Open the CDN version of the site for indexing

For search bots to correctly index content served via CDN, a robots.txt file is required. You must specify which pages to index and which to exclude in the file. Without this, search bots will index all pages, including internal, duplicate, or test pages.

You can use robots.txt in one of the following ways:

Proxy robots.txt from the source

When proxying is enabled, the CDN caches the robots.txt file from the content source and serves it via CDN servers. This speeds up access for bots and reduces the load on the content source.

Before enabling proxying, verify that the robots.txt file is correctly configured. A file with errors can cause the site to be excluded from search engine results.

To proxy robots.txt from the source:

  1. In the control panel, on the top menu, click Products and select CDN.

  2. In the CDN Resources section, open the resource page → Main tab.

  3. In the Search Engine Optimization block, click Proxy robots.txt.

  4. Click Apply. While the settings are being applied, the resource will be in the PROCESSING status. During this time, other settings cannot be applied. The settings will be applied when the resource switches to the ACTIVE status.

Upload robots.txt to the CDN resource

You can upload your own robots.txt file to the CDN resource, even if the source already has its own robots.txt. The uploaded file will handle requests that pass through the CDN. Requests that go directly to the source will use the file on the source.

Before uploading, verify that the robots.txt file is correctly configured. A file with errors can cause the site to be excluded from search engine results.

To upload robots.txt to the CDN resource:

  1. In the control panel, on the top menu, click Products and select CDN.

  2. In the CDN Resources section, open the resource page → Main tab.

  3. In the Search Engine Optimization block, click Custom robots.txt.

  4. Click Select robots.txt and upload the file.

  5. Click Apply. While the settings are being applied, the resource will be in the PROCESSING status. During this time, other settings cannot be applied. The settings will be applied when the resource switches to the ACTIVE status.

Hide the CDN version of the site from indexing

By default, the CDN version of the site is hidden from indexing. You can hide the CDN version of the site from indexing if you previously opened it for indexing.

  1. In the control panel, on the top menu, click Products and select CDN.

  2. In the CDN Resources section, open the resource page → Main tab.

  3. In the Search Engine Optimization block, click Hide from CDN (recommended value).

  4. Click Apply. While the settings are being applied, the resource will be in the PROCESSING status. During this time, other settings cannot be applied. The settings take effect when the resource switches to the ACTIVE status.