Skip to main content

Manage CDN resource search indexing

Search indexing is a process where search bots scan, analyze, and save information about website pages. The better a search bot understands the structure and content of a website, the higher it may rank the site in search results. Higher rankings attract more traffic.

When a CDN is used, a separate website address is created, which provides access to the same content. If a search bot sees both the main address and the CDN version of the site, it may consider them duplicates and lower the site's search engine rankings; therefore, by default, we hide the CDN version of the site from indexing.

If you want search crawlers to index content distributed via CDN, you can make the CDN version of the site available for indexing. At any time, you can once again hide the CDN version of the site from indexing.

Open the CDN version of the site for indexing

For search crawlers to correctly index content served via CDN, a robots.txt file is required. The file must specify which pages should be indexed and which should not. Without this, search crawlers will index all pages, including internal, duplicate, or test pages.

You can use robots.txt in one of the following ways:

Proxy robots.txt from the source

When proxying is enabled, CDN caches the robots.txt file from the content origin and serves it via CDN servers. This speeds up access for crawlers and reduces the load on the content origin.

Before enabling proxying, make sure that the robots.txt file is configured correctly. A file with errors may result in the site being excluded from search engine results.

To proxy robots.txt from the origin:

  1. In the Control panel, in the top menu, click Products and select CDN.

  2. In the CDN resources section, open the resource page → General tab.

  3. In the Search engine optimization block, click Proxy robots.txt.

  4. Click Apply. While the settings are being applied, the resource will have the PROCESSING status. Applying other settings is unavailable during this time. The settings will take effect when the resource changes to the ACTIVE status.

Upload robots.txt to the CDN resource

You can upload your own robots.txt file to the CDN resource even if the origin has its own robots.txt. The uploaded file will serve requests routed through the CDN. Requests that go directly to the origin will use the file on the origin.

Before uploading, make sure that the robots.txt file is configured correctly. A file with errors may result in the site being excluded from search engine results.

To upload robots.txt to a CDN resource:

  1. In the Control panel, in the top menu, click Products and select CDN.

  2. In the CDN resources section, open the resource page → General tab.

  3. In the Search engine optimization block, click Custom robots.txt.

  4. Click Select robots.txt and upload the file.

  5. Click Apply. While the settings are being applied, the resource will have the PROCESSING status. Applying other settings is unavailable during this time. The settings will take effect when the resource changes to the ACTIVE status.

Hide the CDN version of the site from indexing

By default, the CDN version of the site is hidden from indexing. You can hide the CDN version of the site from indexing if you previously made it available for indexing.

  1. In the Control panel, in the top menu, click Products and select CDN.

  2. In the CDN resources section, open the resource page → General tab.

  3. In the Search engine optimization block, click Hide from CDN (recommended value).

  4. Click Apply. While the settings are being applied, the resource will have the PROCESSING status. Applying other settings is unavailable during this time. The settings will take effect when the resource changes to the ACTIVE status.