---
title: "Manage CDN resource search indexing"
sidebar_label: "Manage resource search indexing"
description: "How to manage CDN resource search indexing using the robots.txt file"
sidebar_position: 7
---

import Formbricks from '@theme/MDXComponents/Formbricks'

# Manage CDN resource search indexing

Search indexing is a process where search bots scan, analyze, and save information about website pages. The better a search bot understands the structure and content of a website, the higher it may rank the site in search results. Higher rankings attract more traffic.

When a CDN is used, a separate website address is created, which provides access to the same content. If a search bot sees both the main address and the CDN version of the site, it may consider them duplicates and lower the site's search engine rankings; therefore, by default, we hide the CDN version of the site from indexing.

If you want search bots to index content served via CDN, you can [open the CDN version of the site for indexing](#open-cdn-version-for-index). You can always [hide the CDN version of the site from indexing](#hide-cdn-version-from-index).

## Open the CDN version of the site for indexing \{#open-cdn-version-for-index}

For search bots to correctly index content served via CDN, a `robots.txt` file is required. You must specify which pages to index and which to exclude in the file. Without this, search bots will index all pages, including internal, duplicate, or test pages.

You can use `robots.txt` in one of the following ways:

* [proxy robots.txt from source](#proxy-robots-txt-from-source) — if the source already has a correct `robots.txt` and you want CDN to distribute the same file.
* [upload robots.txt to CDN resource](#add-robots-txt-to-cdn-resource) — if you want to set different indexing rules for the source and CDN.

### Proxy robots.txt from the source \{#proxy-robots-txt-from-source}

When proxying is enabled, CDN caches the `robots.txt` file from the content source and distributes it via CDN servers. This speeds up access for bots and reduces the load on the content source.

Before enabling proxying, verify that the `robots.txt` file is configured correctly. A file with errors may result in the site being excluded from search engine results.

To proxy `robots.txt` from the source:

1. In the [control panel](https://my.selectel.ru/cdn-v3), on the top menu, click **Products** and select **CDN**.

2. In the **CDN Resources** section, open the resource page → **General** tab.

3. In the **Search Engine Optimization** block, click **Proxy robots.txt**.

4. Click **Apply**. While settings are being applied, the resource enters the `PROCESSING` status. You cannot apply other settings during this time. Settings take effect when the resource transitions to `ACTIVE` status.

### Upload robots.txt to the CDN resource \{#add-robots-txt-to-cdn-resource}

You can upload your own `robots.txt` file to the CDN resource, even if the source has its own `robots.txt`. The uploaded file will serve requests that go through the CDN. Requests that go directly to the source will use the file on the source.

Before uploading, verify that the `robots.txt` file is configured correctly. A file with errors may result in the site being excluded from search engine results.

To upload `robots.txt` to a CDN resource:

1. In the [control panel](https://my.selectel.ru/cdn-v3), on the top menu, click **Products** and select **CDN**.

2. In the **CDN Resources** section, open the resource page → **General** tab.

3. In the **Search Engine Optimization** block, click **Custom robots.txt**.

4. Click **Select robots.txt** and upload the file.

5. Click **Apply**. While settings are being applied, the resource enters the `PROCESSING` status. You cannot apply other settings during this time. Settings take effect when the resource transitions to `ACTIVE` status.

## Hide the CDN version of the site from indexing \{#hide-cdn-version-from-index}

By default, the CDN version of the site is hidden from indexing. You can hide the CDN version of the site from indexing if you previously [opened it for indexing](#open-cdn-version-for-index).

1. In the [control panel](https://my.selectel.ru/cdn-v3), on the top menu, click **Products** and select **CDN**.

2. In the **CDN Resources** section, open the resource page → **General** tab.

3. In the **Search Engine Optimization** block, click **Hide from CDN (recommended)**.

4. Click **Apply**. While settings are being applied, the resource enters the `PROCESSING` status. You cannot apply other settings during this time. Settings take effect when the resource transitions to `ACTIVE` status.

<Formbricks />