Customize robots.txt in Shopify

Shopify generates a default robots.txt file that works for most stores. However, if you want to make changes to the default file, then you can add the robots.txt.liquid template in order to make the following customizations:

Add a new rule to an existing group
Remove a rule from an existing group
Add custom rules

The robots.txt.liquid file supports only the following Liquid objects:

robots
group
rule
user_agent
sitemap


Add a new rule to an existing group

Block all crawlers from accessing pages with the URL parameter ?q=:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /*?q=*' }}
  {%- endif -%}
  {%- if group.sitemap != blank -%}
      {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Remove a default rule from an existing group
Remove the rule blocking crawlers from accessing the /policies/ page:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {%- unless rule.directive == 'Disallow' and rule.value == '/policies/' -%}
      {{ rule }}
    {%- endunless -%}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
      {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Add custom rules

Block certain crawlers

<!-- Liquid for default rules -->

User-agent: discobot
Disallow: /

Allow certain crawlers

<!-- Liquid for default rules -->

User-agent: discobot
Allow: /

Add extra sitemap URLs

<!-- Liquid for default rules -->

Sitemap: [sitemap-url]
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...Loading...
Oopps, sorry. This discussion is closed.

If you are a paying customer and you are looking for Tech Support please post a ticket on our Support Tickets page and we will respond promptly.

If you are not a paying customer and you would like to post a pre-sale question or a quote request please use our Contact form.

Why did we close the discussion?

Many of the customers started reporting issues as comments instead of the ticketing system and this started delaying our response time.

We are making this change to serve our customers better. We have ramped up our support team and we are trying to get the reply out on the same day.

Thanks for understanding.

ahaltilse@mailxu.com fritter-brunilda@mailxu.com chinchilla@mailxu.com