From: Mischa POSLAWSKY <perl@shiar.org>
Date: Tue, 2 May 2017 21:35:02 +0000 (+0200)
Subject: robots: host directive to reduce mirror crawling
X-Git-Tag: v1.10~19
X-Git-Url: http://git.shiar.net/sheet.git/commitdiff_plain/ff54b2f125dada0899ac2b12bf4fa9f496d03762?hp=ff54b2f125dada0899ac2b12bf4fa9f496d03762

robots: host directive to reduce mirror crawling

Preferred domain to indicate main site for at least Yandex bots:
<https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml#host>
---