git.shiar.nl
/
sheet.git
/ commitdiff
commit
grep
author
committer
pickaxe
?
search:
re
summary
|
shortlog
|
log
|
commit
| commitdiff |
tree
raw
|
patch
| inline |
side by side
(parent:
03275d1
)
robots: host directive to reduce mirror crawling
author
Mischa POSLAWSKY
<perl@shiar.org>
Tue, 2 May 2017 21:35:02 +0000
(23:35 +0200)
committer
Mischa POSLAWSKY
<perl@shiar.org>
Mon, 29 May 2017 17:23:12 +0000
(19:23 +0200)
Preferred domain to indicate main site for at least Yandex bots:
<https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml#host>
robots.txt
patch
|
blob
|
history
diff --git
a/robots.txt
b/robots.txt
index 4ba41488a278f873796b611151ab7564588f1ab8..15b6ecb586ed18dd64f57c40e774dab89ddd2bc9 100644
(file)
--- a/
robots.txt
+++ b/
robots.txt
@@
-2,3
+2,4
@@
User-agent: *
Disallow: /source/*::*
Sitemap: http://sheet.shiar.nl/sitemap.xml
+Host: sheet.shiar.nl