Mercurial > hg > LGDataverses

annotate DVN-web/installer/dvninstall/robots.txt @ 6:1b2188262ae9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
adding the installer.
author "jurzua <jurzua@mpiwg-berlin.mpg.de>"
date Wed, 13 May 2015 11:50:21 +0200
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
1 User-agent: *
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
2 Disallow: /
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
3 User-agent: Googlebot
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
4 Disallow: /
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
5 #Crawl-delay: 120
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
6 #Disallow: /dvn/faces/javax.faces.resource
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
7 #Disallow: /dvn/OAIHandler
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
8 #Disallow: /dvn/faces/ContactUsPage.xhtml
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
9 #Disallow: /dvn/dv/*/faces/ContactUsPage.xhtml
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
10 #Disallow: /dvn/faces/study/TermsOfUsePage.xhtml
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
11 #Disallow: /dvn/faces/subsetting/SubsettingPage.xhtml
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
12 #Disallow: /dvn/dv/*/faces/subsetting/SubsettingPage.xhtml
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
13 #Disallow: /dvn/FileDownload/
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
14 #Disallow: /FileDownload/
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
15 #Disallow: /dvn/dv/*/FileDownload/
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
16 #Disallow: /dvn/resources/
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
17 #Disallow: /dvn/api/
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
18 #
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
19
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
20
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
21 # Created initially using: http://www.mcanerin.com/EN/search-engine/robots-txt.asp
1b2188262ae9 adding the installer.
"jurzua <jurzua@mpiwg-berlin.mpg.de>"
parents:
diff changeset
22 # Verified using: http://tool.motoricerca.info/robots-checker.phtml