Mercurial > hg > LGDataverses

view DVN-web/installer/dvninstall/robots.txt @ 6:1b2188262ae9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
adding the installer.
author "jurzua <jurzua@mpiwg-berlin.mpg.de>"
date Wed, 13 May 2015 11:50:21 +0200
parents
children
line wrap: on
line source

User-agent: *
Disallow: /
User-agent: Googlebot
Disallow: /
#Crawl-delay: 120
#Disallow: /dvn/faces/javax.faces.resource
#Disallow: /dvn/OAIHandler
#Disallow: /dvn/faces/ContactUsPage.xhtml
#Disallow: /dvn/dv/*/faces/ContactUsPage.xhtml
#Disallow: /dvn/faces/study/TermsOfUsePage.xhtml
#Disallow: /dvn/faces/subsetting/SubsettingPage.xhtml
#Disallow: /dvn/dv/*/faces/subsetting/SubsettingPage.xhtml
#Disallow: /dvn/FileDownload/
#Disallow: /FileDownload/
#Disallow: /dvn/dv/*/FileDownload/
#Disallow: /dvn/resources/
#Disallow: /dvn/api/
#


# Created initially using: http://www.mcanerin.com/EN/search-engine/robots-txt.asp
# Verified using: http://tool.motoricerca.info/robots-checker.phtml