What user agent does Nibbler use?
Nibbler uses "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3" for all requests
I don't want my website to be tested!
You can prevent Nibbler from testing a website and remove any existing reports for that site by implementing a Robots directive (if you don't know what a robots directive is, see the Wikipedia page on Robots).
You'll need to add the following 3 lines to your robots.txt file:
# Make nibbler ignore this site
User-agent: nibbler
Disallow: /
If you haven't already got a robots.txt, just create one in the root of your site (it should be publicly accessible via http://YOURDOMAIN.COM/robots.txt) and include the above lines.
Important notes:
- Nibbler will NOT obey a wildcard user-agent, you need to explicitly deny nibbler.
- No matter what you put on the 'Disallow' line, Nibbler will NOT test any part of a site which has a robots.txt file mentioning nibbler. This is in order to stop anyone manipulating the Nibbler scores.
- If the site in question has an existing report in Nibbler, it won't automatically get deleted. You will need to visit Nibbler, browse to the report and click the re-test button. Nibbler will then detect the presence of the robots.txt entry and remove the existing report.
- If the Nibbler report of a site has already been indexed by a search engine, it be take some time before the search engine updates its indexes to reflect the report being removed. The report may still appear in search engine results for a short time after it has been removed.