Robots.txt and

February 9th, 2006 | by Andrew Ó Baoill |

Over at Gavin’s Blog there’s an interesting observation about the robots.txt file at Now, robots.txt files are not something I’m given to browsing on an ongoing basis, but there do seem to be some strange items on the ignore list – specifically, several documents that no longer appear on the websites of individual state bodies, such as missing documents from the Department of Foreign Affairs’ press office. More interestingly, an entire day of files from the hearings of the Committee of Public Accounts. The items covered on that day are:

Financial Statements 1994-97: Rent Tribunal
Supplementary Welfare Allowances:
Value for Money Examination Department of Social, Community and Family Affairs (Resumed)

Wonder why they went missing?

Sorry, comments for this entry are closed at this time.