Website inside robots.txt

Netter Hack von Alec Bertram, eine Website in robots.txt:


What's going on here: When parsing robots.txt files, search engines ignore anything that's after a hashtag in a robots.txt file - this turns it into a comment. However, a crawler will read anything on the page when it doesn't think it's looking at a robots.txt file, meaning we can put an entire website inside a robots.txt file.

But how is it in a text file? We're using the .txt extension as an alias for PHP, meaning that we can code intelligently and change the content when different URL parameters are present.

And where are the hashtags? They exist, however we're mostly putting the entire website on one line, and using a Javascript Library to hide the hashtags that we do use.