Gepostet vor 11 Monaten, 13 Tagen in
What's going on here: When parsing robots.txt files, search engines ignore anything that's after a hashtag in a robots.txt file - this turns it into a comment. However, a crawler will read anything on the page when it doesn't think it's looking at a robots.txt file, meaning we can put an entire website inside a robots.txt file.
But how is it in a text file? We're using the .txt extension as an alias for PHP, meaning that we can code intelligently and change the content when different URL parameters are present.