This parses robots.txt files.