Robots.txt is a text file that website owners create to instruct search engine robots to scan pages on their websites. The robots.txt file comes under the the robot's exclusion protocol (REP). REP is a group of web standards that operate how robots scan the web, access and index content, and serve that content to users.