Use the robots.txt file to prevent indexing of various areas of your site. September 28, 2023 by John Cartwright How to use the robots.txt file to optimize a WordPress website and block folders.