SEO Toolkit writes your sitemap and robots.txt for you. You decide what goes in them under SEO → Sitemap and robots.
Sitemap
- Open SEO → Sitemap and robots.
- Your sitemap's address is shown at the top:
https://yourdomain.com/sitemap.xml. - Under Include in the sitemap, tick the kinds of content to list (for example Pages and Posts) and untick any you'd rather leave out.
- Submit the sitemap address in Google Search Console and Bing Webmaster Tools.
Pages set to noindex are always left out of the sitemap.
robots.txt
Your robots.txt is at https://yourdomain.com/robots.txt. Webfaze writes the standard rules. To add your own, put them in Extra rules, one directive per line. For example, to keep one crawler out:
User-agent: GPTBot
Disallow: /They're added after Webfaze's own rules.
IndexNow
Tick Tell Bing, Yandex and other IndexNow search engines when a page is published or changed so those search engines hear about changes straight away.
Verify your site with search engines
Under SEO → Titles and meta → Site verification, paste the codes from Google Search Console, Bing Webmaster Tools, Yandex or Pinterest.
Hide a single page from search engines
Open SEO → Pages, open the page, and tick Hide from search engines (noindex). It disappears from the sitemap too.
Press Save after each change.