Two small files in the website's root talk to search engine robots. sitemap.xml is the list of pages you want them to find. robots.txt tells them where they're not allowed to go. The first one helps; the second one, written wrong, can take your whole website out of Google without you noticing.
What you'll achieve
- find your website's sitemap and submit it to Google;
- check that
robots.txtdoesn't block what it shouldn't; - tell
Disallowapart fromnoindexand know when to use each.
What they are, in short
| The file | What it does | What it does NOT do |
|---|---|---|
sitemap.xml | lists the page addresses, with the date of the last change, so Google finds them faster | doesn't guarantee that the pages get into Google |
robots.txt | tells robots which folders or pages not to read | doesn't hide pages: a blocked page can still appear in results, without a description |
the noindex tag | says the page should not appear in results | doesn't work if robots.txt blocks the page: Google no longer reads it, so it doesn't see the tag |
The steps
A. Find the sitemap
- WordPress: open
https://domeniul-tau.ro/wp-sitemap.xml. WordPress creates it on its own. If you have an SEO plugin, it may replace it with its own, usually atsitemap_index.xmlorsitemap.xml; the exact address is in the plugin's settings. - Hand-built website: usually
https://domeniul-tau.ro/sitemap.xml, if whoever built the website added one. If it doesn't exist, they can generate it, or you can write it yourself, following the example below. - You should see a list of addresses. Check that they are the real addresses, on
https://and in the version you chose, with or withoutwww(see Redirect to HTTPS and www).
A minimal sitemap for a small website:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://domeniul-tau.ro/</loc></url>
<url><loc>https://domeniul-tau.ro/servicii</loc></url>
<url><loc>https://domeniul-tau.ro/contact</loc></url>
</urlset>Upload it to public_html from Enhance, the “Files” tab (see Uploading a website with File Manager).
B. Check robots.txt
- Open
https://domeniul-tau.ro/robots.txt. A good
robots.txtfor an ordinary website:User-agent: * Disallow: Sitemap: https://domeniul-tau.ro/sitemap.xmlAn empty
Disallow:means “nothing is off limits”. In WordPress, the file generated by WordPress only blocks/wp-admin/, which is correct.- The red flag:
Disallow: /, with a slash. It means “don't read anything on the website”. It's set on websites under construction and gets forgotten there at launch. - In WordPress, also check Settings » Reading: the Discourage search engines from indexing this site box must be unchecked on a live website.
C. Hide a page from results, the right way
For a page that shouldn't appear in Google (a thank-you page, a test page), use the
noindextag in the page, notrobots.txt:<meta name="robots" content="noindex">In WordPress, SEO plugins have an option for this on every page.
- For private areas (a dashboard, documents for customers), neither
robots.txtnornoindexis protection: set a password.
D. Submit the sitemap to Google
- In Search Console, under Sitemaps, add the sitemap's address (see Google Search Console: connect your website to Google).
If something doesn't work
wp-sitemap.xmlreturns 404. An SEO plugin has replaced it with its own (look for the address in the plugin's settings), or the permalinks aren't saved: in WordPress, go to Settings » Permalinks and click Save Changes without changing anything.- Search Console says “Blocked by robots.txt”. The page is covered by a
Disallowrule. Remove the rule if the page should appear in Google. - Search Console says “Excluded by ‘noindex’ tag”. The page has
noindex. If you didn't add it, check the indexing setting in WordPress and the SEO plugin. - The sitemap contains addresses with
http://or a differentwwwversion. Correct the website address in WordPress, under Settings » General, and the SEO plugin's settings.