Discovering How Your Content Is (and Isn’t) Being Indexed
In the past, Google’s cache was a reliable way to verify how your content was being picked up, but these days, there are sometimes discrepancies between the cached and the actual indexed version of a page. For example, a text-only cache will only display content loaded within the HTML source.
In this article, we’ll go over different ways to determine if and how your content is being seen by Google.
- Google’s “info:” operator
- Google’s “site:” operator
- Chrome’s DevTools (“Elements” panel)
- Alternate: FireFox’s “View Selection Source” feature
- The Fetch as Google tool in Google Search Console
Is Your Page in the Index?
The first thing to determine is whether your page is even indexed. Fortunately, this is easy. Google’s “info:” operator will give you details about a particular page. If the page is not indexed, no information will be displayed.
Simply type “info:” followed by the page’s URL. There should be no space between the colon and the start of the URL. The HTTP/HTTPS protocol is optional.
Be sure to use the canonical version of the URL. If a page exists at multiple URLs, it’s most likely to be indexed at the URL referenced in the canonical tag.
If a URL isn’t indexed, nothing will be displayed.
There are several reasons a page may not be indexed, including:
- Google hasn’t found/crawled it yet.
- It is blocked in robots.txt.
- It has a noindex tag.
- When looking for a noindex tag on the page, be sure to use Chrome’s DevTools (“Elements” panel) to search within the code for the fully rendered DOM, rather than just the HTML source.
Is Your On-Page Content Indexed?
For example, we can verify that the main text on The Search Agency’s Search Engine Optimization page is properly indexed:
If the page being investigated doesn’t appear in these results, this indicates the content wasn’t indexed.
You’ll notice the content we searched for in the above example is bolded in Google’s search result. This indicates that Google knows this content is immediately visible to users on the page. That is, they don’t have to click a “Read More” button or scroll through a carousel to view the content.
If Google doesn’t think your content is immediately visible to users, they’ll still index this content, and the page will still appear in the results for this type of search. However, the content may not appear in the snippet for the result.
That said, Google has stated this won’t be the case when mobile-first indexing rolls out. Per Google’s John Mueller, “So with the mobile-first indexing will index the the mobile version of the page. And on the mobile version of the page it can be that you have these kind of tabs and folders and things like that, which we will still treat as normal content on the page even. Even if it is hidden on the initial view.”
Is Google Technically Able to Render Your Content?
If the steps above show that some or all of the content on your pages isn’t being indexed, the next step is to find out if Google is technically able to render the content.
Fetch as Google
The easiest way to determine if Google is capable of rendering your content is to use the “Fetch as Google” tool available in Google Search Console. Just put in the URL of the page you want to check and click the “Fetch and Render” button.
It may take a minute to process, but once the render is complete, you can click on the URL to see an image of how Google was able to see the page. Here, you can verify that the content you want indexed appears in the screenshot.
Ideally, the screenshot under “This is how Googlebot saw the page” will match the screenshot under “This is how a visitor to your website would have seen the page.”
If there are no blocked resources and the “Fetch as Google” tool still doesn’t display your content, there may be other technical issues on the page, or Google may not understand the framework you’re using. Further investigation is necessary.
If the “Fetch as Google” tool shows that Google is able to see your content, and yet your content still isn’t being indexed, read on.
How Quickly is Your Content Being Indexed?
If a page is indexed and “Fetch as Google” shows that Google can see the content on the page, but the content isn’t being indexed, it may simply take more time (several days or more) for Google to fully render and index the page.
If you publish content frequently, you can get an idea of how many pieces of your recently published content are indexed (and which ones) by using the “site:” operater to do a search of your site, then selecting “Tools” and a span of time from the first drop-down that appears. For example, you can see all content indexed in the past week:
Any delays in getting your content indexed can mean lost traffic. If your content is particularly newsworthy or time sensitive, you may completely miss your chance to appear in the results for relevant searches.
If you’ve made your content fully accessible to Google (i.e. the page isn’t noindexed and it and none of its resources are blocked via robots.txt) and you continue to see inconsistent or delayed indexation of on-page content, there are other options, such as:
- Rebuilding the site to render all crucial content server-side
You May Find These Interesting
What Will Future Data Strategies Look Like? Top 10 Takeaways from our “Managing Complex Change” Series Panel
On August 6th, the agency hosted a panel discussion to dive into what the future data ecosystem will look like, given the intense focus on first party data and how the landscape is changing due to the death of the third party cookie, global privacy regulations and...
This article originally appeared on The Stagwell Group's website. A few weeks back, experts from global brand performance agency ForwardPMX and digital-first creatives Code and Theory led a virtual discussion about key strategies and tactics for Conversion Rate...
Taking place in late July 2020, The Stagwell Group’s Transformation Summit offered a moment in time to look back and reflect on the changes the ad and marketing industry has seen since the start of the COVID-19 pandemic. For brands, the most significant of these...