How to ensure you’re passing Google’s Mobile Friendly Test

UPDATE: April 3rd, 2015 – We’ve published an update for all of our themes that will adjust WordPress’ default robots.txt file to remove any rules blocking the needed files from being loaded by search engine crawlers. Please update your themes as soon as possible.

As you may have heard, Google recently announced that as of April 21st, they are adding a site’s ‘mobile friendliness’ as a new ranking metric for search results. With so much of the web’s traffic and searches being done on mobile devices, this is a great thing for end users as it will make it easier and faster for mobile users to find and consume the info they are looking for. That said, all of the themes in our current line-up are 100% mobile friendly and responsive. Part of our testing process that is done prior to a theme’s launch is to use tools like Google’s Mobile Friendly Test to ensure that not only end users are able to easily access and use our themes, but that search engines can as well.

The Problem

Recently we’ve been receiving a lot of support requests from customers saying that their sites are not passing Google’s Mobile Friendly Test, and inquiring if this is an issue with the theme and how to fix this. In all cases we’ve seen thus far, the root cause of the failing tests is not in fact the theme, but a problem with the site’s Robots.txt file.

The Robots.txt file is a file on your site that gives instructions and sets rules for search engine crawlers and bots as to what pages, directories and files it is allowed to access. If a file, directory, or page is marked as not allowed, the crawlers will respect that and not load/index/crawl those resources.

The problem that we’re seeing is that by default WordPress generates a Robots.txt file that excludes some files and directories that contain javascript and CSS stylesheets that contain the rules and logic for adapting and re-arranging the site content on smaller screens and devices. Without access to those files, the crawlers/bots are effectively seeing an unformatted, un-styled and potentially ‘broken’ page. In order for your site to be loaded and viewed properly by the crawlers, these rules will need to be removed from the robots.txt file.

The Solution

For customers that have not created a custom robots.txt file for their site, and are using the WordPress default, the easiest solution to modify the file is to Update your theme use WP Robots Txt plugin. This plugin allows you to edit the default robots.txt file directly in your WordPress admin. For those who have created/added a custom robots.txt file you will need to edit the file manually.

Once you have updated your theme or have removed any rules blocking the wp-content/themes directory or wp-includes/ directory, make sure to test your site using Google’s Mobile Friendly Test to verify that the crawlers are no longer being blocked by your site’s robots.txt file and that you receive a ‘Mobile Friendly’ result on the test, which should be the case. If you do not, pay close attention to the center column on the results page titled ‘How Googlebot sees this page’ as it will give you the reasons why and if the robots.txt file is the cause, it will show you what is being blocked. This will allow you to adjust your robots.txt file accordingly.