Introducing MobileGrader.com

Mobile Grader by Bluetrain Mobile enables you to see how your website performs on a mobile device. How long does it take for your visitors to see your website on a smartphone? Are you using minified (optimized) code that makes it possible for a mobile browser to show your website quickly? The Mobile Grader will quickly evaluate these and other criteria and then compare your website to our benchmark that we call the Mobile 100 (a list of well-known mobile websites).

We all know the frustrations of browsing the web from a mobile device; we encounter inadequate websites that load slowly, often have broken navigation, require zooming and scrolling on every page, and are generally hard to use. Mobile Grader assesses how your website performs across the broad spectrum of mobile devices including iPhone, Android, and Blackberry smartphones. Our tool delivers an objective, concise report, so you can quickly and easily understand your strengths and weaknesses.

You can test your website right now at www.mobilegrader.com

How does the Score Work?

If you want to understand exactly how we derived your score, read on!

Scores range from 0-100. We consider many factors in scoring and we weigh these factors based on their importance. For example, the download time of the website and the use of Flash affect the usability of your website on a mobile device to different degrees, so they are weighted differently. Mobile Grader assigns a score to each factor and compares those scores to our Mobile 100 benchmark of websites.

Mobile Grader Score Weights

(Enlarge image)

Pass/Fail vs. Percentile Scoring

In order to determine the score per category, the Mobile Grader employs one of two methods. The first method is the simplest and is used for categories that are pass/fail, this includes factors such as whether or not this a mobile website, or whether or not it uses hover states. If a website passes, then it gets an amount of points equal to the full weight of the category, if a website fails the section, then it gets 0 points. For example, if a website contains hover states it will get 0 points out of the possible 5, if the website does not have hover states then it will receive all 5 points for this category.

The second method is based on the percentile rank of the website and is used for sections such as the size of the web page or the total download time. To explain how this is done, here is an example for download time:

The Mobile Grader starts by taking the Mobile 100 benchmark, and calculates the download time for each one. It then orders the list, from fastest to slowest, and determines where the website being tested will end up on that list. If the website being tested falls in between #37 and #38 on the benchmark list of 100, we determine its percentile and then we use its percentile rank to determine how many points to assign. A result of 37 would yield a percentile of 63%. The Mobile Grader then takes that number, 63%, and multiplies it by the assigned weight of that category, which in the case of download time happens to be 20. So 0.63 * 20 = 12.6 points added to the score.

Example for NBA.com
Here is an example of a score breakdown for the NBA website, m.nba.com (as of 12/12/11):

(Enlarge image)

How does Mobile Grader know if I have a mobile website or not?

Mobile Grader visits your website twice – once as a mobile device and once as a desktop browser. It then compares the results to determine if a different version of the website is delivered each time. This is a good method, but it’s not perfect. Here’s why:

In some cases, websites with mobile versions may not differentiate between mobile devices and traditional computers. Instead, they may use a process such as JavaScript Re-Direct, which redirects a mobile device to a mobile website only after loading the traditional website first. Mobile Grader is unable to execute JavaScript, and will not detect a mobile website that uses this technique. If your website uses this process and you’d like to see your mobile score, simplly enter the direct URL for your mobile website.

How Does Mobile Grader Know What My Download Speed Is?

Mobile Grader creates an estimate of your real world download speed by considering three main inputs:

The first is the standard download rates on various networks, including 4G, 3G, 2G, and WiFi. The second, and most important factor is the total size of your website, which includes all of the files required in order to display your website. The third is the number of files that make up your website. While desktop browsers can download dozens of files at the same time, mobile browsers can only download 2-4 files at a time. This means that if a website has 10 files, while the first 2-4 download the other 8 will have to wait. Once the first group is done, the next group of files will download, and so on. Furthermore, the time it takes in between downloading these “batches” of files provides significant overhead because a connection must be closed and re-established before resuming a download. On a mobile network this act of closing and re-establishing a connection with the website takes significantly longer than on desktop websites. Due to this limitation, it will take substantially more time for a mobile browser to load a website with many individual files compared to the load time on a desktop browser. The fewer files you have in your website, the higher your score. It is also because of these three inputs that we need to use more than download speed.

Even the published download speed of a network is not as straight forward as some people may believe. In order for a cell phone network to call itself 3G it needs to meet a minimum standard known as IMT-2000, which, among other things, requires the download rate be at least 200Kb/s (kilobits per second). However, this is a minimum standard, and many cell network exceed this to varying degrees. Sometimes this variance can be as high as 300% according to a study which compares the download rates of the iPhone 4 across multiple networks! That study does not even take into account the fact that different geographical areas have different download speeds that can widen the variance even more.

Download speed on Mobile Grader based on a customized algorithm created by field testing
In order to settle on a final number, we decided it would be best to conduct some real world testing and create a formula that take the 3 inputs described above creating an algorithm that is based on the results. We created the following test websites and tested their download times on multiple types of smartphones in the greater Boston area to see how file size and file count affected download speeds:

(Enlarge image)

Using the data from the tests above, we were able to derive values for the factors to create estimates for real world download speeds. These values are listed in the table below:

(Enlarge image)

Why Shouldn’t I Have Flash?

Adobe has recently announced that it is going to drop support altogether for Flash on mobile devices. Flash support is very hard to use on a mobile device and in many cases, impossible. Mobile Grader finds Flash by looking through all of the files associated with the URL entered, and if one of them is a .flv or .swf file, then it assumes Flash is present. It is very important to realize that even if you have a relatively high Mobile Grader score, if you have Flash your website is going to be very unfriendly to a mobile audience.

What is Optimized Code?

For those of you who are tech-savvy, optimized code is a friendly term for minified code. The source code to your website (the HTML, JavaScript and CSS files) are minimized when it does not contain unnecessary characters such as line breaks (yes, a line break counts as a character), tabs, and excessive white space. When writing code, it is important to include things such as line breaks and tabs in order to help make the code “human readable,” but all of these things should be removed when it is served to the client. Removing these extra characters not only helps your website download faster, it also helps to render the website faster – as on an iPhone, for instance. There is a direct correlation between the size of a JavaScript file and the time it takes the iPhone’s JavaScript engine to process that file. Please see this article for detailed information.

What are Hover States?

Hover states are events that occur when you move you mouse over an item and another action is triggered. An example of this would be placing the mouse over a button on a website and seeing the button change color. Hover states should be avoided on mobile websites, because almost all smartphones cannot detect a hovering pointing device (your finger) unless it touches the screen! Mobile Grader looks at the code of a website for HTML attributes that specify a JavaScript onHover action for a given element in order to determine if a website has hover states.

What Will We Add in The Future?

We have a lot of new ideas for things to add to Mobile Grader in order to make it as useful for marketers as it can be, including a tool that can factor into the score whether or not the user of the website will have to pinch or zoom to read the page clearly.

Finally, it’s important to keep in mind that a mobile grader score only tells part of the picture. In many cases, you shouldn’t be aiming for a perfect score! Learn more about how to use Mobile Grader from a marketer’s perspective here.

If you have any ideas for what we should add please feel free to leave a comment or ask a question in the comment section down below. I hope you were able to learn a useful thing or two from this blog post. Go ahead and test your website at www.mobilegrader.com!