Greetings. The FCC has issued a call for Internet users to test their broadband connections and report the results back to the FCC for analysis.

After inspecting the associated site and testing tools, I'm must admit that I am extremely skeptical about the overall value of the data being collected by their project, except in the sense of the most gross of statistics.

In random tests against my own reasonably well-calibrated tools, the FCC tools showed consistent disparities of 50% to 85%! Why isn't this surprising?

I'm a big fan of Google's M-Lab project (and was involved in the meeting at Google that served as the genesis for the project itself) but I must question the use of the Java-based M-Lab testing tool (and the other Java tool provided by the FCC site) in this particular manner. (Ironically, the FCC site stipulates that the M-Lab tool won't run under Google's own Chrome browser!)

The FCC testing regime provides for no control related to other activity on users' connections. How many people will (knowingly or not) run the tests while someone else in the home or business is watching video, downloading files, or otherwise significantly affecting the overall bandwidth behavior?

No obvious clues are provided to users regarding the underlying server testing infrastructure. As anyone who uses speed tests is aware, the location of servers used for these tests will dramatically affect results. The ability of the server infrastructure to control for these disparities can be quite limited depending on ISPs' own network topologies.

And of course, on-demand, manually-run tests cannot provide any sort of reasonable window into the wide variations in performance that users commonly experience on different days of the week, times of day, and so on.

Users are required to provide their street address information with the tests, but there's nothing stopping anyone from entering any address that they might wish, suggesting that such data could often be untrustworthy compared with (much coarser) already available IP address-based location info.

While these tests under this methodology may serve to help categorize users into very broad classes of Internet service tiers, it's hard to see how their data could be reasonably trusted beyond that level. ISPs may be justifiably concerned that the data collected from these tests by this FCC effort may be unrepresentative in significant ways.

There are certainly methods available to collect meaningful, longitudinal data in manners that would provide genuinely useful insight into the real-world characteristics of users' broadband connections' performance.

I discussed my own proposals along these lines a couple of years ago in:

Practical Issues of the Proposed "Global Internet Measurement Analysis Array"

and:

Proposal for Breaking the Internet Network Neutrality Deadlock

I still feel very strongly that a methodology of the type that I suggested in those documents -- or something similar -- is an appropriate way to collect truly meaningful broadband statistical data, in contrast to the FCC's currently promoted approach that I believe to be of relatively limited value.

--Lauren--

Blog Update (March 19, 2010): Amid Vendors' Finger-Pointing, FCC Says: "We Don't Endorse Our Broadband Speed Tests!"