Surveys are imperfect. They can go wrong in lots of ways: self-selection, too few callbacks, use of quota samples, seasonality effects, etc. And even if they are methodologically sound, a survey may ask questions that are mumbo-jumbo to respondents. One way to do quality assurance on questions is to pre-test them – to see if members of the target population understand what they’re being asked.
In my last December post, I cited an email from Ben Veenhof, the CIUS analyst at Statistics Canada. Ben and I had been discussing competing uses of the terms “broadband” and “high-speed”. He noted Stats Can doesn’t use “broadband” because it “was less familiar to survey respondents (it did not perform reliably in survey testing) and so the term ‘high-speed’ was used on the CIUS questionnaire.”
Garbage in, garbage out
Unfortunately, once surveys are out the door, the questions themselves don’t attract much attention. Even good journalists rarely take an interest in looking skeptically at what was asked of respondents. Instead, they dutifully add the margin of error and confidence level to the end of their stories, figures that add nothing to the average reader’s understanding.
Bad questions aren’t just time-wasters. They can produce results that are highly misleading, adding to the sum of public disinformation circulating about issues that matter. Worse still, the findings from bad research often play to interested parties in the service of special pleading. Let’s look at what happens when a pollster claims a majority of Canadians don’t want their government to “regulate” the Internet – except when they do.
In November and December, reports on a pair of Internet-related surveys hit the press – one in the U.S. fielded by Rasmussen, the other in Canada fielded by Nanos for The Globe and Mail. I’m going to focus on Nanos here and save Rasmussen for a subsequent post. Here’s the general setup question from the Nanos survey:
Some people think that the Canadian government should try to regulate the Internet while others believe that the Canadian government should not try to regulate the Internet. Which of these two views best reflects your opinion?
First of all, the right-wing agenda that has prevailed in Ottawa over the last half-decade has made “regulation” a dirty word. Here’s a typically melodramatic turn of phrase from the consultation paper issued last year in connection with the Tories’ digital economy initiative:
“… the 2010 Speech from the Throne, committed to support SMEs by continuing to identify and remove unnecessary, job-killing regulation and barriers to growth” (p.13, my emphasis).
Let me jump ahead to the American perspective to share the Republican take on the FCC’s recent Net neutrality order:
“The American people […] aren’t asking for yet another government takeover that imposes more job-killing federal regulations and puts bureaucrats in charge of the Internet,” [incoming House Speaker John] Boehner said” (The Hill, Dec 21, my emphasis).
Maybe they all go to the same barber. Bureaucrats?! In charge of the Internet?! Not on my watch. Americans are now going to have politicians in charge of the Internet. (Btw, the right-wing mantra that regulation kills investment is bullshit – coming soon to a blog post near you.)
Second, Internet regulation isn’t one monolithic concept, and the rules involved are painfully complex. My 4th-year students have just begun to learn what an “ITMP” is, and as of last class most weren’t aware that while wholesale rates for broadband are regulated, retail rates are not. Quick: define “Internet regulation” in 25 words or less!
About 1 in 3 of the Nanos respondents (35.8%; N=1017) said Canadian governments should regulate the Internet (I’ll leave aside the issue that the CRTC is not mentioned here because nobody knows what it does). Over half (53.6%) said Canadian governments should not regulate the Internet; and 10.6% weren’t sure.
A dozen other questions follow, a litany of bad things that can be fostered by the Internet, including fraud, stalking, terrorism, breaking and entering, and identity theft. The survey also found that 1 in 20 respondents had personally been the victim of a crime committed using the Internet in 2010.
The interesting part, however, concerns what’s missing. There’s not a single mention in this survey of the problem that afflicts not 5% but 100% of Canadian Internet users: unregulated retail broadband rates, among the highest in the world, the use of data caps, and speeds that are a national embarrassment.
Personalizing regulation: a thought experiment
Nanos relates two other regulation-type questions to i) protecting privacy and personal information, and ii) promoting Canadian content. Let’s take a liberty and imagine this survey also included some attempt to connect lack of regulation with high prices, poor service and anti-consumer behavior.
Here’s a couple of other issues we could slip in, using the same setup as the survey:
For each of the following tell me whether you support, somewhat support, somewhat oppose or oppose the government taking a regulatory role in terms of the Internet.
Making Internet access affordable.
Ensuring ISPs don’t interfere with legal content or misrepresent what they’re selling.
If one or both of these questions had been added to the survey, I’ll bet the pro-regulation responses would have gone up – easily taking the majority back from the anti-reg camp. People care about terrorism and privacy; they also care about whether they’re getting gouged, exploited or duped by service providers. So how did those concerns not make the cut?
What if we called government intervention consumer protection instead of regulation? My surmise is that, apart from not understanding what regulation really is, many consumers react negatively to the idea of a service being regulated, since it suggests restrictions on choice. OTOH, “consumer protection” has a double halo effect: it’s about me (the consumer); and it’s obviously good for me, since I’m going to be, well, protected. Consumer satisfaction ratings indicate clearly that plenty of consumers would indeed like to be protected from certain firms and certain industries as a whole. In the U.S, among the industries consumers hate most, the cable industry is still winning the race to the bottom, with the telcos not far behind (see current ACSI data here).
So what’s the upshot of the Nanos finding?
Our ISPs can put a chart on the boardroom wall showing a majority of Canadians do not want their Internet regulated, dammit! More proof market forces gave us a world-class Internet!
(I note in passing these Nanos surveys are done in partnership with The Globe and Mail, 15% of which is owned by Bell Canada parent BCE, and that if BCE’s purchase of what operated as CTVglobemedia Inc. is finally approved, BCE will still own 15% of The Globe and Mail. Since the CRTC has taken an interest in the evils of vertical integration, it might want to consider what happens to research activities in the press when the press is owned by the phone company.)
Cancon: the regulatory creature that will not die
One other question in the Nanos survey deserves to be hauled into the light of day and given a good shake, the one that goes as follows:
For […] the following tell me whether you support, somewhat support, somewhat oppose or oppose the government taking a regulatory role in terms of the Internet.
Promoting Canadian content like it does for TV and radio
Yessiree, there’s no better way to make Canadians like regulation than to conjure with the motherhood-and-apple-pie bundle of joy that is Canadian content. This time the regulation side takes the majority, by 55.4% (combining support and somewhat support). And the unsure segment drops by half compared to regulating in general, from 10.6% to 5.5%.
As with the other question we just looked at, unwarranted assumptions make it pretty much impossible for respondents to provide a meaningful answer. The problems are of two kinds.
First, the Canadian government has for decades used the twin policies of protectionism and subsidies to promote Canadian content. Most of these are entirely unsuited for “regulating” the Internet: e.g. keeping competition out of the broadcasting system; using quotas in both radio and TV; and requiring BDUs to distribute channels under a rigid set of priorities. Indeed, some of the keystone measures – like keeping out American competitors – will no longer work even in conventional broadcasting.
Second, this question seems to offer something for nothing, i.e., the respondent is not presented with a cost or trade-off for their supportive attitude. Promoting Canadian content means taxpayers are putting $1 billion into the CBC; providing $300 million to the Canada Media Fund through both taxes and BDU fees; being prevented from picking TV channels à la carte; being prevented from seeing world-famous content like the Superbowl ads; enriching half a dozen media conglomerates while the cost of watching TV rose at three times the rate of increase in the CPI through much of the last decade; the list goes on.
How would you like more public services and lower taxes?
In the spirit of our first thought experiment, what if as part of this question respondents were given enough context to allow them to find a real-world trade-off point? Part 2 might probe for what happens to your support when told that promoting Canadian content takes a big chunk out of, what, $1.5 billion each year from your taxes. It might then ask if you would like to have Rogers, Bell et al. offer online “Canadian” content on a priority basis, just like on TV – taking the choice out of your hands.
And the winner is…
The ISPs aren’t the only parties to get something from this survey. The Nanos survey also hands a powerful piece of disinformation to the cultural lobbies. ACTRA, the Writers Guild, Directors Guild and others have been clear that if the CRTC doesn’t regulate the Internet, Canadian culture will be crushed by Apple, Google and their fellow travellers in the U.S. cyber-invasion. This survey helps their case in a big way by suggesting that 55% of Canadians want the government to regulate the Internet in ways that will promote Canadian content. Unfortunately for the underlying logic of this argument, most of those bleeding hearts have been busy consuming un-Canadian content while telling their pollster what they think she wants to hear.
I have one other candidate for inclusion in my imaginary revision of the Nanos survey:
Thinking now about how Canadian ISPs operate free of retail regulation, would you consider yourself concerned, somewhat concerned, somewhat unconcerned or unconcerned about the following:
My ISP is allowed to cut my connection speed permanently in half without informing me and without any reduction in my monthly fee.