WoW! CRTC moves Rogers P2P/gaming file to Enforcement (5)


UPDATE. I noted below – citing a report by Multichannel News on the latest Sandvine traffic data – that streaming video is now the primary driver of network capacity requirements, a reflection of the gradual decline of P2P traffic relative to overall consumer traffic on the Internet. The Sandvine study, the company’s Global Internet Phenomena Report for fall 2011, contains fascinating revelations about just how quickly Internet traffic patterns are changing (pdf here). Two points. First, from a business perspective, the emerging problem for ISPs isn’t congestion of the old-fashioned kind, the one which formed the basis for the CRTC’s ITMP framework. Second, from a technical perspective, traffic flows present new challenges for network engineering in the local access cloud.

Take a look at these big-picture numbers from Sandvine (pdf, p.2):

The four largest Internet services on North America’s fixed access networks, by daily downstream volume, are:

•  Netflix – 27.6%
•  HTTP – 17.8%
•  YouTube – 10.0%
•  BitTorrent – 9.0%

Although BitTorrent traffic as identified here would not account for all P2P transmissions, the lead now established by Netflix is still jaw-dropping. And jaw-dropping by comparison with Web traffic (which seems likely to be eclipsed more and more by the use of apps and other highly specialized platforms, as I discuss briefly in the previous post). Then things get curiouser…

Not only is Real-Time Entertainment increasing dramatically. In addition, the consumption of this content is getting more and more concentrated in a narrow evening viewing band – the new prime time. As Sandvine explains, this trend has implications that will shake up the way ISPs do their load-levelling, not to mention the way they set priorities:

“If this levelling-off of monthly consumption continues, then network operators might be on the cusp of a dramatic shift in how networks are engineered. In a world in which per-subscriber usage is relatively flat from month-to-month, investing to deliver increasing bandwidth no longer makes sense; rather, networks might soon be engineered to deliver a constant quality of experience. To do so effectively, communications service providers will need to look beyond bytes and into metrics that matter, like video quality of experience(my emphasis).

Bell and Rogers must be wondering whether to laugh or cry if they’re paying attention to this analysis. Imagine the leap of faith required of Rogers to get from throttling most of their traffic, to engineering their network so customers can enjoy smooth, satisfying video transmissions – from arch-competitors like Netflix!

Sandvine has other disturbing news for our incumbents and their fondness for screwing around with our transmissions, be it through technical ITMPs or the Commission’s favorite, punishing economic ITMPs, especially in the form of data caps. One of things I like about Netflix is I can adjust my video quality to good, better or best; and when I’m not fiddling with the bitrate, the Netflix servers are doing what they can to keep quality up and data consumption down using adaptive bitrate streaming. Would you rather have that kind of service – or the Rogers kind, whose caps are designed to prevent you from enjoying anybody else’s VoD altogether? (I’m not clear whether Rogers routinely uses variable bitrate streaming: anybody know?)

But this streaming technology comes at a price, as Sandvine explains:

“Rate-adaptive video represents the majority of video bandwidth, with Netflix alone representing 32.7% of peak downstream traffic – a relative increase of more than 10% since spring. This fact is of particular importance to network operators, since it means that most video traffic adapts to network congestion by shifting to lower bitrates and quality, which impacts the subscriber quality of experience. From a network engineering perspective, it means that when capacity is increased, adaptive video simply upshifts to a higher fidelity and fills the new capacity.”

Damned if you do, damned if you don’t.

a) On one hand, as video traffic grows, end-users are going to experience more and more degradation in streaming quality (even though producers, broadcasters and distributors will keep slapping the magic letters “HD” on everything). The policy issue here: allowing data caps and UBB in general, since any use of so-called economic ITMPs will greatly exacerbate problems related to video quality.

b) On the other hand, when traffic drops off, rate-adaptive streams send more data to increase quality, and keep doing so until quality peaks or the network’s available bandwidth gets used up. The policy issue here: the regulator has to stop listening to the ISP bullshit about P2P traffic filling up network bandwidth and start paying attention to the real problem of “congestive competition” from Netflix and OTT services in general.

That issue applies in spades to Rogers, whose constant refrain in response to the Gamers Organization is that P2P is to blame for everything. Ottawa, this is your wakeup call.

<update end>


BitTorrent Throttling by ISPs Worldwide: Glasnost/M-Lab data, Q2/2008-Q2/2010


Exhibit A. Rogers is the world’s worst throttler, having used DPI on 78% of its traffic in Q1-2010.

Exhibit B. Rogers has spent $8 million on network upgrades in 2011, Shaw 35 times more.

Exhibit C. Bell has announced it will phase out throttling on its newly groomed network.

Exhibit D. P2P filesharing is declining as a proportion of global consumer Internet traffic.


Does the CRTC have the guts to stand up for Rogers’ customers?

A mere month or two ago, who would have believed it? Typing “enforcement” in the same line as “CRTC.”

Well, believe it. Some kind of poop is about to hit some kind of ventilation device up in Gatineau. Here’s what we know.

Thursday afternoon, Jason K and Teresa M of the Canadian Gamers Organization (CGO) received a note from the CRTC to update them on their complaint against Rogers. It was the official response to the CGO filing of October 14, which the CGO described as their “final communication and reply on this current file.” The six-page filing provides further detail about the claim Rogers is throttling non-P2P traffic, such as World of Warcraft, and challenges claims made by Rogers about whether their implementation of technical ITMPs is in compliance with the Commission’s 2009 policy. In my previous post in this series, I never got around to spelling out how detailed and exhaustive the CGO’s previous complaint about Rogers was – the one quarterbacked by Teresa and filed back in March, then promptly ignored by the CRTC (pdf of March complaint here). On this round, the Commission has had the sense not to ask for more information – not yet anyway.

A new ballgame?

Three brief sentences was what the CGO got yesterday. Brief and intriguing…

Thank you for your reply comments dated 14 October 2011. The comment period for this record is closed at this time, but the file remains active. Please be advised that, consistent with paragraph 19 of the Canadian Radio-television and Telecommunications Commission’s Telecom Information Bulletin 2011-609 (attached), the Telecommunications Sector has forwarded the above captioned file [i.e. Complaints regarding Rogers’ Internet traffic management practices] to the Compliance and Enforcement Sector for further action.

The note is signed by Andrea Rosen, who happens to be the Commission’s Chief Compliance and Enforcement Officer.

I’ve been exchanging emails with Jason and others in the group, and no one seems confident about next steps, let alone the outcome. On one hand, the very fact a file has gone to Compliance and Enforcement suggests the officials must feel there is probable cause for doing so – i.e. Rogers’ ITMP (its attempts to throttle P2P) is not compliant. That alone would be a small victory for the public interest, even if it led nowhere special.

Rosen may take one or more of the following six actions, according to para 19 of the guidelines (the numbers are mine, added for ease of reference):

(1) request more information from either the ISP or the complainant;
(2) request a compliance meeting with the ISP to discuss the complaint in
more detail;
(3) send a letter to the ISP outlining corrective measures if it considers that the ISP’s ITMP does not comply with the ITMP policy and/or with the Act;
(4) initiate an on-site inspection or independent third-party audit to obtain additional information;
(5) initiate the issuance of a notice of consultation; and/or
(6) initiate a hearing at which the ISP would have to show cause as to why the Commission should not issue a mandatory order, which the Commission could register with the Federal Court

• the mandatory order would direct the ISP to take corrective actions under section 51 of the Act, and could include partial reimbursement to the customer.

There are two footnotes added to this passage (10, 11) that show the regulator has drawn a line between what the Enforcement staff can undertake on their own initiative and what requires “Commission authorization.” At the top of the foodchain, the Commission must approve both the issuance of a notice of consultation (item 5) and initiating a hearing (item 6). The latter is serious stuff: the hearing might result in a mandatory order, which in turn might lead to… partial customer reimbursement.

(Speaking of partial customer reimbursement, I note in passing that a Montreal law firm is attempting to get the court to certify a class action lawsuit against Research in Motion, in order to have as many customers as possible reimbursed for the three-day outage that rocked the planet last week. Imagine if the court found a parallel in Rogers’ throttling.)

Some shrewd advice for the CRTC on what to do next

The CRTC is operating with a lame-duck chair until early 2012. I have little confidence that the incumbent will see the CGO débacle as an opportunity to offer even one small nod in the direction of consumer welfare. Nevertheless, Ms Rosen should be encouraged to move the file briskly up the food chain. That means, first of all, forget about pursuing any action under items 1, 2 or 3. We’ve had all the flaccid requests for information we need; and neither a meeting with Rogers nor a letter outlining corrective measures will make the slightest incremental difference. An ex parte meeting with an incumbent is not exactly an ideal setting for serious fact-finding or stern discipline. And at this stage I don’t believe the Commission has a clue as to what needs to be done to correct Rogers’ non-compliance.

Ms Rosen owes it to herself as a self-respecting federal official, and to the long-suffering CGO complainants, to get serious about collecting the empirical evidence the Commission needs to make a fair judgment. The only way to get real evidence on the public file – that will have credibility with anyone besides Rogers’ senior management – is to proceed under item #4. And not merely an on-site inspection, although that too would be a big improvement over the past. No, what they need now is what the Commission describes as an independent third-party audit. They need a high-profile IP network engineer – somebody like MIT’s David Clark, who as an Internet pioneer/originator, would not only bring consummate technical skills to the job, but would have the added benefit of being an outsider unsullied by the inherent politics. Rogers may be harboring its own IP geniuses on the engineering team; but their senior managers have now drained them of all credibility.

There’s no reason for Ms Rosen to stop there. The guidelines have an “and/or” provision in para 19, so that the Commission can be moving on more than one track to get this file cracked. Indeed, I would argue that it’s crucial for Ms Rosen’s superiors to get involved in a way that leaves no doubt as to their seriousness of purpose. The Executive Director for Telecom and the Chair should give their blessing – as required by the guidelines – to issuance of a notice of consultation (step #5). The Commission would be doing itself a big favor by escalating to this level. It needs to live down a number of misguided actions, not the least of which is the ludicrous attempt it made to keep all the public complaints against ISPs hidden from public view, until Michael Geist wrestled them into the cold light of day last July under an Access to Information request.

The case so far

For clarity’s sake, I listed four exhibits at the top of this post which I’m pasting in again here so I can add some backup for each…

Exhibit A. Rogers is the world’s worst throttler, having used DPI on 78% of its traffic in Q1-2010.

Exhibit B. Rogers has spent $8 million on network upgrades in 2011, Shaw 35 times more.

Exhibit C. Bell has announced it will phase out throttling on its newly groomed network.

Exhibit D. P2P filesharing is declining as a proportion of global consumer Internet traffic.

Worst throttler. Last week a research team that has been measuring ISP use of DPI around the globe released a set of preliminary data. The team is using tools developed by Google for its M-Lab project, whose mandate is described as follows:

Measurement Lab (M-Lab) is an open, distributed server platform for researchers to deploy Internet measurement tools. The goal of M-Lab is to advance network research and empower the public with useful information about their broadband connections. By enhancing Internet transparency, M-Lab helps sustain a healthy, innovative Internet.

Peter Nowak wasted no time getting on the case and pulling out the big news on his blog, under the title The world’s worst throttler (officially): Rogers. The researchers derived these data running their “Glasnost” tests from the distributed server platform provided by Google. They took thousands of measurements involving dozens of ISPs. If you look back at the clipping from the data table at the top of this post, you’ll see percentages in the RH column, which indicate the proportion of tests during which Rogers was detected using DPI for throttling. I’ve only shown Rogers – and the data stop at Q1/2010. But check out those numbers, ending in the last quarter at 78%. In plain English this means that, on a fairly good-sized sample, Rogers was caught throttling over 3/4 of its transmissions. (The flowchart to the right is part of an analytical model developed by Milton Mueller, one of the lead researchers and author of several related papers.)

Does that sound like a lot? You betcha – the highest number measured in the course of this exercise. Does this matter? Oh yeah. The original ITMP framework contains language making it eminently clear that technical ITMPs are to be used sparingly and for a clearly defined purpose. Moreover, if a complaint arises alleging discrimination or preference, the ISP in question must explain why, “in the case of a technical ITMP, network investment or economic approaches alone would not reasonably address the need and effectively achieve the same purpose as the ITMP” (para 43).

Upgrades as last resort. Teresa M (Ressy if you’re a WoW dude) isn’t just a geek and gamer; she also happens to be an accountant. She recently took a close look at Rogers’ Q3 Financial and Operating Results, and made an interesting discovery (the pdf is here if you want to follow along). In the first 9 months of 2011, Rogers invested $8 million in network upgrades and rebuild (see table, p.15).

What are we to make of this number? Well, Rogers and the other incumbents were belly-aching about congestion before, during and after the proceeding on the ITMP framework. We also know from independent sources (Cisco in particular) that residential broadband traffic in Canada is increasing at an annualized rate of about 30%. And if we look at the neighboring column in the Rogers table, we see that the current upgrades figure represents a drop of 43% over the same period in 2010, when the Rogers outlay was $14 million.

Teresa has also examined the latest quarterly financials for Shaw… and the plot thickens (I’ve uploaded that pdf as well). On page 9, we read Shaw spent $277 million on upgrades in 2011, down 4% from $289 million in 2010. That makes Shaw’s current-year network spending 35 times greater than Rogers’ current-year network spending. Are there apples and oranges lurking in here? Sure. First, we don’t know what the baselines are – the prior expenditures that may show these figures in a different light (thus, Rogers’ infrastructure might be in better shape).

Teresa also points out that Shaw’s year-end is August and their financials are for 12 months, whereas Rogers report covers only nine months. In his post, Nowak indicates that in Q1/2010, when Rogers was running DPI on 78% of all samples, Shaw was recorded at 14%. On its face, that suggests Rogers is using DPI as a substitute for network upgrades.

Bell throttles not. Last week Geist broke the news that Bell has informed its wholesale ISP customers that it is going to discontinue throttling on many of the links in its system. Michael quotes the following passage from the letter:

Effective November 2011, new links implemented by Bell to augment our DSL network may not be subject to Technical Internet Traffic Management Practices (ITMP). ITMPs were introduced in March, 2008 to address congestion on the network due to the increased use of Peer-to-Peer file sharing applications during peak periods. While congestion still exists, the impact of Peer-to-Peer file sharing applications on congestion has reduced. Furthermore, as we continue to groom and build out our network, customers may be migrated to network facilities where Technical Internet Traffic Management Practices (ITMPs) will not be applied.

This may sound like a kindly gesture on Bell’s part. Don’t be fooled. It’s more of the same bullshit we’ve heard for years from the incumbents, demonizing hogs and pirates as the source of their engineering problems. The assertion that the impact of P2P on congestion has been reduced is their cute way of being disingenuous, while sticking to the old story. First, Bell fails to point out that P2P traffic itself is what’s declining (see next sub-head). Second, Bell fails to acknowledge that “congestion” is code for OTT competition – and that the most recent data show clearly that legit online entertainment (Netflix et al.) is where the congestion is coming from (e.g. in its latest report, Sandvine says streaming video is the primary driver of network capacity requirements, representing 60% of peak downstream traffic, up from 50% in 2010).

P2P decline. Let’s put one set of figures to bed here – the relative decline of P2P, i.e. relative to global consumer Internet traffic. The best publicly available, regularly updated five-year traffic forecasts I’m aware of are prepared by Cisco, as part of its Visual Networking Index. Here’s what the current edition says (published June 1, 2011):

Global Internet video traffic surpassed global peer-to-peer (P2P) traffic in 2010, and by 2012 Internet video will account for over 50 percent of consumer Internet traffic. As anticipated, as of 2010 P2P traffic is no longer the largest Internet traffic type, for the first time in 10 years. Internet video was 40 percent of consumer Internet traffic in 2010 and will reach 50 percent by year-end 2012 (p.2 of pdf).


It ain’t over till it’s over

What do we conclude from all this? It’s hard to keep the wishful thinking away from the analysis. No reasonable person could look at this evidence and say Rogers was just doing its job. But if the CRTC does get serious, Rogers has nothing to lose by appealing a judgment against it to the Federal Court, or better still, to Cabinet, where the good old boys around the table have a lot less invested in the public interest than the court. And where consumer-oriented policy is not exactly part of the political DNA.

Meanwhile, let’s not forget that the ITMP framework has an economic component; and that the Commission originally wrote that data caps and the like were a good thing (for consumers no less!) because they’re, choke choke, “transparent.” Nowak has a conspiracy theory to the effect that Bell is giving up technical ITMPs as a quid pro quo for being allowed to keep whacking us with UBB. That’s a deal with the devil I find much more worrisome than throttling. Okay, easy for me to say since I don’t do MMORPGs. On the other hand, throttling has no out-of-pocket cost; nor can it prevent another million Canadians from signing up for Netflix.

On the other hand, that’s exactly what UBB can and will continue to do, unless we get the right decision from the Commission next month – or a brave Minister who, in the Clement tradition, will strike down any UBB scheme that smells like price-gouging. UBB got the Commission’s blessing earlier this year because of the Chair’s absurd belief that, in order to keep the Internet running smoothly, Canadians must be discouraged from using the Internet. It’s hard to imagine a more retrograde policy – not just for consumers, but for business people and everyone else. Sadly, it’s not hard to imagine that the incumbents will keep fighting for their UBB windfall, which would not only keep healthy incremental revenues flowing (on markups on the order of 999%), but also give them a very important weapon in fighting off innovative and inexpensive services like Netflix.

May the good guys prevail on the technical ITMPs. But even if they do, I hope Steve and the gang at will be ready to crank up their resources for round two of the big UBB battle.



3 thoughts on “WoW! CRTC moves Rogers P2P/gaming file to Enforcement (5)

  1. Not certain about everything you’ve said here but price gouging and creating artificial devices to continue to do so, I do understand. Having a know nothing and do nothing appointee in government to oversee/enforce also means no accountability!.

Comments are closed.