P2P Streaming and Long Tail: Like Oil on Water

...or Misunderstanding the Long Tail

Here is another problem I have with P2P streaming. It purportedly delivers long-tail content.

I have already discussed how P2P streaming does not work. In fact, it does work, well… kinda, sorta, but it is “the weakest link” among all streaming and all P2P technologies. I believe that means it is destined to end up on the fossil heap thanks to the propensity of the marketplace toward natural selection of better technologies. The 40% extra bandwidth usage required for P2P streaming, and the fact that most connections are asymmetric will kill any chance that P2P streaming will actually work in the marketplace. No ISP will incur 40% additional costs on your behalf even if you are a brilliant billionaire (okay that was a low jab, but appropriate).

Let us for the moment assume that Joost manages to circumvent the laws of either Physics or Economics, and manages to make P2P streaming technology work. Let us assume that Joost or Vudu, or their users are able to convince ISPs to change their offering – make connections symmetric and allow P2P on their networks. (Unlikely – even the University of Ohio has banned P2P on their campus, despite a lot of criticism – but bear with me). Or that Vudu has some technology (Voodoo?) up their sleeve that will make P2P streaming consume a lot less bandwidth. Again, we would have to rewrite our college Physics books, but really have faith, will you? :-) To be fair, Vudu may use P2P streaming only for quick starting of streams, and not for the entire stream.

So, now that we have P2P streaming, let us put some Long Tail content out there to stream. Let us say someone (lets call him Mustafa from San Diego) connects to the Turkish Television Channel on Joost and brings up a Turkish movie. Now, in order to serve this customer in the P2P streaming model, we need peers! Someone, anyone, watching the same movie at about the same time (or within the 2 hours built-in to the buffer)? In an on-demand world? Not too many! Besides Akimbo has cornered the market on Turkish Television! (Sorry, another low jab).

OK, lets suspend our belief one more time. Let us say there are a couple of other people watching the exact Long Tail content within plus or minus one hour from Mr. Mustafa. Mr. Taimur (and of course has a “lame” 400kbps connection) is on the same ISP (defying all logic, he lives right next to Mr. Mustafa) but Mr. Ataturk is on a different ISP (it doesn’t matter where he lives). If you look at the diagram below (extremely simplified for all you “non-technical” people) you will see that Mr. Mustafa and Mr. Taimur connect through the same router or DSLAM, or whatever their ISP uses at the POP.



Unfortunately as you can see, Mr. Ataturk connects to this peer network through the Internet and saves no bandwidth (nor does his ISP). Of course, his ISP is extremely unhappy since all it means is more bandwidth usage. So if you were the ISP, what would you do? I suggest that you have two choices – cap the bandwidth or charge more for it.

In other words, the P2P streaming provider has increased the bandwidth (in comparison to straight streaming delivered from a server) and has then shifted the bandwidth off their network on to that of the users’ ISPs. If you ran an ISP, would you put up with it?

Actually the news is even worse. Most Telco based ISPs use DSLAMs, and the traffic shaping is usually done on the DSLAM itself. So even though they are on the same ISP, Mr. Mustafa and Mr. Taimur, probably would not be able to do much for each other in P2P streaming.

We have not even talked about the QOS (Quality of Service) issues of Streaming Video which of course apply to P2P Streaming Video. I have argued ad nauseum that Streaming quality cannot be guaranteed on the public Internet because of all the nodes in between over which no one has control (and Joost’s technical paper admits that). And in the above example, we have assumed that all the peers are fairly close – what if the majority of these peers lived across the Atlantic? QOS would be non-existent – you can’t wish away latency at those distances.

Of course, Joost could forget about the long tail and focus more on the Fat Tail (the other end of the graph where the big numbers are). Yes, that would work, wouldn’t it? It might even work with P2P streaming assuming some of the physics were taken care of. Wait! There is another technology that serves the Fat Tail quite well – its called Cable TV!

Incidentally, L3 Technology works with both the Long Tail and the Fat Tail. And it uses P2P but in a download and play scenario. I will write next about how P2P really works in a real word scenario. Stay Tuned!

Misunderstanding P2P

Mea Culpa. I have been guilty of some lazy thinking about P2P. And it is unforgivable because my own technology L3, relies greatly on P2P.

If I seem fixated on Joost, please forgive me; they just happen to be in the news a lot these days. Plus, I have used Joost (as well as Jaman) and so my examples often use them. However, Joost will not be alone in having problems with P2P; Vudu, Zattoo, UUSee (China) and a host of others will have similar issues. Jaman, on the other hand is at least technologically sounder (in my opinion), because they use download-and-play and P2P works great with that. But Jaman has some other weaknesses. By the way, I quite enjoyed the videos I have been watching on both Joost and Jaman.

But, I am getting ahead of myself. Lets back up.

A quick pop-quiz. Of the following count those you would classify as P2P:

Kazaa, Skype, Napster, BitTorrent, Joost, Hotmail, YouTube, AIM

If you said all, give yourself a pat on the back. Now, it may sound like semantics, but yes, Hotmail – most email for that matter – is a Peer-to-Peer application. When it is not, it is usually Spam. But I am trying to make a point here. There are P2P applications and there are P2P technologies. Email and to a lesser degree a site like YouTube, are P2P applications. BitTorrent is primarily a P2P technology. Of late, BitTorrent is positioning itself as a distributor of video so it is morphing towards being both – a P2P application as well as a P2P technology.

In my earlier post (7 reasons why Joost could fail), I had mentioned that the creators of Joost were P2P people and so they used it in Joost. In fact, Skype is a P2P application and Joost uses P2P technology – in a sense they are apples and oranges. Hence, I was guilty of intellectual laziness. However, I still stand by the point of the argument in that article – that people use what they know, and that sometimes blindsides them.

Now that we can tell a P2P application from a P2P technology, let us look at two types of P2P technologies that can be used for Internet Video.

P2P Downloads – this is what BitTorrent does. You exchange files with others through your Internet connection. You can download the files any time and watch the video whenever you want (except instantly) – aka download and play.

P2P Streaming – what can I say? This is dead before you can say “Peer”. I had not really given it a lot of thought when I wrote the aforementioned article, yet I could tell there was square-peg-round-hole problem there. Which is why I simply relied on someone else’s quote that they found significant overhead in P2P streaming. I was right, but guilty of not following the thought process through to its logical conclusion.

It is easy enough to do the math. A typical home broadband connection (Cable or DSL) is asymmetric. My connection at home is 1536x256kbps. Which means I can download six times faster than I can upload. For the sake of simplicity, assume that everyone has the same connectivity. This means that if I were to use my DSL connection to watch Joost it would take six peers to service my viewing! Joost freely admits in this presentation by Colm MacCarthaigh – Joosts Network Architect that

Joost servers “top-up” the DSL“bandwidth gap”

In other words Joost will never be a purely P2P application. They will always need servers providing a majority of the bandwidth. My guess is that they may be able to save about 10% of their bandwidth by using P2P (assuming there are peers).

Now this is why the Download and the Streaming models of P2P are very different. In P2P streaming this situation requires six peers as shown above, whereas in P2P downloading you can have as few as one peer – it will simply take six times as long as to get the file. But in the latter case, when you are done downloading you can watch a flawless playback. In the former, you have no guarantee of the quality.

Also, unlike other streaming video offerings, Joost cannot make effective use of CDNs (Content Delivery Networks) to improve quality or save bandwidth. From the same presentation:

We’re willing to peer, but is there much point? Only portions of the long-tail are
peerable.

And

Joost servers also handle the “longtail” (which is still pretty long)

I may be mistaken, but it sounds like they don’t want to peer into other ISPs and so a CDN (which does exactly that) would not be of use to them.

There is another problem with P2P streaming. Since it is streaming, and streaming is real time, other peers have to be watching the same content at the same time as you are. Okay, you can stagger it a little bit, and perhaps that will help the first problem mentioned above (that is, reduce the number of peers I need). Again, since I am familiar with Joost, I know they use a 2GB buffer on my hard disk. So, some of the stream (I would guess about an hour and a half based on the quality) can be saved locally and this could be used to help out all those peers who are watching the same content at the same time as you are. Wait! Stop the presses! That sounds suspiciously like prime time! I wonder if we can get a Walter Cronkite clone!

I hope I have convinced you that P2P works best with downloading. With streaming P2P works hardly at all and if some people are to believed, it is even counter-productive. In fact I am a big fan of Peer-to-Peer distribution, because used correctly it does address the Long Tail. That is what my next article will be about. Watch this space!

P.S. Do read about the Long Tail of Bollywood here.

7 Reasons why Joost could fail…

...and one reason why it probably won't.

Joost is the new, new thing in Internet Video. Pronounced 'juiced', it is an interactive software for viewing video over the Internet using peer-to-peer TV technology, and also has some social networking features that allow chatting and networking while viewing video. Niklas Zennström and Janus Friism, who create and founded Joost, are also founders/creators of Skype and Kazaa).

If they did Skype and Kazaa, they must be smart, right? But very smart people sometimes make very big mistakes. Sometimes it is because they remain glued to their old paradigms. Below I describe some of the mistakes they have made – some in the core concepts of the technology and business model.

1. Joost relies on Streaming Video versus Download & Play. I have written a lot about this here and here and I won’t repeat it. The point that needs to be made is that with Streaming, you cannot guarantee Quality of Service (QOS). Even with excellent bandwidth on both sides, you can still get jittery, jerky and stalled video at least some of the time, because of the all nodes in between that are outside your control. With Cable TV you get excellent quality 100% of the time. This is one of the reasons IPTV has not been successfully deployed although people have been trying for more than a decade.

This is the first mistake made by the creators of Joost. I believe they simply accepted the prevailing paradigm that Internet Video means Streaming Video. Instead, downloaded content guarantees QOS because it is local and today’s storage mechanisms are robust enough to deliver even HD quality video seamlessly. Ultimately, it is not about the cool technology – it is about the user experience.

There is of course another twist to this. A lot content providers prefer streaming, because they feel that if it never leaves the server, it is safe from piracy. So Joost may have been forced to use Streaming Video by the content owners. However, it is a misconception that Streaming Video cannot be copied. There are numerous utilities out there that will download the content behind any streaming link.

2. Worse, Joost uses (or at least plans to use wherever possible), Peer-to-Peer Streaming Video. First, like server-based Streaming, guaranteeing QOS is not possible. It is even harder with Peer-to-Peer (P2P), because there is no control over the bandwidth of the peers. For example, someone streaming to you may suddenly decide to engage in some other activity that affects their bandwidth to you, and the stream delivered by them becomes choppy and jerky. They may even shutdown their device altogether and you lose whatever they were supposed to transmit to you. Can you say "massive overhead"? See also this Wikipedia article on "Peercasting".

According to an unverified source, a Dutch ISP has tested Peer-to-Peer Streaming in the field and found that approximately 40% additional overhead is to be expected in using this technology. It is not clear whether the overhead is in the bandwidth (due to retransmission) or in processing (due to reconfiguration of the sources). However, it makes sense that there is overhead when combining buffers from multiple peers. And if one of the providers becomes unavailable, there will be tremendous overhead in reconfiguring the others.

P2P works best when downloading, as BitTorrent has shown. This is because the download does not have to be sequential like a stream and there is plenty of time to download missing portions of the file later. Unfortunately, the creators of Joost know peer-to-peer best (both Kazaa and Skype are P2P applications) and people have a tendency to use the technology they know best – even if it is sometimes inappropriate.

3. It essentially duplicates Cable TV. Using a broadcast model and channels with a fixed schedule, instead of allowing time shifting and VOD. To be fair some channels on Joost are VOD, but again, in terms of relative volume of VOD content vs. broadcast content it is the same as Cable TV. Another reason why IPTV has not been successful – there is not a sufficiently compelling reason for a consumer to switch. There is the promise of additional content but that has not materialized and even where it has, there has not been so much additional content, that consumers are willing to switch to IPTV (with its disadvantages) and abandon Cable TV.

4. No user contribution possible. Content has to be approved by Joost. This is a big deal. No company in the world can conceivably license all the content in the world. Even if someone had a lot of money (several trillion dollars in readily available cash), they still need the time and resources to contact every content-owner to license their content. If they don’t have cash, they will be spending huge resources on negotiating content rights and royalty payments (possibly several trillion person-days!) – some licensing contracts run into a hundred pages or more! And of course they will be competing with other players who are willing to offer a higher royalty or show a larger installed base (the dark specter of Redmond always looms in the background).

What is needed is a model where a content owner simply adopts the technology and can control pricing, distribution, everything. Like the DVD platform – all the content owners had to do was to create the DVD. They didn’t have to negotiate with anyone because they already owned the rights. I can’t help plug L3 of course, but that is one of is strengths – no rights negotiations are required, the content owner simply convert their video into the L3 format.

5. Their software is open source. At least large portions of it apparently are – see this article. If significant portions of the software are open-source all the code must be open source according to my understanding of the GPL (the licensing protocol for open-source software).

Although this is definitely a weakness, it is not a very big deal. Once they are successful (or perhaps they are doing this even now since they have plenty of money), they can rewrite the code. And frankly, the bigger challenges, as the same article points out, are on the business end.

6. It is a lean forward experience only. A successful video deployment on the Internet will be one that provides both a lean-back and a lean-forward experience with the emphasis on the former. You cannot change the behavior of the vast majority of consumers who are used to and want the lean back experience of watching video on TV. It may work for 25 year olds, but most geeks who design this software forget that not everyone is their age.

7. Joost could be regulated like a Cable Franchise. See this article for a good overview of possible regulation (the authors do make a case against regualtion). Although many companies providing IPTV are arguing that they should not be regulated in the same manner as Cable TV, they may not win that battle. AT&T in the US, MTNL in India, China Telecom, all of them are fighting the respective regulators. AT&T has even gone to congress to pass a bill that would allow them to obtain franchises nationwide rather than from each municipality. Until such a law is passed, Joost has a problem.

…and one reason why it probably won’t. A Venture Capitalist once said that he never looks at the business plan or financials. He simply looks at the quality of the team. According to him, a good team can fix a bad business plan (perhaps even bad technology?), but a bad team can mess up even the best business plan. So, a team like Zennstorm and Fris can fix the problems Joost has. Of course, I think they can only fix it by using L3 technology. :-)

Vindication - (The HD-DVD DRM Blowup)

At last count, the score was 550,000 (A round number coincidentally - but irrelevant). What is relevant is that this number represents the number of pages with the hexadecimal code that, along with some publicly available tools allows you unlock the DRM encryption on HD DVDs. Thousands of users decided to post the number on the Internet as defiance against the MPAA. Here is the rate at which this phenomenon grew:

Google hits
May 2nd 2pm 52,000
May 2nd 6pm 323,000
May 3rd, 5am 550,000


I don’t want to publish the hexadecimal string on my blog, because I don’t want a takedown. But because I also believe that it is immoral if not illegal. But then you have 550,000 other places you can get it. Slashdot for example: http://yro.slashdot.org/yro/07/05/01/1935250.shtml. You can even buy a T-Shirt with the code on it http://www.cafepress.com/umbers.129066329

So what does this prove? Why do I feel vindicated? I’ll tell you. I have always said – any encryption scheme can be broken, if all you need is the correct key and or algorithm. We know that in order to protect content – Audio, Video or for that matter any kind of content, you have to encrypt it. We all have used and some of us know about DVD-CSS – the CSS stands for Content Scrambling System and is used on all DVDs. Listen to this:

In the video world, CSS means an annoying, relatively weak encryption scheme found on most DVD-Video discs.

The program, called qrpff, can simultaneously decode and play DVDs. Qrpff is seven lines of Perl code that, … removes CSS, … [which is] used to provide digital protection to DVDs ... The code was released as the appeals phase of the DeCSS trial is beginning to gather steam. DeCSS is another program that removes CSS and allows for copying or viewing of DVDs.

What is important about DeCSS is that it was created by three people of which two remain anonymous and the third was then a teenager. One can safely presume that these three were teenage collaborators who broke this encryption scheme in their spare time. Apparently in about three months.

This is the scheme that the multi-billion dollar movie industry chose to protect their content worldwide. There are many such examples of encryption keys being compromised either by a hacker or by leaks (usually ex-employees with a grudge/vendetta).

A very simple explanation:

1. Copy protection typically consists of algorithms and keys, which can be discovered and/or broken.

2. So change the algorithm and/or the key as often as possible.

In other words, the answer to protecting content is not a really complicated or powerful algorithm. The answer is a comprehensive copy-protection scheme whose parameters change all the time. So, if one algorithm or key is compromised, you will at most lose a few items of content. Of course our L3 solution has some other features that are important too – such as “fingerprinting”, which allows you to trace the source of a pirated copy of the content.

It is time for content owners to go to the next level to protect their content. Use L3.