Could grid computing restore the Internet growth curve?
New research speculates on what could be the Internet's 'killer application'
Feb 1, 2003
By: Robert B. Cohen
A commonly held view is that Internet growth has stalled and that there
is so much unused fiber in the ground (with only 3% actually lit and in
use) that new traffic can be accommodated for at least the next two to
three years, just by filling existing capacity.
Forecasts that assume the traffic mix on the Internet will remain
unchanged from what it is today suggest that the growth of the Internet
has slowed - and many observers have bemoaned the lack of a "killer
application" that would drive bandwidth consumption and further growth.
But evidence suggests that such an application is emerging. Enterprises
are beginning to adopt grid computing - and if strong trends there
continue, Internet growth could experience a significant upturn within
the next two years.
Doomsayers Is there a bandwidth glut? There is clearly overcapacity.
But much of it is on undersea routes between Asia and the US or the US
Within the US, capacity is not being filled as rapidly as in 1999 or
2000. But some of the largest ISPs are operating at 30% to 35% of peak
capacity or more - a far cry from 3% of capacity. This means seeing
peak hour capacities that are 60% or higher, pushing the envelope on
what engineers consider quality management.
Nevertheless, capital spending has fallen, and it will be some months
before it resumes. If the current economic recovery is slow and
customers cut spending further, IP traffic growth might even slow more
than expected. In Japan, when the 1998 recession was at its worst, IP
traffic growth flatlined. That could happen here if the downturn is
Why is it so difficult to know what is happening? There is no single, unbiased source of information on IP traffic.
In 1995, the National Science Foundation stopped measuring the growth
of traffic on the Internet backbone. Since then, independent ISPs can
measure only the growth of their own IP traffic and discuss their
findings with other ISPs. It's almost as if economic forecasters could
no longer gauge consumer spending and the government had asked each
large retailer to compile its own reports and make adjustments as each
saw fit. This would create chaos for anyone who wanted to measure
quarterly changes in consumer spending. But that is what happened to
the data on the Internet.
So while investment in the Internet grew, there were no objective and
unbiased measures of IP traffic expansion. Each ISP could extrapolate
based on its own experience. But few published statistics were subject
to any kind of peer review. Worst of all, some CEOs claimed that the IP
traffic on the Internet doubled every three months.
A few diligent efforts emerged to create a better picture of what was transpiring.
Andrew Odlyzko and Richard Coffman at AT&T's Labs did several
studies using what limited data was available. They were first to spot
the unusual near 10-fold upswing in traffic in 1995 and 1996 (the
source of the CEOs' mantra that the Internet was doubling every three
months). But their analysis demonstrated that this was a digression
from a more common doubling of traffic every year. Using this finding,
Odlyzko and Coffman proposed a "Moore's Law" for IP traffic, similar to
the one for semiconductor processing power, that defines IP traffic
growth as doubling annually - a growth rate that many ISPs corroborate.
Work by these analysts and RHK contributed to an influential analysis
by JP Morgan and McKinsey done after the dot-com bubble had burst. It
found that IP traffic growth would slow rapidly in the early 2000s
dropping from an estimated 177% in 1999 to just 63% in 2005.
Another study, conducted in 2001 and again in 2002 by Lawrence Roberts
of Caspian Networks, bases its findings on traffic data from 20 of the
largest ISPs. Roberts' exact methodology changed a bit from the first
report to the second, with his most recent forecast predicting an
average of 200% growth annually for the next six years.
Assumptions The JP Morgan-McKinsey study finds growth will slow
primarily because few new applications will spur IP traffic growth
(Figure 1). Corporations and consumers will use more audio and video
streaming, but there will be no earth-shattering innovations. The size
of Web page traffic in overall IP traffic declines rapidly, but the
growth of new computing applications does not change the traffic mix
An analysis by Cohen Communications Group comes to a very different
conclusion because we see a "tranformative" innovation emerging that
will change the IP traffic mix from what the Morgan-McKinsey study
forecasts (Figure 2).
Increasingly, commercial enterprises are linking grids of computers and
databases together for research and decision-making in an application
known as grid computing-and that application has the potential to
greatly expand peer-to-peer (P2P) applications in overall IP traffic,
requiring much more bandwidth.
There is now a great deal of evidence to support this viewpoint. One is
the major effort being made by several firms, including IBM, to support
the growth of grid and on-demand computing.
Another is the large number of corporations that have installed grids
or cluster computing. These include AMD, Airbus, AstraZeneca, BMW,
DaimlerChrysler, Disney, Dreamworks, Eli Lilly, Ericsson, Ford, GM,
GlaxoSmithKline, Intel, JPMorganChase, Johnson & Johnson, Merck,
National Semiconducto, and TRW.
Cohen Communications Group has interviewed a number of experts in some
of these firms who argue that grid computing will be the main new
application to create large IP traffic flows.
Indeed, firms that began by using grids to achieve improvements in
compute power have begun to use them to manage corporate computer
resources and are discussing ways to establish grids throughout their
corporation. Grids are being used in collaborative environments for
decision-making, and product designers will soon be connecting key
partners to facilitate collaboration.
Based on this analysis, Cohen Communications Group has developed an IP
traffic forecast based on the assumption that P2P and S2S traffic (the
traffic driven by grid computing use) will grow at a rate of between
150% and 200% a year from 2004 to 2008, increasing to more than 500
exabytes by 2008. (As a point of comparison, JP Morgan-McKinsey assumes
a P2P and S2S traffic growth rate of 50% to 75% for most of this
decade, growing from .6 Exabytes in 2001 to 30 Exabytes in 2008.)
Assumptions about growth rates for other types of traffic are quite similar to those predicted by JP Morgan-McKinsey.
In the short term, the overall forecasts produced by the two groups are
quite similar. According to the Cohen Group forecast, total IP traffic
growth probably fell to 84% in 2002 and will be close to 100% in 2003,
due to the global economic downturn.
Within just a few years, though, the difference in assumptions about
P2P and S2S growth rates yields a substantial difference in total IP
traffic forecasted. Compared with JP Morgan-McKinsey's baseline
forecast, where IP traffic grows from 1 Exabyte in 2001 to 47 Exabytes
in 2008, Cohen Communications Group forecasts that IP traffic will grow
to 550 Exabytes by 2008 (Figure 3).
Price drops; optical factors Two additional developments support the
growth that Cohen Communications Group forecasts for grid computing.
First, huge price declines for wholesale transit backbone capacity in
2001 and 2002 (and anticipated for 2004-2005) would be passed along to
customers whose bandwidth costs could then fall by 70% or more. This
facilitates using applications that require "big pipes," or high
Second, new optical metro rings and meshed optical switching should
make it far cheaper to transmit IP traffic. This will create an
additional wave of price declines on a per-Gigabit basis. As a
consequence, there is likely to be another major decline in costs,
contributing, in part, to the huge growth in IP traffic that Cohen
Communications Group forecasts for 2005.
Is there additional evidence to support the view that grid computing will be such a key driver of broadband demand?
Efforts to promote grid computing continue to expand. An example is Sun
Microsystems' programs to expand the use of Java through the JXTA
project and to create tools to deal with complex software development
issues for collaborations or 3-D manipulation.
Also, a large number of firms are beginning to use new JXTA software
and tools. And IBM has devoted significant resources to support its
move into on-demand computing and grids, which has become a centerpiece
of IBM's future strategy.
Finally, firms such as Avaki, Entropia, and Platform Computing, are
selling Grid software to a large number of customers. Although many of
these efforts are still in their infancy, they underscore the attention
now being paid to the commercialization of grid computing.
What would success in this area mean to computing and to the economy?
New collaborative software should enhance companies' abilities to have
machines (computers) do more work that until now only humans could do.
This improves productivity, making our economy gain more output from
specific inputs, including from the more productive computers
themselves! Higher corporate profits would help new products and
services come to market. In addition, successful innovative services
could help ISPs shift their business models from charging for transport
to gaining revenues from value-added services, such as collaboration
services, payments, and computational power. This new value-added model
could reinvigorate the telecommunications industry and business in