The Age of Superfluous Information — Part 2

Sorting and searching through information are uniquely human activities because only humans have an external store of information which needs to be accessed and acted upon. The notion of acting on information stored externally is not associated with non-human animals.

The larger the stock of information, the more expensive it is to search through it to locate the precise bit that is relevant at any particular instance. To make the task of searching more tractable, ordering the information in some fashion—called sorting—becomes paramount. Computer scientists have worked on the problem of sorting and searching for decades with phenomenally successful advancement in our understanding in this regard.

(Volume three of Donald Knuth’s magnum opus The Art of Computer Programming is devoted to Sorting and Searching. I did not get past the first volume on The Fundamental Algorithms, leave alone tackling the third volume in my graduate computer science courses.)

This is a continuation of my earlier piece on the age of superfluous information. I argue here that in post-industrial world and increasingly so in the future, sorting and searching through information will occupy the role that manufacturing did in the growth of the old economy.

The libraries of the world contain an ever-expanding stock of information, much of which is very rapidly being added to the humongous stock already existent on the world wide web. That stock is growing rapidly as the flow of information is turning into a flood as the internet spreads its tentacles into every nook and cranny of human activity. Billions of people today have access to information equivalent to hundreds of millions of books on the world wide web. Compare that to just a hundred years ago when the average human had access to half a dozen books worth of information at most. When taken to such extremes, quantitative change amounts to qualitative change. The world of information is not what it used to be. The challenges therefore are qualitatively different.

When the quantity supplied of any good is in excess of anything reasonably required or demanded, the variable of importance is quality. Basically a person’s information needs are really very simple. One can only read so much, listen to so much talk and music, watch so much video, and wish to know only so much about what is going on around in one’s neighborhood and in the world at large.

Here is the result of some simple arithmetic I did just now. I estimated the total stock of information available today. Then I divided it by the maximum rate at which information can be scanned by a human. The result: a person will take about 18 billion years to merely scan the information to exhaust the current store of information. At the current rate of increase of information (an accelerating rate, I might add), or flow, a person would require 44 additional years for every passing hour. Compare the 18 billion years to the current estimated remaining lifetime of the sun: a mere 5 billion years.

The bottom line is this: there is already so much information out there that even if no additional information were generated, each one of us could be occupied a little longer than forever to finish it. Information, as we well know, is a non-rival good. That is, my “consuming” a particular piece of information will not diminish the amount available to you. Compare this to a rival good such as food. Stock of food is enough to last the six billion humans for about 3 months. In other words, if we produce no additional food, all together humans would finish the stock in three months. Or, a single human can therefore finish this in 1.5 billion years. But it is not so in the case of information. Each of us would take the estimated 18 billion years to finish the information we already have before we ask for more.

Clearly, for an average human, about 0.00000000001 percent of the total information stock is more than enough. About 99.9999999999 percent of the available information is worthless. So how does one go about searching out the teaspoonful of useful information from the oceans of available information. That is the challenge and therein lie the opportunities. That is why firms like Google will make the big bucks. The opportunity is not so much in making information available but making the right information available.

Which brings me to the point which I started off with. Searching is only part of the story when it comes to information. The other part is sorting. If one can sort the information along some relevant dimension, then you have meaningful information. What is meaningful can only be defined in the context of the entity processing the information. From the same stock and flow of information, different entities define different subsets that are relevant and meaningful. This subset can be labeled private information as opposed to the vast store of public information. Private information is the top of the sorted list of public information. Internalizing the private information leads to what we can call a stock of knowledge associated with the individual.

It is useful at this point to remind ourselves of the distinction between information and knowledge. Information is a public good the stock of which is growing exponentially. Knowledge is a private good and its primary raw material is the private information which is a very vanishingly small subset of the available public information. Even though public information has no known bounds, there are limits to how much private information can processed by a human brain and thus there are limits to the acquisition of the private good we call knowledge.

Conflating knowledge and information is distressingly too common these days and so I would like to dwell on this distinction for a bit. Some say that today we have a knowledge economy. It is trivially true because it has always been a knowledge economy ever since humans evolved brains capable of processing information into knowledge and began using knowledge to organize and coordinate economic activities. What is novel is the unfathomably huge stock of information we have available today. What distinguishes one individual from another today is the capacity to figure out what is relevant information and to internalize it efficiently into knowledge. That capacity is one of the basic skills imparted by what we call education.

To summarize the story so far: from the vantage point of an individual, this is an age of superfluous information; only a tiny fraction is relevant and meaningful; searching through the information can be automated but efficiently sorting for relevance is a private skill; imparting that skill is a primary function of education.

Next time I will explore the role of education in an age of superfluous information.