The World is (Information) Fat

“Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it.”
— Samuel Johnson quoted in Boswell’s “Life of Johnson.”

If you come to think about it for a moment, what we really want is knowledge, not information. (Recall what the business school guru said: what people want is not a quarter-inch drill but rather a quarter-inch hole.) The good news is that there is a lot of information out there. The better news is that the cost of accessing that information has been dropping exponentially. But the bad news is that the cost of searching through the vast stock of information to satisfy your knowledge needs is increasing.

First a brief aside on the distinction between knowledge and information. People use the terms interchangeably but they must be distinguished if we wish to reason with some degree of clarity about our information-suffused modern society. A telephone book has information about names and numbers, but it does not ‘know’ telephone numbers. A human brain ‘knows’ a phone number, in contrast. Outside the human brain, it is information and organized within the structures of a human mind, it is knowledge. Dr Johnson appreciated the distinction very keenly. Information is what economists call a public good, while knowledge is a private good. This has important implications which need not detain us right now.

Back to the good news about information. Here are some fun facts about information.

Fun Fact #1: There is a heck a lot of information in the world today. The stock of information is stupendous. Naturally so because human activity primarily produces information. The increased production of information has been intensive and extensive.

Intensive because more of what we do is recorded, whether in government files, in private sector databases, or in your own record of your private life such as in blogs and photo albums, and so on. You have lots more bits of information associated with you than was associated with your ancestors.

Extensive because first there are more of us on the planet today, and second, because more of us are doing stuff that produces more information. A much larger percentage of our six billion human population is engaged in generating and processing information than before. Millions of researchers and scientists do stuff that generates truck-loads of information. Commercial, governmental, and personal information grows unbounded.

Fun Fact #2: A heck of a lot the available information is available online. Some of it is on the world wide web accessible through the internet, while much of it is in the deeper web not generally accessible to the average web surfer. As time goes on, a greater percentage of the total stock of information will be online.

Fun Fact #3: The stock of information is increasing exponentially. And consequently, the stock of online information is also increasing exponentially. Exponential increases are fairly dangerous things. This is a not-so-much fun fact we will come back to bite us. Flow (increase in the stock) of information is threatening to become a tsunami.

Fun Fact #4: Regarding the quality of information online: as the quantity is increasing, the variance is increasing and the average is decreasing.

Let’s just focus on books although we could as easily tell a similar story about movies, or books, or research papers, or photographs, or blogs, or usenet postings. A century ago, there were very few books published compared to today. Given the high cost barrier of publishing, only those works which had some enduring quality made the grade. Therefore the average quality of available printed matter was high. Today, there millions of titles are published because of both greater supply (more writers) and greater demand (more readers), and because the cost of publishing (relative to average incomes) has fallen. The quality of the average book, I believe, is lower than before.

It is my contention that the best book of today is better than the best book of yesterday, and that the worst book of today is worse than the worst book of yesterday. Just a hunch and I don’t have hard data to support this hunch.

Allow me to make a hand-waving argument about quality and quantity. Let’s take photographs. When I used an analog camera (print or slides), I used to take a lot fewer pictures than I do today with my digital camera. But I rejected a lot lower percentage of pictures in those analog days. Today, I throw away most of what I take with my digital camera but I end up with much higher quality “best pictures” than before. Taken as a whole, my digital pictures are on average “lower quality” because I click a lot of pictures with greater abandon today given the low average cost of each click.

Greater variance and lower average quality coupled with an immensely larger stock leads us to the bad news about today’s information age. But first, a few more fun facts.

Fun Fact #5: The cost of accessing information is going down.

Let’s just say “google.” Enter some keywords and you will get about four million hits, give or take a few million. Marginal cost to you: nearly zero (assuming that you have a connected computer at your disposal.)

Another way of putting this is to say that the “channel capacity” has increased. The information can flow through to you through a vast pipe if you need it.

Fun Fact #6: The cost of identifying the information is going up.

So you do get four millions hits in less than 0.4 seconds when you do that search on Google. But, unless you are very lucky, or have been very clever in specifying the search, it will take you a lot of time to sort through it all to find the information you need.

For any given stock of information, the lower the average cost of accessing the information, the higher the search cost for any specific required information. Here I would like to enunciate what I call the Information cost complementarity principle: for a given cost, quality and quantity are complementary. You can have high quantity but will have to put up with low quality; or you can have high quality but it would cost you. (Compare with the folk wisdom: “Good, fast, cheap: Choose two.”) Another way of putting it: the cost of searching and cost of sorting through the results of the search are complementary.

I model the principle after Neils Bohr, the father of quantum mechanics. He stressed the importance of the complementarity principle and held that it has wide applicability in areas far removed from physics. The principle says that knowledge of one aspect of a system precludes knowledge of certain other aspects of the system. (Ref: Steven Weinberg’s Dreams of a Final Theory, pg. 74.) Heisenberg’s uncertainty principle is an instance of the complementarity principle: absolute knowledge of a particle’s position (or momentum) precludes absolute knowledge of the particle’s momentum (or position.)

Fun Fact #7: The cost, therefore, of obtaining knowledge has gone up.

Recall that we really are not interested in information for its own sake; we are interested in knowledge. Being given four million hits in return for a search is about as helpful as being thrown both ends of a rope when one is drowning.

The claim that the cost of knowledge has gone up simultaneously with a dramatic decrease in the cost of information is clearly counter-intuitive. Counter-intuitive but consistent with the facts. Low quality information is cheap. That leads to what I call “information obesity” and is inconsistent with “knowledge health.”

For all of our evolutionary history, food was not easily available and so we have evolved to subsist on a low calorie diet. Suddenly (in evolutionary time-scales) in developed countries, calories are abundant because low quality high-calorie foods are cheap. High quality low-calorie foods are expensive. So the poor in rich nations such as the US suffer differentially more than the rich from obesity. This is in contrast to poor nations where the rich are obese. Obese people are calorie-rich but health-poor.

I contend that one can be information-rich and knowledge-poor. And further that in an information overloaded society, the poor people will be information-rich and knowledge-poor, and the rich people will be information-poor but knowledge-rich.

After all this talk, let me come to the point that I want to make: There is a big opportunity managing information overload. Create a filter which will let only the top quality information through and people will beat a path to your door. You may say that it is a Super Filter which filters out not just spam but low quality non-spam content as well. In the past, portals which gave you everything (when there was not very much of anything) were a big hit. Now (when there is much too much of everything) portals which give you access to absolutely selective exclusive stuff will make it big.

In one specific area — education — I have figured out how to manage the information overload problem. Perhaps we should talk about it later. Or perhaps not.

Follow-up Post: The Age of Superfluous Information and The World is Information Fat Followup.