Intro to Information Theory

Information is everywhere, but that's obvious and everybody is saying it.  In a certain sense, at the very heart of experience, there is nothing other than information.  But that's a story for another day.  So what do we mean by information?  Certainly, there is a sense of relationship, of correlation, of geometry.  But can we make it precise?

We can.  Information Theory is extremely well developed and mathematically rigorous.  Here, we will briefly explore the basic intuitions that lie behind it.  In the future, we will explore its many consequences.

Everything starts with probability theory, more particularly, with the main object of probability theory, the random variable.  A random variable is a variable which, lo and behold, takes on a random value from some predefined set of values.  The number of hours a day a person spends in traffic is a random variable, with values taken from positive real numbers between 0 and 24.  A more abstract random variable is the configuration of human civilization in a particular point in space and time.  If we 'measure' this configuration, then we find that the random variable takes on a value of, say, the exact current configuration of our society today, measured by some macroscopic qualities (population density, lifespan, distribution of wealth, velocity of goods, etc.).  But this is only one of the many possible values that the random variable configuration of human civilization could take on.  Indeed, many an author have made their fortune exploring other possible values of this wretched-though-magnificent random variable.

So the neat thing about a random variable is that it comes with a probability distribution.  That is, if we measure the random variable repeatedly, some values may be more likely to appear then others, and so we can talk about the probability of the random variable taking on any one of its many possible values.    We call this set of probabilities a probability distribution over the random variable.

Consider a coin toss.  The result of a toss is a random variable, whose possible values are 'heads' and 'tails'.  If it is a fair coin, then there is an equal probability for both possible outcomes (its probability distribution is uniform).  On any given toss, you literally have no idea what is going to happen.  It's either heads, or tails, but you can't do any better than that.  So naturally, when you flip the coin, and find out that it has landed heads or tails, you will be pleasantly surprised, because you had no inkling as to which it would have been before hand.  We might say, "the probability distribution of a fair coin exhibits maximal surprise."

Consider now a coin which is not fair, but which, by a clever trick of its design, comes up heads 90% of the time when flipped.  If you knew this, then you won't be surprised at all when you find out the outcome of the flip, most of the time.  So this probability distribution has considerably less surprise, because it's almost always heads.


Now this surprise-quantity more commonly goes by the name of entropy.  That is, the entropy of a probability distribution over some random variable is a measure of how little one knows about the outcome of any measurement on the variable.  So its like an uncertainty.  But it is simultaneously a measure of how much knowledge one can gain by measuring the variable, given how little one knew before hand.  And so in an interesting sense, the entropy of a variable, its surprise content, is a measure of its information content.  A probability distribution which is uniform is most uncertain, having maximal entropy, and the most information to gain from measurement - the most surprise.

But that's not the whole story.  When we talk about information, we almost always refer to information between two random variables.  We are not so much interested in what someone says as we are in how much of what is said is heard and understood properly by the receiving party.  Suppose there was a mysterious light on the wall that flashed colours when you flipped your coin.  You soon notice that every time it flashes blue, you flip a heads.  It's flashing blue now.  Will you be surprised when you flip heads?  Not at all.  The flashing light, by being correlated to the outcome of your random variable, reduced the entropy of your random variable.  It used to be a surprise, but now that you have the light, its not.  So there is mutual information between the light and the coin.

Mutual information then is defined in this precise way, as the reduction in entropy (uncertainty) of one variable, given the value of another.  We can theoretically measure the information content between any two variables, or more, by sampling their respective probability distributions, and calculating how much each variable reduces the entropy of the others.  In the ideal case, where there is maximal information, the entropy of a variable is reduced completely when given the other variable.  So the maximum mutual information between any two variables is the entropy of the variables themselves! (Actually, it is the lesser of the entropies of the two variables).  That is to say, something which has a low entropy to begin with (ie. you have some degree of certainty as to its outcome), is necessarily restricted in how much information it can share with another variable, compared to something which has a higher entropy. 

In most cases, this result, that the maximal mutual information between two random variables is the lesser of the entropies of the variables, has a lot to do with the number of possible outcomes of a variable in the first place.  A coin toss, regardless of its probability distribution, has only two possible outcomes, and so is obviously limited in the amount of information it can share.  But similarly in large systems, with many possible outcomes, a distribution which maintains a low entropy necessarily restricts the information sharing capacity of the system.

So there is an interesting trade-off.  Something which is well organized, about which there is little uncertainty, which is accurate and precise and efficient, ie. something which has low entropy, is fundamentally restricted in the amount of information it can share with another variable, by virtue of the fact that it is so organized and predictable!  On the other hand, something which is initially more uncertain, which may be less reliable and effective but certainly more opportunistic and versatile, ie. something which has higher entropy, is able to share far more information with another variable, by virtue of the fact that it can be found in its other states with greater probability.  That sort of trade off permeates everything.

I think about these ideas most often in the context of protein signalling in the cell, where proteins take on particular shapes and are sometimes well ordered and sometimes totally disordered, and one-way-or-another these structural differences relate to their functionality and give rise to the marvels of our existence.  But on the other hand, consider our political organization, and the chains of beaurocracy, clamping down on entropy everywhere they can with enormous energy expenditure.  The reduction in entropy is catastrophic in the face of how much information our governments need to handle if they are to do their jobs.  We need far more disorganization in our society, because only with a higher internal entropy can we properly encode the enormous wealth of information about ourselves and our environment that is necessary to keep track of in the maintenance and evolution of the (supposedly) most complex and intelligent species on the planet.  Libertarians have been saying this for a while but I think I just grounded it in physics. 

Information theory is cool.

Buckminster Fuller - A Guide to Spaceship Earth

So I'm currently reading Buckminster Fuller's "A Guide To Spaceship Earth."
It is absolutely fantastic.

Part of what captivates me so much about good ole' Bucky is his no-bullshit-tell-it-like-it-is-you-must-be-crazy-if-you-think-the-universe-is-not-profound attitude.  Let's take for instance the title of the document I am reading.

Consider our position.  Here we are, on a fantastic space-craft (that is really what the Earth is), ripping through seeming emptiness at thousands of miles per hour around our enormous mother-ship, the life-giving sun.  Not only are we on a space-ship zipping through space at such incredible speeds we can't even fathom them, we are on a space-ship that has been zipping through space at such incredible speeds we can't even fathom them for a duration (4 billion years!) that is probably even more difficult to fathom than the speed we're zipping at.  And to think, if it weren't for astronomy and geophysics, we wouldn't even know!!  Talk about a well engineered space-craft.  Can you even feel all that acceleration?

So that's one hell of a situation to be in.  I mean, it's a hell-enough-of a situation to be in even if you don't know about the cosmic magnitude of the circumstance, ie. that the unvierse has pooled its resources and energy budget into some exuberant structural and systems engineering on a small rock around a medium sized fire-ball somewhere along the outer spiral arm of this here Milky Way Galaxy.  But to come to terms with the facts of the matter, that HERE WE ARE, heirs of Universal Energy and Cosmic Becoming, zipping through god-knows-what-the-stuff-called-SPACE-really-is, on this fantastic Spaceship, is an altogether profound situation.

I mean what the fuck?!

Seriously.  What are we supposed to do?!  Here is this fantastically engineered space ship, equipped with full regenerative capacity and practically perennial propulsion, carrying on its own business of zipping and zapping while all sorts of myriad interesting things occur within the ships compartments and flight decks, virtually oblivious to the literally stellar circumstances of the craft they ride on.  And of course, for those that have become privy to at least the galactically local aspects of the circumstance, there isn't a guide or a manual or an instruction set or a help-forum to be found.  Wouldn't it be nice if there was an Earth-Overflow.org, hosted by advanced alien civilizations, offering help and advice and direction to us newbs in the experiment of intelligent life?

So seriously, what the fuck?!

What do we do?  Well, first of all, we must collect observations.  We must get to know what kind of ship we're on, what its engine consists of, how it operates, what its relation to the mother-ship is, and so on.  We have done a relatively good job of that.  Relative to whom?  Let's say the kangaroos and pythons.  I can only imagine that the Cetaceans have a better grip on the thing than we do, despite not being able to actually get a physical grip on anything, flippers and slippery skin and all.  But given that our spaceship is over 70% water, its conceivable that organisms exploring the water might have a better handle on the systematics of the thing than those confined to land.   Of course we're no longer confined to land, and now we fall freely from space, which is a fun-recreational-quasi-cosmic milestone that will probably lead to the death of many young adventurers tired of terrestrial-adrenaline-rushes and searching for the next kick, before it leads to productive exploration of the atmosphere and beyond.  I hope our parachute technology improves considerably, or that RedBull figures out how to actually engineer wings (I bet you it's coming, otherwise they might get sued for false advertising).

Anyways, the next step is more important.  We need to apply our observations.  If we sum the observations as such: "We are members of a species of organism which has evolved along with other organisms out of the abundant but finite resources of this planet, developing a novel form of generalized intelligence and visuo-kinesthetic capacities which allow us to manipulate and abstract upon the myriad resources of our space ship, providing many-fold opportunities for innovation and consumption.  However, we are growing at an exponential pace and ARE CERTAIN to over deplete the space-ships resources such that it will no longer be able to sustain its crew, who will be doomed to die a hollow death spinning thousands of miles per hour around a sun which can't provide fuel any more effectively than it has for the past few billion years", then we find that not only is our circumstance spectacular, it is somewhat dire, for, as any competent human being with internet access and an awareness of global events knows, we are living in a geo-political shit storm, where all of the above facts are, if not harshly denied, then blatantly ignored.

The shit-storm has everything to do with mentality.  As Bucky points out, much of it is based on thermodynamics, which established the concept entropy, which basically states that, left to itself, the organization of a system must decrease, that is, it must dissipate energy.  In other words, systems are running down.  Psychologically, this translates into the following: "If all systems are running down, then the Earth system is running down, and since there may not be enough to go around, there will certainly not be enough to go around in the future.  Thus we must protect our own and fight every man for himself.  Like in Nature, only the strong survive"

And so we have wars and republicans and democrats and trivial political flings filled with useless jabbering and basically the whole system has gotten up, exchanged its face with its ass, and sat down again.  Completely useless.  Of course, despite entropy being a universal concept, its implementation here is faulty, for while entropy runs systems down when they are LEFT TO THEIR OWN DEVICES (ie. lacking in innovative design and energy input), it may actually encourage self-emergence and novel organizational capacity in the context of innovative design and energy input.  Take the origin of life itself, for example!

So here then is our current position.  Geo-politically, we are Sisyphus pushing his rock perennially up the hill, and no matter what Camus has to say on the subject, we are getting nowhere, and certainly not enjoying ourselves.  On the other hand, armed with 1) an emerging theoretical understanding of the nature of non-equilibrium thermodynamics and self-emergence (where entropy runs systems up instead of down) and 2) an emerging practical understanding of open-source-cross-platform-free-exchange-of-ideas and self-emergent collaboration in the computer sciences, there may be some hope for us after all.

I think Bucky's vision is extraordinary.  Given enough human ingenuity, we can design ourselves out of any mess.  We need only to assemble and do.  It would also be helpful, of course, if the political system stopped interfering with every form of progress that emerges under (around?) the sun.  But in the open-source world, politics becomes irrelevant.  Border's fall down.  True power is put back in the hands of the people, so long as they are willing to share.  And with that infrastructure growing, at an accelerated pace, there may be hope for us astronauts after all.

Design will save the world.  Enjoy the rest of your flight here, on Spaceship Earth.

Memory vs. Novelty and the Impact of Belief

The human memory system is a profound process that encodes environmental information as physiological signals.  In doing so, it must employ an encoder which manages the task of actually converting the information, which comes in as electrical signals from sensory organs, into a form that it can store, long term, and access with relative ease.

This is a profound task.  Anyone who has ever worked with an audio file knows a thing or two about this task.  Music comes out of a musician in a continuous, analog stream of pressure changes in the air.  Technological equipment must somehow capture these pressure changes, using some sample rate (to catch changes every thouandth of a second, or 44 thousandth, since it cant catch allll of them right down to ultimately small time), and encode the changes as a digital signal, which is stored as a .wav, for example.  This file, despite the losses of information that occurred when the analog signal become digital, is considered lossless.  But the problem is its huge!  The human brain does not have room for lossless data.

Computers have a handy solution for dealing with .wav, namely, to convert it to mp3.  If you take a look at the file sizes, an mp3 is about a tenth the size of the .wav it came from, despite both having the same song and for the same amount of time.  The difference lies in the concept of 'compression.'

The basic ideas emerged from a man named Claude Shannon in the middle of the century.  Shannon was an electronic engineer.  His master's thesis, which was published in 1937, demonstrated that Boolean algebra could be used to build computers, and has been considered the most significant master's thesis ever written (consider that when you're studying the subtle effects of protein A on the localization of protein B's less prominent cuzin, Protein Ba, in some subcompartment of the nucleus in the presence of chemical C and D at room temperature in rainbow trout because it may have something to do with cancer).  But the important ideas about information came out a decade later.

Basically, Shannon defined the entropy of some amount of information to quantify the extent to which one piece of the information could not be predicted from the others.  The higher the entropy, the less predictable the information.  But the power of the idea emerged as a consequence: if there is redundancy in a set of information (parts can be predicted from other parts), then the information can be compressed into a less redundant form.  The entropy of the information is then equal to the entropy of the information compressed into its smallest, non-redundant form.

With music, instead of encoding every piece of the music as it came in through the microphone, we can represent those parts that are similar to earlier parts (say, the same guitar note) by pointing to the earlier instance of the part, instead of actually storing it again.  Naturally, it is far easier to store a 'pointer' in memory than it is to store the whole thing we're 'pointing' to, and thus the file can be compressed significantly.


If you speak to an audiofile, though, you will find that they want nothing to do with compressed music.  It is too 'lossy' - it is missing information, they say.  Naturally, they are right.  The guitar tone that was repeated, that we compressed by pointing to the original instance, may not actually have been exactly the same each time.  In fact, it probably wasn't.  The subtle differences, be it in the way the note was picked, or the way it flowed out of or into other parts of the melody, are lost in the compressed version, and those with trained ears can literally hear this loss in the way the track plays - it doesn't sound real enough.

For most of us, the compressed mp3 file does just fine.  It is efficient and effective in re-presenting a piece of music.  What can we say about the way the brain stores information about itself and its environment?  What algorithms does the brain use to compress such information?  Is the compressed version sufficient?

Information comes into our brain by way of neurons (brain cells) linked to sensory organs (eyes, ears, skin, nose, tongue).  It travels as electricity down neuronal wires and flows through complex neural circuits, the structure and dynamics of which 'process' the information.  What this actually means is a raging mystery, and understanding this mystery sits right up front in the auditorium of science's goals.

Lets gloss over the technical neuroscience and consider how we remember things.  By the time we're an adult, say, we have experienced numerous contexts and objects and processes, have assimilated them into our understanding, structured them relative to one another, assigned associations between them, grouped them under labels and classifications, regrouped them, redefined their associations, restructured them, and so on.  The net result is our psychological person - the sum of our memories, valuations, expectations, and opinions.  Any new information comes into the brain through an already established network comprising an individual psychological person.



So how should information form the environment at this stage of the game be remembered.  Ideally, if the information is identical or particularly similar to other memories (to other experiences), then we need not store them twice, and can simply refer to the previous encoding as part of the memory of this 'new' experience.  Similarly, if we expect certain things to happen, and our expectations are corroborated by the outcome of events, then perhaps we need not even remember the new event at all, simply our expectation, and the fact that our expectation is true.


Now, I hope you can see the danger in this last assertion.  Indeed, this fact lies at the heart of all Human bias.  We see what we expect to see, we hear what we want to hear, we feel what we want to feel.  More often than we might suppose, we witness something which does not adhere to our expectations.  Nevertheless, we massage into being a new memory that supposes it actually did happen according to the way we expected it to, and thereby save energy on memorizing new and unexpected information.  Partial neglect breeds higher efficiency.


It seems reasonable that the brain should evolve like this - to be able to store new experiences using less energy by assimilating them into old, expected, experiences.  To compress information by squeezing it into a framework, a belief system, a system of expectations.  To consolidate one's perspectives and confirm one's own biases.  To be right.  It is easy to know you are right.


But to be wrong?  Novel information is energy expensive.  It requires more activity, more attention, more Being, to assimilate the new.  It suggests that we were not, in fact, completely right.  Suggests that there is more to learn (that there is infinite to learn), that the world is dynamic and we must be too.  Suggests that stagnation can corrupt the soul.  Does it suggest that we should just forget the old?  Let go of memory of past and future?  Be entirely open and aware of the moment, only memorizing with perfect accuracy what Is, Now?