Posts Tagged ‘Bayesian’

June 9, 2012

We showed in Chapter 6 that side information Y for the horse race X can be used to increase the growth rate by the mutual information I(X;Y). We now extend this result to the stock market.

Here, I(X;Y) is an upper bound on the increase in the growth rate, with equality if X is a horse race. We first consider the decrease in growth rate incurred by believing in the wrong distribution.

Thomas A. Cover & Joy A. Thomas, Elements of Information Theory


Distance between Words

June 4, 2012

Which pair is more different?

  • keyboard | keyb`ard
  • keyboard | keybpard
  • keyboard | keebored

Of course in mathematics we get to decide among many definitions of size and there is no “correct” answer. Just what suits the application.

I can think of two approaches to defining distance measures between words:

  • sound-based — d(Hirzbruch, Hierzebrush) < d(Hirzbruch, Hirabruc)
  • keyboard-based — d(u,y) < d(u,o)

Reading on online fora (including YCombinator, tisk tisk) the only distance functions I hear about are the ones with Wikipedia pages: Hamming distance and Levenshtein distance.

These are defined in terms of how many word-processing operations are required to correct a mis-typed word.

  • How many letters do I need to insert?
  • How many letters do I need to delete?
  • How many letter-pairs do I need to swap?
  • How many vim keystrokes do I need?

and so on—those kinds of ideas.

inter-letter interaction effects

If we could get conditional probabilities of various kinds of errors — like

  • Am I more likely to mis-type ous while writing
    • varoius
    • precarious
    • imperious
  • ? There could be some kind of finger- or hand-based reason, like if I’ve just been using right-handed fingers near my ous fingers, or that I have to angle my hand weirdly in order to hit the previous couple strokes in some other word?
  • Am i more likely to mis-type reflexive as reflexible when the document topic is gymnastics?
  • Am i more likely to make a typo in google if I’m typing fast?
  • What if you can catch me mis-placing my hand on the homerow/ how dp upi apwaus fomd tjos crazu stiff? That’s almost like just one error. (It’s certainly less distance from the real sentence than a random string of characters of equal length.)
  • Or if I click the mouse in the wrong place before correcting my spelling? d(Norschwanstein, Ndorschwanstein) or d(rehabilitation, rehabitatiilon)
  • Am i more likely to isnert a common cliche rather than what i actually mean after a word that begins a common cliche/

A Bit Of  Forensics

EDIT: Once I got about halfway throguh this article, I stopped correcting my typoes, so you can see the kind that I make. I was typing on a flat keyboard, asymmetrically holding a smallish non-Mac laptop (bigger than an Eee) with my elbows out, head down — except when I type fast and interchange letters, with perfect posture, “playing the piano” with my ten finger muscles rather than moving my wrists — at an ergonomic keyboard with a broken M. I actually don’t recall which way i wrote this article. I may hav eeven written it in shifts.

Here are some nice ones as well. Look at the comments section. By the posting times (and text) you can see that the debate was feverish—no time for corrections and the correspondents were steamed up emotionally. Their typoes really have personalities—for example Kien makes a lot of errors with his right middle finger moving up. (did → dud, is → us, promoted → promotied, inquisition → iquisition, mean → meaqn, Church → Chruch, because → becuase, Copernican → Ceprican, your → you, clearly → cleary) but also some errors of spelling with no sound-distance (Pythagoras → Pythagorus) and uses both the sounds disingenious and disingenuous. Letter-switching, ilke I do, is common; a few fat-fingers (meaqn) or forgotten letters, but this iou stuff seems unusual and possibly characteristic of something.

Other participants make different sorts of errors, or at least with different frequencies (they’re relatively more likely to omit or switch letters than to use the wrong letter, for example). But let’s just focus on Ken because so many errors of the typoes are localised to that right middle finger. I wonder if Ken has a problem with that finger? Or maybe his keyboard is shaped in such a way that it’s difficult to correctly strike those keys specifically? (Maybe certain ergonomic keyboards would fit this — or an Eee Pc with the elbows out and “pigeon-toed” hands. But why would the errors then be localised to the right middle finger? It’s more mobile than pinky & ring fingers and we’re not taught to stick it to the homerow like the index finger.) I rule out the theory that his right hand hovers above the keyboard rather than sitting on the homerow because then he should make similar errors with yuiop and maybe bnm,.hjkl; as well. Also, notice that he doesn’t make comparable errors with ewr as with iou. How do we know he sits symmetrically? I have a tough time deciphering why there are more errors with that finger on a first read-through.

We could find more of Ken’s writing here and see how he types when he’s less agitated. I bet there are no Ceprican’s there but Pythagorus would still be. As for Chruch? Hmmm. Don’t know.

Big Data vs Models

Now the big-data-ists (the other half of Leo Breiman’s partition of statistical modellers -vs- data miners) would probably say “Google has a jillion search results including measurements of people correcting themselves and including time series of the letters people type — so just throw some naive Bayes at that pile and watch it come to the correct answer!” Maybe they’re right.

If someone wants to mess around with this stuff with me — leave me a comment. We could grab tweets and analyse typoes within differnet text-…[by which tool] was used to send the tweet. For example the Twitter website means it was keyboard-typed, certain mobile devices have Swype, other errors we might be able to guess tha tis …[that it’s] a T9 mobile keyboard.

  • Could we tell if a person is left-handed by their keyboard mistkaes?
  • Could we guess their education level/
  • Could we tell what tweeting platform they used by their errors rather than by 
  • Could we tell where they’re from? Or any other stalky information that advertisers/HR want to know but web browsers want to hide about themselves? (Say goodbye to mandatory drug testing in the workplace, say hello to your boss getting an email when a statistics company that monitors your twitter feed guesses you smoked pot last night based on the spelling and timing of your Facebook posts.)

I Need A Speech Bubble To Appear Over My Head When I Talk So I Can Diagram the Bayesian Uncertainty In My Statements

May 2, 2012

December 4, 2011

The actual science of logic is conversant at present only with things either certain, impossible, or entirely doubtful, none of which (fortunately) we have to reason on.

Therefore the true logic for this world is the calculus of Probabilities, which takes account of the magnitude of the probability which is, or ought to be, in a reasonable man’s mind.

James Clerk Maxwell (1850), quoted in Plausible Reasoning (1994)

Irrationality in Economics, and “Subjective Probability”

October 6, 2011

I gave this talk several years ago, but you know what? It’s still pretty decent.

Irrationality in Economics  

The title is misleading. Like many of my titles, it’s meant to grab attention rather than be exactly correct.

I was trying, with this talk, to convince college freshmen to switch from Philosophy to Economics. And you know, Philosophers are always talking about Rationality — is there even such a thing, and if so what does it consist of? Econ provides more than one concrete prescription for Rationality — more on that below.


“We are recorders and reporters of the facts—not judges of the behavior we describe.” —Alfred Kinsey 

I actually think that economists and psychologists could do more to prescribe healthy, effective behaviours and thoughtstrategies for people to follow. But the recommendations should be based on empirics, e.g.

  • “buy experiential goods, not durable goods”;
  • “purchase with cash instead of plastic”;
  • “beware these 4 common investing mistakes made by novices”;
  • put crisps and fudge in a drawer, not in plain sight”

—not on a general model of “optimal” behaviour.

Theorists, though, don’t have the necessary understanding to make normative evaluations. Not yet, at least. But they can approach the deep Utility Theory questions in the spirit of the above quotation. They can model behaviours and thoughts, and inquire as to how they are internally structured — without the prejudice of inherited mathematical aesthetics.

What do I mean by ‘inherited aesthetics’ ? One example is substituting the mathematics of probability for a separate theory of human figuring.



One parsimonious shortcut economists tried, which didn’t work out, was to use probability mathematics to explain how people think about the future. If we can conceive of people’s beliefs as mathematical probabilities, then regular microeconomics + more maths = a new, better theory of behaviour.

For example, curved preferences over wealth would manifest themselves in probabilistic situations such as lotteries, insurance, betting, investing, employment in risky jobs, and love & sex risks.

But. People don’t think that way. They don’t make accurate calculations about Poisson distributions, Beta distributions, Bayesian priors, Aumann agreement theorems, and so on. I guess evolution either built us for something different or else we’re just misshapen clay with limited resources to Bayes our way to rationality.

I speculate that the way people think about probability — dubbed “subjective probability” by Leonard Savage — is shaped very differently from what mathematicians usually consider “natural” axioms — transitivity, commutativity, reflexivity, independence of irrelevant alternatives, monotonicity, and so on. But who knows? The correct theory doesn’t exist yet.



The word “irrationality” I definitely ab-used.

Economists come up with a theory of how people behave and say it’s “ideal” or “rational”. People don’t actually think like that, so then we say they’re “irrational”? That doesn’t make sense. The theory was just wrong; an incorrect description. They perform sub-optimally according to some guy’s theory of the world, of their value system, and of how they should think. But since we don’t really know how people really think, how they experience the results of their choices, or how we should evaluate discrepant self-reports of how good a decision was, we can’t say what’s rational.

Like so, although it took the Ellsberg Paradox, Allais Paradox, and other results to disprove the accepted theory which naïvely united Probability and Utility, those results are not the point. The point is that we have to conceive a more realistic model of people’s mental models before Economics can draw valid conclusions about what people “should” do.

May 1, 2011

Contrary to common folklore, causal relationships can be distinguished from spurious covariations using inductive reasoning.

Judea Pearl, Causality

The radiolab story “It’s Alive” made vivid the claim of Geoffrey West and Luis Bettencourt that a city’s size determines how fast people walk in that city. West & Bettencourt have written that people earn more in large cities, waste less, file more patents, and commit more crimes — and that city size is the main determinant of all these things. The charming Cosma Shalizi has recently published a 15-page paper that rebuts them. From the abstract:
Re-analysis of the gross economic production and personal income for cities in the United States, however, shows that the data cannot distinguish between power laws and other functional forms … and that size predicts relatively little of the variation between cities. The striking appearance of scaling in previous work is largely artifact of using extensive quantities (city-wide totals) rather than intensive ones (per-capita rates).
(Sorry if that’s hard to read. Horizontal axis = log( city population ). Vertical axis = pedestrian speed in m/s, give or take a standard dev. Solid & dashed lines are two fits proposed by Bettencourt & West.

February 27, 2011

Radiolab got it wrong

Probability of God

August 28, 2010


Apparently if you go beyond the Amazon preview, Unwin just makes up some numbers, plugs them into the Bayesian machine, and — poof! — calculates the probability he wants for God’s existence, 2/3 (which he later ups to 95%).

Doing this with numbers I agree with yields a probability more like 10^−17.  That sounds really low but I guess the point is, when you make up lots of numbers, you can reach any conclusion you want.

Essentially, Unwin made up an arbitrary number of points to assign to various evidence for/against God, and then also made up what the evidence said about those criteria.

For example, the existence of human moral evil is evidence against God, and he says that current levels of moral evil militate for odds of 2:1 against God (while current levels of moral good militate for 1:10 for God).  My response, in two words:  genocide and … ?

My question coming out of this is:  are quants unscientific?  Unwin was a quant and got hired for primo positions.

Also, dumb assertion in Wilmott:  Mao, Hitler, and Stalin were all half-educated — to the point of charisma but not to the point where they realize “the madness and helplessness of it all”.  The author’s solution?  ”Managing society requires the world to embrace randomness.” (sic)  That itself sounds half-educated to me — like somebody who treats randomness and risk as an epsilon symbol.

So are the quants like some mathematically sophisticated quasi-skeptics who really believe in magic?