Archive for January, 2011

Why recommendation engines matter

January 31, 2011

“The Internet makes everything available, but mere availability is meaningless if the products remain unknown to potential buyers.”

(possibly attributable to Gavin Potter, a Netflix Prize contestant; possibly to Jordan Ellenberg, who writes for Slate and Wired)

Advertisements

Movie Ratings

January 30, 2011

Another unmeasurable distance is ★★★ movie ratings.

Movie ratings are drawn from the set {★,★★,★★★,★★★★,★★★★★} and related by the total ordering >:

  • ★★★★★ > ★★★★
  • ★★★★ > ★★★
  • ★★★ > ★★
  • ★★ > ★

and the transitivity of > gives the rest of the relations.

TWO PLUS TWO ≠ FOUR

However ★★★★ is not the same thing as 4, because 4 comes with all the baggage of being an integer. Baggage like the usual metric whereunder

  • |42| = 2,

whereas |★★★★−★★| ≠ ★★.

If one naïvely assumed {★,★★,★★★,★★★★,★★★★★} ≅ {1,2,3,4,5}, that would mean

  • |★★★−★|  =  |3−1|  =  |★★★★★−★★★|  =  |2|.

Which would be wrong.

HOW I THINK PEOPLE USE ★★★ RATINGS

There’s no reason to believe that the distance between ★★★★★ and ★★★ is 2 or that it’s the same distance as between ★★★ and ★. I believe there is a wider gulf between ★★★ and ★ for most people.

It depends on the person but creo q’ a lot of people basically only use three, four, and five stars. Mostly they just use four and five stars because they only watch movies they like.

Then when faced with two choices (★★★★ versus ★★★★★) they may think back to other movies they’ve rated, and wish they had a finer scale of gradation, or just something else to say about them — like in an orthogonal direction.

People use ★★ and ★ but not no creo very judiciously. It’s kind of like the hotness scale … but that’s another topic.

WHAT’S NEXT

I actually have a long, in-depth critique of the ★★★★ system—which also suggests better ways to do surveys in general. But I’ve gone on too long already so let me just preview that critique by saying:

Bad data in, bad recommendations out. Don’t blame yourself, Netflix Prize contestants.

PS: Really wanted a subjunctive mood while writing this. Thanks a lot, English Language. Not.

Smoking in Bars

January 29, 2011

You know how there are smoking bans in various bars, pubs, and restaurants?

How can the government do that? I mean, the location belongs to the proprietor. You know?

edit: I know why they do it — I just mean, like how is that legally possible? Smoking is not generally outlawed, it’s just outlawed inside my property. Wait, but it’s my property! How can you only make something illegal inside my place and not everywhere.

edit 2: I’m talking about legally, not morally.

The Hotness Scale

January 28, 2011

One example of a total ordering is the “hotness” scale from 110.

Because of the widespread disagreement about the meaning of the numbers, the only thing one can infer based on a man’s rating of a woman is that she is more attractive than those who score below her.

YOUTH

The Hotness Scale derives, I think, from a need to explain one’s tastes to peers and hear them justified.

It typically surfaces in sleepover conversations like this:

  • Chris (secretly likes Kelly Russell): Who do you think is hotter: Liz Jones, or Kelly Russell?
  • Dave: Are you kidding?! Liz Jones is waaaay hotter.
  • Chris: Oh, yeah. I mean, obviously. I was just checking. I just meant, you know, that I think Kelly Russell is like maybe a 7.
  • Dave: Are you crazy?! She’s like a 2.
  • Chris: Come on, 2 is like people who have skin grafts. 2 is people who were burned in fires.
  • Dave: Whatever. Maybe.
    (sheepish retreat to celebrity hotness)
    But man, Jane March is the hottest woman on the planet.
  • Chris: No way, Jenny McCarthy is hotter.

I’m a little embarrassed to admit (though I’ll still admit it) that when my best friend and I started using the hotness scale, we scored girls in different categories, like

  1. tan
  2. boobs
  3. personality
  4. legs
  5. I forget what else
  6. overall score

Yeah, we were really cool. (Also we were really twelve.)

DISPUTES

Fast forward to college. We guys were joking about using the ten-point scale, which by then was passé (although I did once use the phrase “a Bloomington 6 is a hometown 9”). We were trying to answer, what is the difference between a 6 and a 7 anyway? And is the distance between 6 and 7 greater or less than the distance between a 9 and a 10?

Everybody had taken calculus by this point so statements involving derivatives were bandied about (even though none of us meant to use real numbers … it was calculus as metaphor).

One guy proposed that each number should correspond to a decile —

  • 1 to the ugliest decile,
  • 10 to the hottest decile,
  • and so on.

 Someone else said that one’s initial reaction put a girl either in

  • the >4 (most of the time) or
  • <5 — but that since no one would ever hit on someone in the latter category,
  • in fact 1≈2≈3≈4.

Another said that he never assigned a 10 to anybody because that would mean he had met his wife. Um, yeah … we were still super cool.

Also contentious was whether each of us accepted the truth of whatever our own numerical ranking was. All I know is that whatever I said my score was, I secretly hoped it was 2 points higher.

There was a lot of inconsistency to the scores, which is why I’m bringing this up under the topic of rank without distance measure. Although I would wager that transitivity is violated, so perhaps this scale does not have a rational basis.

PEOPLE CAN BE ATTRACTIVE IN DIFFERENT WAYS

As I’m writing all of this I desperately want to jump ahead to partial orderings. But I haven’t defined them yet and I refuse to link to Wikipedia, so I’ll have to put that topic off.

Suffice to say that attraction is a perfect jumping-off point for one further generalization I want to make in order to get mathematics into bed with human experience.

Not everyone can be ranked side-by-side against everyone else. People can be attractive for different reasons (multiple >’s) and some people you just aren’t comparable. All of these reasons and more are great justifications for switching to posets.

POSTSCRIPTA

I’d be interested to hear other interpretations of the hotness scale, or other scales of attractiveness, and how they evolved over time. What measure, if any, do you give guys when you’re in your 30’s or 40’s?

PS When I was 24, my girlfriend told me that “24 is a very hot age”. Ha ha.

PPS Tim Ferriss claims to be able to quantify the difference between a 6 and a 9. Tell that to Jimi Hendrix.

Unmeasurable Distances

January 27, 2011

I wrote earlier about the many different ways to measure distance. One way I didn’t include is unmeasurable distance.

Sometimes A is

  • tastier,
  • sexier,
  • cooler,
  • more interesting,
  • or otherwise better endowed

than B … but it’s impossible to quantify by how much. No problem; just say that A≻B but that |A−B| is undefined.

It’s still the case that if A is sexier than B and B is sexier than C, it must follow that A is sexier than C.

Symbolically: A≻B & B≻C A≻C.

This concept opens up many parts of human experience to the mathematical imagination.

I will also express my view on moral rates of income tax using orderings ≻.

Oh, and if you’re into this kind of thing: using orders instead of measurable quantities kind of saved the economic concept of “utility”. Kind of saved it. At least instead of talking about 174.27819 hedons, nowadays you can just say X is lexicographically preferred to Y. Ordinal utility instead of cardinal utility.

January 26, 2011

wowgreat:

95988-NebelWald_rund2_2-3 (by holger lippmann)

January 25, 2011

Theory is easy. Data are hard.

Bill Becker

1 2

Lawyers enable complex laws

January 24, 2011

Let’s assume that (i) there is an optimal number of lawyers and that (ii) said number is between 0% and 100% of the population.

  • The cost of adding a lawyer to society is that a smart, articulate person no longer works in a productive capacity. Maybe there are additional costs in the form of extra frivolous lawsuits or overly complex laws.
  • The benefit of adding a lawyer to society is that more complex laws can be implemented. Sometimes this is desirable because surgical justice is desired. It can be worth spending billions of dollars on precision and nuance, if the cost of being wrong is great enough.

For example, corporations object to Sarbanes-Oxley because the implementation is too expensive. Companies above a certain size (for we Americans are always willing to regulate large corporations) have to pay $1 million extra per year in order to comply with Sarbox regulations, each.

Of course, everybody is in favor of “transparency” since that word sounds so pleasantly costless. But — is it worth the cost? What are regulators doing with all of that information and how does it benefit the common man?

I don’t think there are incentives to reduce the complexity of the law. People generally seem to agree that the law is too complex but it’s not a focus issue — it won’t swing enough people’s votes. That’s bad, because it means legislators are pushed only one way, with no countervailing force — in other words, they’re not optimizing on this parameter.

I’m not saying that we need to eliminate all laws — but I wish there were a way for legislators to be rewarded at least somewhat for reducing the national lawyer total.

January 23, 2011

wowgreat:

88850FlowerCircles_13_grid1 (by holger lippmann)

January 23, 2011

In this paper an exhaustive characterization of financial markets was given.

  • Dependence: Autocorrelation in returns if largely insignificant. (Exceptions being at the tick level and annual returns.)
  • Distribution: Approximately symmetric, increasingly positive kurtosis as the time interval decreases and a power-law or Pareto-like tail.
  • Heterogeneity: Non-stationary (clustered volatility).
  • Non-linearity: Non-linearities in mean and (especially) variance.
  • Scaling: Markets exhibit non-trivial scaling properties.
  • Volatility: Volatility exhibits autoregressive conditional heteroskedasticity. Long-range dependence of autocorrelation, log-normal distribution, non-stationary, non-linear and scaling.
  • Volume: Distribution decays as a power law, also calendar effects.
  • Calendar effects: Intraday effects exist, the weekend effect seems to have all but disappeared, intramonth effects were found in most countries, the January effect has halved, holiday effects exist in some countries.

from Characterization of Financial Time Series by Martin Sewell.via maxdama