Bayesian statistics in the NYT

The NYT has a book review of THE THEORY THAT WOULD NOT DIE: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines and Emerged Triumphant From Two Centuries of Controversy, a new book by Sharon Bertsch McGrayne. I have not read the book, but the article is a good introduction to the Bayesian approach to statistical analysis, or at least what it claims to do in theory.

The theorem itself can be stated simply. Beginning with a provisional hypothesis about the world (there are, of course, no other kinds), we assign to it an initial probability called the prior probability or simply the prior. After actively collecting or happening upon some potentially relevant evidence, we use Bayes’s theorem to recalculate the probability of the hypothesis in light of the new evidence. This revised probability is called the posterior probability or simply the posterior. Specifically Bayes’s theorem states (trumpets sound here) that the posterior probability of a hypothesis is equal to the product of (a) the prior probability of the hypothesis and (b) the conditional probability of the evidence given the hypothesis, divided by (c) the probability of the new evidence.

Consider a concrete example. Assume that you’re presented with three coins, two of them fair and the other a counterfeit that always lands heads. If you randomly pick one of the three coins, the probability that it’s the counterfeit is 1 in 3. This is the prior probability of the hypothesis that the coin is counterfeit. Now after picking the coin, you flip it three times and observe that it lands heads each time. Seeing this new evidence that your chosen coin has landed heads three times in a row, you want to know the revised posterior probability that it is the counterfeit. The answer to this question, found using Bayes’s theorem (calculation mercifully omitted), is 4 in 5. You thus revise your probability estimate of the coin’s being counterfeit upward from 1 in 3 to 4 in 5.

Getting started with data visualization

Having spent the last year working a lot more with data than ever before, I have a new-found appreciation for skillful data visualizations. I have begun to explore some of the more elaboriate visualizations that are possibile with R, Excel and Processing, but I certainly have a long ways to go. Here's a good blog post from Enrico Bertini over at Fell In Love With Data about how to get started with data visualization.

How to Become a Data Visualization Expert: A Recipe

1) Study a Lot
2) Steal
3) Criticize
4) Produce
5) Seek Discomfort

I'm hoping to bring these skills to bear on future research and job opportunities, so I can advance not only my ability to draw inferences on data (social, political, or otherwise) but also present the data and my inferences more skillfully.

Mac OSX Lion review

I downloaded and installed Mac OSX Lion this morning and have been messing around with it today. I don't think I can say any more in a review than John Siracusa of Ars Technica said in his 19-page review but a couple of things that I think are worth noting are listed below.

1. Its pretty remarkable how much Lion borrows from iOS (Apple's mobile OS). Instinctually, I would think a mobile OS is inspired by a desktop OS but with the ubiquity of smart phones and tablets, it makes perfect sense that the flow of innovation is now flowing both ways. This is abundantly clear in features like the new Launchpad which resembles the iOS Springboard, or the central focus on the Mac App Store.

2. The added track-pad gestures are quite great. Since I project my desktop from Macbook Pro to an LCD screen with external mouse and keyboard, I feel like I'm losing out on some functionality by using a traditional mouse. Some of the new gestures can be replicated with a 2-click external mouse, but many that require multiple fingers on the track-pad are best used either with the Macbook Pro track pad or external Magic track pad.

3. While I like to delude myself that I can multi-task well on the computer, I know there are time where the ability to focus my attention on one task is important. The new full-screen feature for almost all apps is great for this. There were many work-arounds for this sort of thing before, but with Lion this feature is available on almost all apps, and you hardly lose any functionality while in full screen mode.

I was also happy with the quick (and relatively cheap) download and upgrade through the App store. Unlike some others who had long waits or slow downloads, from start to finish my download and upgrade took around 40 minutes. I would be much more willing to upgrade or download apps and utilities if the process was this seamless. Looking ahead, it might become even less cumbersome with over-the-air updates in iOS 5 and whatever future iterations of Mac OS we can look forward to.

On metaphors

The Chronicle of Higher Education has a review of James Geary's I is an other, a new book on the place of metaphors in language. This quote from Geary was well received -

Metaphor conditions our interpretations of the stock market and, through advertising, it surreptitiously infiltrates our purchasing decisions. In the mouths of politicians, metaphor subtly nudges public opinion; in the minds of businesspeople, it spurs creativity and innovation. In science, metaphor is the preferred nomenclature for new theories and new discoveries; in psychology, it is the natural language of human relationships and emotions.

Although I wouldn't go as far as someone like Derrida, I think there is definitely value in the work of someone like George Lakoff who advanced the idea that metaphors are a conceptual (not merely linguistic) construction. His forays into political science, through things like conservative concepts of the paternal state versus progressive concepts of a maternal state, are pretty useful.

William Faulkner on winning the Nobel Prize

"I believe that man will not merely endure: he will prevail. He is immortal, not because he alone among creatures has an inexhaustible voice, but because he has a soul, a spirit capable of compassion and sacrifice and endurance. The poet's, the writer's, duty is to write about these things."

William Faulkner at the banquet honoring his Nobel Prize in literature in 1949.

Education and economic output

In "Hollowing out of the ivory tower" Tim Black addresses Prof. Alison Wolf's Does Education Matter? Myths About Education and Economic Growth. Prof. Wolf's biggest criticism is that higher education has become far too mechanized and is merely an instrument to drive economic growth. She warns

An increase in the quantity of graduates will neither create a dynamic, wealth-producing economy nor will it create the conditions for the emergence of lots of dynamic, wealth-producing individuals. Universities are not what they are currently being cracked up to be.

This leads to another problem

Higher education has become central to the economic wellbeing of nations and individuals’, that it is becoming increasingly difficult to recall what the purpose of institutions of higher education might be. Their autonomy as academic bodies, in which one ought to be free to pursue an interest in a subject area to a higher level, has been effaced by their thoroughgoing instrumentalisation as drivers of economic growth and social mobility.

I'd say the correlation between education attainment and income potential is pretty well documented (its a pet example for instructors when picking social science data upon which to draw elementary formal models). However, this idea that the unintended consequence of viewing education attainment as an economic activity rather than a humanistic or perhaps political activity is worth note. Prof Wolf has written this book in the context of the UK but I think this warning readily applies to most advanced higher-education contexts, including the United States.

Last Train Home (2009)

Last Train Home (2009) is a Chinese documentary dedicated to migrant workers in China. It follows a family working to produce consumer goods for Western consumers that end up destroying the lives of the producers who hardly get paid a dime for their labor. It features heart-wrenching scenes of a father and mother committing themselves to arduous factory work only to see their son and daughter once a year during the Chinese New Year (only reaching them after being herded like cattle on crowded trains traveling between provinces). The entire film captures very well the emotions of the bleak reality of migrant workers' lives. The "low-hanging fruit" of China that has catapulted it to global economic success hardly benefits from these successes.

Although the film is wrought with pain and strife, it comes highly recommended as a fantastic piece of work in film-making and also as a worthwhile look into migrant workers lives in China. I can only imagine that the struggles featured in this film are characteristic of struggles of workers around the world.

Taking down Michael Lewis

Michael Lewis has amassed quite a reputation on the back of his 2010 book The Big Short. I remember reading the book and listening to coverage on shows like This American Life's The Giant Pool of Money in 2010 to try and understand the financial crisis. This piece, however, has a lot to say about what Lewis missed in his book.

Lewis’ desire to satisfy his fan base’s craving for good guys led him to miss the most important story of our age: how a small number of operators used a nexus of astonishing leverage and camouflaged risk to bring the world financial system to its knees and miraculously walked away with their winnings. These players are not the ugly ducklings of Lewis’ fairy tale; they are merely ugly. Whether for his own profit or by accident, Lewis has denied the public the truth.

I can certainly understand the argument that Lewis was writing for a different audience (as perhaps Ezra Klein was in this piece, which was also rebuked) but I think after a cursory understanding is established, these kinds of nuanced complications are absolutely necessary for better policy building and informed decision making (including personal voting behavior) in the future.