Tag Archives: data mining

Numbers, Recommendations and Unforseen Connections

11 Feb

Two blog posts caught my eye today. They touched on a similar theme — how to do you get a computer to account for taste?

The first was Stephen Baker’s BusinessWeek blog post with a take on the Apple iTunes Genius recommendations.  His initially incendiary claim (apparently unresearched beyond his own iTunes collection) is that the Genius recommendations — are rigidly race-based. Asking iTunes to make a playlist based on Aretha Franklin, he says, generates a playlist composed entirely of black R&B musicians because the database was compiled with that category rating highest. Taking a step back from his first blog post, Baker writes: 

“When I do a search on Aretha, the system starts with what it knows about her: R&B. And it lines up an entirely R&B playlist. How does it know she’s R&B? Many of the commenters say that the analysis starts with no groupings provided by the programmers, and that it is based entirely the study of user behavior. (Ie. What other artists do we group with Aretha on our playlists, and what other music do Aretha buyers purchase?)

“Based on what I’ve seen so far, I’m sticking my initial guess: The programmers started out Genius by putting the artists in their boxes—R&B, Folk, Classic Rock, World music, etc. It’s a little crude, but you have to start somewhere.”

Then there was a post by George at Fast Horse on the Idea Peepshow blog. George reports on the challenge NetFlix faced building customer recommendations based on the movie Napoleon Dynamite.  Apparently, it is almost impossible to create an algorithm to predict accurately what movies you’d like based on a high rating of the movie Napoleon Dynamite.  NetFlix has gone so far as to offer a $1 million prize to anyone who can improve their movie recommendation engine, and much of the challenge is accounting for taste in quirky movies.  The common thread: the drive to get computers to understand our behavior well enought to predict what we’ll do next — or at least what we’ll buy. 

It’s nothing new, of course. Much of Amazon’s initial success came from a recommendation engine based on collaborative filtering — the idea that population of people who purchased the one book might be interested the other  books that same population has purchased. The techniques have continued to grow in sophistication. And they have continued to    please and confound marketers–seeking the perfect pitch–and consumers–simply seeking something new.    

Baker says that this is the goal — to put us in categories to see how we are similar in what we buy, what we choose to do    and even what our risks might be for disease.  He writes:   

“Traditionally, marketers and politicians have organized us along traditional demographic lines: Income, ethnicity,    neighborhood, etc. But with more data about our activities, they can start to create new “behavioral tribes.” The old    boundaries break down…In music, I imagine the same thing will happen. Once the data comes in, Aretha and others will break    out of their boxes.”   

Maybe so…more likely, they’ll break out of the old boxes only to hop into new ones.       

But there is a lot of fun in all this.  Search and social media are helping people find like-minded partners-in-crime in ways  no one could have predicted back when I was in college in the latter half of the 80s, wondering why I could never hook up with fellow Neil Diamond-listening sci-fi fans.   

On the other hand, the more we seek and find people and experiences “like us”, the more we yearn to be surprised.  The reason Napoleon Dynamite fascinates, Boing Boing is a wonderful blog, Super Bowl ads disappoint, and why print news media may well live on is that they give us something we didn’t ask for.  Something we didn’t specify in our search terms, preferences or    previous actions. Something we didn’t even know we wanted.  

Crunching the numbers is worth the effort — the better they get, the less money will be wasted on marketing to the wrong    people at the wrong times. But breakthroughs happen when you surprise people.   I’d make sure there’s a seat at the table for creativity, ingenuity and intuition.  You never know what’s going to happen … and that’s the point.

Advertisements
%d bloggers like this: