Grand Challenges can be useful

Fields are to some extent driven by trends. For instance, in networking nowadays there are lots of "clean slate" proposals floating around. A few years back, the trend was to figure out how to innovate without disrupting the already successful network (eg TCP friendly congestion control, network tomography etc). Other trends include things like p2p, overlay nets etc.

Fields like networking, which are more synthetic (let us invent a DHT so we can do cool things with it) than analytic (eg let us discover how gravity works) are especially vulnerable to the herd mentality because we have to create novel things from thin air (rather than picking up something that already exists, e.g. a cancer cell, and studying how it works). I say this although my friends in analytical fields would say that certain "sexy" problems attract a lot of people who ride the trend winds.

So, how to motivate new work, so that it does not create too many people working on a single field? Applications can be great forcing functions (the term is from DC himself, though from a very different conversation). Grand Challenges that target some pie in the sky application are one way to force focus, without having everyone work on the same piece. This year's IDEAS competition is a great example. The challenge was TB Drug Adherence.

There are several noteworthy things to note about the design of this challenge:

1. It is big enough of a problem that it cannot be solved just by the IDEAS entrants
   
2. It is not just a sexy thing: ie the topic is not HIV-AIDS but TB drug adherence. This helps concrete work & to demotivate people who simply join for fashion
3. It is endorsed by someone big or charismatic enough
4. There is a big enough prize to create motivation for people who really want to work
5. The problem spec is generic enough for people from different fields to contribute in their own ways.

2 & 5 are probably the most important parts in designing challenges. 5 - because you dont want 100 different p2p implementations all of which work better than each other in slightly different applications. 2. because hype is usually why challenges turn out to be empty wastes of effort.

Other successful challenges: putting man on the moon (well, the real challenge was, can democracy do it faster than communism). The Ansari X Prize - this one is too specific, but it achieves its aim by offering a high-enough prize money.

Homework for the month: What would be good challenge applications in networking?