Sunday, August 22, 2010

Failure is an orphan

An eye opener! That's the least to say about this inspiring article for Prof. Stan Szpakowicz in Computational Linguistics journal.

According to the article, one project out of a hundred produces results that justify the investment. However, we tend to count ourselves among the 1%. That's because we need to show we outperformed others in order to publish more in order to have a thriving career in research. That's a compelling reason for us NOT to invest more time on something that produced negative results.

But WAIT. "Suppose you have set up an experiment carefully and in good faith, but still it comes up short. That’s not a positive outcome. Maybe your intuition has let you down. Maybe this cannot work. Wait, maybe you can prove that it cannot work?" THAT would be a useful outcome. But you need to get it published.

The problem boils down to peer reviews which aggressively reject failures. "A forthright admission of the inferiority of one’s results—despite the integrity or novelty
of the work—is a kiss of death: no publication. There must be improvement ... conformance to reviewers’ expectations is an asset. Indeed, we write so they are likely to accept". But we're talking about ourselves. WE are the reviewers. If we -the authors- insist on striving for success and run away from every negative result, then we -the reviewers- will check papers for signs of success.

An experiment carefully thought out, a systematic procedure, an honest evaluation—these are the ingredients of good science. It is not mandatory for the results to be positive, though it certainly lifts one up if they are. In areas where empirical methods dominate (e.g. computational linguistics) people try things which fail at the experimental stage. This may be due to lack of rigor, but often there are deeper, unexpected, and intriguing reasons. We can learn a lot if we analyze scientifically why an intuitive and plausible experiment did not work. Then again, to know what leads to dead ends in research surely can warn others off paths which take us nowhere. Simply put, a negative result can be a useful lesson.

Q: Philosophy aside, I want to publish a serious, worthwhile negative result I've obtained. Where to go today?

A:
Here's a non-comprehensive list in different disciplines

2 comments:

  1. really amazing topic..... :)

    *but could u explain to me more what does negative results in research mean?

    ReplyDelete
  2. @Muhammed.. Suppose you had an idea to improve the quality of web search (i.e. get search results which are more relevant to user's query). When you implemented, experimented it, and evaluated the results, you discovered that your idea made the results worse. This is what negative results mean.

    ReplyDelete