May 22, 2011

A must-read: Empiricism is not a matter of faith

Found a great article "Empiricism Is Not a Matter of Faith". It's about computational linguistics, but it can very well be applied to any discipline. The main idea is that most of our empirical science is not reproducible and verifiable:

Our inability to reproduce results leads to a debilitating paradox, where we as reviewers and readers accept highly empirical results on faith. We do this routinely, to the point where we seem to have given up on the idea of being able to reproduce results. This is the natural consequence of faith-based empiricism, and the only way to fight that movement is with a little bit of heresy. Let’s not accept large tables of empirical results on faith, let’s insist that we be able to reproduce them exactly and conveniently. Let’s insist that we are scientists first and foremost, and agree that this means that we must be able to reproduce each other’s results.
Suggestions for reproducibility (I re-interpret them for non-computational linguistics areas):

1. Release early, release often. This one is about software code due to the nature of computational linguistics. But it can also be applied to data, results, and other research procedures.

2. Measure your career in downloads and users. Maybe not downloads, but definitely users. And definitely not citations. Why care about citations if nobody outside academia is using what we produce?

3. Ensure project survivability. I'd say, it's important to think about expansion of one's research. Can others use it beyond ritual citing? What are the outcomes of one's project beyond a publication?

4. Make the world a better place. This one deserves another quote from the paper:

We are fortunate to do what we do: even if it takes many hours and causes great personal stress, in the end the work is challenging and satisfying, and compared to how most people in the world live and work, we are leading charmed and privileged lives. ... it seems like we really ought to try to give back as much as we can to the greater public good."