It really depends upon the task. For each task (e.g. POS tagging), I maintain a list of current open source implementations. When I attack a problem that involves this task, I weight the tradeoff between how easy the tool is to use with how accurate the tool is. Most of the time, I favor ease-of-use. If I need high-accuracy, I start hand-rolling my tools and keeping them around for later.

p.s. I don't actually use NLTK or Weka that often.