From what I gather its no so much a curse word and more simply a word to express annoyance.

Oh.. Well poop :lol

YEAH!

In Japan, they only have a small amount of words that could even be considered curses or insults. How grave someone swears depends more on the situation not the word itself. For example, I once read that baka can mean anything from idiot to well, whatever insult needed in the situation.
I remember that in earlier years, TVN rather chose some more extreme cursing for their subs than needed. But then again, it's all a matter of reading the situation.

If you need to learn the many ways Baka can be used, I suggest watching Kiva. Between Megumi and Yuri you'll get every variation :lol