There are a variety of issues that factor into the question of whether bigger is better. Certainly clipping, from a lack of power, is a bad thing. And, if I understand these things properly, if its the usual class A/B, the bigger the amp, the more power is available in class A before the transition to B, the transition being a source of audible artifacts. And it is commonly accepted (and I agree) that tube amps seems to have more oomph than equivalent rating solid state equipment. And amps interact with the complex loads that speakers present, so often results differ from expectations.
But the issue I keep coming back to is that of expense. For some people, cost is no object in the pursuit of audio perfection (regardless of whether it is an achievable goal). For most people though, bigger may be better, but it is also much more expensive. And audio perfection simply doesn't matter that much to most people, not least, because it is unachievable. Most people do want good sound, but only audiophiles are willing to spend much on it. Everyone else is willing to live with whatever 200 bucks will get you. But even audiophiles have budgets. What you already own is infinitely cheaper than the alternatives, and if you like it, blowing a fuse now and then may be fine. Or maybe turning it down just a little bit will also be fine.
If the OP decides that he doesn't want to blow fuses anymore, and is willing to save up his cash, more power to him (literally). If not, also more power to him (now metaphorically).