There's the general problem of "Things are slow for no reason", which I don't think is specific to the Task Manager or OSX.  But I have noticed that when I go to the Task Manager to try to figure out if I have a tab using lots of memory or something, the browser process always seems really busy.  Eventually I took a sample, and it was mostly in updates of the Task Manager itself (I think).  I verified this by correlating with top, with the Task Manager up browser process would ping-pong between 35% and 45% CPU, but when I closed it it dropped to 10% then faded to 6-7% (I am still unhappy with that, but grumble).

I can see why this happens, but it seems like this state could cause mis-leading bug reports, etc.  I don't have any immediate suggestions on how to fix it, but my guess is that it probably doesn't get much attention, so there might be minor changes which would have an impact.