Polygamy and Adultery

I've been keeping merely a mildly interested eye on the Mormon sect of polygamists that were broken up recently in Texas. I've seen some of the wives on Larry King and have read several articles. Most recently I read this article on Triablogue.

What interests me is the notion that polygamy is rightly vilified as wrong in much of the American culture. It's certainly illegal. Adultery is also illegal in most states, although the definition of adultery differs slightly. However, the ban on adultery isn't enforced. What makes us think that polygamy can be enforced aside from the fact that civil marriage at least requires a license.

I'm no polygamist and I think polygamy is wrong. But let my submit this:

1a) Polygamists, by definition, commit to the marriages they make.
1b) Adulterers, by definition do not commit to relationships.
2a) Evidentially, the families of polygamists typically raise their children to be otherwise very law-abiding and responsible citizens.
2b) Evidentially, the children of adulterers abandoned to one parent or another or grandparents or multiple homes raise a high percentage of their children to be troubled citizens.

So, which one is worse civilly? They're both wrong morally, but one seems to be more civilly tolerable than the other. Why would we as a culture put up with adultery while we condemn polygamy?