Paul Martin and the slippery slope

Here's yesterday's vaunted Paul Martin quote (cribbed from Sullivan):

"We will be influenced by our faith but we also have an obligation to take the widest perspective -- to recognize that one of the great strengths of Canada is its respect for the rights of each and every individual, to understand that we must not shrink from the need to reaffirm the rights and responsibilities of Canadians in an evolving society."
"The second argument ventured by opponents of the bill is that government ought to hold a national referendum on this issue. I reject this - not out of a disregard for the view of the people, but because it offends the very purpose of the Charter."
"The Charter was enshrined to ensure that the rights of minorities are not subjected, are never subjected, to the will of the majority. The rights of Canadians who belong to a minority group must always be protected by virtue of their status as citizens, regardless of their numbers. These rights must never be left vulnerable to the impulses of the majority."
"We embrace freedom and equality in theory, Mr. Speaker. We must also embrace them in fact."

Can somebody explain to me why this logic does not also apply to a minority of Canadians of a particular religious persuasion who wish to practice, say, polygamy? Let me answer my own question partly: the rest of PM's speech addresses the strong legal basis for legalizing gay marriage, particularly recent court decisions that have dealt specifically with the question of "civil unions" versus full-blown "marriage."

I can accept gay marriage on the basis of PM's argument (though I prefer to justify it on purely libertatian grounds). But is there no danger that the this logic will lead to a future breakdown in the definition of marriage? That is, if we accept the argument that the Charter enshrines the rights of a minority group against "the impulses of the majority", what of members of the Muslim community, whose religion specifies the right of a man to have multiple wives, or the beliefs of the Mormons in the polygamist colony of Bountiful, BC?

I use polygamy as an example because it is a fear-stoking topic of the far right (and because I can't imagine the boundaries of marriage being pushed much further). But I'm drudging it up again because of one of Martin's key phrases and one of MacDuff's posts today.
PM: "...the need to reaffirm the rights and responsibilities of Canadians in an evolving society." If Trudeau and Kiefer Sutherland's grandfather were debating legalizing homosexuality -- full stop -- not so long ago, where will we be in another 35-40 years in an "evolving society"? Will there come a day when polygamy rights activists grab the public's imagination, or (more likely) the support of the courts? And will the gay marriage act be the impetus?

Not scaremongering, here -- just thinking aloud.