FWIW, in PA, it can't be brought up in court as evidence that you refused to take a polygraph. Nationally, we have a right to remain silent, so I imagine it's not much different elsewhere, but IANAL, so there could be some tricks cops/prosecutors can play in other states. There's no connection to the fifth amendment.
