Rumors  be damned ... a frustrated Joe Simpson is telling friends and  colleagues the stories that he's switched teams are absurd.

There have  been several reports out recently that Joe has told his family he's gay and  that's why his marriage fell apart. There are also reports about several guys  he's rumored to be dating/hooking up with.

Sources who speak with Joe  tell TMZ ... Jessica Simpson's daddy says it ain't so ... that  the stories are "ridiculous."

The people we spoke with are split -- some  believe him, and some think he's just not ready to come out publicly.