While the story eventually normalizes with both groups, you have somewhat of a difficult decision at the start of Persona Q: Shadow of the Labyrinth. Do you go with the Persona 3 crew, or the gang from Persona 4?
I know a few people who agonized over the situation, but for me it was simple -- Persona 4. While I think that P3 is technically a deeper game, the setup is more interesting, and the gloomier tone melds more with the SMT universe, I can't resist the cast of P4.
Yosuke, Chie, Yukiko, Teddie, Rise -- the Investigation Team is classic at this point and I couldn't give them up. On the flipside, while they do fit the more lighthearted tone of Shadow better, it's really interesting seeing the cast of P3 with the new animation style, interacting with the new characters and the entire cast. It's pretty much a Persona fan's dream.