Problems with online

When i play online with my friends, the servers are strange. With one friend we can all join however when one of my other friends sets up a server only a few of us can join and others are excluded. We just get kicked from the server or are rejected before we can even join. I was wondereing whether anyone knew why this was and if there was anything we could do about it so that we could all play on the other server.

Any help would be greatly appreciated.