Currently if custom match has already started, I can join party and be there in observer slot but cannot actually view game.

I think allowing observer to join game at later point is useful, I tend to waste lot of time in lobby when my friends are already in custom game.