Is your home dir encrypted? If so, for your first ssh session you will have to provide a password. The second ssh session to the same server is working with auth key. If this is the case, you could move your authorized_keys to an unencrypted dir and change the path in ~/.ssh/config.
What I ended up doing was created a /etc/ssh/username folder, owned by username, with the correct permissions, and placed the authroized_keys file in there. Then changed the AuthorizedKeysFile directive in ~/.ssh/config. to :
AuthorizedKeysFile /etc/ssh/%u/authorized_keys
This allows multiple users to have this ssh access without compromising permissions.
