Is your home dir encrypted? If so, for your first ssh session you will have to provide a password. The second ssh session to the same server is working with auth key. If this is the case, you could move your authorized_keysauthorized_keys to an unencrypted dir and change the path in ~/.ssh/config.
What I ended up doing was createdcreate a /etc/ssh/username folder, owned by username, with the correct permissions, and placed the authroized_keysauthorized_keys file in there. ThenThen changed the AuthorizedKeysFile directive in /etc/ssh/config. to :
AuthorizedKeysFile /etc/ssh/%u/authorized_keys
AuthorizedKeysFile /etc/ssh/%u/authorized_keys
This allows multiple users to have this ssh access without compromising permissions.