My scenario was that I have a NAS server on which I created a backupbot user, after the creation of my primary account, which was able to log in to initially create the backupbot user. After fiddling with sudo vim /etc/ssh/sshd_config, and creating the backupbot user, vim can create, at least on Ubuntu 16.04, and based on your ~/.vimrc config, a swap file left from your vim session's editing of /etc/ssh/sshd_config.
Check to see if: /etc/ssh/.sshd_config.swp exists, and if it does remove it and restart the sshd daemon:
$ sudo rm /etc/ssh/.sshd_config.swp
$ sudo service sshd restart
This magically resolved my issue. I had previously checked all my permissions and even the RSA fingerprints of the public and private keys. This is strange and probably a bug with sshd OpenSSH_7.4p1 Ubuntu-10, OpenSSL 1.0.2g 1 Mar 2016, specifically this version:
OpenSSH_7.4p1 Ubuntu-10, OpenSSL 1.0.2g 1 Mar 2016