Run the command from a non-X terminal.
CTRL+ALT+F2 should get you to a normal console. Login as your normal user, and then run the command there.
xmodmap only affects X sessions. so it needs DISPLAY to work outside the X server, like this:
DISPLAY=:0 xmodmap