Roughly speaking HSM is supposed to ingest or generate some secret material (key) and then never export them through the command interface. The keys can only be used according to their configured policy. In order to make sure the keys are really non-exportable, HSM hardware is designed to be tamper-resistant and resist physical key extraction attempts.
Clearly, tamper-resistance is required to protect the raw key material from a physical attacker. However, I am considering what actual security advantage is gained by doing so. Assume that physical tamper-resistance measures are perfect. Then there are roughly two classes of attackers:
Remote attackers who can only talk to the HSM through the command interface. Here, physical security is unimportant, and the key is protected as long as the firmware refuses to export the key.
Local attackers who have physical access. Then assuming the physical anti-tamper measures are perfect, sure the attacker can no longer export the raw keys. However, the attacker can unplug the application server, plug in their attacking device, and talk to the HSM directly through the command interface and use the keys however they want.
Now, we can certainly configure key-usage policies such as "sign but not decrypt", but since our own application need to use that key too, it follows that any (application-level) useful usage of the said key must be allowed by the policy. Thus, while the attacker is technically prevented from e.g., using the RSA2048 signing key as a P-256 scalar then encrypting data with it, the attacker won't be interested in doing this anyway, no?
It seems like what we really need is delay of access (so transient physical access where the attacker is forced to leave in a short amount of time does not lead to key compromise) and tamper-evidence, and that can be cheaply accomplished by e.g., reinforcing the enclosure and does not require expensive tamper-resistant hardware.
Of course, HSMs are used for a reason. What am I missing?