When did Lee Harvey Oswald first break the law?

Generally speaking, you don't need to complete a crime to be charged with it, as you can simply be charged with attempting to commit the crime. And even then, you don't need to pull the trigger and miss; it's enough if you intend to commit the crime and take substantial steps toward doing so.

An attempt to commit crime is composed of two elements: First, the intent to commit; second, a direct, ineffectual attempt towards its commission. It may be further defined as consisting, perhaps, of three elements; the intent to commit the crime, the performance of some act toward the commission of the crime, and the failure to consummate its commission. It falls short of the thing intended.

Shipp v. State, 81 Tex. Crim. 328, 333 (1917).

If the same events unfolded today, they would be prosecuted under the federal attempted assassination statute, so the gunman would commit an offense once he had taken any step “constituting a substantial step in a course of conduct planned to culminate in his commission of the crime.” United States v. Duran, 96 F.3d 1495, 1508 (D.C. Cir. 1996).

Duran had purchased a rifle, ammunition clips, and an overcoat large enough to conceal the rifle (all of which he would eventually use in firing toward a man on the White House lawn who resembled the President), then traveled to Washington, D.C. and stood by the White House gate with the rifle and ammunition concealed. The jury had ample evidence, then, from which to conclude that Duran purchased weapons, ammunition, and the overcoat, traveled from Colorado Springs to Washington, D.C., and stood in front of the White House for several hours with the weapons and ammunition on his person, with the purpose of assassinating the President. Furthermore, the jury was entitled to find that these acts constituted "substantial steps" taken in pursuit of this objective.

I don't know that much about Oswald, but my understanding was that he bought the gun months earlier, for a totally separate crime, so the purchase likely wouldn't trigger criminal liability in connection with the JFK assassination.

Instead, I'd think that his request for a friend to drive him to pick up the gun would constitute a substantial step sufficient to trigger criminal liability. If not, bringing the gun into the depository would certainly be enough.

This answer expands on the other answers, which I don't disagree with, to answer a secondary part of the question.

I understand that planning and/or intending to commit a murder is in itself a crime, but I'm more interested in what physical act he might do that was irrefutably a crime (without the need to delve into his mind).

None.

All relevant crimes have an intent requirement.

Even once he fired the shot, there is an intent requirement that requires delving into his mind. If the case had gone to trial, he could have argued, for example, that he was just cleaning his rifle and accidentally happened to fire shots at the President. And, if a jury believed him, that would provide a legal basis to acquit him.

Proof is distinct from when someone has first committed a crime. It isn't terribly uncommon for law enforcement to lack proof that someone committed a crime for decades after it is committed, or even never. A huge share of all crimes are never "cleared" (i.e. "solved or rendered moot").

When he first took a step towards the murder with the intention of committing that murder

You say “without the need to delve into his mind”, but that is precisely the finder of fact’s role. For any crime with an intentionality element, the jury must “delve into his mind”; just as they would have to do if he had been arrested after the assassination — they would have to determine if he intended to kill the President by firing a rifle at him three times.

So until he squeezed the trigger he hadn't committed a crime?

No, he had committed a crime when a) he decided to murder the President and b) he had taken a concrete step towards fulfilling that intention. At that point he had committed attempted murder; once the President died, he had committed murder.

That is the law. At what point in the chain of events, the prosecution has enough evidence to convince a jury beyond reasonable doubt has a practical relevance, but many crimes are committed which can’t be proved every day. Different juries might be convinced at different points, and that’s fine — reasonable people can differ on the facts, but the law is what it is.

If a police officer has seen him with a rifle that morning, could he have (legally) stopped him?

Once the police officer had probable cause they could effect an arrest.

One definition of the standard derives from the U.S. Supreme Court decision in the case of Beck v. Ohio (1964), that probable cause exists when “at [the moment of arrest] the facts and circumstances within [the] knowledge [of the police], and of which they had reasonably trustworthy information, [are] sufficient to warrant a prudent [person] in believing that [a suspect] had committed or was committing an offense.”

Is it suspicious that Lee bough or owned a gun? No, lots of people do that. Is it suspicious that he concealed it in a bundle of curtain rods to bring it to his place of work which was right next to the route that the Presidential motorcade would pass? Probably. Is it suspicious that he set himself and the gun up in a window that was a perfect sniper’s nest to shoot at the said motorcade? Yes.

First time if he bought a gun with the intent to shoot somebody. If he bought a gun for target shooting at a gun club it would have been legal. At this point if a police officer had arrested him, it would have been awfully difficult to prove he committed a crime (attempted assault); attempted murder would have been even harder to prove.

With every step further to the murder, it would have been more obvious and easier to prove he wanted to commit a crime. Setting up a point to shoot from would have been attempted assault beyond reasonable doubt.