[release/10.0] Enhance createdump to detect alt stack execution

[hoyosjs]

Fixes .NET 10 version of #126981

Description

createdump's UnwindNativeFrames fails to capture the original thread stack when a crash occurs on a thread using an alternate signal stack. The native unwinder's monotonic-SP guard breaks we cross crosses the signal trampoline back to the original stack, because the SP legitimately decreases. This causes the unwinder to stop early, omitting the original stack memory from the dump.

The fix uses libunwind's unw_is_signal_frame in PAL_VirtualUnwindOutOfProc to detect signal trampoline frames. When a signal frame is detected, UnwindNativeFrames allows a SP decrease, enabling the unwinder to cross back to the original stack and capture its memory.

Customer Impact

Minidumps collected via createdump for crashes on alternate signal stacks are missing the original thread's stack memory. This makes the dumps incomplete and difficult to debug - native frames below the signal handler are absent from the stack walk, and you can only get the managed stack separately via clrstack. Watson and WinDBG both fail to do this automatically.

Regression

We added the SP monotonic check ~7 years ago to prevent corruption unwinding issues.

Testing

The following scenario was tested as proxy of customer's issue: Pinvoke into native library with some frames before hitting a nullref on a secondary thread. Pre-fix the repro shows the early bail unwind. The fix captured the full unwind across the signal trampoline, identifies the libc trampoline, and includes original stack memory in the dump. I also validated the fix works both with and without dwarf unwind info in the crashing native library.

Risk

Low. The change is narrowly scoped to createdump's native unwind path and some of the DAC's lazy state machine unwinding. unw_get_proc_info reuses the same cache unw_step populates - no additional remote memory reads. If unw_is_signal_frame returns false due to some libc not marking the trampoline correctly, behavior is identical to before - no regression.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR fixes createdump’s native stack unwinding when a crash occurs on an alternate signal stack (sigaltstack/SA_ONSTACK), ensuring the unwind can cross the signal trampoline back to the original thread stack so the dump includes the missing stack memory.

Changes:

• Extend PAL_VirtualUnwindOutOfProc to report whether the current frame is a signal trampoline using libunwind’s unw_is_signal_frame.
• Update createdump’s UnwindNativeFrames monotonic-SP guard to allow a one-time SP decrease immediately after a signal trampoline frame.
• Plumb the new PAL API parameter through createdump’s PAL shim/wrapper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/coreclr/pal/src/exception/remote-unwind.cpp	Detect signal trampoline frames (via unw_get_proc_info + unw_is_signal_frame) and return the result to the caller.
src/coreclr/pal/inc/pal.h	Update PAL_VirtualUnwindOutOfProc declaration to include the isSignalFrame out parameter.
src/coreclr/debug/createdump/threadinfo.cpp	Relax SP monotonicity check to permit crossing from alt stack back to original stack when the prior frame is a signal trampoline.
src/coreclr/debug/createdump/createdumppal.cpp	Update the dynamically-resolved PAL function pointer typedef and wrapper to match the new signature.

[janvorli]

LGTM, thank you!

[steveisok]

/ba-g Known issues