__malloc_margin was used as a variable name just because it's the name that avr-libc uses for the same purpose, namely to tune the minimum distance between the stack and heap during an allocation that the break to increase. By the way, you can see this same kind of check happen in avr-libc's implementation of malloc (which also takes sbrk's role.) Why is __malloc_margin 4K ? No really scientific reason. It's about 4% of memory, so it's not large, and it's completely inconceivable to me that someone would try to put local variable(s) approaching 4k in a stack frame on a system that has 96kb total memory. 4K may not be an appropriate default though; more thought should go into that. Having it in a variable like this allows it to be tuned at runtime; usually there would be a header that provides an extern declaration for it.

__malloc_margin was used as a variable name just because it's the name that avr-libc uses for the same purpose, namely to tune the minimum distance between the stack and heap during an allocation that the break to increase. By the way, you can see this same kind of check happen in avr-libc's implementation of malloc (which also takes sbrk's role.) Why is __malloc_margin 4K ? No really scientific reason. It's about 4% of memory, so it's not large, and it's not completely inconceivable to me that someone would try to put local variable(s) approaching 4k in a stack frame on a system that has 96kb total memory. 4K may not be an appropriate default though; more thought should go into that. Having it in a variable like this allows it to be tuned at runtime; usually there would be a header that provides an extern declaration for it.

I called abort() in a sketch and dumbed the following via an arm version of the binutils objdump with -dS command-line options.

I called abort() in a sketch and dumped the following via an arm version of the binutils objdump with -dS command-line options.

Regarding abort()

It occurs to me in some cases abort() may be an appropriate way to handle this. If you believe that you can characterize the system (or the sketch if you like) at runtime so well that malloc() failing can be regarded as a major oversight, then calling abort() in sbrk() could make sense. Or at least providing the option to do that.

In the comments you asked:

What does abort() exactly [do] on the Due?

Well, Newlib documentation says:

Before terminating your program, abort raises the exception SIGABRT (using ‘raise(SIGABRT)’).

and then further down:

Supporting OS subroutines required: _exit and optionally, write.

The SIGABRT and _exit parts are more or less standard behaviour. So it will call your signal handler for SIGABRT if you have one.

So then what does _exit do? It loops forever.

I called abort() in a sketch and dumbed the following via an arm version of the binutils objdump with -dS command-line options.

Here's the relevant abort() code:

00081b98 <abort>:
   81b98:   b508        push    {r3, lr}
   81b9a:   2006        movs    r0, #6
   81b9c:   f000 fb5c   bl  82258 <raise>
   81ba0:   2001        movs    r0, #1
   81ba2:   f7fe fec5   bl  80930 <_exit>
   81ba6:   bf00        nop

bl <raise> would in turn call your signal handler. and bl <_exit> calls exit which displays as:

00080930 <_exit>:

extern void _exit( int status )
{
   80930:   e7fe        b.n 80930 <_exit>

00080932 <_kill>:

    for ( ; ; ) ;
}

It appears the optimizer has combined some things here, but essentially _exit has an instruction that jumps (branches) to itself, equivelent to for(;;);.

I half-expected the abort(); code to disable interrupts before calling _exit(). If you do plan on abort() in your _sbrk() that might be something to think about. You may also want to provide a global flag that can be inspected from SIGABRT that allows the signal handler to do something diagnostically appropriate specifically for _sbrk() failures.
Perhaps sbrk should set a flag in a no-init data section and then set the watchdog timer and let it expire (or provide the SIGABRT handler with the information to do that). Lots of things to consider.