Port recent OOM-handling and memory leak fixes from bellard/quickjs

[past-due]

Fixes #1066

Ports:
bellard/quickjs@db3d3f0
bellard/quickjs@3dc7ef1 (except changes to build_backtrace)

[bnoordhuis]

One comment, otherwise LGTM.

[bnoordhuis]

This change looks kind of dubious to me. Ceteris paribus, it's better to have a backtrace.

[past-due]

I believe the reason this was part of the upstream commit is that all but one (?) of the reasons JS_NewString (JS_NewStringLen) might fail are due to an allocation / OOM situation.

EDIT: And since the case of an invalid message is already handled above:

    msg = JS_NewString(ctx, message);
    if (JS_IsException(msg))
        msg = JS_NewString(ctx, "Invalid error message");

The only case where msg is still an exception at this point should be if JS_NewStringLen failed due to allocation failure (OOM). Avoiding the call to build_backtrace in this case seems reasonable (it looks like it tries to allocate plenty of things itself)?

[bnoordhuis]

My thinking is that it's better to try and fail than not try at all, so can you move it outside the block again?

[past-due]

My thinking is that it's better to try and fail than not try at all, so can you move it outside the block again?

Done. 👍

Of note: Since it has diverged from bellard/quickjs, I did not fully audit build_backtrace to ensure it properly handles all memory allocation failures internally - would be worth looking at in the future.