ASM x64 function pointer not returning the good value

An int is 32 bits, but rax is a 64-bit register. A function that returns int will place its return value in eax, which will typically zero out the high half of rax. So if cmp returns -1, which is the 32-bit number 0xffffffff, then rax will contain 0x00000000ffffffff. This is not a negative 64-bit number, so test rax, rax will not set the sign flag.

Try using test eax, eax as your test instead.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top