ASM x64 function pointer not returning the good value

An int is 32 bits, but rax is a 64-bit register. A function that returns int will place its return value in eax, which will typically zero out the high half of rax. So if cmp returns -1, which is the 32-bit number 0xffffffff, then rax will contain 0x00000000ffffffff. This is not a negative 64-bit number, so test rax, rax will not set the sign flag.

Try using test eax, eax as your test instead.