Comparing Operands

When reverse engineering binary, it is quite often we encounter branch instructions such as cmp and test.  The idea of comparison is really performing a subtraction of the 2nd opperand against the 1st opperand and dropping the result and setting the sign flag, overflow flag, signed flag, and zero flag.  Today I want to share how to determine the potential value of the 1st and 2nd opperand by looking at the flags.  It is actually very simple.

for signed comparison
let 1st opperand = x and 2nd opperand = y. 
let cmp x y == x-y

zero flag is set when two opprand are identical in value. zero flag = 1

sign flag is set when the result of x - y is negative.  Therefore x < y, sign flag = 1

to extract anything useful from the overflow flag, we have to also look at the sign flag as well.  When the overflow flag is set, it is notifying us the result of x - y have result in an overflow.  The fact that sign flag is 0, this tells us the result of x - y is positive.  In order to get an overflow positive value, it must be -x + -y such that it integer wrapped around to positive.  Therefore -x - y can result in the overflow, and this lets us know the 1st opprand is negative, and the 2nd opprand is positive.

so if we have both overflow flag = 1, and sign flag = 1, then it is telling us x - y resulted in a negative value, and knowing the overflow flag is set, we know that x + y such that it wrapped around to negative.  So therefore, y must be negative, to achieve x - -y == x + y.  Therefore, this tells us 1st opprand is positive, and 2nd opprand is negative.

for unsigned comparison (both opprand contains only positive values)

if zero flag is set, then opprand 1 and opprand 2 are equal.

if carry flag is 0 (no overflow occured for unsigned), and zero flag is 0, then it tells us x - y resulted in a positive value.  So the 1st opprand must be bigger than 2nd opprand, X > Y to achieve this result.

if carry flag is 1 (overflow occured for unsigned), and zero flag is 0, it tells us a small X - a large Y, such as 1 - 10000000000 would result in a negative value, but since there is no negative values for unsigned, it gets wrapped around starting at 0, so therefore X < Y.