using a ternary operator inside of a preprocessor

I don’t think you understand how a macro works; you don’t declare a new variable max2 inside it. The preprocessor is a simple text substitution system.

Your macro expands to:

int maximum = (int max2, a>=b ? max2 = a : max2 = b);

This is a syntax error. A max macro is very simple, all you need is:

#define max(a, b) ((a) >= (b)) ? (a) : (b)

This will then expand to:

int maximum = ((a) >= (b)) ? (a) : (b);

Which will give you the result you want.

That said, there already are macros named min and max in your <stdlib.h>… I think.