What is the encoding technique called used to store negative numbers in the comouter's memory?

Negative integers are typically encoded using two's complement notation. That is, to convert the sign of any given integer, the bits are flipped (as per ones' complement) and 1 is added. Thus to convert the value +1 to -1 using 8-bit notation:

00000001 = 11111110 + 1 = 11111111

And reversing the process (-1 to +1):

11111111 = 00000000 + 1 = 00000001

In hexadecimal notation, the value 0x80 is the largest negative integer (-128) and 0x7F is the largest positive integer (+127). That is, 10000000 and 01111111 respectively.

Note that ones' complement was used in the past and while some systems still use it they are few and far between today. One of the problems with ones' complement is that +0 and -0 (00000000 and 11111111 respectively) would be treated as being two separate values, when the value 0 is neither positive nor negative. In twos complement, flipping the sign of zero becomes:

00000000 = 11111111 + 1 = 00000000

Note that the actual value is 100000000, but the most-significant bit (the overflow) is ignored because there can only be 8-bits in an 8-bit value. The same is true of 16-bit values, 24-bit values, etc. When all bits are filled with 1s, adding 1 wraps the value back to 0 again.