Paying Attention?

Here is a curious English construct I recently encountered:

Have I not been paying attention?

Yes, you have not been paying attention.

No, you have not been paying attention.

According to some handy-dandy reference guide, the question is in the "present perfect contiguous negative interrogative" case. That's the easy part.

Here's the hard part: What is the difference between these two answers? They both seem to say the opposite of each other, and they both seem to say that the questioner has not been paying attention...curious!

Of course, if a person were trying to say that the questioner HAS been paying attention, they could say,
"Yes, you have been paying attention." Or "No, you have been paying attention." Curiouser!!

Is there an English major out there that can make sense of this?