You're still seeing karma as a means of restoring fairness to the universe, it's not. It's a means of teaching, it is the great guess and check table of existence. When a mind accumulates karma, it is because the mind acted the best they could given their understanding of the situation, and good or bad they are still forced to reap what they sow. This is more basic than people understand it as, mistakes are possible and righteousness is dubious. What you "deserve" can often still feel totally unfair.

Let me draw an example. Do you think karma would simply turn off when you're blacked out drunk? Are you no longer responsible for your actions? Does your karma turn off because your emotions get too intense to think rationally and you, for example, act violently and go red and come back and realize you've hurt someone? In a court room, there's such thing as not guilty for reason of insanity. In life, all actions come back around.