Humans can explain their reasoning when you ask them. The best we have with NNs is a fancy version of first-order sensitivity analysis.

Humans cannot really explain their reasoning. It has been shown that they usually invent an explanation that matches what has already been decided in their black-box.They mostly use post-hoc explanation, but cannot explain the true decision mechanism.

If you make them believe that they choose another decision, they will create another explanation on the fly.

This! Decision making is mostly "irrational". I wish it would be different.

Decision making in humans is a mix of slow and fast thinking. Fast thinking - like reading, writing, physical reflexes - requires little effort and is best suited for repetitive situations.

Slow thinking - such as reasoning - is more expensive so we only use it when we are faced with a new situation for which we have not 'cached' the answer. It entails the creation of a mental model and imagining outcomes. The exact mix of the two modes is tuned by evolution.

If all we had was the slow system we'd be much less efficient at survival.

Yes. Humans make mistakes, humans with computers make mistakes faster. Computers without humans make mistakes even faster.