The skills that engender competence in a particular domain are often the very same skills necessary to evaluate competence in that domain—one’s own or anyone else’s. Because of this, incompetent individuals lack what cognitive psychologists variously term metacognition, metacomprehension, or self-monitoring skills. These terms refer to the ability to know how well one is performing, when one is likely to be accurate in judgment, and when one is likely to be in error.
Several lines of research are consistent with the notion that incompetent individuals lack the metacognitive skills necessary for accurate self-assessment. Work on the nature of expertise, for instance, has revealed that novices possess poorer metacognitive skills than do experts. In physics, novices are less accurate than experts in judging the difficulty of physics problems. In chess, novices are less calibrated than experts about how many times they need to see a given chessboard position before they are able to reproduce it correctly. In tennis, novices are less likely than experts to successfully gauge whether specific play attempts were successful.
We propose that those with limited knowledge in a domain suffer a dual burden: Not only do they reach mistaken conclusions and make regrettable errors, but their incompetence robs them of the ability to realize it.
~ From “Unskilled and Unaware” by Justin Kruger and David Dunning
Learning a martial art is inevitably a process of trial and error. To a limited degree, we are all inventors of our own unique style of martial arts.
Some ambitious individuals choose to go further. Rather than building on the experiential framework provided by a living martial arts expert, these innovators attempt to create a superior new system from first principles.
Is it harmless creative expression, or dangerous folly? How do we distinguish the brilliant martial inventor from the incompetent Kung Fu crackpot? Security expert Bruce Schneier suggests the following approach:
When someone shows me a [new style] by someone I don’t know, my first question is, “What has the designer broken?” Anyone can design a security system that he cannot break. So when someone announces, “Here’s my security system, and I can’t break it,” your first reaction should be, “Who are you?” If he’s someone who has broken dozens of similar systems, his system is worth looking at. If he’s never broken anything, the chance is zero that it will be any good.
I agree with Bruce in theory. In practice, I do not know how to demonstrate that a martial art style has been conclusively broken. What do you think?