Abstract

Although advances in the underlying theory of a subdiscipline of AI can result in impressive increases in the performance of systems that employ such an underlying theory, this sometimes seems to be almost "by accident." The reason for this impression is that the impressive new advance in performance sometimes seems to be a feature merely of the specific example tests that are being demonstrated. And this impression is further strengthened when one notes that, in general, a localized theoretical advance is only rarely suflicient to increase the overall performance of any complex system. As a result of all these considerations, researchers who make theoretical advances are left desiring some method that would demonstrate that the advance really does have general, overall positive consequences for their system's performance.