Abstract

Advances in the underlying theory of a subdiscipline of AI can result in an apparently impressive improvement in the performance of a system that incorporates the advance. This impression typically comes from observing improved performance of the new system on some test problems. However, the improvement in performance may be for only the problems used in the testing, and performance on other problems may be degraded, possibly resulting overall in an degradation of the system’s performance. This comes about typically when the incorporation of the new feature increases the resources required overall, but the feature has bene- fits on only some problems (e.g., those used to test the new new system). In general, a localized theoretical advance is only rarely sufficient to increase the overall performance of any complex system. Therefore, researchers who make theoretical advances, also need some way to demonstrate that an advance really does have general, overall positive consequences for system..