Abstract

Although hierarchical approaches are evidently important to reinforcement learning, most existing hierarchical RL models either do not involve automatically developing hierarchies (i.e., using pre-determined hierarchies; e.g., Dayan and Hinton 1993, Sutton 1995, Pre-cup et al 1998, Parr and Russell 1997, Dietterich 1997), or involve only domain-speci c processes. Models in the latter category rely on domain-speci c knowledge or procedures and are thus not generic or autonomous; for example, Lin (1993), Moore and Atkeson (1994), and Singh (1994). The problems of such hierarchies include in exibility (because the characteristics of the domain can change over time) and lack of generality (because domain-speci c hierarchies most likely vary from domain to domain). This is true even when limited learning is used to ne tune mostly pre-determined hierarchies (e.g., Parr and Russell 1997, Dietterich 1997)