An autonomous agent in the real world should learn its own sensor-motor coordination through interactions with the environment; otherwise the behaviors can not be grounded and they can easily be inappropriate in the variety of the environment. The sensor-motor signals are usually complex time sequence, therefore the cognitive action system of the agent has to handle them.In this paper, we propose a computational model of the cognitive action system that consists of a sensor space HMM-SOM, a motor space HMM-SOM and (...) connection mapping between the two HMM-SOMs. A HMM-SOM can be recognized as a set of HMMs that are placed in a SOM space. It can model a set of complex time series signal in a self-organizing manner.We apply this HMM-SOM based cognitive action system on vision-motion and auditory-articulation signals. The experimental results show that this system is basically capable of constructing sensor-motor coordination structure in a self-organizing manner, handling complex time series signals. (shrink)