To date, much of the work on Responsible Innovation (RI) has focused on the ‘responsible’ part of RI. This has left the ‘innovation’ part in need of conceptual innovation of its own. If such conceptual innovation is to contribute to a coherent conception of RI, however, it is crucial to better understand the relation between responsibility and innovation first. This paper elucidates this relation by locating responsibility and innovation within Emmanuel Levinas’ phenomenology. It structures his work into three ‘stages’, each described in terms of their leading experience and objectivation regime. This analysis identifies a need for constant innovation of political and technological systems, originating from and motivated by our responsibility to others. It also shows the relation between responsibility and innovation to be threefold: foundational, ethical and structural. These insights could help RI to avoid some pitfalls of ‘regular’ innovation, and provide moral grounding for important aspects of RI.