Wittgenstein composed five original musical fragments during his transitional middle period, in which he employs musical notation as a means by which to convey his philosophical thoughts. This is an overlooked aspect of the importance of aesthetics, and musical thinking in particular, in the development of Wittgenstein’s philosophy. We explain and evaluate the way the music interlinks with Wittgenstein’s philosophical thoughts. We show the direct relation of these musical examples as precursors to some of Wittgenstein’s most celebrated ideas (the push beyond the inner/outer picture, the problem of rule-following, and aspect perception). We also show how important broad aesthetic notions such as style and composition concerning the very possibility of philosophizing were to Wittgenstein.