A lot of philosophical energy has been devoted recently in trying to determine if mathematics can contribute to our understanding of physical phenomena. Not many philosophers are interested, though, if the converse makes sense, i.e., if our cognitive interaction (scientific or otherwise) with the physical world can be helpful (in an explanatory or non-explanatory way) in our efforts to make sense of mathematical facts. My aim in this paper is to try to fill this important lacuna in the recent literature. (...) My answer to the question of this paper is negative. As I will argue, there are serious problems with the main reasons for believing in the first place that it is possible to have physical understanding of mathematical facts. (shrink)