Standard lore holds that magnetic forces are incapable of doing mechanical work. More precisely, the claim is that whenever it appears that a magnetic force is doing work, the work is actually being done by another force, with the magnetic force serving only as an indirect mediator. However, the most familiar instances of magnetic forces acting in everyday life, such as when bar magnets lift other bar magnets, appear to present manifest evidence of magnetic forces doing work. These sorts of counterexamples are often dismissed as arising from quantum effects that lie outside the classical regime. In this paper, we show that quantum theory is not needed to account for these phenomena, and that classical electromagnetism admits a model of elementary magnetic dipoles on which magnetic forces can indeed do work. In order to develop this model, we revisit the foundational principles of the classical theory of electromagnetism, showcase the importance of constraints from relativity, examine the structure of the multipole expansion, and study the connection between the Lorentz force law and conservation of energy and momentum.