We use deep Chandra and HST data to uniquely classify the X-ray binary (XRB) populations in M81 on the basis of their donor stars and local stellar populations (into early-type main sequence, yellow giant, supergiant, low-mass, and globular cluster). First, we find that more massive, redder, and denser globular clusters are more likely to be associated with XRBs. Second, we find that the high-mass XRBs (HMXBs) overall have a steeper X-ray luminosity function (XLF) than the canonical star-forming galaxy XLF, though there is some evidence of variations in the slopes of the sub-populations. On the other hand, the XLF of the prototypical starburst M82 is described by the canonical powerlaw (αcum ∼ 0.6) down to LX ∼ 1036 erg s−1. We attribute variations in XLF slopes to different mass transfer modes (Roche-lobe overflow versus wind-fed systems).