How did the mass of 108−1010M⊙ super-massive black holes at the center of massive galaxies in the local Universe build up? Did the bulk of the growth happen in an optically luminous AGN phase? Or did a substantial fraction of SMBH growth occur in a dusty, obscured phase, visible as a luminous infrared galaxy? Has there been substantial SMBH growth in a low luminosity or radiatively inefficient regime after the more luminous AGN phase? These are particularly important questions, given the tight relationship between the mass of galaxy bulges and their SMBHs, suggesting that the formation and evolution of galaxies and SMBHs are intimately linked. We use the multi-wavelength data in the NDWFS Boötes field to address this issue. We have performed an x-ray stacking analysis of ~20,000 red galaxies at z = 0.2−1 to show that the average nuclear accretion rates in these sources are low and decreasing with time. Given the long timescale, significant SMBH mass growth could occur in this regime. We also investigate the nature of an extreme, obscured population of AGN-dominated luminous infrared galaxies which are likely to host SMBHs undergoing a period of rapid and substantial growth.