Starting as early as the Spanish colonial caste system, the ancestors of modern Latinos faced discrimination that led Whites to view them as a non-White racial group. The discrimination Latinos faced resulting from the caste system limited their social mobility, helping create a belief that Latinos were incapable of assimilating into colonial society. Other types of formal and informal forms of discrimination against Latinos (e.g., the California Land Act, Operation Wetback, the Zoot Suit riots) had a similar effect, reinforcing beliefs about the inability of Latinos to assimilate as well as creating an impression that Latinos fail to adhere to Anglo-American norms. This chapter traces various historical institutions that have helped shape how White's perceive Latino identity and ultimately shape the way that animus is expressed toward Latinos today.