Ordinary probabilities are real numbers, and ordinary entropy is a real number too. Building on ideas of Kontsevich, we develop an analogue of entropy in which both the probabilities and entropy itself are integers modulo a prime number p. While the formula for entropy mod p is quite unlike the formula for real entropy, we prove characterization theorems for entropy mod p and information loss mod p that are very closely analogous to the theorems over the real numbers, thus justifying the definition. We also establish a sense in which entropy mod p is the residue mod p of real entropy.