Cocoa is an important agricultural product that plays a crucial role in local communities in South America. In Brazil, it is traditionally grown in agroforestry systems, which are more sustainable and contribute to biodiversity conservation. However, the recent expansion of intensive monocultures in tropical forests poses significant threats to this activity. Using historical data on land use and cocoa productivity at the municipality level from Brazil’s primary cocoa-producing states, we show that maintaining and restoring forest cover are positively correlated with cocoa productivity, particularly in areas with less anthropogenic disturbance. This highlights the dependence of cocoa production on ecosystem services. Recent data reveal that in municipalities where local agriculture is less reliant on cocoa, only larger farms have benefitted from increased forest cover, probably due to their greater dependence on fragments of natural habitat for ecosystem services. In municipalities that are more reliant on cocoa, the effects of forest cover were not detected, while strong negative effects of forest fragmentation were observed in both small- and large-scale farms. We emphasize the importance of preserving natural forests near cocoa plantations to optimize productivity in Amazon and Atlantic Forest agroforestry, especially in deforested areas.