Government cohesiveness is known to moderate retrospective voting. While previous work on this topic has focused on characteristics of the government, we build on the literature on clarity of responsibility and the literature on valence to argue that the extent to which government and opposition are ideologically distinct also moderates retrospective voting. Two alternative expectations follow from these two theoretical perspectives. While the clarity of responsibility framework leads to the expectation that a larger difference between government and opposition will strengthen retrospective voting, the valence literature presumes that retrospective voting is stronger when ideological differences are small. Using the data of the Comparative Study of Electoral Systems (CSES) project, we find evidence that is in line with the clarity of responsibility framework: the higher the degree of ideological polarization between government and opposition, the larger the effect of retrospective performance evaluations on the vote.