How the time reference of a sentence is processed based on the grammatical marking of the verb has already been explored in several languages with grammatical tense and aspect. It can also be grammatically expressed according to the reality status of the event (whether the event exists in time, realis mood, or not, irrealis). This study reports results from an acceptability judgment experiment in Paiwan, a Formosan language which exhibits a realis-perfective/irrealis distinction. By placing realis-perfective and irrealis markers after deictic past or future time adverbs and manipulating the grammaticality of the sentences, we asked which temporal concord violation (i.e., realis-perfective or irrealis) was harder to detect. The temporal concord violation of the realis-perfective marker induced greater processing difficulties (interactions between time reference and mood marking revealed lower accuracy rates and longer reaction time), but not the irrealis marker, in line with previous hypotheses. These processing difficulties may be partly due to the Paiwan realis mood marker which also encodes perfective aspect meaning. The reanalysis of the design of previous studies indicates that the interaction with perfective aspect also led to additional processing cost, suggesting that perfective aspect marking plays a crucial role in the processing of time reference.