We analyze the cross-correlation function of the soft and hard X-rays of the atoll source 4U 1636-53 with RXTE data. The results show that the cross-correlations evolve along the different branches of the color-color diagram. At the lower left banana states, we have both positive and ambiguous correlations, and positive correlations are dominant for the lower banana and the upper banana states. The anti-correlation is detected at the top of the upper banana states. The cross-correlations of two atoll sources 4U 1735-44 and 4U 1608-52 have been studied in previous work, and the anti-correlations are detected at the lower left banana or the top of the upper banana states. Our results show that, in the 4U 1636-53, the distribution of the cross-correlations in the color-color diagram is similar to those of 4U 1735-44 and 4U 1608-52, and confirm further that the distribution of cross-correlations in color-color diagram could be correlated with the luminosity of the source.