Hot carrier based methods constitute a valuable approach for efficient and silicon compatible sub-bandgap photodetection. Although, hot electron excitation and transfer have been studied extensively on traditional materials such as Au and Ti, reports on alternative materials such as titanium nitride (TiN) are rare. Here, we perform hot hole photodetection measurements on a p-Si/metal thin film junction using Ti, Au and TiN. This material is of interest as it constitutes a refractory alternative to Au which is an important property for plasmonic applications where high field intensities can occur. In contrast to Au, a TiN/Si junction does not suffer from metal diffusion into the Si, which eases the integration with current Si-fabrication techniques. This work shows that a backside illuminated p-Si/TiN system can be used for efficient hot hole extraction in the IR, allowing for a responsivity of 1 mA/W at an excitation wavelength of 1250 nm and at zero bias. Via a comparison between TiN and other commonly used materials such as Au, the origin of this comparably high photoresponse can be traced back to be directly linked to a thin TiO2-x interfacial layer allowing for a distinct hot-hole transfer mechanism. Moreover, the fabrication of TiN nanodisk arrays is demonstrated which bears great promise to further boost the device efficiency.