Currently two planetary systems around pulsars are known - discovered by the pulse timing technique, an indirect method. These planets were a surprise and gave rise to diverse planet formation scenarios, some of them very different to the common planet formation models around solar type stars and thus physically very interesting. Furthermore, neutron star planets are not only interesting themselves but also to study properties of the poorly understood neutron stars. After a summary about the current state of pulsar planets and the theoretical formation models, we present our own direct-imaging search for thermal emission of neutron star planets using the VLT. The project sample includes the fascinating radio-quiet isolated neutron stars, which are some of the closest and probably youngest neutron stars we know. Companions around them can only be found by direct imaging. Detecting planets around neutron stars by direct imaging differs significantly from using this technique for other, e.g. solar type, stars. As great advantage there is no need to reject the starlight of the primary.