Supermassive black holes are found in most galactic nuclei. A large fraction of these nuclei also contain a nuclear stellar cluster surrounding the black hole. Here we consider the idea that the nuclear stellar cluster formed first and that the supermassive black hole grew later. In particular we consider the merger of three stellar clusters to form a nuclear stellar cluster, where some of these clusters contain a single intermediate-mass black hole (IMBH). In the cases where multiple clusters contain IMBHs, we discuss whether the black holes are likely to merge and whether such mergers are likely to result in the ejection of the merged black hole from the nuclear stellar cluster. In some cases, no supermassive black hole will form as any merger product is not retained. This is a natural pathway to explain those galactic nuclei that contain a nuclear stellar cluster but apparently lack a supermassive black hole; M33 being a nearby example. Alternatively, if an IMBH merger product is retained within the nuclear stellar cluster, it may subsequently grow, e.g. via the tidal disruption of stars, to form a supermassive black hole.