Shortly after William Adams’ death, his son Joseph was summoned by Shogun Hidetada. Fulfilling his own father's promise to Adams, Hidetada conferred on the young child Adams’ title, his samurai privileges and the Hemi estate, including the villages, rice fields and servants.

Adams’ illegitimate children fared less well. Richard Cocks records that in February 1621 Coshuro, Adams’ interpreter, came with her child Cowjohns to claim financial support. He gave her two tayes and offered to pay school fees, if the child were put under English guardianship. Again, in April of the same year, Cocks paid out two tayes and one mas for a kimono each for another who claimed to be Adams’ child and again to Cowjohns.

Richard Cocks’ nemesis came in the shape of Richard Fursland, a severe disciplinarian who had been posted by the English East India Company to Batavia [today's Jakarta], as President of its Council of Defence. Fursland had been put in charge of dismissing or punishing all incompetent or degenerate employees in the Moluccas and in Banda, Sumatra and Pattani. When Fursland turned his attention to the Hirado factory, he was shocked by the reports of extravagant feasts, dancing girls, concubines and illegitimate children. In the spring of 1622, Fursland recalled Cocks, Eaton and Sayers to Batavia for punishment, but Cocks simply ignored the command. Only Japan's monsoon season prevented a furious Fursland from immediately despatching a vessel to Hirado.

Cocks’ reprieve did not last long. In the summer of 1623, one of Fursland's vessels, the Bull, captained by Joseph Cockram, sailed into Hirado Port. Captain Cockram handed Cocks a letter from Fursland repeating his order to report to Batavia. The captain had specific orders to inspect the factory's accounts and was astonished at the enormous deficit. He reported back to Fursland that there were no funds to maintain the factory any longer and, on 25 July 1623, the Council of Defence decided to close it down.

Cockram announced the Bull would depart for England on 22 December 1623. On the appointed day, the English factors’ concubines, illegitimate children, Lord Matsura himself, several Dutch friends and many townsmen with their wives and families all came down to the wooden quayside to say farewell.