Tobit Potter's life was never easy. Born in 1709 as the illegitimate son of the Vineyard Wampanoag Elizabeth Uhquat, Potter appears to have spent most of his childhood shuttling back and forth from the mainland colonial household where his mother was an indentured servant, to other English homes where he himself worked. Uhquat finally gained her release some time after Potter's ninth birthday, but that brought no end to her son's troubles. She could not bear the cost of supporting him and therefore committed the boy to serve Edward and Mary Milton of Tisbury while she returned to her natal village of Christiantown. The Miltons, exceptionally good employers, instructed Tobit in reading and Christianity, and he proved receptive. The missionary Experience Mayhew wrote of Potter, “he said once when he came from Meeting that hearing the Minister mention those Words, If my Father and Mother forsake me, the Lord will take me up; he was glad to hear this, for that he thought he had no body to take care of him.”

Doubtless there were many times when Potter invoked the comfort of this passage. He fell ill after four years with the Miltons, prompting his masters to transfer him to yet another English family. However, the change of environment did nothing to improve the boy's health, so he was sent to Christiantown to be nursed by his impoverished mother.