Armstrong moved to Dereham in early October 1850 with Nellie, and his two daughters, Helen, aged 6, and Louise (Lilly), aged 2. Within a few weeks his first son, John, was born. A further son, Herbert (Bertie), in 1853, and daughter, Gertrude, in 1855, were to follow. As a thanksgiving for the safe birth of his son, he presented a Bible to the church to be placed on the eagle lectern. While he was dedicated to his parish work, Armstrong was also a family man; he had felt isolated from his relations when at Crowle. Dereham was somewhat nearer, being 100 miles from London, and the developing rail network made family visits much easier. He was particularly close to his father, whose opinion he always valued, and they exchanged weekly letters. In 1853, he felt that as his high church practices were becoming accepted in Dereham, he was in a strong enough position to introduce daily services in Lent and carry on with them thereafter. However, before embarking on this innovation, he was anxious to have his father's approval and wrote to ask his opinion. His reply was very diplomatic, saying that his son should do what he felt was right, but also that he should proceed with caution, partly because of all the extra commitment he was letting himself in for. When he was sorting his father's papers after his death, Armstrong found that all his letters to his father had been carefully saved. Sadly, they do not now survive.

In May of the year following the move to Dereham, his parents came to stay and were pleased with all they saw. This may have been the first time that they met their grandson and Armstrong's pride in the boy was partly due to the pleasure he knew an Armstrong heir would give his father. The next month he stayed with them in Southall and went up to London to see the Great Exhibition. In November he went to see his parents’ new home in Wimpole Street, and to see the lying-in-state of the Duke of Wellington and watch the funeral procession.