In October 1856, Thomas Roberts was discharged from the 17th Lancers Army Regiment after twenty five years’ service. He and his wife Esther and their five year old daughter Ann Elizabeth Roberts, then settled in Brighton, living at No 16 West Hill Street.
Thomas worked as a domestic servant, most probably for one of the towns wealthy residents. On 26th February 1860, Frederick Goodacre Roberts, Thomas and Esther’s son was born. He sadly died aged one year, on the 18th March 1861. The 1861 census shows the Roberts family living at No 16 West Hill Street. Thomas, aged 47, a servant and Chelsea Pensioner, Esther aged 43, was a schoolmistress. Their daughter Ann was then eight years old.
By early 1863, Esther was falling ill to tuberculosis and it is rumoured that Thomas might have had an extra marital relationship with his future wife Mary Ann Walder. On the 14th November 1863, Esther Roberts died at No 19 Viaduct Terrace, London Road in Brighton, leaving Thomas with a 12 year old daughter. In early 1864, he and his daughter left Brighton for good and moved to Stoke Newington in London where he wed Mary Ann Walder in July 1864. She had given birth to an illegitimate child in December 1863.