On 18 July 1866, Lord Sefton married Hon. Cecil Emily Jolliffe (1838–1899), the fifth daughter of William Jolliffe, 1st Baron Hylton. They had five children:
Lord Sefton was appointed a Knight of the Garter in 1885. In 1886, he built Abbeystead House in the forest of Wyresdale, Lancashire as a 'private shooting lodge on a grand scale'. On his death in 1897, his titles passed to his eldest son, Charles.
The family seats were : Croxteth Hall, Lancs ; Abbeystead House, Lancs ; Sefton House, Belgravia Sq., London