He married firstly, Lady Elizabeth Joanna de Burgh (22 February 1826 – 26 February 1854), daughter of Ulick de Burgh, 1st Marquess of Clanricarde, on 17 July 1845. She was the granddaughter of former Prime Minister George Canning. They had six children:
^"Christmas Festivities". Leeds Mercury. West Yorkshire, England. 26 September 1872. Archived from the original on 28 June 2017. Retrieved 30 October 2014 – via Genes Reunited. ... the Earl of Harewood and Mr. William Middleton Esq., Chairman of the Leeds Board of Guardians, supplied a large quantity of evergreens. At the Infirmary those patients who were able...
^"Workhouse, St James Hospital". Leodis – A photographic archive of Leeds. UK: City of Leeds. Retrieved 30 October 2015. ....the newly built workhouse, which later became part of St James Hospital. The foundation stone was laid on Easter Monday, April 5th 1858 by William Middleton, Chairman of the Board of Guardians. This building is now occupied by the Thackray Medical Museum.