He married Ada Isabella Williams in 1901. The marriage produced four recorded children. His elder son, Arthur Dyfrig James, became headmaster of his father's alma mater, Christ College, Brecon in 1931. The younger of his two sons was Robert Leoline James, who in 1953 became headmaster of Harrow School.
References
^"The Very Rev H. L. James". The Times. 19 January 1949. p. 7.