Roughly speaking, an endoscopic group H of G is a quasi-split group whose L-group is the connected component of the centralizer of a semisimple element of the L-group of G.
Langlands, Robert P. (1983), Les débuts d'une formule des traces stable, Publications Mathématiques de l'Université Paris VII [Mathematical Publications of the University of Paris VII], vol. 13, Paris: Université de Paris VII U.E.R. de Mathématiques, MR0697567
Shelstad, Diana (1983), "Orbital integrals, endoscopic groups and L-indistinguishability for real groups", Conference on automorphic theory (Dijon, 1981), Publ. Math. Univ. Paris VII, vol. 15, Paris: Univ. Paris VII, pp. 135–219, MR0723184