Spencer started his career as an assistant coach for Gardner–Webb in 1961.[3] In 1966, he joined his alma mater, Shepherd, as the line coach.[2] In 1967, he became an assistant coach for Bridgewater.[4] The following year, in 1968, Spencer was named head football coach.[5] In seventeen seasons as head coach he finished with an overall record of 56–93–3. His best season came in 1980 when he led his team to a 6–3 record and a Old Dominion Athletic Conference (ODAC) title.[6][7] He resigned following the 1984 season.[7][8][9][10] From 1985 until his retirement in 2002 he served as the linebackers coach through four coaching tenures: Joe Bush, Dan Antolik, Max Lowe,[11] and Mike Clark.[12]
Spencer also served as the head track and field coach during his tenure with Gardner–Webb.[13]