Down never made the majors as a player, making it only to AAA ball (1971 and 1973). Down later went on to coach at the University of Nevada, Las Vegas from 1979 to 1984. He went to professional baseball in 1985–88 as a special assignment coach and roving hitting instructor for the Angels organization. He returned to the Yankees organization in 1989, beginning the season as the club’s roving minor league hitting instructor before being promoted to manager at Triple-A Columbus for the final 16 games of the season when Bucky Dent was named manager of the Yankees. In 1990, his Double-A Albany club was at a 24-21 mark when he was promoted to manage Columbus in June after Stump Merrill was named Yankees manager. In 1990–92, he guided Columbus to three consecutive International League Western Division Titles and two consecutive Governor’s Cup Crowns. His Clippers set a franchise record with 95 wins in 1992. Down’s Columbus team was in first place in 385 of 391 games while he was manager. He later became the hitting instructor for many major league clubs.
In 2002, the Yankees hit 223 home runs, second most in the American League and second highest single season total in franchise history.
The Yankees led the majors in batting average in each of his first two seasons as a hitting coach (1993–94). In 1994, the team's batting average of .290 was the highest Yankees' average since 1936 (.300) and the highest in the majors since Boston hit .302 in 1950.
He helped guide the 2000 Dodgers to a franchise record 211 home runs.
He led the 1996 Orioles to a then major league record 251 home runs in his first season with the club.
From 1990 to 1992, he served as the manager of the Yankees' Columbus (AAA) team of the International League. He led the 1992 club to a 95-49 record and a Governors' Cup Championship. The 95 victories were a franchise record and were the most wins in the IL since 1960, when Toronto went 100-54.
In three seasons at Columbus, he directed the Clippers to three straight International League West Division titles and two straight Governors' Cups. In three years, he won 242 games and had a winning percentage of .619.
He compiled a 122-273 record (.309 percentage) in six years as a minor league manager.
Spent seven seasons in the minors as an outfielder. Hit .257 with 33 home runs, 247 RBI and 54 stolen bases in 745 games. He was placed on Montreal's major league roster on September 30, 1971.
Down also served as an advance scout for the San Francisco Giants and hitting coach for the Spokane Indians of the Northwest League.