The castle was built in 1175, by William II of Blonay (died 1197), a ministerialis in Savoy, on the site of a fortified structure that dated from 1095. It has served as the Blonay family seat since that time, aside from a brief interruption in the 18th century.
The oldest remaining portion of the structure is the sturdy, square castle keep, which has been strengthened, and seen a row of buildings attached to create another courtyard. Only two of four towers remain, around an irregular rectangular plan. There is a forecourt with a Renaissance-style loggia, from 1677 on the western edge, and a chapel dating from the 15th century, with stained glass windows from 1577. Residential quarters built between the 18th and 20th centuries have undergone renovations and alterations. By the 19th century, a moat had been filled in.
Gustave Courbet created a painting of the castle in about 1875, during the artist's self-imposed exile from France, living in neighboring La Tour-de-Peilz.