Duffy was born in Rathcabbin, County Tipperary, the youngest of three children of John and Jane (née Nolan). His father died when he was seven year old and, after completing his education at Rathcabbin N.S., he took over the running of the family farm from his uncle. Duffy joined the Irish Volunteers at a young age and as a member of the 4th Battalion of the Offaly Brigade of the Irish Republican Army he played an active role during the War of Independence. An opponent of the Anglo-Irish Treaty, he took the Republican side during the Civil War and was imprisoned at various times in Birr Castle, Templemore, Portlaoise and the Curragh. Duffy married Emily Harding (1897-1986) in June 1924 and had eight children.