This area, formerly part of Guilford and known as East Guilford, was first settled about 1650 on land bought from the Nehantic and Mohegan Indians in 1641. With population increasing, settlers here sought separate parishes and the Society of East Guilford was incorporated in 1707, the Society in North Madison, called North Bristol, in 1753.

The Town of Madison, named for President James Madison, was incorporated in 1826.

This was once a center for fishing, shipping, shipbuilding, farming, and crayon manufacturing. Famous people born here included Thomas Chittenden, first Governor of Vermont,; philanthropist Daniel Hand; artist Gilbert Munger; and chief sponsor of the Civil War ironclad warship Monitor Cornelius Scranton Bushnell.