The Milton Green is a triangular greensward just east of the east branch of the Shepaug River. At the northwest corner of the green the river flows under Milton Road before it turns to the west where it enters Milton Pond, now drained. The green is bounded by Milton on the north, Headquarters Road on the west and a gravel lane on the east. It remains completely open. There are no trees, flower beds or benches embellishing it.

At the south end are two 19th-century school structures: the 1855 Milton Academy and the 1896 Milton District School, the latter with a distinctive tower and belfry. Both structures are converted to other uses. Milton had a great many industries in the late 18th and early 19th centuries associated with water power. Evidence of their existence is difficult to find today, but the civic and residential community in the center near the green has remained essentially intact. The absence of commerce and industry gives the center a rural, agrarian appearance in keeping with its 18th-century origins.

Trees have never been planted on the green.