Alice Chesley Tasker and her husband, Edward A. Tasker, founded the Chesley Memorial Library with a generous donation in the early 1950s.  When Alice Tasker died in 1950, she left a stipulation in her will that a specific parcel of land be turned into a memorial library.  The land, located at the intersection of Route 4 and Old Mountain Road (now known as Route 43), was provided by Dana C. Woodman.  A stipulation in the will stated that the monies and land would revert to the original holders if the library was not completed within three years.

Alice Tasker's original contribution of $20,000 was augmented by her husband Edward Tasker, with an addition $6,000.  In addition to her original donations, Mrs. Tasker further bequeathed $2,000 for the purchase of books and materials and $5,000 "to be prudently invested and the income to be used on an annual basis to purchase books."  Her will further stipulated that the building was to be named the Chesley Memorial Library and dedicated to the memory of her parents, George and Mary Chesley.  This building opened on October 21, 1954.

In 1991 Joseph Grano donated over $100,000 to the library in honor of his wife, Theodora Kalem Grano, for the construction of a new library wing. On September 22, the new wing opened to the public.  The addition, which doubled the size of the library, provided room upstairs to house the adult collection of the library.  The downstairs was not finished off, but provided an area to renovate for the children's collection and a meeting room in the future.

In 1995 the town voted, via a warrant article, to appropriate $20,000 to renovate the downstairs of the library.  Another $20,000 was granted in 1996.  The library also received $19,245, through the Library Services and Construction Act Title II Construction grant, to make the entire library handicapped accessible.  The work commenced, and that year the new Children's Wing opened.