Dr. Samuel MacKenzie Elliott was a pioneering ophthalmologist who counted John Jacob Astor, Peter Cooper, Henry Wadsworth Longfellow, Horace Greeley, and John James Audubon among his patients. He built this house around 1840 from locally quarried stone, and supposedly designed more than 21 other similar structures (which are no longer standing, as far as I can tell). Dr. Elliott was a fervent abolitionist, and is said to have sheltered fugitive slaves in the cellar of his house (although I'm not sure if he ever lived in this particular building, or if he just designed it).
After Dr. Elliott's death in 1875, the New-York Tribune (founded by Mr. Greeley) remembered him as "emphatically one of the men who impart the element of the picturesque to common affairs. A person of very strong, original, eccentric character. A man of positive genius in his profession."