From the NYC Landmarks Preservation Commission's 1976 landmark designation report:
St. Peter's Church, Chapel, and Cemetery form a pleasant and charming enclave in the heart of the old town of Westchester in the Bronx, formerly part of Westchester County. The two strikingly picturesque Gothic style buildings in their quiet graveyard setting dominate the neighborhood and are a tangible reminder of the rural past of this section of the Bronx. The county of Westchester was formed in 1683 and the borough-town of Westchester was named the county seat. . . .
The parish of St. Peter's, one of the oldest in New York City, was organized in 1693 following an act of the Colonial Assembly "for settling a ministry and raising a maintenance for them in the County of Westchester." It was not until 1700 that the town meeting house, previously used for religious services, was abandoned, and a church was erected. . . . The church was situated on the town green adjoining the county court house and jail, the same site as that of the present structure [completed in 1855; interior pictures here].