The skinny brown thing embedded in the wall is an old milestone, probably dating from 1813. It once stood a couple of blocks away on the Kingsbridge Road (today's Broadway), just south of what is now 204th Street, where it informed travelers that they were "12 Miles from N. York", that distance having been measured from City Hall. When the Kingsbridge Road was widened sometime in the latter half of the 1800s, the road workers tossed out the old milestone. It was rescued by William B. Isham, who had it built into the retaining wall beside the entrance to what was then his estate and is now Isham Park.
You can see some cool old photos of the stone and the wall here. (The linked page claims that the stone was originally an 18th-century Post Road mile marker, but that seems quite unlikely. Richard J. Koke, who authored an extensive study of the city's milestones in 1950, had concluded by 1964 that the stone was not that old.)
Did you know ahead of time that it was there, or did you spot it on your travels?
I didn’t notice it when I first passed by last year, but then read about it several months ago while researching something else. So I made a note to stop by next time I was in the area.