On a mountaintop above Olean stand the Rock City rocks: house-sized boulders of quartz conglomerate, an ancient beach turned to stone and then split into a maze of streets and passages you walk between, the largest exposure of the stuff in the world. Not a gem to collect but a whole city of it to wander. A private roadside park since 1890.
Quartz conglomerate
See it on the map
Other places on the map within a couple of hours.