The ballast is different over different parts of the system. On the Highland it appears a fairly dark gray as I documented in this post.
In this post, I covered my latest approach in ballasting the layout, using real stone ballast from the same ridge in Plainville that the New Haven used. I think it's a very good match. There is a little brown. It's magnetic, so there's some iron in in, and I think that's what gives it the brownish tinge over time. I might experiment with a brownish wash for that effect.
In photos of the Shoreline, there is more brown in the color of the ballast. So when we do Chris' layout I'd like to go sift ballast from the Saybrook Tilcon quarry instead.
Here's the New Haven Railroad Ballast Chart from 1917, updated through 1921:
And this one is updated through 1959:
