Memory allocation shows up as a hotspot during profiling.

Consider ::swap semantics to avoid making a copy of the set, and clearing the set of layers that need updating.  This would also allow removal of individual calls to remove layers from the LayersThatShouldPushProperties while pushing properties.