Color Label filtering should allow removing non-labeled groups

Problem
When filtering by color label it doesn’t have an option to refine the filter. In particular, a Group still pops up!

If I want all the brown layers, I can narrow it down to only the brown layers plus their parent groups. If I want to only select all the brown layers by shift + clicking then I then must deselect all the groups.

Proposal
Include an option to have No Groups (pictured). This would make Color Labels much more useful for obtaining only the desired results.