When filtering by color label it doesn’t have an option to refine the filter. In particular, a Group still pops up!
If I want all the brown layers, I can narrow it down to only the brown layers plus their parent groups. If I want to only select all the brown layers by shift + clicking then I then must deselect all the groups.
Include an option to have No Groups (pictured). This would make Color Labels much more useful for obtaining only the desired results.