Hide private knowledgebase articles from certain staff?

[threshold]

I have parent knowledgebase categories cat1 and cat2 that I have made private.
Cat1 is set for only abc123 template and cat2 is for xyz456 template.

The problem is that all staff can see all articles and categories, even tough they are set like this:
abc123 staff member belongs to abc123 group, which can only view department abc123 tickets.

So, I have ab123 staff belonging to abc123 group, which can only view ab123 department and I have knowledgebase categories and articles that are assigned to abc123 template. But xyz456 can see the abc123 knowledgebase articles and categories. Clear as mud? Any help is appreciated.