Help with making Villagers follow you if wearing custom armor?

As the title says. Is there a way to make villagers follow you while wearing the armor and stop if you take a piece off? I've already managed to get potion effects and avoid wither with armors but I can't get this to work.