how do I stop a mob from spawning when another of the same is already spawned in?

A simple way is just to use the 'does entity exist of (type) in (radius)' function to check whether there's already an entity of that type nearby. 

If you want a more configurable method, you can make a local number variable, and run an entity iterator that checks for your custom entity in the same radius, but instead increases the number variable by one each time it finds your entity. You can then spawn in the entity based on how many are nearby, and set a threshold at which they no longer spawn.