Blocks updating when adjacent block updates.

Hello!

I have created a tree which has fruits grow on its leaves. I set the block to tick randomly and then on update tick, it'll replace the block with the one with the fruit. All is working well, except that whenever I change any of the neighbouring block, the original one updates automatically. (I set a procedure where upon right click on the leaves, it'll replace the leaf block with a fruitless version, then spawn the item (fruit). So let's say I have two blocks on top of each other. Whenever I right click and harvest the top block, it works well, but when I harvest the bottom one, the block above gets updated and it grows instantly, then because of that, the block at the bottom gets updated again.)

This allows for infinite harvesting which I kinda don't want. 

Is there anyway to fix this?

Thank you!