Only swing hand on right click when there is valid action

I have been trying to create a seed-type item, but for some reason my item makes the player swings their hand whenever I right-click on any block, which vanilla seeds don't do.

The current solution is have a global "Player right clicks block" trigger set up, then cancel it if the player is holding the seed but isn't right-clicking on a farmland:

However with this current implementation, if the player is holding the seed and right-clicks on a door or anything that can be right-clicked on, no hand-swing animation will be played.

Is there a way to enable the hand-swing animation iff the right click actually triggers something?