In my humble opinion, The phantoms don't annoy me that much to be honest.

I usually go to sleep in-game before it becomes night(I'm scared of the Night), so that's why I dont find them annoying. Find them pretty funny looking to be honest lmao.