For any podcaster looking to grow their show, a key part of that is growing your audience. This means keeping your existing audience and adding to that with new listeners. Ideally, the best way to do this is have your existing audience recommend your show to potential new listeners, since that kind of recommendation still remains one of the most effective.
But oftentimes podcasters can hinder this through no real fault of their own, because they try to appeal to everyone and keep all their existing audience happy. That means acting on every bit of feedback, even if it's not valid, experimenting with format because two listeners said they prefer it, changing publishing days, and more. And while I completely understand why you feel you should make those changes, what you're actually doing is listening to the few and that impacts the many.
So, instead, really learn who your captive audience is, who's always contacting you, sharing your podcast, offering solid feedback, etc. How are you ensuring they continue to be the loyal listeners that will help you attract the new ones? Are you asking them online? Sending surveys, having a feature on your website that lets them send voice messages, etc. These are really effective ways of making a show your real target audience wants and loves, and they'll help it grow beyond where it is now.
For the other listeners? Well, they're more than likely not your target audience anyway, so don't be afraid to let them go so you can concentrate on the ones that are. Until the next time, happy podcasting.