Gabrielle Harbowy, who is both an editor and writer, knows about "unsellable" stories. As an editor, she's acquired and edited novels that had been rejected so many times, the authors were despondent (and some of these have gone on to receive great reviews and award nominations). She's also rejected stories that other editors have picked up.
My own experience of being both author and editor is that things look different "from the other side of the desk." Yes, making editorial decisions about someone else's work has given me insights into why my own editors do that they do, and especially Not To Take Things Personally.
I should know that nothing’s really “unsellable.” I should know it because I see other authors sell pieces that they love but don’t expect the market to welcome. I should know it because I sometimes acquire them. I should know it because I see manuscripts I reject get picked up and lauded by other (and sometimes bigger!) publishers.
And yet, when it comes to my own work, where I’m the author and not the editor, it’s hard to keep that editor objectivity and ego separation from the work. Just like you can’t tickle yourself, you also can’t be objective about your own work. That’s been an interesting and challenging lesson to learn, but I know I’m a stronger writer for having learned it.
Sometimes stories don't sell because they really aren't good enough, because they have unsurmountable flaws. But sometimes they're wonderful stories that don't fit the market...or have not yet found the editor who will love them. As much as I applaud the proliferation of publishing options that includes self-publishing, I know all too well from my own experience that (most of the time) I cannot tell the difference.
Having a manuscript rejected is a learning experience, part of a dialog. Early in my career, I learned that form rejection letters were unhelpful. Not discouraging/infuriating but lacking in useful information. Was the story too amateurish? Wrong for the market? Showing great promise but the editor didn't have time to write a note? Then I started getting those personal notes, either scrawled at the bottom of those form rejections or as letters in themselves. Once I got past being upset, I started appreciating what I was learning. As I've gotten more experience -- and I've been at this writing business for three decades now -- I've gotten a better sense of when to quietly slip a story into the trunk and when to set it aside, looking for the right connection-to-an-editor. Always, always, I go on to the next project.
I would never advise a writer not to go immediately to self/indie publishing, because most of the time, I barely have a clue what's the best for me, let alone someone else. For me, dividing rejected stories into "better left unseen" and "I have faith in this one, it's too good to languish" is essential. And for that, I need to try -- and often fail -- to sell it to a professional market.