My gut reaction was that it doesn't. I have never pressed a copy of Susanna Clarke's Jonathan Strange and Mr Norrell on someone because it was a great book by a woman, but because it is a great book. Likewise, I have never given someone a copy of Pat Rothfuss's The Name of the Wind, because it was really well done for something written by a man. Just saying those phrases sounds ridiculous.
Except... I am writing my dissertation on the religious writing of late medieval and early modern holy women. And while the project was conceived as a way to do scholarship on Joan of Arc, and not as a specifically feminist project, the fact remains that for my dissertation, the fact that the writers I am studying are women matters. Their gender influenced what they wrote, the way they were treated, and the way they were and are read. And so I thought that maybe I only care about a writer's gender when I'm being a scholar. Except, the other scholarly project that I have been working on is a study of Shakespeare in Sandman. In terms of that project, I could really care less that Will Shakespeare and Neil Gaiman are both men. So that's no help.
And I'd like to say that it doesn't matter whose name is on the cover, only whether the words inside are good. When I go to the bookstore, the name of the author only matters to me if I am looking for a book by a friend, by an author I have read before and loved, or for a new author who was recommended to me. I certainly don't choose a new book based on the author's gender. Except that people do. Hence, J. K. Rowling, rather than Joanne.
But just because an author might sell better if she wrote under a man's name wouldn't change the fact that, under the above categorization, she would still be writing women's literature. So I'll just stick to my first thought. It would be a really stupid way to organize a bookstore.