Literary fiction is a label that, in the book trade, refers to market novels that do not fit neatly into an established genre (see genre fiction); or, otherwise, refers to novels that are character-driven rather than plot-driven, examine the human condition, use language in an experimental or poetic fashion, or are simply considered “serious” art.
Generally, genre or category fiction is more plot-driven, while literary fiction is more character-driven. In either case, the story needs an arc or shape. However, I think of commercial or category fiction as being driven largely by action than by the character’s introspection. In literary fiction one usually finds more detail about how each character thinks and feels as it relates to the storyline. I’d be interested to know how others define the difference.
Wikipedia defines it this way:
Genre fiction follows specific genre conventions or models, using tropes, structures, plot points, and archetypes to tell a story. Genre fiction includes romance, mystery, crime, horror, science fiction, and western. Each genre has a set of reader expectations. For example, a romance novel usually ends “happily ever after.” The girl gets the guy or the guy gets the girl after overcoming great odds. That may be a simplistic model, but you get the point.