The more I think on it, the more my thoughts head to the inevitable conclusion that real design skills are something granted to you by either a higher power, genetic makeup, or sheer luck. More to the point, neither god nor my genes have granted me any worthwhile natural knack for design, and that the only way to obtain such is by having some epiphany involving hitting my head, doing psychadelics, or divine intervention.
I keep hoping that I'll lay down one night, head full of inspiration that I can do naught with visually, and wake up with a profound understanding of layout, color, and aesthetics that I had lacked the night before. As if the information could sink into my brain by osmosis by laying on the Everest-like mound of design books I've purchased and not yet intuited.
What I need is a formula that I can use to plot beauty. Follow a flowchart to design nirvana. If there was just a path I could follow to get out one great intentional design, I'm sure I could replicate the process. I think.
But I'm discovering that there isn't a process to perfection. You either know the path or you don't. For some the path is a golden road leading directly to perfection. For others the path is windy and filled with briars, but they get to the end of it. For me, woodland animals laughing at my lack of direction.
I'm not ready to give up just yet, though. I might need to study under a guru (where will I find a master??) for a few years, but I'll get it. And when I finally do, I'm sure I won't be able to explain how it works.
I know how you feel. To complicate the issue, not only does skill with design seem to require some sort of natural gift -- the judging of designs itself is totally subjective. Looking at a design made by a master, you may perceive beauty and perfection where all I see is confusion and ugliness.
I believe the following applies where love==art:
Why's it gotta be so darn complicated?
If it were simple either a) it would be boring or b) you would already be at the point you describe and are able to perceive the limpid beauty of the design.
On other avenues I've looked for the same thing you talk about. A good design, or poem, or any other work of art, seems to arrive perfect, but our execution is more a stumbling in the direction of the perfection we see.
You're looking for a perfection that doesn't exist. I think you have unrealistic expectations of what is possible... you want some sort of surreal super-reality, the kind of things that can be described, but not realized. Perfection is something to strive for but rarely reached.
I think one of the big problems is that the books you're reading are often poorly written and aren't telling you want you want to know. From what I've seen, the BEST ones talk about facts, like color schemes, and maybe how to create color schemes, but they don't teach WHY those things are true. Often people just don't know. Without knowing the why, you can't create your own things, only mimic others'.
A real master would be able to look at an arbitrary thing and tell you why it's bad, how it could be improved, and what the tradeoffs would be. Books might give a specific example and rip it apart, but don't generally give you the tools to do the same (if you even agreed with their point of view).
I've been advocating the scientific method as a fundamental problem solving technique lately. It can be slow, but tried and true, and gives you a simple outline of a plan of attack. If there's some design you want to improve:
1) Think about a potential cause for the problem
2) Determine a fix for the problem and what the effect should be
3) Try it and see if it has the expected effect
4) Repeat
Then just keep track of the things that worked and the things that didn't. I don't think design is something where given problem has a solution that works in all cases, but having a set of things to try or think about can't hurt.
The things you don't intuitively get are the hardest. You have to really work to get your brain to think the right way.