The Rules of Writing

Psh, nonsense! There's no such thing as rules when it comes to writing! Right?

Well, yes and no. As is the case with many creative pursuits, the "rules" exist in an incredibly grey area that's more of a three-dimensional spectrum rather than a list of checkboxes.

To some degree I believe that grounding your writing in a set of rules is what makes any author successful -- or at the very least, it makes a successful author even more successful. Almost every writer will believe certain literary conventions hold more weight than others, but very few believe in an all-encompassing set of principles that will work for any writer.
As a result, the "rules of writing" are less about discerning what's right and wrong and more about researching, understanding, and ultimately settling on a set of rules that are right for you.

Obviously the most commonly agreed upon rules relate to things like spelling and grammar. Presenting literature in one consistent, easy to understand format for a global audience is something that's hard to argue against. But even these things can be broken and tweaked by an author to varying degrees of effect.
I consider myself a very pragmatic person, and I have absolutely no qualms about using "incorrect" grammar/spelling at times if I feel it conveys my point better to a general audience. That's not to say that all of my mistakes are intentional (goodness, no. I still feel guilty about the amount of re-editing plenty of my old stuff needs), but I'll often do things like string two words together (like alright, or checkbox -- which my browser likes to underline in red) when the common usage sounds more fluid and direct than the dictionary entry. Spelling and grammar aren't set in stone. Language evolves, after all; but 99% of the time you want to ground it in the rules everyone commonly agrees upon.

But then you get to things like characterisation and storytelling. There is a very strict formula to these things, believe it or not, and understanding it is key to constructing a compelling piece of writing.
The important thing to understand about the structure of storytelling is that it isn't a guide all stories must follow in order to be satisfying -- it's a structure that forms itself naturally almost as a by-product of a good story being told.
  I've talked briefly in the past about structure and pacing, and the graph I included in that post is a pattern that will almost always apply to any good story as it unfolds. Tension builds, danger increases, action hots up, and emotions boil. There must be a constant increase of emotional engagement with the reader as the narrative progresses. If there's a steady decrease, stories feel unfulfilling. If you've ever read a book or watched a movie that felt like it ended with a whimper rather than a bang, it's probably a result of this engagement curve progressing in the wrong direction.
  If a story is all action, all emotion, all tension all the time, then it becomes tiresome, and the dramatic thrills cease to be impressive. This is what happens when you watch a mind-blowing hollywood special effects extravaganza and find yourself completely unimpressed when something explodes in a (non-)dramatic fireball for the dozenth time. On the opposite end of the spectrum, a consistently low amount of emotional engagement results in something that feels slow and boring.

So with all we've learned about structure and pacing in mind, storytellers have come up with a pretty consistently engaging pattern for how to set out your story. You want emotional engagement to rise, peaking in a series of climaxes that increase in intensity until you finally arrive at the big dramatic conclusion.
Now, this doesn't always apply to every story, and you certainly don't have to plan out everything to adhere to this curve, but if you write with it in mind you will write an engaging story.

Goodness, this post is dragging on, and I've barely even talked about half of what I wanted to mention. I think this might be a blog post that warrants a sequel at some point in the future, where I talk about genre-specific rules, the line between the cliché and the bizarre, and the fact that you can count every single plot type ever conceived of on the fingers of two hands.

My overall point to conclude the initial segment with is this: Storytelling may have no hard and fast "rules", but what it does have is a series of observable patterns that are consistently applicable to many of the most successful stories we've seen over the years. A proactive author will study these patterns, understand how and why they work, and decide on how they want to use this understanding to shape their own writing.