My software engineering philosophy is very simple; to paraphrase Albert Einstein: "Make things as simple as possible, but no simpler"*. I am a fan of of design patterns in software engineering, and I am well versed in the fundamental patterns (GOF, Fowler), and others I have discovered over the years, but I don't tend to use design patterns as a starting point. I find that when I start with these basic design goals, patterns will suggest themselves:
The risk of starting with a pattern is that you will design your code to fit the pattern rather than the problem you are trying to solve. Also, often what is a design pattern in one language is simply a feature of another!
When it comes to writing the code itself I like to start with an implementation that errs on the side of clarity, even if it means sacrificing conciseness and performance at first. At the public interface to my library I try to be as forgiving as possible with accepting inputs--though setting clear boundaries for failure--and as strict as possible with my output. There will always be time to refactor in the future, and there will be ample opportunity to test, measure and optimize as needed.
I regard refactoring as an essential part of the design process. For anything other than the most trivial project, even assuming static requirements, just going from design to working prototype will often expose design flaws. What works for a prototype may not be adequate for production, and what works for production will probably fail under scale. At the minimum, refactoring is the best path between these phases. More generally though I prefer to refactor as often as practical; discovering the design as I go, while improving maintainability, and overall structure. Refactoring is partially why I tend to prefer statically typed languages for large scale systems; IDE tools can make small refactorings trivial, and larger refactorings tractable. However, even though a compiler can use static type data to prevent a lot of errors, it cannot detect errors in logic which are usually far more serious, so a test suite with a high level of code coverage is an essential precondition for refactoring.
*"It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience". - Albert Einstein, at Oxford, June 10 1933