Drupal Development Best Practices: Techniques (Part II)

As we talked about in the last post about Drupal Development Tools, the process of making Drupal websites is a lot like cooking and requires great tools, techniques, and secret sauce to make it all happen. Here are some of the techniques we use on our projects:

The Techniques

Due Diligence and Drupal Development Plans

Just as breakfast is the most important meal of the day, writing an initial Drupal development plan is the most important part of any project. It might seem like paperwork or busywork, especially to seasoned developers where Agile is Cowboy spelled backwards, but it has been my experience that every extra hour you spend planning saves you several hours at the end of the project - plus all the headaches. This process is pretty straight forward for small sites - identify the major content types, views, and contributed modules you need - and then start building. It usually takes a couple of hours thinking it through and often helps to talk it through with another developer.

For larger sites, the process of building a Drupal development plan should be a more involved one where you consult with all of the developers, themers, and designers involved in the process. For sure a lot of the Drupal implementation is obvious (this is a node!) , but it is the hard stuff (especially the custom code and how that all fits together) that will throw you for a loop. At Chapter Three we believe that the best development process is an iterative one and items will be changed, modified, and blended throughout the process, however, its much easier to change something if you know the structure of what you are changing in the first place. There are many ways to put together a project plan, but here is a basic Drupal development template we have used to give you some ideas.

Sandbox Environments

As I mentioned previously, we use SVN version control for all our projects and to take full advantage of this we give every developer their own sandbox copy of the code located at projectname.chapterthree.com/ ~username. This allows the developer to work in isolation and only commit up changes when they are ready to have them reviewed. We have an automatic SVN post-commit hook that fires on each commit to update our general development instance (project.chapterthree.com) with the latest and greatest code. There are occasionally SVN conflicts that need to be resolved, but its much safer than trying to work off the same general copy of the code - in fact, our SVN post-commit hook explicitly forbids commits from any non-sandboxed copy of the code.

The database is shared across sandboxes for convenience sake - Drupal places nice with multiple administrators - but we try to do as much configuration in the code as possible (see below). It is easy enough to segment the databases if needed and for more advanced database development we do. The files directory is likewise shared (as a symlink) for development convenience stake, but we never store anything critical in that directory. Moreover, in cases of remote developers, we can easily export the SQL + files and checkout the SVN so they can work in their own environments. We have a script for that too.

Dealing with Database Changes: Export It to Code

Let's face it - Drupal's database driven storage model does not provide a clear distinction between content and configuration, which makes all of our lives difficult. We cannot selectively copy parts of the database and push it through from development to staging and then to live without blowing away existing content and configuration. This technique can work in isolated, absolute cases, but it is at best unreliable and is only limitedly auditable and cannot cleanly be stored in SVN. Instead, at Chapter Three, we avoid this problem as much as possibly be exporting our changes to code. Variables can be set via update hooks with variable_set() and hook_update_N() and a host of modules (Views, CCK, Imagecache, Chaos Tools, Block) allow exportable and importable elements in code. The Features module, bless its code, makes this process much easier (especially if you are using contexts) and we have a few internal techniques, like exporting specific views to specific files, that allow for easily updates and management of the code. This makes pushing changes easy - just update SVN and run update.php (or drush updatedb).

Quality Assurance: Have a Designated Person Involved

The process of delivering quality code to your client is something that the right set of tools can help with. In our last post, we talked about we use Simpletest and Selenium to do automatic testing of site functionality. This automation helps to improve things a lot, but we have found it is important to have a real person take a look at the code as well. In our process we have the primary developer of a piece of functionality do a first pass and then hand it over to a dedicated quality assurance person.

At Chapter Three, we employ Ray Thompson as our full time Quality Assurance specialist. Ray approaches each project with "fresh eyes" and maintains an "user perspective" for each piece of functionality. Ray has developed his own process - adapted for general best practices - to test features and functionality, identify problems, and report back to developers - using descriptions, screenshots, and video clips - the items that need to be resolved. During the creation of our development plan we create a separate Quality Assurance plan which helps Ray run through the project's user scenarios. We find this combination of testing - some automatic and some manual - to be the best way to guarantee a fabulous final product.

Development -> Staging -> Live Deployment

Just as it is important to be disciplined about sandbox development practices, so too is it important to be disciplined about our deployment practices. At Chapter Three we follow a disciplined approach where all development happens on the development server, code is pushed to staging for testing and evaluation, and when everything is approved it is pushed live. To help us manage this process we have several custom scripts:

• code_merge_staging.php - This script takes the code from our SVN development branch, merges it onto the production branch, and then updates the staging site with the new code. By default this all happens under once branch, but can be configured per branch if we have several open branches (perhaps one for tweaks to a live site and another for new feature development).
• db_sync_staging.php - This script takes a copy of the live database and overwrites the staging database, as well as syncing the associated files from live to staging. This syncing is particularly helpful for testing since it allows us to test our new code against the latest and greatest from the live server. We do a few tweaks to the export script (like not dumping the cache tables) to improve performance since that is often a large and unnecessary thing to migrate over.
• code_push_live.php - This script takes the current, tested code from the SVN production branch and exports it into the production box. The code is pushed as an SVN export to keep the production code all in sync - once you push it, it all goes live immedaitely.

Prototype Sprints

The first week of any project is critical because it defines the structure and tone for everything that comes later. A solid Drupal development plan is a good start, but after that is written comes the all important prototype sprint. The goal of this sprint is to build a functional prototype while the specifications are fresh in our minds from the development plan phase. We typically start with the hardest problems first, since its a poor idea to leave incorporating a newly released module or integrating an unfamiliar API till the end.

Ideally, these sprints are done as a group - with two or three people working on each one. This simultaneously lets us get several people up to speed on a project and allows us to work collaboratively, sharing information with each other during the process. In cases where the timeline is short - and many of our projects are on short timelines - the sprint is critical to removing blocking issues and getting the project development process (content entry, theming, advanced custom development) rolling. This is how we built a site in a day and how we start the process for even the largest of sites.

Look for the last blog post in this series on Drupal Development secret sauce in the next few weeks!