Drupal Development Best Practices: Tools (Part I)

In a past life, I used to teach cooking classes and as a class exercise at the end of the first class I would ask my students to cook a particular dish, using a collective set of ingredients, and bring it to the next class. Each student approached the problem differently, using their own techniques and tools, and they all mixed in their own "secret sauce" to improve on the collective ingredients. It was interesting that despite starting with the same ingredients, the students ended up in wildly different places and with different quality. The difference came from the little things - the tools, the techniques, and each chef's secret sauce.

Drupal development, in a lot of ways, is like cooking. For the most part, it starts with the same basic ingredients (Drupal core + contributed modules) and often follows common recipes (make content types, create views, write custom modules), but where you end up as a final product matters a lot on the tools, techniques, and secret sauce you work with. At Chapter Three we have been doing Drupal development as a team for over three years and have chosen our tools, refined our techniques, and perfected our secret sauce in making great Drupal websites. Over the next three blog posts, we will talk how we do the work we do.

The Tools

SVN for Version Control

The ability to selectively save and revert your work is something I often wish for when cooking and find extremely useful when coding. At Chapter Three we use SVN version control for all of our projects, big and small. This allows us to effectively manage projects with multiple developers and allows us to integrate external developers into our client process. While we recognize there are other version control systems out there with more features (Bazaar, Git), we choose SVN because of its relative simplicity and familiarity to our developers, themers, and clients.

Trac for Code History and Bug Tracking

To complement SVN with each site we roll out we also include a separate installation of Trac. This open source package allows us to see a timeline of our SVN commits and get visual diffs between revisions. This is helpful in auditing specific pieces of code (what happened and when) and provides a lot of assisstance in tracking down problems that might arise (if the module was broken on Friday, what changes were made to it on Wednesday and Thursday?) To manage our credentials between Trac instances we use our project management Drupal instance to control the HTTP authentication for Trac.

Drush

I whole heartedly agree with @ronald_istos that "drupal without drush is like life without mobile phones .. you know you managed just fine but now you just can't imagine how". Although our average site containing roughly 30 modules, installing and updating all of the modules is made really easy with Drush. With each site rollout we also deploy a symlinked wrapper around Drush that identifies the project that needs updating, switches context to the developers sandbox, and pushes up any changes to SVN. This makes updating modules on a Drupal site as easy as drush updatecode. To quickly turn on and off modules the commands drush enable module and drush disable module does wonders to increase productivity.

Beyond the module management, Drush is a venerable swiss army knife of Drupal functionality allowing a developer to log directly into the MySQL database with the credentials the Drupal installation uses (drush sql cli), runs cron (drush cron), run any pending database updates (drush updatedb) and much much more. It is really easy to extend Drush's functionality and we have found this very helpful when doing extensive data imports and exports. All in all, the centeralization of all of this functionality to the command line (now not even a module!) is very powerful and indespensible to our development process.


Devel, Drupal for Firebug, Coder, et al: Drupal Development Modules

As a developer platform, there are many useful developer modules written for Drupal. Any given development project at Chapter Three will include several of them, but almost every project ends up having a few modules always installed. They include:

  • Devel Module - This is the defacto standard for a Drupal development module and includes essential functionality for generating test content, doing performance testing, examining SQL queries, developing forms, switching user accounts, identifying theme functions, dumping variables using the Krumo debug tool, and much more.  It probably should be named the "Devel Suite" instead and we use it on every site.
  • Drupal for Firebug - This module and its corresponding Firefox extension provides an excellent Drupal developer experience for those already using Firebug on a daily basis. Through a fair bit of XUL Magic, Drupal for Firebug hooks into the Firebug tabs and adds a new tab for "Drupal". It slows you all of the nodes, users, views, and forms used in each page load and provides intuitive coloring to identify what items have been added, changed, or deleted. You can easily output custom debugging messages using the firep() function and have it show up, in order of execution, in the general messages. I wrote the module and the extension to help our development process at Chapter Three and now many people use it everyday (~100,000 downloads so far). 
  • Coder - This module does more than just improve your code style, it also helps to enforce Drupal best practices on every line of code you write. We run this on much of our code and it helps to quickly find obvious security issues, identify PHP warnings, and enforce comment completeness. Starting soon all over our SVN commits will be run against coder module's rules to gaurantee quality code structure.

There are many more developer modules - Deadwood for upgrading D5 to D6 modules, Masquerade for user testing, and hundreds more. We use these modules on a case by case basis as the project or client needs require.

Xdebug

The Xdebug debugging package helps to identify PHP errors by providing stack and function traces for our developers. As a complex application, Drupal often has errors that occur many layers deep in the Drupal architecture and being able to track back and pinpoint where the original problem occurs can be critical to efficently tracking dow a problem. The functionality Xdebug provides is specifically important when the technical error that PHP gives is in a common function - like node_load() - but the source of the problem is a function that happened several steps previous. The Xdebug backtrace is far more helpful and makes our development process more efficent.

Simpletest and Selenium - Automatic Testing

Making sure that we deliver high quality products, free of critical bugs, is a complex task that requires a lot different processes. Part of our solution is to deploy automatic testing on both a functional level (via Simpletest) and a user experience level (via Selenium). A lot has been made of automatic testing going into Drupal 7 core (and for good reason), but much of that same functionality is available in Drupal 6's Simpletest contributed module. We roll this module - along with custom tests we write on a case by case basis - into all of our large development projects and use the Simpletest framework to gaurentee critical functionality throughout each stage of our project. Part of our development strategy is identifying early the critical paths and functionality for each project and baking in the tests nessesary to support them. The Simpletest functionality does a good job at making sure critical PHP and SQL functionality is working at the code level.

Since a web application is more than just PHP and SQL, to ensure quality of the user experience side of things we use the Selenium testing package to make sure things work at the browser level. This testing helps to identify errors (like broken Javascript or forms hidden due to CSS issues) that are not detectable by a Simpletest framework alone. There are a number of ways to program Selenium tests, but the easiest is to use the Selenium Firefox Plugin to record a macro of user activity and then automatically play it back across many different browsers. This way we can regularly test each "user scenario" under different conditions to gaurentee critical site functionality.

jMeter, KCachegrind, Apache Bench: Performance Testing

We often work on large sites that expect to receive a lot of web traffic (either on a continual basis or on a burst basis at key moments) and use several tools to make sure our sites and their supporting server architecture can succesfully handle the high sever load. To get an initial idea of the kind of load our server can handle we use the Apache Bench tool which tests Apache to see how many requests per second it can handle. This can help identify immediate improvements with the server stack and can test the theoretical limits of the server setup.

To provide a more real world testing situation, we use a tool called jMeter which records actual click-paths of user scenarios in order to simular the actual user activity of thousands (or tens of thousands) of concurrent users. The big idea here is that each site has different usage patterns and in order to have a complete performance test you need to simular your actual user conditions. After we run our tests we refine our code and server architecture to increase overall performance. In order to efficently target our improvements we use a tool called KCachegrind that profiles individual functions and libraries for memory and CPU usage and the Devel SQL Query log to identify slow queries. We make our improvements and then test again  to be assured we have hit our goals and can handle the amount of traffic (and a bit more) needed by our clients.

Look for the next blog post in this series on Drupal Development techniques in the next few weeks!