Matt Cheney's blog

As we talked about in the last post about Drupal Development Tools, the process of making Drupal websites is a lot like cooking and requires great tools, techniques, and secret sauce to make it all happen. Here are some of the techniques we use on our projects:

The Techniques

Due Diligence and Drupal Development Plans

Just as breakfast is the most important meal of the day, writing an initial Drupal development plan is the most important part of any project. It might seem like paperwork or busywork, especially to seasoned developers where Agile is Cowboy spelled backwards, but it has been my experience that every extra hour you spend planning saves you several hours at the end of the project - plus all the headaches. This process is pretty straight forward for small sites - identify the major content types, views, and contributed modules you need - and then start building. It usually takes a couple of hours thinking it through and often helps to talk it through with another developer.

For larger sites, the process of building a Drupal development plan should be a more involved one where you consult with all of the developers, themers, and designers involved in the process. For sure a lot of the Drupal implementation is obvious (this is a node!) , but it is the hard stuff (especially the custom code and how that all fits together) that will throw you for a loop. At Chapter Three we believe that the best development process is an iterative one and items will be changed, modified, and blended throughout the process, however, its much easier to change something if you know the structure of what you are changing in the first place. There are many ways to put together a project plan, but here is a basic Drupal development template we have used to give you some ideas.

Sandbox Environments

As I mentioned previously, we use SVN version control for all our projects and to take full advantage of this we give every developer their own sandbox copy of the code located at projectname.chapterthree.com/ ~username. This allows the developer to work in isolation and only commit up changes when they are ready to have them reviewed. We have an automatic SVN post-commit hook that fires on each commit to update our general development instance (project.chapterthree.com) with the latest and greatest code. There are occasionally SVN conflicts that need to be resolved, but its much safer than trying to work off the same general copy of the code - in fact, our SVN post-commit hook explicitly forbids commits from any non-sandboxed copy of the code.

In a past life, I used to teach cooking classes and as a class exercise at the end of the first class I would ask my students to cook a particular dish, using a collective set of ingredients, and bring it to the next class. Each student approached the problem differently, using their own techniques and tools, and they all mixed in their own "secret sauce" to improve on the collective ingredients. It was interesting that despite starting with the same ingredients, the students ended up in wildly different places and with different quality. The difference came from the little things - the tools, the techniques, and each chef's secret sauce.

Drupal development, in a lot of ways, is like cooking. For the most part, it starts with the same basic ingredients (Drupal core + contributed modules) and often follows common recipes (make content types, create views, write custom modules), but where you end up as a final product matters a lot on the tools, techniques, and secret sauce you work with. At Chapter Three we have been doing Drupal development as a team for over three years and have chosen our tools, refined our techniques, and perfected our secret sauce in making great Drupal websites. Over the next three blog posts, we will talk how we do the work we do.

The Tools

SVN for Version Control

The ability to selectively save and revert your work is something I often wish for when cooking and find extremely useful when coding. At Chapter Three we use SVN version control for all of our projects, big and small. This allows us to effectively manage projects with multiple developers and allows us to integrate external developers into our client process. While we recognize there are other version control systems out there with more features (Bazaar, Git), we choose SVN because of its relative simplicity and familiarity to our developers, themers, and clients.

Trac for Code History and Bug Tracking

To complement SVN with each site we roll out we also include a separate installation of Trac. This open source package allows us to see a timeline of our SVN commits and get visual diffs between revisions. This is helpful in auditing specific pieces of code (what happened and when) and provides a lot of assisstance in tracking down problems that might arise (if the module was broken on Friday, what changes were made to it on Wednesday and Thursday?) To manage our credentials between Trac instances we use our project management Drupal instance to control the HTTP authentication for Trac.

Drush

I whole heartedly agree with @ronald_istos that "drupal without drush is like life without mobile phones .. you know you managed just fine but now you just can't imagine how". Although our average site containing roughly 30 modules, installing and updating all of the modules is made really easy with Drush. With each site rollout we also deploy a symlinked wrapper around Drush that identifies the project that needs updating, switches context to the developers sandbox, and pushes up any changes to SVN. This makes updating modules on a Drupal site as easy as drush updatecode. To quickly turn on and off modules the commands drush enable module and drush disable module does wonders to increase productivity.

Beyond the module management, Drush is a venerable swiss army knife of Drupal functionality allowing a developer to log directly into the MySQL database with the credentials the Drupal installation uses (drush sql cli), runs cron (drush cron), run any pending database updates (drush updatedb) and much much more. It is really easy to extend Drush's functionality and we have found this very helpful when doing extensive data imports and exports. All in all, the centeralization of all of this functionality to the command line (now not even a module!) is very powerful and indespensible to our development process.

It is a good Drupal development best practice to embed each of your views in code instead of storing them in the database. This technique will allow your views code to be managed by your version control system which makes pushing up changes to views from development to staging to live a much cleaner process. Additionally, since the views code is being loaded from disk instead of the the more expensive database you will see a modest performance improvement. All in all a pretty good thing.

However, getting this to work in practice can be a little tricky. The embedded views are *sizable* data objects and the standard implementation of the Views 2 hook to do this (hook_views_default_views()) leads to the creation of a *giant* function where each of your views is copied and pasted into your module file. Making changes to individual views later can be a real hassle.

To help solve this issue I took a cue from views implementation of organic groups and adapted a technique to allow each view to be stored in its own separate include file which is automatically detected by the module. The win here is to allow easy updating of each view (just delete its file and copy the new export over it) and not have to worry about a lot of other cruft. To make this happen, four steps:

As a matter of best practices with privileged user account passwords, it is important to avoid using the same password for multiple sites. The reasons for this should be pretty clear, but when you are working in a collaborative environment with multiple different sites this is easier said than done. It is really hard to remember many different complex passwords and to share them (for User 1) across your development team. An encrypted "password safe" file can be one solution, but the problems to this kind of approach should be obvious.

A better solution, that we use at Chapter Three, is to setup our own Open ID server to create our own OpenID accounts and then use Drupal's OpenID support to credential ourselves into Drupal. This allows us to easily grant each of our developers access to the User 1 account (or their own admin account) without requiring that they share (or even know) the standard Drupal User 1 password. The use of OpenID has other benefits including being able to easily use SSL authentication, globally change a compromised password, and being able to quickly remove access from developers at the end of the project without having to update everyone with new User 1 credentials.

To get this working you first need to locate and install an OpenID server package to your development server. We use Community ID, but any mature package should do just fine. There is a development version of a Drupal package as well (http://drupal.org/project/openid_provider). Be aware there is a current bug with the Drupal's OpenID module that makes it sensitive to the HTML source positioning of the "rel" attribute on the OpenID provider definition. For some OpenID implementations (including Community ID) Drupal requires a patch to Drupal or a modification to the output of the OpenID server to change the ordering to be Drupal approved (we opted for the later). For extra points, you can configure a wildcard ServerAlias rule with Apache (ex: ServerAlias *.id.chapterthree.com) to allow certain OpenID servers to create cool subdomain IDs (ex: matt.id.chapterthree.com instead of id.chapterthree.com/identity/matt).

Afterwards, on each Drupal site you want to use your new credential with you need to enable the OpenID module (its in contrib in 5.x and core in 6.x) and then enable your OpenID for each account (with the OpenID Identities Tab on the user edit page, see blow) you want to use your new OpenID account with. The User Login block will now have a neat link to login with OpenID (instead of the standard Drupal login) and you can throw away your password file.

It should come as no surprise that San Francisco is a great place to talk about technology. The area has fostered a ridiculous amount of innovation and there are always plenty of people interested in getting together to talk about the next cool thing. We have seen wonderful growth in the Drupal Project and its been great to see that growth translate to an increasingly vibrant local Drupal community here in San Francisco. It is a wonderful time to be doing Drupal in the Bay Area and I wanted to share a few highlights:

Drupal User's Groups: SF and Berkeley

There are two vibrant user's groups in the Bay Area. The San Francisco group meets monthly and regularly gets 50+ attendees. Hosted by John Faber in the PariSoma Co-Working space each meeting has several different presentations from community members. Across the bay, long time Drupal contributor Tao Starbow runs the monthly Berkeley User's Group well attended (40+) by UC Berkeley staff (and members of the east bay community). I try to regularly attend and speak at this groups and have recently given talks on security (feat. Neil Drumm), using organic groups to build an academic community site, and the panels module. Outside of the immediate Bay Area, there are also vibrant user groups in Sacremento and Santa Cruz. Information about these groups and their events can be found on groups.drupal.org.

Drupal Camp: BADCamp 2007, 2008, 2009

For the last three years, the Bay Area has played host to what we affectionately called BADCamp (Bay Area Drupal Camp). Started on the UC Berkeley campus in 2007 by Tao Starbow, the camp regularly draws 200-350+ people to attend sessions by Drupal visionaries, masters, entrepreneurs, rockstars, poets, and prodigies. The event is free to everyone and helps lower the barrier to entry so others can learn and join the wonderful Drupal party party.

The last BADCamp was held this past weekend at the San Francisco State downtown campus. The entire event was conceived, planned, produced, and executed in a span of about 2 weeks and was a true testament to the vibrancy and agile nature of the Drupal community. The back story here was after we finished our Drupalcon SF 2010 proposal, Kieran mentioned he had a venue downtown San Francisco and everyone agreed it sounded just about right to host an event. A couple weeks later we proposed some sessions (including a session on Panels and the End User by me and one on WYSIWYG Editors by Jen, Sun Microsystems and Acquia sponsored some drinks, and the magic just happened.

Chapter Three has been doing a number of training sessions around the Bay Area and as part of the BADCamp we proposed to hold free all day introduction to Drupal class. The class began with Kieran introducing Drupal and giving a demonstration on installing Drupal with the Acquia installer, John Faber discussed content and content management, Chris Bryant talked about site building and Views, and Jen Lampton and myself talked about Drupal theming and "Advanced Topics" to close out the day. We had over 100+ people attend the introduction training and got some great feedback from all the sessions. Total attendance was over 200 with a lot of new faces. Look for lots more events like this in the future.

Drupalcon SF 2010

As we mentioned earlier, the Bay Area Drupal Community has come together to propose to host Drupalcon North American 2010 in the world class city of San Francisco. We launched our public proposal website last month and have been working hard to refine and improve our proposal since then. There has not been a lot of movement on this from the Drupal Association (they are hard at work on Drupalcon Paris), but I think the selection process will get picked up and finalized soon.

To keep up to date with our progress, follow the DrupalconSF Twitter and check out our proposal website at http://drupalconsf2010.org/. Nota bene, we were able to secure additional dates at the Moscone Center that during the 3rd week of April which avoids major holidays + conferences and allows a Drupal party we can all attend in downtown San Francisco.

Chapter Three has a long history of bringing together the rockstars to make wonderful things happen and as participants in the open source process we understand that sometimes making the coolest things takes a community. We have an amazingly vibrant Drupal community in San Francisco - and the surrounding Bay Area - and so it was that we decided to come together to propose something awesome - Drupalcon 2010 in San Francisco.

We have been working hard over the past couple of months - meeting every Wednesday in the Chapter Three office or at Parisoma - to plan, prioritize, and finally propose to the Drupal Association that the next North American Drupalcon be held in late March 2010 in downtown San Francisco at the Moscone Center. It is a lot of work to run a great conference, but we have the right people and the right attitude to produce a wonderful conference in the city we love.

San Francisco is a world class city and a great place to hold a Drupal conference. We are home to Silicon Valley, one of the most vibrant technology communities in the world, and with thousands of web developers, designers, and writers nearby we can guarantee tremendous local buzz and attendance. San Francisco has always been an incubator for startups and new technological innovation and as we look forward, as a Drupal community, to the next few years innovation on the web - what better place to have that conversation that at Drupalcon in San Francisco?

San Francisco is also a center for culture, cuisine, and creativity. Our location offers an ideal combination of cross-pollination with other innovators online, business connections for the expanding Drupal marketplace, and a stimulating and pleasurable experience for conference attendees. The evening party scene will be off the hook and plenty of attention will be paid to hosting events that have more to do with code than cerveza. As the Drupal community grows - and the attendance at Drupalcon is measured in thousands (instead of hundreds) - special care needs to be paid to making sure we maintain our distinctive community feel and give core developers a chance to discuss and plan the next big things for Drupal.

This initiative is only a proposal and the final decision over where Drupalcon North American 2010 will be located is up to the Drupal Association. However, if you are interested in getting involved with our planning process or want to keep up to date with our progress please checkout the proposal website - designed by the wonderful Nica Lorber - and follow us on Twitter.

We spend a lot of time at Chapter Three talking about how we do our work and developing the project management and developer tools we need to make sure everything runs smoothly. Our team has grown considerably in the last year and its important to us to continue to keep the bar high and have each member of the team produce high quality work the Drupal way. We see our design and development work as more than just meeting a specification for a project, but rather something that fits into a larger framework of how Drupal design, development, and deployment should be done. Best practices are wonderful things.

It is in that spirit that we would like to announce the Chapter Three Experts program as an initiative to involve high quality people in our company to help make sure we keep a high level of Drupal excellence. In practical terms, Chapter Three Experts advise on large scale architectural decisions, help design and develop custom functionality for our client and company projects, and provide tier-three Drupal support and advanced training to our development team. Chapter Three will coordinate our Drupal outreach efforts with our experts to help maximize our contributions and community involvement.

To kick off the program, I would like to formally welcome Neil Drumm to our team as a Chapter Three Expert. Neil is a longtime Drupal contributor who is the branch maintainer for Drupal 5, maintains the ever useful api.drupal.org, and is developing the very awesome Dashboard module for the new Drupal.org. He is a wonderful person and Chapter Three has had the privilege of sharing office space and working on several projects with him over the last few years.

I have always been impressed with the contrasting mix of craziness and competence in the Drupal community. Only in our wonderful world can a rockstar create industry leading educational videos, those who cannot vote or drive can still patch core, the sexiest man in Drupal is also one of the baldest, and somehow skills in customizing websites translates to customizing beautiful fixed gear bicycles. Deciding to start Chapter Three with Josh and Zack was one of the best decisions I have ever made and so much better than the stuffy alternatives.

As a company who regularly handles half a dozen client projects, its not always an easy thing to take time off. However, Drupalcon DC was shaping up to be a fantastic event and we decided to bring the entire company to share in the community and learn from the best. Chapter Three also had our own rockstar moments with Matt presenting our work on PBS Engage, Josh demonstrating crowd pleasing techniques to Handle Asynchronous Data with AJAX/AHAH, Jen talking about her experiences running BADCamp, and Matt (with Neil + Greg + Ezra) doing a security double header session with plenty of live demonstrations.

To help spread the good word, Drupal evangelist Zack took the stage to present "Drupal Lunch" (feat. Kieran Lal) and educate DC based CTOs and technology decision makers about the benefits of using Drupal. Josh (and the rest of the Drupal Dojo) ran an all day session talking about screencasting and Drupal education. Jon released the Live Update module and we all threw down for patches and strategy at the code sprint.

The nightlife at Drupalcon is always very unique and Chapter Three was happy to host an epic party Thursday night at the Asylum Rock and Roll Lounge. The following night we combined two wonderful elements (chx + a limo) and headed to Virginia to spend the evening playing lasar tag with Dimitri, Charlie, and our flat hat awesome friends at Zivtech. Drupalcon was fantastic and a big thanks to Bonnie for making it all rock. It's a great community of which to be a part, I woudn't have it any other way.

You can't go wrong with a Rock & Roll Lounge that welcomes the Harley Davidson and serves a fully complemented vegan brunch. Please join Chapter Three on Thursday night from 8pm onward at the Asylum Rock & Roll Lounge for drinks, eats, and general Drupal mayhem. The bar is a hot minute from the Adams Morgan Metro which is four steps down the red line from the conference venue.

nota bene - there will sadly be no pinatas on account of some previous incidents and corresponding strong options from the bar staff.

A couple years back, Chapter Three made a commercial where I embraced my tangential family legacy by quickly firing off a few rounds to highlight Drupal offering "rapid prototyping like you've never seen". Since then Chapter Three has done plenty of prototyping and has built dozens of sites, some on extremely tight timelines, but yesterday sets the company record for speed of design, development, and deployment of Drupal.

In just over 10 hours using the entire team assembled in our Humboldt County office, we ran a full Drupal design process, simultaneously delegated the configuration and coding tasks - including deploying the new Panels 3 module and getting the MP3 audio integration working. As we do with all of our sites, we made sure to include our Search Engine Visibility best practices and some Views 2 powered custom administration interfaces. We were even able to fix a few bugs with Audio module and contribute the work back as a patch.

So behold, Grown Kids Radio. The site was developed for Dan Finnerty, our exceptionally talented financial manager, who also doubles as the equally talented DJ Spinnerty. With some other cool kids, he has a radio show on Pirate Cat Radio, 87.9 FM and can now use the power of Drupal to spread their "neck snapping beats" around the world. Listen to their show live each Thursday from 12 to 2 PT and once they finish the content migration check out their past episodes. Thanks to Drupal for making it all possible.