Josh Koenig's blog

Just a note to all kindly Drupalists and your followers. I'll be appearing at SxSw interactive to talk about Drupal in the Cloud, sporting an updated presentation which includes info on how we're using BZR to create a "cloud platform", where all that's going anyway, plus details about our forthcoming Mercury on-demand service.

I am looking forward to seeing all sorts of great folks in/around the conference. If you're going to be in Austin this coming weekend, drop me a comment and let's coordinate! You can mark the session on your planner right here.

For those unable to attend, there will be some video and other media, and my slides will be posted online as always. See you in the Lone Star state!

After well over 1,000 hours of development and thanks to the help of many legendary open source engineers, Chapter Three is proud to announce the release of PANTHEON Mercury 1.0, the quick-start server environment for those looking for the best in Drupal performance. You can launch Mercury now with our free Amazon Machine Images (AMI) or follow our install instructions yourself using any Debian Jaunty server. Commercially supported hosting options will be announced in the coming weeks.

What is Mercury?

Mercury is a standardized best-practice server configuration (aka "stack") for running your Drupal website that takes the best of the collected community practices, combines them with cutting-edge open-source tools for high-performance hosting, and delivers it all in a complete package. With Mercury, you can launch a speedy new Drupal server in less than five minutes.

In addition to deploying the required technologies to host Drupal (Apache, MySql, PHP etc), Mercury includes a number of "optional" elements that are rapidly becoming must-haves for any ambitious installation. We give you Varnish for bulletproof protection from the Digg/Drudge effect, Memcached as an application cache to keep your logged-in/admin pages zippy, and Apachesolr to deliver faster and more relevant content search results.

Mercury also implements Pressflow, a high-performance variant of Drupal core similar to the version that runs drupal.org itself (handy comparison chart) and provides critical support for high-availability and high-performance requirements.

What's New in 1.0?

Changes to the hosting stack in 1.0 have been restricted to bug-fixes and minor tweaks with one big exception: we've implemented the BCFG2 system configuration management systems, meaning users who launch today can take advantage of innovations and fixes that land months from now. Lemme lemme upgrade ya!

The system remains 100% open-source, and there's no "mothership" or lock-in here. Our BCFG2 setup works by pulling configuration from our public Launchpad repository and serving as its own master. You're free to turn it off once the system launches, merge your own changes, or simply continue pulling updates as they become available.

We think this is a game-changer for Drupal hosting. With this innovation, we can offer a system which remains free and open-source, where you keep root access, but where we can also deliver incremental improvements and new services in a completely transparent fashion. Simply put: if you start with this version, expect to stay on the bandwagon as we continue pushing the envelope with best-practice development, deployment and hosting infrastructure.

Already using Mercury?

If you've been braving our beta releases we'd love to learn from your experience. Please take a few minutes to complete this 8 question survey. This feedback will help us understand how people are using Mercury and what we should focus on in the future to improve it.

Standing On The Shoulders of Giants

Our tagline is "Stand on the Shoulders of Giants" and we're serious about it. None if this would be possible without the enormous efforts of literally thousands of brilliant individuals going back to Linus Torvalds, RMS, and beyond. Somewhat more specifically, we'd like to thank:

• David Strauss of Four Kitchens, who basically wrote the whole 1.0 roadmap into his Scalable Drupal Infrastructure presentation, is responsible for Pressflow, and has be an continuous source of moral support and technical inspiration (and vice-versa) throughout this process.
• Our open-source comrades from #varnish, #bcfg2 and #libcloud, particularly desai, sojl, phk, bjorn, polvi and pquerna.
• Greg Chaix and Damien Tournaud, who helped pioneer the use of Varnish with Drupal and provided invaluable initial support and documentation.
• Dries, of course.
• Khalid Baheyeldin of 2bits who's articles comprise a solid corpus of "how to" for high-performance Drupal, as well as Matt Westgate, Robert Douglass, Steve Rude, Narayan Newton, Jeremy Allare and the whole crowd of usual suspects from the high-performance group for the code and knowhow.
• Our early adopters, for being brave, not listening to the "not ready for production" disclaimers, and running real websites with this thing. Without all the real-world feedback, we'd probably still be too nervous to call it 1.0. ;)

The list could go on for quite a bit longer. Really, it wouldn't be possible without you, so if you've so much as clapped at a presentation, give yourself a pat on the back. Thanks to everyone who helped make this possible!

One of the great things happening this year at DrupalCon North America — Moscone Center, San Francisco, April 19th - 21st; sign up now — is the community is organizing to offer a number of focused trainings in advance of the conference itself. For those looking to build their Drupal skills and come away with practical knowledge, the pre-conference training sessions offer solid takeaways on a number of subjects.

This is a great time to be training. As our fearless leader Dries has said, the need to develop more talent is more pressing now than ever. It reminds me of the early days of the Drupal Dojo: more ninjas please! Lullabot has clearly been setting the pace here for years, but with Drupal continuing to grow there's simply more demand out there than any one shop can satisfy. Chapter Three has been taking up that challenge by offering regular training events in San Francisco, which has been a good experience for us and our students so far. It's exciting to see others following suit, like our good friends at Zivtech.

Anyway, I'm particularly proud to be helping to organize a session on Drupal performance and scalability. This is a hot topic that I'm happy to see demystified. The more people we can get up to speed on core best-practices, the better we'll be able to drive the next round of Drupal growth, which is going to include a lot of Enterprise-scale projects with intense uptime, scalability and performance requirements.

It's also something people seem to be hungry for; after all, everyone wants their internet to be fast. Last Friday at the Silicon Valley Drupal Users Group I gave a presentation on Drupal in the Cloud, and spent almost two hours doing Q&A about Cloud hosting in general, but really about performance and scalability in particular. It's not a simple thing, but the good news is that with the amount of information-sharing that goes on in the Drupal universe (e.g. via the high performance group), over the past few years some real best-practices are emerging.

If spending a whole day talking to the likes of myself, David Strauss, Matt Westgate, and other luminaries about these issues sounds exciting, sign up now. Space is limited, and I'm pretty sure this one will sell out quick!

More great video keeps emerging, this time from the fabulous folks at Four Kitchens, who helped organize the wonderful Drupalcamp in Austin last November. My colleague Matt Cheney and I were lucky enough to attend and present. I gave my talk on "Drupal in the Cloud" and Matt covered all things CCK, and the cameras were rolling...

Introduction to CCK from Four Kitchens on Vimeo.

Drupal in the cloud from Four Kitchens on Vimeo.

You can see more as it's added via the Four Kitchens Vimeo page, or watch the conference site for updates.

As my hombre Rob Purdie has already blogged, the fabulous Drupalists at NodeOne have posted some really fantastic videos from the most recent DrupalCamp Stockholm online. I gave three presentations there, but the one that was on tape is a talk about how we've been working with some Universities to build sites with Drupal:

How Berkeley and Stanford University Use Drupal (Joshua Koenig) from NodeOne.se on Vimeo.

It's a mix of business practices and general tips on how Universities (and other big organizations) can do a lot with Drupal as a platform, not just for building a single site, but maybe hundreds! Even though I think I could have given a slightly better talk — jetlag and a cold don't put me at the top of my game — it's still got a lot of good information about how we've been making this whole "running a consultancy" thing work.

And check out the other videos too. They're quite high quality (I hope we can do as well for DrupalCon SF), and you can see Ken Rickard explaining Domain Access, Addi talking about Drupal as a Movement, and experience the force of nature that is MortonDK.

This comes on top of the already (in)famous video of the Kitten Killers doing their take on the Drupal song. It really was an amazing conference. I'm definitely looking forward to showing our Sweedish friends some reciprocal hospitality come April!

While attending DrupalCampNYC7 this past weekend at NYU Poly, one of the most eye-popping demos I saw was by Roger López, showing off some functionality hiding in the Chaos Tools Suite to empower you to create awesome AJAX functionality, including modal dialogs/forms, without writing a single line of javascript. You can read Roger's presentation notes here, which I strongly recommend.

Needless to say, I was immediately jazzed by the possibilities of a clean programmatic mechanism for handling data exchanges of this nature. It's complex work, and doing it on a one-off basis is tricky and prone to error. This is why we're all so lucky to be in the same coding community as Merlinofchaos: not only do we get killer apps like Views, we get the infrastructure he writes "for himself" that goes along with it!

I was also sort of surprised/chagrined that I'd been working on Panels all year long and never known about this wondrous gem of an API. But that's a problem I know how to solve: I thought, "hey, this should be a sample module included with ctools by default."

So, on the plane from LaGuardia to New Orleans (where I'm presenting on Friday at Do It With Drupal), I took my own stab at it. This isn't a lot more code than Roger presented, but I cleaned up the namespace and used a few more of the ctools functions.

The sample module shows how to perform simple operations like text replacements, table manipulation, and a modal login form. I'd love to hear other use-cases that would be worthwhile to include in a sample module (chime in on the issue thread). I'm also looking forward to digging in further to add new ctools_ajax/modal commands, so if you've got any great ideas let me know.

It's a great time to be a Chaos Wizard!

As a long-time Drupalist, I have to confess a wandering heart. Over the past couple years, I've grown more and more interested in other software and bigger-picture issues, branching out from the trials and triumphs of developing sites with my favorite CMS and looking increasingly at issues of integration and platforms. However, recent news has brought Drupal itself right back into the center of my thinking. With all respect to the Wu, I have to bite off a venerable hip-hop-ism in light of the recent run of news: Drupal Rules Everything Around Me.

The two bits of news I'm most excited about came through last week. First was the relaunch of Whitehouse.gov on Drupal. A lot of the reporting on that event (both pro and con) was typically fuzzy technology journalism — even prompting some correctional responses to the FUD — but overall I think the NY Times Circuits Blog got the essence of the story right:

To some, the White House shift to Drupal from a proprietary software package represented a serious seal of approval for open-source software.

“While open source is already widespread throughout the government, its adoption by the White House will almost certainly give permission for much wider uptake,” wrote Tim O’Reilly, who publishes books and holds conferences about open-source software.

The truth is that the use of Open Source code (Drupal or otherwise) for the whitehouse.gov site probably says very little about what functions that site will exhibit. Given my back-story with the Howard Dean campaign I admit I get a thrill from seeing our technology win out, but the site itself isn't going to change radically: it will remain a tightly controlled, top-down public relations website for the President. The "political ramifications" here are small, if not nil.

However, the business ramifications are very real. The addition of an ultra-high-profile site to the roster of Drupal adopters, likely the biggest signal launch so far, is the latest in a series of signs that Drupal is growing past the initial "early adopter" user base that's been driving the market for the past three or four years.

Following on that event, Dries published a blog to let us all know that Drupal is in the visionaries quadrant in the latest Gartner research report on "Social Software in the Workplace." This is a very good sign for future adoption of the software, and a major coup for Acquia's marketing efforts:

This is another important step towards increasing enterprise/institutional adoption of Drupal as a platform. Along with the other great developments we're seeing with "Drupal products" (or distributions, if you prefer) and strong growth in the training sector, 2010 looks to be a very bright year for Drupal. It's an exciting time to be a member of this community, and a participant in this rapidly evolving market.

With great pride, and after six alpha-level releases, I'm announcing of our seventh iteration on the Project Mercury stack, finally baked enough to call "beta". This release is a milestone: our course is now locked in for the 1.0 release.

At this point, we know that many people are using the Mercury EC2 image (or the same stack) in production environments, and we've tuned this release conservatively to prevent it from breaking down under heavy load. We've also verified that the stack will work under a resource-constrained VPS (e.g. one with 1/4th the RAM of a small EC2 image), which gives us more confident that this configuration is stable.

You can find the EC2 image in the US and EU availability zones with the following manifests:

• chapter3-storage/PANTHEON-pressflow-mercury-0.7-Beta.manifest.xml
• chapter3-storage/PANTHEON-pressflow-mercury64-0.7-Beta.manifest.xml

Thanks to all the people who have provided critical encouragement and feedback as we've developed this toolset over the past three months. Over 300 Mercury instances have been launched so far!

The next steps are zeroing in on 1.0 release, making the image available in other places (e.g. RackSpace Cloud, VMware marketplace, etc) and working to get a "hosted" version of it online through a number of partners.

More exciting news to come. Stay tuned, as always, and let us know what you're thinking about.

It was a great time as always at BADCamp. Chapter Three brought in copious coffee from Ritual Roasters and supplied bagels for morning munchies and I was excited to give my "Drupal in the Cloud" talk, as well as a more in-depth presentation on making Drupal fast and scalable. Matt and Jon rocked worlds with their talks digging into what we're calling "The Panels Paradigm" around here, and Squiggy introduced a room full of folks to the nuts and bolts of theming in Views 2.0.

That plus all the conversations had, almost had and interrupted in the hallways and at Jupiter... well, it's nothing short of exhilarating. We'll likely have more reflections and reactions from the team, but below you can find links to our presentations.

Also, huge credit is due to Tao Starbow and (doing it on her own time, not ours) Jen Lampton. Without their leadership, effort and organization, this event simply wouldn't be possible.

I realized at one point during the weekend just how big the ripples were from having a first-class camp three years running: there's no way we'd have the local community confidence to bring DrupalCon North America to San Francisco next spring without this event. Hats off.

And of course, the slides:

• Matt Cheney: Panels 3.0: The Power of Chaos Magic
• Josh Koenig: Drupal in the Cloud!
• Josh Koenig: High Performance Drupal
• Jen Lampton: A Best-of-Breed WYSIWYG Editor
• Squiggy Rubio: How To Use The Views UI and Theme A "View"
• Jon Skulski: Nodes, CCK and Panels

You can see a full list of other presentation slides here. Many more worth reviewing.

See you all next year!

I had a great time in Portland hanging out with the community at DrupalCamp PDX. Personal highlights for me included winning the race to loan Sam Boyer my laptop development environment for his Panels/Page Manager presentation, getting stuck for more than half of the lunch hour answering questions about PANTHEON and EC2 after my talk, and getting to meet mikey_p, heyrocker and other IRC personalities I've known and loved in person.

Of course, I'm also looking forward to next weekend's BADcamp where I'll be giving my presentation again, along with a more in-depth talk about high-performance Drupal. However, for those of you who are curious or want a sneak peak, I've posted my slides from last weekend on slideshare: