solving obvious problems

This year’s Drupalcon was the biggest Drupal Party to date with over 800 attendees and many different sponsors (including Chapter Three of course). Drupal rocked Boston all week and gave everyone in the community a chance to catch up and keep abreast of some of the cool new stuff going on in the Drupal World. Plus, there was some scrumpcious and tasty vegetarian food on the scene.

In my Drupalcon rockstar moment, I took the stage and gave a session with Drupal Genius Neil Drumm and Digital Newspaper Extraordinaire Ken Rickard on Using External Data Sources with Drupal (slides here). The general idea is that Drupal is marvelous for content management, but becomes even better when it integrates with data from external sources and Drupal allows a lot of different ways to make that happen.

Ken kicked off our session by laying the groundwork for why using external data is important and transitioned into talking about some of the basic tools (drupal_execute, drupal_http_request, database switching) and techniques (lazy instantiation, hook_search) needed to make it all happen. He has a lot of good experience around using external data sources from his work in the newspaper world and provided a detailed orientation to the 150 odd people who attended our session.

Neil talked about his work with a wonderful organization called Map Light that records and analyzes information about campaign contributions and congressional votes. For that project Neil had to import *millions* of pieces of information and developed a couple modules to assist him in his work. Job Queue is his module that helps to import external data in batches (instead of all at once) and Import Manager allows administrators to monitor and manage the overall import process.

For my part, I concluded the session by discussing a few specific case examples of how to integrate external data into a Drupal website. I started with an example, which was dear to my heart as a librarian and admirer of philosophy, was to use the Swish-E Module (which I maintain on Drupal.org) to full text index a folder of philosophical essays in .PDF format and make them available via the Drupal search. The second example (see below) was probably the coolest and involved some data mining magic to use the Data Miner API to take a Drupal user’s MySpace name and automatically download their picture from MySpace for display on their user profile. This general technique I was calling “Data Enrichment” and could be used to enhance the data a Drupal site has around either users or specific pieces of node content. Careful consideration here needs to be made to respect both the terms of service and privacy of your users. The final example I presented had to do with some client work Chapter Three did involving the migration of 20,000+ pieces of raw HTML content through a cool audit system set up as several Views and managed by the Workflow System. This system seemed to have a lot of practical value for people and I got a lot of questions about it after the session.

All in all, Drupalcon was awesome and we are looking forward to Drupalcon Europe later this year and Drupalcon San Francisco in 2009.

Just a little to the left please. Flip it around. Put that on top of this. Call it by a different name. It is the little changes, that seem trivial and small, that often end up being real headaches to make and support our clients in making. Do we really want to try to build capacity with clients by teaching them to adjust #weight in hook_form_alter?

The Drupal Theming System is pretty powerful and, when done right, can offer a good avenue for our clients and their staff to edit, modify, and change their own website content. Its a lot easier to modify HTML files than Drupal module files.

A good example of where this kind of process is needed is on the user registration page. There are a lot of little bits of language and ordering to change and add, but to do so in Drupal module code can get a little hairy. Observe our technique to abstract the user/register form into a flat template file (while maintaing most of the other good Drupal goodness).

Step One: Create a theme override in your module code for the user/register form that executes a _phptemplate_callback to use a separate template file.

Step Two: Expand the theme override function made in step one to remove the titles and descriptions Drupal provides for the form elements. We do this in the theme function (instead of in a hook_form_alter) to preserve the original field titles so they can be used as part of any error messages coming out of form validation.

Step Three: Create "rendered" versions of each of the form elements and set them as variables that can be passed to the template file.

Note: This can also be done with a generic foreach loop (similar to the one in step two) that renders each form element automatically.

Step Four: Create a template file in your site's theme directory to build the user/register form with the customized variables we defined in step three.

Note: This file needs to be the same name as specified in the _phptemplate_callback (example: user_registration_form.tpl.php).

The drupal magic here is that the user registration form is now uniquely customizable by anyone who can edit the theme template. This allows for customized "prompts" for each profile field element, without changing the site-wide field name in admin/user/profile, and it allows for customization of the username and email titles and descriptions.

This technique will need to be modified to support external modules that modify the user/register form like LoginToboggan. It also needs to take into account things like "required" fieldstates.

Much like the parent who gives their child the keys to the family car, there comes a time in our client engagements where we set up administrative accounts for our clients to begin managing their own content. At Chapter Three we call this Capacity Building and try to make the process as smooth and intuitive as possible. The problem is the default Drupal administration interface makes this difficult.

In general, the Drupal administration navigation menu (see right) is simply too complex to be really intuitive. There are all sorts of options (read noise) available that have little relevance to the particular administration needs of our clients. Some of this menu can be distilled down through permissioning, but with general permission categories in Drupal like “access administration pages” its not possible to remove all the options. Plus, many of the most relevant options are buried several layers deep in the navigation. Its unlikely our clients will regularly find them on their own.

In the past we have build a static “administration page” where we listed the major administration pages and provided links to each, but with our latest project (that had a number of different editors and content managers who all needed to update the site) we wanted to improve our business practices and give our clients a better way to manage their website. The solution we came up with was to provide a constantly present top menu that would only appear for site administrators and would allow them easy access to their specific administration options in a nice drop down menu. We populated the menu through the Drupal menu system with the 20 or so administration pages that were needed to manage the site. This menu display was accomplished by using the SimpleMenu module and can be seen below.

To manage the individual pieces of content on the site we built one, general purpose “Content Management View” using the wonderful Views module. The idea here is to provide a consistent UI to manage each type of content and to allow site administrators to filter, search, and sort that content for their editorial and review purposes. Setting this interface up as a View allows us a lot of flexibility later to add additional administrative options and support different content types.

The end goal is to make it as easy as possible for our clients to easily work with the Drupal CMS and to manage their specific content. This kind of system is something we intend to refine and deploy in future sites.

Like a fresh cup of coffee with extra soy creamer, a daily morning trip to drupal.org to browse Drupal Planet or see recent activity helps to start the day off right. The current infrastructure issues not withstanding, if I have an internet connection I can get my open source blue alien fix. But what happens if I am a long way from an internet connection? Say, in the middle of a fishing village in rural Mexico or in the woods during a 24 hour bike race. In that case, you need to sling that data.

The general idea behind slinging data is that if you have a stable internet connection somewhere in the general area (within 20 or 30 miles) you can “sling” that data through a wifi bridge to your actual destination. At the ends of the bridge should be a pretty nice wifi antenna. We used 900MHz Yagi antennas which fold down nicely and are easy for transport. The antennas can be mounted on a number of things, but we choose (for ultra portability) to mount them to large wooden rods secured by christmas tree stands. The tricky party in setting this up is to make sure that both antennas point at each other. This is pretty easy if you are just shooting wifi within your immediate line of sight, but for longer connections (like between the chapter three office and the hills of berkeley across the bay) its probably best to use GPS coordinates to set up the connection.

Once you have two antennas pointing at each other, you need to attach the antennas to a couple of generic wireless routers. We used the trusted WRT54G routers with the detachable antennas and connected the routers to the antenna using some RF cabling we found online. On a software level, we immediately replaced the standard Linksys firmware with the industrial strength DD-WRT firmware created by Brainslayer. This allowed the power frequency to be boosted to 84 mW and the routers to be associated and placed into bridge mode (a built in option). Once they were powered on and associated, wow, bam, suprise, a wifi link was activated.

Of course, with that setup the wifi range is weak. Like 1 mile, maybe 2 miles. The real secret sauce is 4 W 915 L Series Antennafier. If you hook that amplifier on both ends of the connection, set its power to full, and start the bridge your wifi signal strength will dramatically increase. Over a two mile link, this setup was able to get a better connection than a normal a normal connection between the two wifi routers in the same room! Under optimal conditions (like in the desert or by from a boat on the ocean) this could send internet 30+ miles. Under non-ideal conditions, a your range will sadly be limited to 15 to 20 miles. So find some remote locations, invite some friends, and get your do fix on a boat, beach, or backyard.

Our office in San Francisco is pretty cool, but to commemorate our first year in business we decided to roll down the coast and do a little Drupal development somewhere even more wonderful than San Francisco. So we got ourselves some airline tickets, got hooked up with a great house with glass walls right on the beach, invited a few friends, and moved the operation to a small fishing village about two hours by dirt roads north of Cabo San Lucas. Why do development with Drupal in an office when you can do it on the beach?

We are equipped with some WIFI internet broadcast from an amped twenty foot omni-directional antenna, a couple bags of vegetarian food brought from rainbow grocery in San Francisco, Zack’s overflowing iPod, and all the fresh oranges, mangoes, avacados, cantaloupes, and strawberries we can eat. It’s a perfect setup to spend the next two weeks building websites, rocking emails, doing a little skype with our clients, and having a two week long Drupal beach party. There are definitely plenty of distractions and we might have some complications from an incoming hurricane, but there is a certain tranquility in coding from a beach in Mexico.

I am sure it’s a common problem. You are a rocking Drupal software engineer and you are talking to a particular cute guy or girl at a pretty awesome party. Naturally, the conversation turns to open source software projects and you boast that you work on Drupal - the coolest content management system out there. Since this boast is bound to be followed up with questions like “what is Drupal?” or “why is Drupal cool?” - wouldn’t it be grand (instead of trying to explain just how cool the garland color picker is) to whip out a portable USB key, stick it in an available computer, and show them the wonders of Drupal right then and there?

Install Ubuntu onto a USB Key

• Download Ubuntu and mount [mount -o loop ] the image
• Use fdisk [fdisk /dev/sda] to delete any existing partitions [fdisk: d], add a new partition [fdisk: n,p,1,*enter*,+700M], set its type to FAT16 [fdisk: t,6], and make it active [fdisk: a, 1], add another partition [fdisk: n,p,2,*enter*,*enter*], and save all the changes [fdisk: w]
• Install the FAT16 filesystem [mkfs.vfat -F 16 -n usb /dev/sda1] and the ext2 filesystem [mkfs.ext2 -b 4096 -L casper-rw /dev/sda2]
• Install the Linux System Tools [syslinux -sf /dev/sda1]
• Mount the USB Key [mount /dev/sda1 /mnt] and copy over Ubuntu [cp -rf casper disctree dists install pics pool preseed .disk isolinux/* md5sum.txt README.diskdefines ubuntu.ico casper/vmlinuz casper/initrd.gz install/mt86plus /mnt]
• Copy this special file to the USB key [tar xf usyslinux.tar ; cp syslinux.cfg /mnt]
• Run Lilo [lilo -M /dev/sda]

These instructions and system file are adapted from this tutorial. For the purposes of these directions the USB drive is assumed to be at /dev/sda.

Install and Configure the LAMP Stack

• Install the stack [apt-get install php5-mysql mysql-server php5 apache2]
• Enable Apache’s mod-rewrite [ln -s /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/rewrite.load]
• Allow .htaccess overrides in apache’s configuration file [vim /etc/apache2/sites-available/default] by changing all “AllowOverride None” configurations to “AllowOverride All”
• Edit your /etc/rc.local file [vim /etc/rc.local] to add the following line [mount /dev/sda2 /mnt], run the command on the command line [mount /dev/sda2 /mnt], and add a web root directory [mkdir /mnt/webroot]
• Edit the Apache configuration file to point to your new webroot [vim /etc/apache2/sites-available/default] to change all references to /var/www to /mnt/webroot. ** Note - its important to setup a separate webroot on the ext2 system since apache is not a fan of the FAT16 filesystem **

Install and Configure Drupal

• Create your MySQL database [mysql -u root], set the appropriate permissions, and copy over your Drupal database (or install a new one)
• Copy over the Drupal code files to your webroot at /mnt/webroot
• Do any other Drupal configuration needed

Set up “Kiosk Mode” in Linux

• Create a shell script [vim /home/ubuntu/kiosk.sh] to execute [/usr/bin/firefox —fullscreen http://127.0.0.1] ** Note - sometimes it takes a second for apache to load and the script may need “sleep 30” at the beginning to give apache time to load **
• Configure Ubuntu at (System —> Preferences —> Sessions) to run the kiosk.sh script at Startup
• Install the AutoHide Firefox Extension
• Add a line [rm /home/ubuntu/.ICE*] to your GDM runtime to remove ICE Authority File Locking Issues [vi /etc/gdm/PreSessions/Default]

To boot off of the USB key, insert the USB key into a computer and set its BIOS to boot from the USB key. Wait a couple of minutes for it to boot up and enjoy the Drupal.

After spending a few months bouncing around San Francisco coffeehouses and our all too familiar homes, Chapter Three now has a brick and mortar headquarters. Our 1000 square ft office space is located at 20th and 3rd St. in the Dogpatch District of San Francisco. Check us out: Google Maps · Yahoo! Maps

A former cannery, the fruit slicers and can labeling machines have been replaced by laptops, complimentary organic juice and ever present indie rock. Since we enjoy surrounding ourself with genius, we have offered Neil Drumm (of Advomatic) a regular workspace so he can join our daily Drupal party.

This is another step forward for Chapter Three and we are excited about utilizing our office space to be even more productive and collaborative. We have lots of space and tasty snacks so we invite any Drupal contributors in the San Francisco area to stop by for the day and see the office.

I have a friend who is attending graduate school and mentioned that the graduate students in his program each had collected a sizable number of different academic documents (term papers, book and article lists, obscure or important pdf journal articles) and wanted to share them together in what the librarian in me would describe as a digital “group library”. He asked me if I knew of any possible solutions. I recommended Drupal.

In about an hour of research + configuration, we pulled together a few cool drupal modules and got a digital library up and running. The digital library stores its records in custom CCK fields, were presented to users using some slickly configured Views, and used tags all over the place with a cool tag cloud from Tagadelic. With a bit of tinkering, I was able to use some Word2TXT and PDF2TXT extraction tools to generate searchable text for the document and save it to the node type using a nodeapi hook on submit/update. I also turned on the ability to leave comments and discuss documents and the RSS feed to let people know when new documents are uploaded.

The whole experience left me feeling really positive about the potential to develop some really cool digital library functionality using Drupal and maybe even move towards a “Drupal Digital Library” install profile. There are a few cool open source digital library projects (including my favorite, Greenstone), but I think Drupal has a lot to offer not only in the storing + sharing of documents. Plus, there is some great potential to develop a community of people sharing + commenting on those documents.

As we’ve posted about before, the past year has seen almost 100% growth in the Drupal economy. Several great new companies and several good coders have emerged, and a new ethic of training and local groups has taken hold, and yet there just aren’t enough savvy web developers to meet the demand for Drupal-related services.

Like every professional services shop that plays heavily — our in our case works almost exclusively — in the wonderful world of Drupal, we’re feeling that talent shortage. And it occurs to me that poaching talent from our “co-opetition” and/or headhunting people with existing IT jobs — two popular ways to build out a development team — isn’t a long-term solution to this problem.

It Has Begun!

To that end, I’m excited to announce the formation of The Drupal Dojo, a community group dedicated to increasing the proficiency of apprentice and journeyman-level developers. The Dojo will work through the groups.drupal.org forums, and an IRC channel.

A middle-tier developer community is good for the project as a whole (#drupal intimidates a lot of people), and for enterprising individuals like myself, the idea of being able to build stronger relationships with up-and-coming developers makes obvious business sense. Seems like a classic win-win. I’ll be posting on group progress as it evolves.

I’ve arbitrarily said we need five learners and at least one other “expert” to start out. If you’d like to participate, subscribe now and help me get the ball rolling.

UPDATE:
Well that’s a surprise! 72 hours after creation we have 175 subscribers. The ratio of “master” to “apprentice” may be a bit off, but I think with the number of participants we’re bound to have some good results.

Tremendously successful web startups tend to solve very obvious problems. They innovate in how they do things, not in what they do.

Take for example:

Sold for $1.65B, now the ninth most trafficked website on the internet. Their innovation? Making it stupidly simple to put your videos online and share them.

Making a successful business is incredibly hard, you’re nearly always best off keeping things as simple as possible. Most people I talk to seem to think the success has more to do with the right idea with the right people at the right time, i.e. luck… it is that, but much more. Just as important (if not more) as those obvious factors is something much more subtle. In a startup a million and one things can go wrong. The wrong hire, the wrong market, wrong brand, wrong back end technology… etc etc. If you get just one thing wrong chances are you’re toast.

But people win… some routinely, the ones that seem to play the game with ease. The savvy ones have read the rules better than anyone and apply strategy where as others rely on just their guts and friends. The general two-bit understanding of the business horizon for the technology sector reads as follow: backed by the drum beat of Moores law, technology will steadily advance at an exponential rate, and in our lifetimes we will see innovation flooding all technology gaps until all floats atop a seamlessly perfect techno-utiopian first class world.

It is in the quickly flooding narrows of sillicon valley that the nerd-cowboys catch their waves and jostle for control of the floods, navigating their carbon fiber racing yachts and suborbital starships from opportunity to opportunity. But it is not brawn, speed, or fitness that counts the most in this race, it is your timing.

As my Bulgarian fencing coach Iana Dakova could demonstrate with ease on her students, the best fencers were those who were not competing through their blades and finessed footwork, it was the ones who had such impeccable physical game that they could afford to think and strategize, in the middle of the bout, in between the tenth of a second second actions and reactions. These were the ones who could always kick your ass. The ones who could turn your own sense opportunity, your own sense of timing, and spin it’s momentum back around and impale you with it so quickly and cleanly that you knew you were through even before the blade hit, and the buzzer sounded, and the judge or anyone else watching had time to even think about what just happened.

Since YouTube.com got bought for 1.65B everyone i’ve heard talk about it mentions something like “But it’s just a stupidly simple website that people used to upload video, how could it be worth so much!” But think back a bit. In 2005 it was nuts that nobody had yet figured out how to make video work the web. We had 40% broadband penetration, and our juicy 25 and under target market spending more time on the web than on any other medium… the kids who used to watch more tv than anyone and watching more and more each year were suddenly ramping back the tube watching in favor of online games, websites, and im’ing. But nobody figured out how to get them video. I guess in the mentality of the time it could be rationalized that “The web is so much richer a medium than tv. Kids don’t care anymore about video”. But that’s a load of crap. Kids were only online because the content available in a medium no richer than FM radio and newspapers, was better than the extremely well produced video content pumped through the now ignored tv’s in their living room.

It wasn’t that they didn’t want video, it was that they found internet non-video more interesting than TV. But as soon as YouTube figured out how to create internet TV they had a new daily destination, and one little dot-com miracle came true.

Ok, but couldn’t that be explained as simply right place right time?

No, I don’t think so. People have been putting video on the web since the web took off.

So, what did YouTube get right?

I think the biggest reason YouTube was successful was something so incredibly simple that when I explain it you’ll probably think I’m full of crap (which often times I am). YouTube won because they encoded their content as flash files. Hear me out… The norm for video content at the time was to offer it up in a stream-able format such as Real Video or WMV which were theoreticaly easier to copy protect. The only problem was that they hardly ever worked. Do you remember buffer hell? Of course video on the web never took off, it never worked. But YouTube did theirs in flash, with a big play button and a pre-loader, so videos would load instantaneously and would never stop halfway through. And so people watched them, uploaded their own, told their friends, and within half a year YouTube had a daily audience of more than 100M.

YouTube won because they solved the simplest problem, literally how to put video files on the internet in a way that other people could actually play them. And that was stupidly ingenious. And that is why they won.