Pragmatic IT

IT Infrastructure and Software Development from the Customer's Perspective

Whither the CBC

The CBC like many media outlets has been in crisis recently. On top of repeated cuts in government funding over the last twenty years, they’re suffering from the collapse of advertising revenue to traditional media, due to the television services’ dependency on ad revenue.

(U.S. readers: The CBC is the government-funded public television and radio broadcaster in Canada, like the BBC in the UK or the ABC in Australia.)

To me, the role of the CBC is to provide channels for Canadian cultural expression. Market forces will inevitably regurgitate what is produced by the massively-productive American media machine, so we need someone to provide the infrastructure for cultural dissemination. Nothing in that says that the CBC has to be a radio or television network in the traditional sense.

In the post-network, post-newspaper world, I think there are a lot of exciting opportunities for public broadcasters to facilitate even greater promotion of Canadian (or British, or Australian, or South African or New Zealand or …) culture. So here’s my road map for CBC over the next decade. It consists of two themes, distribution and production:

It should be obvious that people don’t think of network television or radio as their primary way of getting information or entertainment anymore. Music comes on a iPod. Video comes from YouTube. I want to listen to my local CBC radio morning show on my iPhone, because that’s what I take with me when I walk my dog. And I want to view CBC TV on all those devices, too.

CBC should stream all its content, in local timezones, and in open source formats. This means CBC should support open source projects that develop and/or package codecs and whatever other software is needed to ensure that users of all platforms have access to CBC audio and video content: Mac, Linux, Windows, iPhone, Blackberry, Android and other phone platforms. A lot of this technology is available already, but much of it is only accessible to us hard core geeks. It needs to be really easy for anyone to use.

Even more exciting is the possibility for production: Enable the public to produce their own high-quality podcasts and video. This means, again, funding open source projects to develop and package audio and video production tools. Having done a bit of screencasting, I know it’s not easy for the general public to do today, but we’re not a long way from it being easier.

The CBC, or Canadian universities, could also produce a series of screencasts or videos to show people how to produce content: A sort of on-line journalism and documentary production school. And there’s no reason to limit this to news and documentaries. The CBC, NFB and universities

This is technically feasible. I believe it’s economically feasible within a budget that CBC has or could get. Maybe they can cut real costs (rather than cutting services) from the traditional parts of the broadcast system by moving programs around on the Internet, instead of via satellite or other dedicated pipes. Maybe they can increase their budget, by showing the government they’re more relevant than they’ve been in years (not likely with Harper as PM, but he’ll be gone someday).

Showing Lists of Events in Drupal 6

I wanted to have a page in Drupal that showed upcoming events for the Parent Advisory Council of my son’s school, with a tab making it easy to show past events. The idea was to have one place to go to find out about upcoming events, or to find previous events so you could get materials from the event (e.g. handouts provided by speakers).

Drupal has a pretty good module for showing calendars, but it shows them on day, week or month views in a calendar. My users wanted to see a list of events. Our events are more spread out, so a list or table view is actually easier to work with.

So to summarize, we want:

  • A menu in Drupal's primary menu for "Events"
  • The menu takes the user to a page that shows upcoming (in the future) events
  • There is a tab on the page for "All Events". Clicking the tab shows all events, paged with 20 events per page
  • There is also a tab for "Upcoming Events". Clicking it takes you back to just the future events
  • Events have to have a start and end time, and may have an additional description and attachments (e.g. agenda, minutes, hand-outs)

The high level overview of what I did is:

  1. Install and enable some contributed modules
  2. Create a "content type" for events. This defines what information you're going to store for each event
  3. Define a "view" and two "displays" for the view. The view and its displays are partly a query of the information you want to appear, and partly the definition of the menus that get you to the information. The format of the information is defined by your theme, which I won't talk about in this post
  4. Put the menu in the right place
  5. Create some events and test it out. Testing is really important. It's almost impossible to get this right the first time

A small warning before you get started: The smallest error in anything can lead to very bizarre behaviour, or simply to some part of the page not showing up, with no apparent connection to the error. It took me days to get this working (with a lot of side-tracks into Drupal esoterica). Don’t expect to get it right the first time, even with detailed instructions.

Here’s what I did. I don’t guarantee that I’ve covered every single thing, so please let me know by commenting on this story if I missed anything. (I assume you know how to do Drupal basics like install modules.)

  1. Install the Content Creation Kit (CCK), Views, Date, and Advanced Help modules. Date module version 6.x.2.1 won't work for the use case we're trying to implement. The filter relative to a date (e.g. "now") is broken, which means you won't be able to show upcoming events. Use date-6.x-2.x-dev (see [http://drupal.org/node/426990][1]) until the fix gets rolled into the release version
  2. Enable the "Content" part of CCK
  3. Enable all parts of Date except "Date Locale" and "Date Tools", unless you want them for other reasons. Also, if you're running on PHP5 you don't need "Date PHP4"
  4. Enable the "Views" and "Views UI" parts of Views

Now, create the content type for events:

  1. Administration-> Content Management-> Content Types
  2. Click on the "Add content type" link at the top of the "Content types" area
  3. Enter the name "Event" and type "event". There are many options to set, and the help text is pretty self-explanatory. Under "Workflow settings", "Attachments", make sure they're enabled (since we said that was a goal of what we're doing). Most of the other options depend on your needs, although under "Workflow settings" it's likely you'll want to turn off "Promoted to front page" under "Default options"
  4. Click the "Save content type" button at the bottom of the page
  5. Click on "Manage fields" beside the Event content type
  6. Under "Add", "Existing field", select "Datetime: field\_event\_datetime..." You may have to scroll down to see the existing fields. If you don't see an existing field called "field\_event\_datetime", then create one under "Add", "New field"
  7. Click "Save"
  8. You will be presented with another page allowing you to set a bunch of options. Set them according to your needs
  9. Click "Save"

Now create the view. You really should read the Drupal Views documentation, which you have available because you installed the Advanced Help module. Read it now. Click on Administer-> Site building-> Views and you’ll be given a link to the documentation near the top of the “Views” area

Now, here’s my summary of what’s important in the Views documentation:

  • A view can have several displays. The display determines where the view is seen, and can also override some parts of the view's default settings. In the rest of this post, I use "view" and "display" in these specific meanings
  • Set up the common aspects of the view first in the Default display. In our case, that means the output format (a table), the fields to be viewed, and some of the filters
  • Be very careful when editing non-default displays to click "Override" before editing anything that you want to be unique to the display. If you don't explicitly click "Override", you'll change the default which will affect other displays that use the default value

So what we have to do is:

  1. Create the view
  2. Set up the default for the view the way we want it
  3. Create a "Page" display for the "Upcoming Events" tab
  4. Move the menu to the right place
  5. Enter some test events
  6. Check that the "Upcoming Events" tab works
  7. Create another "Page" display for the "All Events" tab
  8. Check that the "All Events" tab works

Here are the steps:

  1. Administer-> Site building-> Views
  2. Click the "Add" link at the top of the "Views" area
  3. Put "event\_list" in the "View name" field, and "Event List" in the "View tag"
  4. Make sure the "View type" is node and click "Next"
  5. This brings you to the Views UI page. I found this page to be extremely unintuitive. You have to be patient and always remember to scroll down after clicking on something, because many links simply change what's displayed below what's likely visible when you click on the link

Set the defaults for the view:

  1. Under "Basic Settings", click "Unformatted" under "Style". Scroll down. You'll see some radio buttons. Click "Table" and click "Update". Ignore the error message that appears about no fields defined. We'll get to that later
  2. Beside ""Use pager", click "No". Scroll down. Select the "Mini pager" radio button and click "Update"
  3. Beside "More link", click "No". Scroll down. Select the "Create more link" check box and click "Update"
  4. Under "Fields", click on the "+" beside "Fields"
  5. Scroll down. Click on the check boxes for "Content: Event Dat and Time... From date". Don't check the box for the corresponding "To Date" field. Both are included in the "From date". Click on the check boxes for "Node: Body", "Node: Title", and "Upload: Attached files". Click "Add"
  6. You'll now be presented with a number of pages to allow you to set options for each field. For "Event Date...", accept the defaults by clicking "Update"
  7. For "Node: Body", accept the defaults by clicking "Update"
  8. For "Node: Title", click the "Link this field to its node" checkbox and click "Update"
  9. For "Upload: Attached file", click the "Link this field to download the file" checkbox and click "Update"
  10. Now, put the fields in the order you want: Click the up and down arrows by "Fields". Scroll down. Drag "Title" up to the top and click "Update"
  11. Set the sort order: By "Sort criteria", click the "+". Scroll down. Select "Content: Event date and time" and click "Add"
  12. In the next page, select "Ascending", so the next event will appear at the top of the list. Click "Update"
  13. Choose the nodes you want to display, specifically the published event nodes. Click the "+" beside "Filters". Scroll down. Select the checkbox for "Node: Published" and "Node: Type" and click "Add"
  14. On the page for "Node: Published", click the "Yes" radio button and click "Update"
  15. On the "Node: Type" page, click the "Event" checkbox under "Node type" and click "Update"
  16. Click "Save". We have our default view set up.

Create the “Upcoming Events” page:

  1. Click the "Add Display" button (the drop down box above it should read "Page" already)
  2. Under "Basic settings", click on "Page" beside "Name". Scroll down. Type in "Upcoming Events" and click "Update"
  3. Click on "None" beside Title. Scroll down. Click "Override". Enter "Upcoming Events" and click "Update"
  4. Under "Page settings", click on "None" beside "Path". Scroll down. Enter "events/upcoming\_events" and click "Update"
  5. Click on "No menu" beside "Menu". Scroll down. Select the "Default menu tab" radio button. Enter "Upcoming Events" under "Title". Click "Update"
  6. On the next page, click on the "Normal menu item" radio button, and enter "Events" under "Title". Click "Update"
  7. Click on the work "Filters". Scroll down. Click on "Override" and then "Update". This overrides the filter selection but makes a copy of the two filters that we set as default
  8. Click on the "+" beside "Filters". Scroll down. Click on the checkbox beside "Date: Date (node)". Make sure you get the right one. There are three fields that start with "Date:". Click "Add"
  9. Scroll down until you see a heading that says "Date field(s)". Select the checkbox beside "Content: Event Date and Time... From date". Click "Update"
  10. On the next page that appears, under "Operator", pick "Is greater than or equal to". Under "Date default", enter "now". Click "Update"
  11. Click "Save". Note that after clicking "Save" you're again editing the default settings, not the upcoming events display. Be careful to select the display you mean to edit. I got that wrong lots of times

Set up the menus correctly:

  1. The above puts the menu under the site navigation menu. We want it under the Primary Links menu. Click on Administer-> Site building-> Menus
  2. Click on "edit" beside "Events"
  3. Under the "Parent item" heading, select "" from the drop-down list [1]: http://drupal.org/node/426990

Building a Website for a Grassroots Group

As I’ve been blogging about recently, I’ve been building a Drupal-based website for my son’s school’s Parent Advisory Council (PAC). As a geek, the technical nature of putting together a content management system like Drupal for a site is interesting, and I learned a lot. However, what really interested me is the social aspect of it.

You can’t have a business it seems without having a website. That must mean there’s value for a business to have a website. But is there value for a grassroots group to have one? If technology truly is neutral, then grassroots groups must be able to benefit from websites. (I’m not sure technology is inherently neutral, but that’s another topic.)

When we started, the PAC has a static HTML website updated infrequently by a parent who knew some HTML and how to FTP files to his server. It was out-of-date, since the parent’s children had moved on to high school and he was obviously not so active in the PAC. Members of the PAC did want some place to post meeting dates and minutes, and to publicize their activities, so I offered to help out.

We set up a committee with four members. I built a test site under my company’s site (jadesystems.ca) and we started to work.

Right from the start a few people really took to it and started to use it in ways that I couldn’t keep up with. This included translation into Chinese, especially after I enabled the translation modules in Drupal.

I also set myself up to do screencasts, and posted a number of them. It’s still early, but I’ve got some feedback from people that they find them very useful. For grassroots groups, where you don’t site face to face with people every day, I think screencasts are going to prove to be an excellent tool.

I think the main lesson I’ve learned so far is this: I like Drupal and it’s allowed me to do a lot. However, I can’t help feeling that I’m writing in assembly language for content management. What I mean is that I’ve had to learn too much about how the system works to get it to do what I want. We’re still a long way from grassroots groups being able to set up a website without someone with technical skills.

My dream would be something analagous to blogging. You simply go to a website and create your own community site. The task is inherently more complicated than setting up a blog, so it will never be as simple, but the skill set should be relatively similar. You should be able to pick and choose based on functions the user wants (e.g. a list of upcoming events), not functions based on how the programmer delivers them (e.g. a Views module with an obtuse and fragile configuration screen that lets you cobble together a list of upcoming events if you bang your head on it long enough).

More on this later.

Parent Advisory Council Website

The Parent Advisory Council (PAC) website is live at http://hastingspac.ca/ and is in a relatively stable form. Much work remains on the translations, but the bones of the site are there and it’s up to everyone else to put up the content (muscle).

I think we’ve got a pretty good site for a volunteer group. Our goal is to increase participation in the PAC to benefit the education of our children.

I’ll be posting more about the technology behind the site (Drupal and content management systems) this morning.

I’d love to hear comments and ideas you might have about the site.

Time Comparison in Ruby, especially on Rails

I had a bizarre failure in some test cases. The test case looked like this:<pre>marker = Time.now
t.create(args)
assert t.created_at.between?(marker, Time.now)</pre>In other words, take the time, create an object, and make sure it was created between the first time and now. Obviously true, but my test was failing.

The reason: Rails keeps the create time to the second, but Ruby keeps time to fractions of a second (depending on the resolution of your hardware clock). Both would have the same time (to the second) but the marker would have some microseconds more than t.created_at, so the assertion would fail.

The Ruby documentation actually warns you about this, but in way that’s cryptic enough that it might not be obvious why your simple test case is failing in such a bizarre way.

The solution? Mine was make the time an integer and convert it back to a time using Time.at:

marker = Time.at(Time.now.to\_i)  
t.create(args)  
assert t.created\_at.between?(marker, Time.now)

This doesn’t really merit a blog post, except that I Googled for solutions to the problem and didn’t find much.
VMware Server 2 and Ubuntu 8.04 LTS

Some irritating behaviour that I couldn’t resolve pushed me to the big upgrade of my virtual infrastructure. I was running VMWare Server 1.0.5 on Ubuntu 6.06 on a Dell SC440. Periodically all the VMs would just lock up for about five minutes. All the VMs would freeze, and even the time (as reported by date) would fall back by five minutes. The host ran just fine, and reported that nothing was happening (e.g. top(1) reported 100% idle).

I was having to endure too many frantic calls from my son that he couldn’t get to lego.com, so it was time to do something.

Some Internet research turned up that I would need VMware Server 1.0.6 at a minimum for Ubuntu 8.04, so that meant I would need to do VMware first, and therefore go to VMware Server 2.

The upgrade to VMware Server 2 went fairly smoothly, but I had a couple of problems that sucked far more time than the solution eventually warranted:

  • The management interface didn't work. When I connected to vmhost:8222 I got the grey VMware background, but it didn't show the login window. I solved it by some combination of restarting the VMware management server on the VMware host (sudo /etc/init.d/vmware-mgmt restart) and clearing the cache in FireFox
  • Once the management interface was up, I couldn't get the console to open on many of my VMs. The error window told me to look at log files and report the problem. The VMs were version 4 of the "hardware". I upgraded the VMs' "hardware" to the current version (7 I think) from the management interface, and was able to open the console
  • When I was installing VMtools, I was unable to bring the network interfaces back up. I spent much time trying to figure out what was wrong with VMtools before I realized it was a simple as my DHCP server had gone away. My DHCP server is in a VM, but I don't know if anything about the upgrade is what shut down the DHCP server software

The upgrade of Ubuntu went smoothly and quickly. The big gotcha for me was that I forgot to download the kernel headers before I rebooted the upgraded server. When I went to rebuild VMware Server for the new kernel, I couldn’t and since my network depends on VMs, I had to do some manual network configuration to get out to the Internet and download the kernel headers.

P.S. The reason I decided I should try the upgrade is because the time handling in Linux kernels is now much more friendly to running in a VM, whereas the Ubuntu 6.06-era kernel wasn’t. I haven’t been running long enough to know if the upgrade has fixed the freezing problem.
Tika and Solr

This is just a quick note to document another experience with Solr.

Background: To index Word, Excel, PDF and other “unstructured” documents, Solr uses Tika, another Apache project. Tika comes bundled in Solr and is ready to run in Solr. However, if you want to run Tika individually (e.g. you don’t trust your installation, or you’re just curious) you have to copy a few .jar files around (Java experts who can manage class paths will probably tell me there’s a better way to do this).

I did <pre>cd [Your path]/apache-solr-nightly/lib
cp commons-io-1.4.jar commons-codec-1.3.jar [Your path]/apache-solr-nightly/example/solr/lib
cp ~/.m2/repository/org/jempbox/jempbox/0.2.0/jempbox-0.2.0.jar [Your path]/apache-solr-nightly/example/solr/lib</pre>(I have no idea where ~/.m2 came from. It may have been when I ran the Tika build.) Then I could run <pre>java -jar tika-0.2.jar</pre> in that directory.
Solr and Rails

Well, after some long diversions I have Solr working in some simple test cases with Rails. The long diversion was partly caused by not understanding what was offered by the Rails Solr plug-in, so I’m going to give an overview here, and a link to detailed instructions for Solr in Rails at the end of this post.

The Rails plug-in for Solr from git://github.com/mattmatt/acts_as_solr.git includes a complete installation of Solr. You don’t need to install Solr separately. (My “long diversion” is that I rushed off and installed Solr separately, and spent a fair bit of time getting it running due to my ignorance of how it worked.)

If you want to index Word, Excel, PDF, and other types of documents, there is a bit of additional configuration to do. To index those files types you have to get a nightly build of Solr from [here][1], and copy some files and directories as described in the link at the end of this post. You have to add the following lines to example/solr/conf/solrconf.xml:

 <requestHandler name="/update/extract"
class="org.apache.solr.handler.extraction.ExtractingRequestHandler"> 

[1]: http://people.apache.org/builds/lucene/solr/nightly/

Rsync iPhone

If you’re using the Cydia installer on your iPhone, there’s a package for rsync. Just open Cydia and search for “rsync”. You won’t find a “BSD subsystem” package, because they say Cydia comes with BSD. It might, but it doesn’t come with rsync hence the extra installation.
Creating Screencasts on Ubuntu

I’m building a web site using Drupal 6 for my son’s school’s parent advisory council. The idea of the site is to facilitate community participation. Since we’re a volunteer group and we aren’t at the same workplace every day, I thought screencasts might be a great way to help people learn how to use the site.

Archive