Pragmatic IT

IT Infrastructure and Software Development from the Customer's Perspective

Microsoft Considering Again Buying Yahoo

Microsoft is again considering buying Yahoo. According to one MS executive, the acquisition would allow them to “obliterate AOL.” AOL still exists?

As a Yahoo shareholder, I hope MS buys. If I were a MS shareholder, I’d be afraid, very afraid.
A New Computer -- Installing VirtualBox

I installed VirtualBox from the Ubuntu Software Centre on my new computer. I had already copied over my VM – it came with all the other files when I copied everything under /home.

When I figured out how to run VirtualBox under Unity (it’s under Accessories), it came up and knew about all the VMs I had on the old machine. When I started my Windows VM, it complained that it couldn’t find “/usr/share/virtualbox/VBoxGuestAdditions.iso”. Sure enough, the iso wasn’t anywhere on the new machine.

However, for my purposes I didn’t need the iso. I just unloaded the iso from the virtual CD device for the virtual machine, and tried restarting. It worked.

I would need the guest additions iso sometime, so, with the virtual machine running, I went to the Devices-> Install Guest Additions menu. It asked me if I wanted to download the file from the Internet, and I said “yes”.
A New Computer

The backlight died on my Lenovo x300 a couple of weeks ago, so I bought a Dell Vostro with Ubuntu pre-installed. Here’s how I got from one to the other:

The Dell website said I was getting Ubuntu 11.04, but out of the box the computer had 10.10 on it. My first try upgrading to 11.04 failed. When it tried to reboot, it said the device for / wasn’t ready. Fortunately, I somehow got to the old grub screen that let me choose which image to boot, and at the bottom there was an option to return the box to factory state.

I tried that, and it worked. I had about 10 minutes of the spinning “wait” cursor before it started doing something, but by being patient I got the box back to factory state.

So I went through the process of updating 10.10, reboot, then upgrade to 11.04 again. This time I saved the sudoers file. I was asked whether to keep the old one or use the new one as part of the installation.

I’m going to force myself to use Unity for a while. That’s causing me some grief, but I’m already finding so of the tricks that make Unity more productive (try right-clicking the “Applications” or “Files & Folders” icons in the Launcher bar).

To get my old files over from the old computer, I connected a 1 TB USB drive to the old laptop, shut down Evolution, virtual machines, and any other applications that might be updating files while I copied, and did:
Network Team for Data Centre Relocations

I had a real “Doh” moment this week. We’re about 80 percent of the way through relocating a 500 server data centre. Things have been going pretty well, but right from the start we’ve found we were under-staffed on the network side. We have a pretty good process in place, with what I think is just the right amount of documentation. The individuals working on the network team are excellent. We even brought in one more person than we had planned for, but we’re still struggling with burnout.

The light bulb went on for me a few days ago: Here, like other IT organizations of similar size and purpose, we have about ten times as many server admins as we do network admins. We have a bigger pool of people on the server side to draw from when we organize the relocations, which typically happen over night or on the weekend. While we rotate the network guys for the nights and weekends, it’s a smaller pool. They do shifts more often, and they more often have to come in the next day, because they have to plan for the next relocation on their list and it’s coming soon.

Another factor is that, in our case, the network team started intense work earlier than everyone else. We’re occupying a brand new cage in a data centre, so the network team had to build the whole data centre network first. They did three weeks of intense work before we moved any servers at all.

As we hit six months of intense work for the network team, the strain is showing. We’re going to try to rearrange some work and delay what we can. Other than that, we’ll probably have to suck up the remaining 20 percent of the project somehow.

In the future, I’m not sure what I’d do. One approach, if I had enough budget, would be to hire a couple of contract network admins well in advance. Tell them they’re going to work Wednesday to Sunday, and often at night. Train them up ahead of time so they’re as effective as your in-house people. Then give most of the nasty shifts to the contractors.
The Data Centre Relocation Calendar

I’m past the half-way point in relocating a 500-server data centre. The servers are a real variety – a typical medium-scale business data centre. We’re using mostly internal resources, supplemented by some contractors like myself with data centre relocation experience.

I chose not to come in and impose a relocation methodology on everyone. There are a lot of reasons for that, some of which were out of my control. Rather than using a methodology out of the can, I tried to foster an environment where the team members would build the methodology that worked for them.

This turned out to be quite successful. One of the items that emerged fairly late for us, but was very successful, was a shared calendar in Microsoft Outlook/Exchange. (The tool wasn’t important. It could have been done in Google Calendar. Use whatever your organization is already using.)

The shared calendar contained an event for every relocation of some unit of work – typically some service or application visible to some part of the business or the public. Within the calendar event we put a high-level description of what was moving, the name of the coordinator for that unit of work, and hyperlinks to the detailed planning documents in our project shared folders. The calendar was readable by anyone in the corporation, but only my team members could modify it.

What struck me about the calendar was how it organically became a focal point for all sorts of meetings, including our weekly status meeting. Without having to make any pronouncements, people just began to put the calendar up on the big screen at the start of most of our meetings. We could see at a glance when we might be stretching our resources too thin. The chatter within the corporation that we weren’t communicating enough diminished noticeably.

Based on my experience, I’d push for a calendar like this much earlier in the project. We built it late in our project because I had a lot of people on my team who were reluctant to talk about dates until they had all the information possible. We got so much value in having the calendar that I think it’s worth it to make a calendar early in the planning stage, even if the dates are going to change.
Linuxcon 2011 Part II

I went to a lot of cloud computing-related talks at Linuxcon 2011. One of the better ones was by Mark Hinkle of cloud.com.

One of his slides showed what he considers the five characteristics of cloud computing. Two important ones for him are self service, and a measured service. I think those are two useful criteria for distinguishing between a VMware cluster and a cloud that is distinct from a VMware cluster.

It was clear listening to all the talks, including Mark’s, is the role of open source in the large clouds. Basically, anyone big is building their service on the open source cloud stacks. Of course, there are a number of open source cloud stacks. One of the challenges is to pick which one to use.

Fortunately, there are serious supporters behind the three main stacks. Eucalyptus has a company called Eucalyptus Systems backing it now, headed up by Marten Mikos of MySQL fame. Cloudstack has cloud.com which is part of Citrix. And the OpenStack project is backed by Rackspace and NASA.

One factor that seems to be important is the hypervisors supported by the cloud stack. OpenStack supports the most right now.

Something that struck me listening to the talks is that the cloud, like so much in IT, isn’t a slam dunk solution by itself. You need to know what problem you want to solve, and then figure out how to use the cloud to solve it, if indeed the cloud is a solution to your problem.

Related to that insight, it’s clear that unless you solve the problem of monitoring your infrastructure with Zenoss or Nagios, and of provisioning it with Puppet or the like, then you’re not going to see much benefit from the cloud.

Linux 2011 Part I

Linux is 20 years old this year, and Linuxcon was in Vancouver, so I had to sign up. The conference ended yesterday. There were a lot of good speakers. As a bonus, we also got to hear some poor guy from HP give a keynote about HP’s great WebOS play, at almost exactly the same time as his company was killing the product line.

What I was looking for, frankly, was a business opportunity for a small consultant/system integrator like Jade Systems to use Linux to help businesses with 1,000 servers, give or take a zero at the end. The most obvious opportunity I came away with is storage.

I’ve written before about the cost of enterprise storage. There are tremendous opportunities with hardware solutions like Backblaze’s storage bricks, and the software that will make it all work is Gluster. Install Gluster on a couple of boxes with storage, and you have synchronous replication (locally) or asynchronous replication (over the WAN). It provides what you need to store your virtual machines and move them around your data centre as load or availability needs dictate. It can be your large, reliable, network attached storage device for all your spreadsheets and documents.

Gluster grew out of the needs of a supercomputing project at Lawrence Livermore Labs in 2004 and have an impressive list of users today. They’re working to integrate with the OpenStack cloud computing stack to provide a complete cloud storage solution for OpenStack.

This is certainly a solution that could support a business case.

Making DVDs

My son’s class at school raised and released some salmon this spring, and he had a project to produce a video about it. I offered to edit the raw video together. That spun into a project where I ended up putting together four short videos. Then I put them all on one DVD with a menu. Of course, it wasn’t easy. It turned into another episode of all my spare time for two weeks being spent trying to do something useful with a computer.

First I had to find the right tools for Ubuntu (10.04 in my case). It turns out that Kino, the video editor I had used before, is no longer under active development. It looks like everyone is using OpenShot now. It’s in the Ubuntu repository, but I found there were sufficient new features with the more recent version that I followed the instructions to get it from their PPA repository.

I found OpenShot to be quite intuitive. There’s also pretty good documentation. I think I observed a few random crashes, so it’s worthwhile to remember to save your work frequently. There is an autosave feature as well, but you have to find it and turn it on. It isn’t on by default.

OpenShot will take your completed video project and turn it into a DVD image ready to burn. That’s very slick. My earlier attempts at producing DVDs led to a lot of command line fiddling, and I found it very easy to burn DVDs that didn’t actually work. OpenShot made it easy.

The next part of my challenge was to put four videos on the DVD and stick a menu on the front. I found documentation here and here and elsewhere on how to do it (the colour scheme is a killer). However, it turns out that making DVD menus is very picky and error prone, at least for me. I never got anything working consistently.

Finally I found DVD Styler. Again, it’s in the Ubuntu repository so it’s easy to install. It has a GUI and lets you set up a menu, including automatically doing the typical “Play all or episode selection” scenario if you have multiple videos.

There were a couple of tricks I discovered along the way:

  • Some of the original videos were shot in HD. OpenShot can't deal with HD. I had to use ffmpeg to convert the format to ordinary MPEG-2 video ("ffmpeg -i input.mp4 -target ntsc-dvd -y output.mpg" if you're making a DVD for North America)
  • OpenShot by default makes a DVD. You don't want that when you're planning to use DVD Styler to make a menu. I had to go back and re-export my videos from OpenShot as "mpeg2" videos
  • DVD Styler sets up the DVD menu buttons to use "auto" navigation. That didn't work on my cheap DVD player. I had to go into the properties window for each menu button, and under "focus", change the target of each button to the explicit value I needed (this description will make sense when you look at the properties window for the buttons)

DVD Styler lets you simply preview the menu, create an ISO file, or go straight to burning the DVD. I created the ISO so I could test it with video players (I used VLC and mplayer). Using the players didn’t expose the menu button problem I mention in the second bullet above, but otherwise was a worthwhile step.

I didn’t try to do anything but the default menu background in DVD Styler. If you’ve done it, please comment here with your experience.

Unfortunately, I can’t post the final product. Since it was done at school, they’re very careful not to release anything publicly when they don’t have all the parents’ permission. Makes sense, of course.

Rhythmbox Wouldn't Rip to .m4a

I had a problem yesterday where Rhythmbox wouldn’t rip a CD to .m4a format. I went to the Preferences window to set the format, and it wouldn’t appear in the drop-down. Curiously, if I clicked the Add button it would show the existing formats, and .m4a was in that list.

After much flailing, I discovered that you need to have both gstreamer0.10-plugins-bad and gstreamer0.10-plugins-bad-multiverse installed. I had removed the multiverse one a few weeks ago when I thought I was cleaning up after a previous round of flailing to get something around music working.
Debugging Windows Shares and Samba

I wrote here about a problem I had connecting to a Windows share that was using DFS from my Ubuntu 10.04 laptop. Turns out I was wrong about the issue being related to authentication.

The issue was that my server was sending back a string without the Windows file separator at the end (backslash “"). I simply patched the code at the relevant place to check if the file separator was missing and put one in if needed. Not the most elegant patch, but probably less likely to impact any other code.

I’m being careful not to call this a bug in Samba. The file server was an EMC NAS implementation. Fairly common, but not a standard Windows server sharing files. Perhaps the EMC is out of spec on this point, or perhaps the spec is ambiguous on trailing file separators. Given the number of devices out there, I think it’s reasonable to make the Samba client code handle the case, regardless of what the spec says.

I fed the patch to Ubuntu, because that’s where I originally logged the issue. Building Samba was remarkably hassle free, as was getting the source from their git repository, thanks to good documentation for both. My thanks to the Samba team for a great product.

Archive