Categories
Tool of the Week WLUG

“Useful” command line tools

A coworker was doing some work on a server we’re building up, and wanted to kill a bunch of processes. The killall binary wasn’t installed for some reason (default etch install, probably just missing the package), but he found a killall5 binary instead.

For those of you who don’t know, killall5 is the SysV version of killall. It’s quite a bit more literal about it’s functionality than the killall most of us are used to – it will, without taking any command line arguments, prompting if you are sure, or any indication of what is about to happen, send a kill signal to all processes.

I was in the middle of saying “Don’t run that” when my coworker did. Oops. Good thing we were still building the server and it was on the build desk next to him.

This got me thinking though. Are there any other command line tools that are similarly dangerous as killall5? That is, they will do something terminal to your system, without prompting for help or confirmation?

Categories
Xen

Resizing domU partitions with Xen Enterprise 3.2

I recently had to resize the root partition of a sarge domU under Xen Enterprise 3.2. The various xe tools and the xen enterprise console let you resize the disk image, but they don’t resize the filesystem for you. If it’s not the root filesystem, you can resize it yourself once you reboot, but you can’t resize a mounted filesystem

There are a couple of XenSource forum posts dealing with this, but they seem to be oriented around Xen Enterprise 3.1, as they don’t work with 3.2. The basic premise is fine – find the device ID relating to the disk image on the Xen Enterprise dom0, and resize it, however the mechanics have changed.

The block devices you get access to under the XE dom0 are actually disk images, with an intact partition table. Attempting to access them directly will fail, because the standard partition resizing tools expect a partition, not a disk image. So you have to use losetup and an offset to bind that partition to a new loopback device, then you can start working on it.

It’s possible there is a much easier way to do this than below, and I’m sure it’s probably changed with the new storage manager codebase under XE 4.0, but I need to document this somewhere.

First up, shutdown the domU, then resize the xvda, either through the XE 3.2 console or via the xe command:

[code] xe vm-disk-resize vm-name=Test disk-name=xvda disk-size=11000[/code]

Now of all, get the right UUIDs from XE:

[code]

# xe host-vm-list
NAME: Test
uuid: 184bea51-9733-43b0-af02-fa6493235218
state: DOWN

# sm info

…..

——> VDI ID: [184bea51-9733-43b0-af02-fa6493235218.xvda]
Name: [NULL]
Descr: [DESCR]
Device: [/dev/VG_XenStorage-1ae909da-1302-45db-939b-1a2e26e78573/
LV-184bea51-9733-43b0-af02-fa6493235218.xvda]
Shareable: [0]
Virtsize: [11000 MB]
Parent UUID: [1ae909da-1302-45]
State: [0]
Type: [0]
[/code]

The “Device” path in there gives us the appropriate device on the dom0 host. In order to keep this displaying nicely, I’m going to refer to it us /dev/path/to/xvda from now on. This is a disk image, not a partition image, which we can prove with the following command:

[code]
# fdisk /dev/path/to/xvda -ul
Disk /dev/path/to/xvda: 11.5 GB, 11534336000 bytes 255 heads, 63 sectors/track, 1402 cylinders, total 22528000 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/mapper/xvda1 63 10474379 5237158+ 83 Linux
[/code]

Note that /dev/mapper/xvda1 above is probably a long string with some UUIDs embedded in it. I’ve shorted it for visibility.

This fdisk command showed us that at offset 63, there is a partition, which has 5237158 blocks (or the 5GiB partition configured by default). We can use this offset with the losetup command to bind a device node inside this disk image. I use /dev/loop99 below because I’m sure it’s not already in use. Disk sectors are 512 bytes, and this partition starts at sector 63, so we need to look at offset of 63 * 512 bytes:

[code]
# losetup /dev/loop99 -o $((63 * 512)) /dev/path/to/xvda
[/code]

Done! We now have a device at /dev/loop99 that corresponds to the xvda1 partition inside the xvda block device. We can run a filesystem check on this:

[code]
# e2fsck -f /dev/loop99
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop99: 35722/655360 files (0.3% non-contiguous), 132621/1309289 blocks
[/code]

The partition xvda1 is still only 5GB in size however. So, you’ll need to resize the partition using parted

[code]# parted /dev/path/to/xvda
GNU Parted 1.6.19
….

Using /dev/path/to/xvda
(parted) print
Disk geometry for /dev/path/to/xvda: 0.000-11000.000 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 5114.443 primary ext3
(parted) resize
Partition number? 1
Start? [0.0308]?
End? [10997.6216]?
(parted) p
Disk geometry for /dev/path/to/xvda: 0.000-11000.000 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 10997.622 primary ext3
(parted) q
[/code]

And now, we finally run another fsck and then run resize2fs before tearing down the block device mapping
[code]

# e2fsck -f /dev/loop99
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop99: 35722/1409024 files (0.3% non-contiguous), 156364/2815383 blocks
# resize2fs /dev/loop99
resize2fs 1.35 (28-Feb-2004)
Resizing the filesystem on /dev/loop99 to 2815992 (4k) blocks.
The filesystem on /dev/loop99 is now 2815992 blocks long.
# losetup -d /dev/loop99

[/code]

Now you can restart the domU, login and check your disk space:

[code]
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 11084028 448004 10636024 5% /
[/code]

If you boot the domU and get a fsck failure, and you run fsck and it tells you the filesystem is larger than the partition, make a note of how many blocks fsck things the filesystem should be, destroy the domU, rerun losetup and e2fsck, then run resize2fs again and specify the number of blocks fsck output. Run e2fsck again, losetup -d, then start the domU again.

Categories
advocacy

Xen partially merged into linux kernel

It’s been a long time coming, but the first parts of Xen have been finally merged into upstream codebase for the linux 2.6.23 kernel.  Announcements here, here and here.  Worth reading Simon Crosby’s announcement, as it actually has some information about what is and isn’t being merged. Specifically, this is the merge of the paravirt_ops part of Xen, something that VMWare has had in the mainstream kernel for a while now, and it’s only 32 bit at this stage. From Simon’s blog:

“The effort for XenSource has been led by Jeremy Fitzhardinge, who has tirelessly tracked the developing kernel versions, while adding the Xen guest support for SMP guests, with fast paravirtualized block and network I/O. Next up is 64 bit support, according to Jeremy, who is also working on Dom0 support.”

While this isn’t a full merge of the entire Xen codebase into the kernel, it should still make it a lot easier to build xen-aware kernels.

Also merged into 2.6.23 is lguest, Rusty Russel’s linux-only paravirtualised hypervisor. I’ve been following Rusty’s blog posts on his development of this, although I was never sure of his reasoning for starting work on it. It may just have been a “lets see if I can make one” sort of approach.   Between kvm, lguest, uml and xen, there is now a lot of choice for virtualisation under the linux kernel. Not all approaches are the same – xen and lguest are paravirtualised while kvm is full virtualisation, and UML just runs a new linux kernel in user-space.  Xen and KVM will support windows (and other OS) guests, lguest and UML will only support linux guests. KVM requires VT or AMD-V chips, Xen requires them to install windows (but not linux), and lguest and UML don’t make use of them at all.

Categories
linux Tool of the Week WLUG

Miro – Internet TV

Miro, formerly known as Democracy TV, made its first public release a few days ago.  It’s available at http://www.getmiro.com/. Miro is like a blog aggregator for video sources such as YouTube and Google Video, as well as provider content such as various news  and science tv channels, The Onion.

Installing it was trivial under Ubuntu, although it conflicts with the blackdown JRE. You can install the sun jre instead to get around this.

Categories
linux WLUG

make-kpkg and recent 2.6 kernels

I keep hitting a bug in make-kpkg when building recent kernels, and I always forget the fix. So, blogging it here for ease of reference

[code] make-kpkg uses version.h to get UTS_RELEASE. UTS_RELEASE has
moved to utsrelease.h.

Right after you get the error, modify
debian/ruleset/misc/version_vars.mk

-UTS_RELEASE_VERSION=$(shell if [ -f include/linux/version.h ]; then \
– grep ‘define UTS_RELEASE’ include/linux/version.h | \
+UTS_RELEASE_VERSION=$(shell if [ -f include/linux/utsrelease.h ]; then \
+ grep ‘define UTS_RELEASE’ include/linux/utsrelease.h | \

And rerun your make-kpkg. The above is not a valid patch, you’ll have
to hand change it.

Joel

[/code]

Original post was found at http://lkml.org/lkml/2006/7/16/109

Categories
advocacy

XenSource ‘Simply Virtualize’

The XenSource ‘Simply Virtualize‘ tour made it’s way to NZ this week, with a 3 hour set of presentations at Microsoft House yesterday afternoon. We had a good catchup with John Glendenning from XenSource on Monday, but I can’t talk about most of that.

The presenters at the event were XenSource, Sun Microsystems, Microsoft, Platespin and ExpresData hosted the event. The tour seems to be done in conjunction with IBM elsewhere, but was done with Sun here in NZ.

John Glendenning from XenSource gave a brief introduction to XenSource and Xen Enterprise, and also some points about where development is heading. One point is the planned interoperability with Veridian, the Microsoft virtualization stack. The Microsoft presentation followed, which focussed entirely on MS’s various virtualization technologies – presentation, application, server, etc. It had some interesting aspects, but was a tiny bit out of place I thought.  Sun had a good set of slides on their AMD-V platforms, including their new blade infrastructure which will support opteron, xeon (when they come out later in the year) and ultrasparc blades. Platespin then gave a fairly quick, but comprehensive overview of their P2V / V2V and capacity planning and management tools. James Johnstone from ExpressData finished up with a demo of Xen Enterprise and Windows XP guests running on a SunFire x4200 M2.

Things that I got from the various presentations:

  • Microsoft’s SoftGrid Application Virtualization software suite looks damn useful, and I can think of at least one site we could have used it this year.
  • Sun are coming out with Xeon-based servers soon, as well as AMD’s quad-core range being on the horizon
  • Sun have a blade range that has a fully modular IO system – the PCI infrastructure is abstracted away from the blade.
  • Platespin’s PowerConvert and PowerRecon products look very useful, and they are aggresively adding new features.

From Platespin’s presentation and some of the points that MS came out with it is very clear that the virtualization technology you choose for server virtualization is definitely not the final decision to make, nor should it be. The virtualization ecosystem is massive already – mostly due to the number of ISVs VMWare has on board. These ISVs are now targetting Xen Enterprise as a platform as well, and are bringing their already mature technology to focus on the alternative platforms. This gives Xen Enterprise quite a bit of credibility, as the management tools don’t have to be rebuilt – vendors like Platespin, Leostream, Mountain View Data, Marathon etc can target Xen with relative ease.

Categories
NSP WLUG

Feisty Fawn and Software RAID

It turns out there’s a race condition in Feisty Fawn, which can cause software RAID sets to not be set up on boot. This is problematic if you have your root partition on software RAID

Bug #75681 discusses this in some detail, although there are several suggestions on how to fix it.

I first hit this bug on a local machine, then had to the same upgrade on a machine in a different country. Needless to say I wanted to get it right. I’m archiving my notes here as I’m sure I’ll need them eventually. This race condition has probably already been fixed in Feisty, but it’s not worth risking on a remote machine.

First of all, there is some new management involved in setting up software RAID under feisty, so you need to make sure you read the documentation for the mdadm package. Every time an initramfs is generated it will generate a warning:

[code]
update-initramfs: Generating /boot/initrd.img-2.6.20-14-generic
cp: cannot stat `/etc/udev/rules.d/85-brltty.rules’: No such file or directory
W: mdadm: unchecked configuration file: /etc/mdadm/mdadm.conf
W: mdadm: please read /usr/share/doc/mdadm/README.upgrading-2.5.3.gz .
W: mdadm: no arrays defined in configuration file.
W: mdadm: falling back to emergency procedure in initramfs.
[/code]

Following those instructions, you are told to check the configuration in /etc/mdadm/mdadm.conf and compare with the output of /usr/share/mdadm/mkconf. Once you’ve done that, you can remove /var/lib/mdadm/CONF-UNCHECKED and re-run update-initramfs -u -k all to regenerate your initramfs images.

The particular race condition that I mentioned above occurs because udev hasn’t had time to stabilise before mdadm tries to create the array, which means mdadm can’t find the devices and fails. The fix suggested in the bug report is to insert udevsettle into the initramfs at an appropriate point, and recreate the initramfs images:

[code]
# echo “/sbin/udevsettle –timeout=10” >> /usr/share/initramfs-tools/scripts/init-premount/udev
# update-initramfs -u -k all
[/code]

This works, at least as of today. I don’t know if the bug is actually still a problem or not – I didn’t want to risk it./

Categories
General NSP WLUG

Debian Etch and apt-proxy issues

Debian Etch (4.0) was released on Monday, and I have to say I wasn’t at all prepared. I’ve got about 70 machines that will probably need to be upgraded to Etch at some point in the near future. I could leave some of them running sarge, but I’ll definitely have to upgrade most of these servers.

We use an apt-proxy internally, to improve apt performance. It works well, aside from a couple of bugs that cause it to lock up every now and then. While running some upgrades on out of the way servers today, I discovered that the version of apt in sarge really doesn’t play very nicely with an etch repository served by apt-proxy running on an etch server. It seems that Ubuntu is fine, and trying to update a sarge server via an apt-proxy running on an etch server is ok too.

Once the etch client has been upgraded, the etch apt-proxy works fine. So, looks like a key issue. The version of apt in sarge doesn’t have the archive security stuff in it, and has no way of checking whether the keys are intact – BUT, it still seems to care, and will timeout and eventually fail.

It turns out that installing a copy of apt from the sarge backports solves this. You’ll also need the gnupg package, but the one from sarge is OK
[code]
wget http://backports.org/debian/pool/main/a/apt/apt_0.6.46.4-0.1~bpo.1_i386.deb
wget http://backports.org/debian/pool/main/d/debian-archive-keyring/debian-archive-keyring_2006.11.22~bpo.2_all.deb
dpkg -i *.deb
apt-get update
[/code]

Categories
advocacy NSP Xen

Xensource and VMWare performance comparison

I was discussing Xensource with a potential client a few weeks ago, and was fairly surprised when they pulled out a performance comparison of VMWare and Xen, which showed VMware massively outperforming Xen in several tests. On further inspection, it was fairly obvious that VMWare’s tests used the open-source version of Xen, and were running windows based tests on it. This might be a fairly typical enterprise environment, but they weren’t really playing a fair game – Xensource’s product range include a PV driver set for windows which drastically improves performance. This driverset isn’t available under the open source version of Xen.

The comparison the client had been given also had some other data included, some of which was misleading, and some of which was just plain wrong. It included statements such as ‘Xen does not support live migration’ when it does (and what’s more, the open source version supports it natively, so it’s not a bolt-on to the product), and a point stating that Xen had no management consoles available on one page, and a price comparison of VMware, Xensource and VirtualIron on the next. Xensource provide a commercial management console for Xen. Huh.
After a bit of digging, I found the original VMware published report that this comparison was drawn from. Yes, VMware didn’t run a fair test, and yes, given that unfair test, Windows under VMWare ESX massively outperforms Xen in some areas, primarily I/O related.

We mentioned that this report was being circulated to Xensource at about that time. They must have been getting the same heads-up elsewhere, because within a few days of that they had published a performance comparison of Xen Enterprise and VMWare ESX themselves, and even gotten approval from VMWare to publish it! Roger Klorese links to the report from his blog. The report is here.
The report shows that the gap between VMWare ESX and Xen Enterprise performance is negligble in most cases, and Xen Enterprise outperforms VMWare ESX considerably in some areas. It’s definitely a much closer race than VMWare’s report would have you believe.

Categories
Uncategorized

Puppet and per-domain configuration

I started deploying puppet at work, after installing it a couple of weeks ago and playing with it. I’ve got it doing some pretty basic management so far:

  • SSH key management
  • Timezone and Locale
  • NTP
  • Minimum required packages
  • /etc/resolv.conf

None of this is overly complicated, but it is very useful for me.

SSH key management

This is actually really trivial at this stage – I have an authorized_keys file which I push out to all hosts, thus allowing anyone with a key matching one in that file to log in as root. I’ll eventually change this to having user profiles managed by puppet, but for now this is good enough.

Timezone and Locale

For some reason, these two things get missed an awful lot when configuring new servers quickly, particularly if they are VMs (when debootstrapping a server, you don’t get prompted for these things). I manually symlink /etc/localtime to the right timezone file, and I use debconf-set-selections to pick the right locale then install localeconf

NTP

Once the timezone is set correctly, I install and configure ntpd. I give it a simple config using a few of the nz.pool.ntp.orNTPg servers and an internal one, and call it done. We seem to get a lot of calls from customers about time/date being wrong on their email/reports/something, so this will fix that.

Minimum required packages

Other than my preferred editor, there are a large number of packages that all hosts should have installed, no questions asked. less. tcpdump. mtr. strace. nmap. host. dig. And the rest. Puppet installs these for me now, rather than having to wait till I need the tool to install it. Puppet will happily install packages from a range of sources, and autodetects which one to use based on your OS version. Of course, package names will probably change (ssh vs openssh-server, etc), but that can be dealt with on a per-OS override.

/etc/resolv.conf

This one caught me out quite badly. I figured I was being really smart by pushing out a resolv.conf to all sites, until I then realised I was overwriting references to any local DNS servers. Up until now I had one node definition – the default one. Everything else was being autodetected at run time.

I could depart from this model and have a per-node definition which included the searchpath and the local resolvers to use, but that means creating a node definition for every new node I add. That’s not too much work, granted, but it’s more than I want to have to do for something as basic as /etc/resolv.conf. I could also create a per-site class and have every node include that class, which would save some work, but still require a per-node definition.

It looks like it’s currently impossible to have a wildcard definition of a node, eg:

[code]
node ‘*.example.com’ inherits base {
$searchpath = ‘example.com’
$nameservers = ‘192.168.0.1’
}
[/code]

So I signed up to the puppet mailing list and asked how to do this, and predictably I worked out how to do it about 30 seconds later. Puppet uses the ruby ‘facter’ library, which lets you get various facts about the system you are running on. One of these facts is the domain name.

Now I have a couple of switch statements in my resolvconf class which specify the searchpath and nameservers based on the domain name of the node being configured, which are then used to fill out a template. Magic!