Categories
jabber MetaNET

Jabber.meta.net.nz has moved

I bit the bullet and moved jabber.meta.net.nz onto its new server last night. DNS records have changed, however the TTL was set to 60 seconds a long time ago so this shouldn’t be a problem.  If you’re having problems connecting please verify that you’re actually connecting to jabber.meta.net.nz and that whatever IP you are resolving that to matches the authoritative one – 219.88.251.36.

The new server is running ejabberd, and while this is much better software than the previous version of jabberd2 it was using, it still has some quirks. The most noticeable one is the godawful time it takes to start up when using internal authentication, and a bug when trying to use an ODBC backend which results in roster information being lost entirely. I’ll keep working on these two points, with hopefully only minor outages over the next week or so.

Other new features include an XMPP.net federated SSL certificate, rather than me buying an SSL cert specifically for jabber, and we’ve pulled jabber.meta.net.nz into the web 2.0 age with the addition of a jwchat web-chat client, which makes use of ejabberd’s http binding features.

Categories
linux MetaNET

Restarting linux software raid arrays

I have a server with different sets of RAID1 arrays, across two different sets of disks. The first set of arrays, on the first pair of disks, always starts fine. The second set never starts on boot. I don’t reboot this server very often, so it’s not often a problem, however I figure it’s worth looking into.

For starters, if you have an existing software RAID array that is merely not running, you can assemble it with mdadm like so:

[code]
mdadm –assemble /dev/md8 /dev/sdc1 /dev/sdd1
[/code]

mdadm also has a config file which is referred to in the system boot process, in order to work out where your arrays are. In theory this is all supposed to be self configuring, but I’ve seen enough cases where some arrays don’t start that it looks like using this config file is a recommended step.

You can generate a useful representation of your existing arrays with

[code]
mdadm –detail –scan
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=…
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=…
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=…
[/code]

You can use similar syntax but specifying the specific devices used in each array, instead of the UUID, if you like. In theory, with this information appended to /etc/mdadm.conf, next time I reboot this server it should bring all the RAID devices up for me.

Categories
General NSP Xen

Xenserver Enterprise + Fibrechannel storage = live migration

I finally got round to booking time at the IBM demo center to have a fiddle with Xen Enterprise on shared storage. They’ve got a good range of entry-level kit at the demo center, but the important bits (for me anyway) were a Bladecenter H chassis with some HS21 blades fitted with Fibrechannel HBAs, and a DS3400 Fibrechannel storage array.

The IBM DS3000 series of arrays looks really promising. There’s three current variants – SAS, iSCSI and Fibrechannel attached, along with an EXP3000 expansion shelf. All four systems will scale to 48 SAS (14.4 TB) drives over 4 shelves, and an IBM firmware announcement I read online the other day strongly suggests they will support SATA drives in the very near future (36 TB using 750 GB disks). And the DS3000 series are cheap. Performance is pretty good, but you can only stack the controller with 1GB of cache – which really highlights the “entry level” bit of these SANs. The SAS attached option is compelling as well – you get SAN functionality, very close to FC levels of performance (3 Gbps peak HBA throughput for SAS compared to 4Gbps for FC), at a fraction of the cost. The DS3200 allows 6 single-connect or 3 dual-connect hosts, and as soon as IBM get round to releasing a SAS switch, that limit will disappear.

I’d used the DS3400 management software in the past, so setting up a LUN within the existing array took about 5 minutes, including creating the host to WWN mappings; and I had two Xenserver Enterprise installs already set up on a matched pair of blades from a previous attempt at this with the DS3300.

Xenserver Enterprise supports shared storage, but as of version 4.0.1, it only supports NFS or iSCSI shared storage officially. I’d had problems getting the openiscsi software iSCSI stack that Xenserver ships with to communicate successfully with the DS3300 however, and ran out of time. On the other hand, FC shared storage is just not supported at all yet. There’s a forum article explaining Xensource’s position on the support, which also links to a knowledge base article describing how shared storage works in Xenserver. The article was pulled with the release of 4.0.1:

“We took it down because, with the level of testing and integration we could do by initial release of 4.0, we couldn’t be any further along than a partial beta. There are business reasons we couldn’t ship a product as released while describing some of the features as “beta”, and it is hypocritical for us to officially describe a way of using the product yet describe that as “unsupported.” For that reason, until we are ready to release supported shared Fibre Channel SRs, we’re not going to put the KB article back up.”

The article describes the overall setup you have to have in place to have FC shared storage working – namely array, LUN and zoning management on the SAN and HBA, and locating the device node that maps to the corresponding LUN. And then there are two commands to run, on one node of your Xenserver pool, and your shared storage system is up and running.

At this point it took about another 5 minutes to create a Debian Etch domU, and verify that live migration between physical hosts was indeed working.

I set up a slightly more complicated scenario, which I’m planning on using with some other software to demonstrate a HA / DR environment, but which also enabled me to do one of those classic cheesy live migration “but it still works” tests – I connected a Wyse thinclient through to a terminal server domU, migrated the TS between physical hosts, and demonstrated that the terminal session stayed up. A ping left running during this time experienced a brief bump of about 100 ms during the final cutover, but that’s possible attributable forwarding-path updates in the bridge devices on the Xenserver hosts. Either way it works well. Moving a win2k3 terminal server took about 15 seconds on the hardware I was working with.
Overall, I’m quietly encouraged by this functionality and it’s ease of set up. I’m perhaps a bit underwhelmed by it too – it ended up being such a nonevent to get it working, that I’m annoyed I didn’t get round to setting up the demonstration months ago.

Categories
General NSP WLUG

Benchmarks of an Intel SRSC16 RAID controller

One of our clients gave us an Intel server with an Intel SRSC16 SATA RAID controller and a 500 GB +hotspare RAID1 set up on it, to install XenServer Express 4.0.1 system. While building the system up for him, I noticed abysmal write perfomance. It was taking around 29 minutes to install a guest from a template, a process which basically involves creating and formatting a filesystem and unpacking a .tar.bz2 file into it. Inspection of the system revealed that the controller lacked a battery backup unit (BBU), and thus the writeback cache was disabled. Also, the firmware on the controller disabled the on-disk cache as well, and the controller listed disk access speed at 1.5Gbps, which I’m presuming means it was operating in SATA-1 mode, so no NCQ either. The controller has 64MB of cache.

I persuaded the customer to buy the BBU for the system, and then ran some quick bonnie++ benchmarks, which I know aren’t the best benchmark in the world, but show a good indication of relative performance gains. Results are as follows:

Note: I didn’t do the tests right either – not specifying a number of blocks of files to stat results in those tests completing too soon for bonnie to come up with an answer. So, the output below only really shows throughput tests, as the sequential create/random create tests all completed too soon. Changing the disk cache settings requires a reboot into the BIOS mode configuration tool, so I’ve avoided doing this too many times. Changing the controller cache settings can be done on the fly.

[code]

RAID controller writeback cache disabled, disk cache disabled:

Version  1.03      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.loca 512M  1947   4  2242   0  1113   0 10952  18 36654   0 169.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.locald 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ ++

RAID controller writeback cache enabled, disk cache disabled:

Version  1.03      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 512M  7938  19  9195   1  4401   0 28823  50 41961   0 227.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.locald 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

RAID controller writeback cache disabled, disk cache enabled:

Version  1.03      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.loca 512M 19861  47 17094   1  9870   0 28484  47 41167   0 243.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.loca  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

RAID controller writeback cache enabled, disk cache enabled:

Version  1.03      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 512M 38633  95 40436   4 15547   0 32045  54 42946   0 261.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.locald  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
[/code]

Enabling the only the controller write-back cache (64MB in this case) roughly quadrupled the write throughput in all cases. Enabling only the disk cache provided nearly 8 times the performance on it’s own. And enabling both together increased write throughput by about a factor of 20.   I suspect the tests weren’t large enough to actually tax the cache systems on the disk or controller however, as I was running them in a Xen domU with only 256 MB of ram, and actually just wanted some quick results.

I know they aren’t really representative of anything, but here’s a test that is semi-representative: Installing another copy of the Xen domU via a template took 2minutes 55 seconds with disk cache enabled, and 2 minutes 30 seconds with disk cache and controller cache enabled (I didn’t test this with just controller cache enabled as that would have required a reboot and manual intervention, and I wasn’t onsite at that point).  Prior to enabling the disk cache and controller cache, this was taking nearly 30 minutes.

While the above shows that a combination of the controller write-back cache and the disk cache shows the best improvement, merely enabling the disk cache on it’s own had the biggest single effect. Of course, the disk cache isn’t backed up by a battery, so there’s the risk of losing the data that is in the disk cache at the time. The Intel documentation for the controller implied that this is limited to the sector that is being written at the point of powerfailure.

When I get some free time and a SCSI or SAS server, I’ll do some similar benchmarks up for that.

Categories
jabber MetaNET WLUG

Upgrading jabber.meta.net.nz

I’ve finally gotten sick of the jabber.meta.net.nz services having a negative impact on the rest of the server due to the transports chewing up lots of ram, and so have put together another server and will start migrating things over. The new server is a dual 1GHz Coppermine P3, with 2 GB of ram and 36 GB of SCSI RAID-1 disks.

This seems like a great opportunity to tidy things up again. The server stack has been a bit messy since we upgraded to jabberd2, partly because of the server codebase, and at least partly because of gentoo. So, the new plan is to move jabber.meta.net.nz to ejabberd, sometime soon.  Along with the move will come a new webbased chat system ( jwchat ), and newer versions of the transports if they are available.

Ejabberd seems to be a pretty robust system. Jabber.org moved their system to it several years ago, and jabber.org.au made the switch more recently. I’ve seen some pretty impressive benchmarks regarding message latency / cpu loading / ram usage under high protocol load and it beats jabberd1.x and jabberd2.x hands down. Not that jabber.meta.net.nz really notices, or compares with jabber.org or jabber.org.au in terms of volume. We’ve currently got 112 clients connected, and someone noticed we had 203 clients connected a couple of weeks ago, but I have no idea what the high water mark for client connects is, much less any idea about message throughput statistics.

I aim to have the following done in the next few weeks:

  • Install and test base system – ejabberd, transports
  • Install and test jwchat
  • Script the migration of registration information and rosters from the jabberd2 pgsql database to something ejabberd can read.
  • Update the website with better information
  • Setup transports to prevent OOM issues (ulimit/regular restarts)
  • Test migration of registry/rosters
  • Pick a day and cutover.

The trickiest bit will be the migration of roster info from pgsql to ejabberd’s format, but I’ve already done at least three quarters of that already, as I considered doing this migration back in March.

Categories
NSP Xen

VMWare responds to XenSource, MS Threat?

Five weeks or so ago VMWare accounced massive price cuts for their existing products. Today I read that VMWare are redefining their product range completely.

The article has more detailed information, but the basic gist is that VMWare is releasing a new entry-level SKU, not manageable via VirtualCenter (and thus not suited for enterprise, but entirely suited for sites that just want one or two hosts), with SATA support.   SATA support has tradiationally been missing from VMWare’s products, and the presence of SATA support was one of the selling points for other virtualisation vendors such as Xensource and VirtualIron.

I asked in my last post “Has VMWare seen the writing on the wall?” I guess it has. It’s not end-game yet, of course: Gartner analysts peg virtualisation use at under 10% of the server market (I don’t have an exact reference for that, but there’s lots of references at 6% or so during late 2006). This is merely the point where VMWare realises that it’s not enough to be the big fish in the pond – Citrix and Microsoft are now in there too.

Categories
linux NSP WLUG Xen

iSCSI for SCSI device passthrough under Xen Enterprise

I recently had to add a SCSI tape drive to a Xen Enterprise server, and needed to use BackupExec under one of the Windows domU’s as the backup software. Luckily, Greig did this a few months ago already using the iSCSI Enterprise Target, and put his notes up on the WLUG Wiki here.

I hit one problem however – when using NTBackup to test the system, it would write about 20 MB to tape, then fail. Greig pointed out he’d only ever used BackupExec, which was the software that was going to be used finally anyway, so I installed that and then it worked fine.

Also, going one step further, it’s possible to use the same technique to push USB mass storage devices over iSCSI to domUs. As Xen Enterprise doesn’t have a nice way of passing USB mass storage devices through to domUs yet, this is a very good solution in the interim

[code]

Target iqn.2007-04.com.example:tape0
Lun 0 H=4,C=0,I=0,L=0,Type=rawio
Type 1
InitialR2T No
ImmediateData Yes
xMaxRecvDataSegmentLength 262144

Target iqn.2007-04.com.example:usb0
Lun 1 Path=/dev/sdb,Type=fileio
[/code]

As it refers to the scsi device, which will change if you unplug and re-insert a USB block device, it makes a lot of sense to use udev to map your USB mass storage device to a specific /dev entry.

I don’t have a feeling for how robust this is yet.

Categories
General

Touchgraph Google Browser

Saw the TouchGraph Google Browser mentioned on an irc channel today. Touchgraph make graph based visualisation, showing spatial relationships and correlations between data. The google browser uses results from a google search as the input; Touchgraph have Facebook and Amazon visualisations as well, but I guess their main business is visualising data warehouses and the like.

Categories
linux NSP Xen

Has VMWare seen the writing on the wall?

As seen here, VMWare is set to slash the pricing for VMWare ESX and VirtualCenter for the SME market – three copies of ESX and a limited copy of VirtualCenter, for the nice round price of $3k US – a significant price cut. This is hot on the heels of XenSource’s new pricing and acquisition by Citrix (XenSource also has a current promotion of “buy one, get three” as an introduction to the new XE 4.0 pricing model.)

Has VMWare seen the writing on the wall? They look a bit defensive at times. Or maybe VMWare has just realised they’ve neglected the SME market for too long. In NZ especially, most businesses are in the SME bracket, and just can’t afford VMWare’s prices – it’s cheaper to buy a new server machine in most cases.

Xen Enterprise’s new price tag of $2.5kUS may be outside their reach as well, but most SMEs don’t need the enterprise features present in XenSource’s flagship product. XenServer, at $750 US, fits right in the sweet spot, along with VirtualIron’s flagship offering.

Categories
linux NSP Tool of the Week WLUG

NUT: Network UPS Tools

I was tweaking the UPS rules at a client’s site, when I noticed that the base NUT configuration that we use didn’t really do a hell of a lot. The example config files had some hints as to what were needed, but unless I missed something fundamental, didn’t have the full picture.

After a bit of searching, my laptop battery ran out so I couldn’t carry on working onsite. I did get far enough to make some notes, but I have since lost the site I referred to, so can’t put proper attribution. It looked something like this one though, and was also dedicated to setting up NUT on a Mac, so I figure that will do.

I’ve since returned to this issue, and after fighting with serial and USB cables, have finally completed and tested it all. My configuration is on the WLUG wiki at the NutNotes page.