Categories
jabber linux Tool of the Week

Using monit for system and process monitoring

One of the servers I maintain is the jabber server at jabber.meta.net.nz. This is a free public service, anyone can use it, and it does get quite a wide range of use – for a long time we seemed to be very popular for south american users, possibly because of the web based clients and the range of transports to other protocols we support. We typically see between 50 and 100 concurrent users, depending on time of day and week, but the active account base is normally in the low thousands.

The transports themselves cause me a lot of problems. In the past they’ve been downright buggy, crashing all the time, but with the current codebase for all four protocols in use (AIM, ICQ, MSN and Yahoo) all being in python, we don’t seem to have as many outright crashes. We do have slow memory leaks however, which prompted me to move the services to a new server a while back. Part of me was hoping that the memory leaks were caused by the gentoo system I was using initially, but this doesn’t seem to be the case.

So, I needed to either fix these memory leaks, or to work around them. Enter monit. I’ve heard about monit quite a bit, but never really looked into it other than thinking it might be interesting. I really wish I’d looked further ages ago. It’s easy to set up, is designed specifically to monitor and restart services, and it solved my memory leak problem in about 5 minutes.

Here’s a snippet from the config file:
[code]
check process aim-transport with pidfile /var/jabberd/pid/aim-transport.pid
start program = “/etc/init.d/aim-transport start”
stop program = “/etc/init.d/aim-transport stop”
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 300.0 MB for 5 cycles then restart
group transport
[/code]

This is pretty self explanatory really. If CPU usage of this process gets too high, alert, then restart if it stays high for 5 cycles. And if the ram usage is over 300 MB for 5 cycles (a cycle is 2 minutes by default), restart the process. Problem solved. Or rather, the symptoms are solved, but that’s good enough for me at this stage

NoteThis is old, but somehow didn’t get posted

Categories
jabber MetaNET

Moving from ejabberd’s internal database to postgres

During the initial import of data to the {{post id=”jabbermetanetnz-has-moved” text=”new jabber.meta.net.nz server”}}, I noticed that the roster information failed to be migrated if I was using an ODBC backend – it would just fail entirely. I didn’t think too much of it until I tried to restart ejabberd, and noticed that it spent around 5 minutes initializing it’s mnesia database. There’s around 220,000 roster entries in the database, and that’s after I culled around 9000 accounts that hadn’t been used in more than 180 days.

So, I decided to have another look at moving the content to an SQL backend. I noticed that one of the many commands available from the ejabberdctl cmdline tool is an export2odbc command. It didn’t seem to work at first, but it turns out I had the parameters slightly wrong – this post set me straight. So, I started importing all the tables, including the roster table, into my database

I then noticed that it appears to be doing one, sometimes two deletes, for every insert. I have an empty database, so there’s no real need for this, and there’s 220,000 entries. During the 30 minutes or so I spent playing round with other ways to do this import I only got to around 115,000 entries added! The export command had output the content on the smallest number of lines possible, so after free ringtones makerfree funny voice ringtonesdownload free cingular ringtonesfree mp3 nextel ringtones,nextel mp3 ringtones,mp3 nextel ringtonesringtones gratis,1100 gratis nokia ringtones,c115 gratis motorola ringtonesdownload free new ringtonescarrier cellular free ringtones usfree nokia ringtones tracfonefree nickelback ringtones,nickelback ringtones,nickelback ringtones rockstardownload ringtones for sprint phone,sprint download ringtonesfree lg ringtones verizon vx6000box music ringtones sony,sony music box ringtonescell free phone ringtones samsunginfo motorola personal remember ringtones t731,info motorola personal remember ringtones,info motorola personal remember ringtones v66totally free cell phone ringtonescomputer download free ringtones1 hour loan payday,payday loan in 1 hour,1 hour payday loansavings account payday loanno fax payday advance loanquick low interest payday loan,loan payday quick,quick pay payday loanno bank statement payday loanbank loan no payday statementloan payday until,loan payday say until wordpress,loan until paydayfast payday loanquick cash payday loanline loan payday,payday loan on lineaafes loan paydayadvance cash net payday usacash advance loan payday internet,cash loan payday,instant cash payday loancash until payday loanpayday loan on lineemergency loan paydaycheap payday loanpayday loan uk,loan payday uk,payday loan in the ukpayday loan companyfee loan low paydaypayday loan calculatorfast easy payday loanbad credit guaranteed loan paydayfax loan missouri no paydaypayday loan in georgiaadvance fax loan no paydaypayday loan store chicagodefault on payday loancanadian loan online payday,canadian payday loanapproval guaranteed loan paydaypayday cash loan,loan payday,payday advance loancash fast loan paydaychicago in loan payday store,chicago loan payday storeadvance cash loan payday today,payday payday loan cash advance loan,advance cash loan payday a bit of sed and grep magic I had a file which contained merely the relevant inserts. It’s still running slow, but it’s running a lot quicker than previously.

I know postgres’s dump functions tends to prefer COPY rather than INSERT because it is so much quicker, but that’s the point at which my familiarity with the COPY command ends, and I certainly didn’t want to spend the time to mangle the SQL I have into a couple of COPY commands.

Categories
jabber MetaNET

Jabber.meta.net.nz has moved

I bit the bullet and moved jabber.meta.net.nz onto its new server last night. DNS records have changed, however the TTL was set to 60 seconds a long time ago so this shouldn’t be a problem.  If you’re having problems connecting please verify that you’re actually connecting to jabber.meta.net.nz and that whatever IP you are resolving that to matches the authoritative one – 219.88.251.36.

The new server is running ejabberd, and while this is much better software than the previous version of jabberd2 it was using, it still has some quirks. The most noticeable one is the godawful time it takes to start up when using internal authentication, and a bug when trying to use an ODBC backend which results in roster information being lost entirely. I’ll keep working on these two points, with hopefully only minor outages over the next week or so.

Other new features include an XMPP.net federated SSL certificate, rather than me buying an SSL cert specifically for jabber, and we’ve pulled jabber.meta.net.nz into the web 2.0 age with the addition of a jwchat web-chat client, which makes use of ejabberd’s http binding features.

Categories
jabber MetaNET WLUG

Upgrading jabber.meta.net.nz

I’ve finally gotten sick of the jabber.meta.net.nz services having a negative impact on the rest of the server due to the transports chewing up lots of ram, and so have put together another server and will start migrating things over. The new server is a dual 1GHz Coppermine P3, with 2 GB of ram and 36 GB of SCSI RAID-1 disks.

This seems like a great opportunity to tidy things up again. The server stack has been a bit messy since we upgraded to jabberd2, partly because of the server codebase, and at least partly because of gentoo. So, the new plan is to move jabber.meta.net.nz to ejabberd, sometime soon.  Along with the move will come a new webbased chat system ( jwchat ), and newer versions of the transports if they are available.

Ejabberd seems to be a pretty robust system. Jabber.org moved their system to it several years ago, and jabber.org.au made the switch more recently. I’ve seen some pretty impressive benchmarks regarding message latency / cpu loading / ram usage under high protocol load and it beats jabberd1.x and jabberd2.x hands down. Not that jabber.meta.net.nz really notices, or compares with jabber.org or jabber.org.au in terms of volume. We’ve currently got 112 clients connected, and someone noticed we had 203 clients connected a couple of weeks ago, but I have no idea what the high water mark for client connects is, much less any idea about message throughput statistics.

I aim to have the following done in the next few weeks:

  • Install and test base system – ejabberd, transports
  • Install and test jwchat
  • Script the migration of registration information and rosters from the jabberd2 pgsql database to something ejabberd can read.
  • Update the website with better information
  • Setup transports to prevent OOM issues (ulimit/regular restarts)
  • Test migration of registry/rosters
  • Pick a day and cutover.

The trickiest bit will be the migration of roster info from pgsql to ejabberd’s format, but I’ve already done at least three quarters of that already, as I considered doing this migration back in March.