{"id":59,"date":"2008-03-09T11:57:15","date_gmt":"2008-03-08T22:57:15","guid":{"rendered":"http:\/\/www.meta.net.nz\/~daniel\/blog\/2008\/03\/09\/moving-from-ejabberds-internal-database-to-postgres\/"},"modified":"2008-03-21T00:01:09","modified_gmt":"2008-03-20T11:01:09","slug":"moving-from-ejabberds-internal-database-to-postgres","status":"publish","type":"post","link":"https:\/\/www.meta.net.nz\/~daniel\/blog\/2008\/03\/09\/moving-from-ejabberds-internal-database-to-postgres\/","title":{"rendered":"Moving from ejabberd&#8217;s internal database to postgres"},"content":{"rendered":"<p>During the initial import of data to the {{post id=&#8221;jabbermetanetnz-has-moved&#8221; text=&#8221;new jabber.meta.net.nz server&#8221;}}, I noticed that the roster information failed to be migrated if I was using an ODBC backend &#8211; it would just fail entirely. I didn&#8217;t think too much of it until I tried to restart ejabberd, and noticed that it spent around 5 minutes initializing it&#8217;s mnesia database. There&#8217;s around 220,000 roster entries in the database, and that&#8217;s after I culled around 9000 accounts that hadn&#8217;t been used in more than 180 days.<\/p>\n<p>So, I decided to have another look at moving the content to an SQL backend. I noticed that one of the many commands available from the ejabberdctl cmdline tool is an export2odbc command. It didn&#8217;t seem to work at first, but it turns out I had the parameters slightly wrong &#8211; this <a href=\"http:\/\/www.ejabberd.im\/node\/1259\">post<\/a> set me straight. So, I started importing all the tables, including the roster table, into my database<\/p>\n<p>I then noticed that it appears to be doing one, sometimes two deletes, for every insert. I have an empty database, so there&#8217;s no real need for this, and there&#8217;s 220,000 entries. During the 30 minutes or so I spent playing round with other ways to do this import I only got to around 115,000 entries added! The export command had output the content on the smallest number of lines possible, so after <u style=\"display:none\"><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/free-ringtones-maker.html\">free ringtones maker<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/free-funny-voice-ringtones.html\">free funny voice ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/download-free-cingular-ringtones.html\">download free cingular ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/nextel-mp3-ringtones.html\">free mp3 nextel ringtones,nextel mp3 ringtones,mp3 nextel ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/ringtones-gratis.html\">ringtones gratis,1100 gratis nokia ringtones,c115 gratis motorola ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/download-free-new-ringtones.html\">download free new ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/carrier-cellular-free-ringtones-us.html\">carrier cellular free ringtones us<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/free-nokia-ringtones-tracfone.html\">free nokia ringtones tracfone<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/nickelback-ringtones.html\">free nickelback ringtones,nickelback ringtones,nickelback ringtones rockstar<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/sprint-download-ringtones.html\">download ringtones for sprint phone,sprint download ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/free-lg-ringtones-verizon-vx6000.html\">free lg ringtones verizon vx6000<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/box-music-ringtones-sony.html\">box music ringtones sony,sony music box ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/cell-free-phone-ringtones-samsung.html\">cell free phone ringtones samsung<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/info-motorola-personal-remember-ringtones.html\">info motorola personal remember ringtones t731,info motorola personal remember ringtones,info motorola personal remember ringtones v66<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/totally-free-cell-phone-ringtones.html\">totally free cell phone ringtones<\/a><a href=\"http:\/\/financiallyfreewomen.com\/wp-content\/1\/computer-download-free-ringtones.html\">computer download free ringtones<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/1-hour-loan-payday.html\">1 hour loan payday,payday loan in 1 hour,1 hour payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/savings-account-payday-loan.html\">savings account payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/no-fax-payday-advance-loan.html\">no fax payday advance loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/loan-payday-quick.html\">quick low interest payday loan,loan payday quick,quick pay payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/no-bank-statement-payday-loan.html\">no bank statement payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/bank-loan-no-payday-statement.html\">bank loan no payday statement<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/loan-payday-until.html\">loan payday until,loan payday say until wordpress,loan until payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/fast-payday-loan.html\">fast payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/quick-cash-payday-loan.html\">quick cash payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/line-loan-payday.html\">line loan payday,payday loan on line<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/aafes-loan-payday.html\">aafes loan payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/advance-cash-net-payday-usa.html\">advance cash net payday usa<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/cash-loan-payday.html\">cash advance loan payday internet,cash loan payday,instant cash payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/cash-until-payday-loan.html\">cash until payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/payday-loan-on-line.html\">payday loan on line<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/emergency-loan-payday.html\">emergency loan payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/cheap-payday-loan.html\">cheap payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/loan-payday-uk.html\">payday loan uk,loan payday uk,payday loan in the uk<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/payday-loan-company.html\">payday loan company<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/fee-loan-low-payday.html\">fee loan low payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/payday-loan-calculator.html\">payday loan calculator<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/fast-easy-payday-loan.html\">fast easy payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/bad-credit-guaranteed-loan-payday.html\">bad credit guaranteed loan payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/fax-loan-missouri-no-payday.html\">fax loan missouri no payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/payday-loan-in-georgia.html\">payday loan in georgia<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/advance-fax-loan-no-payday.html\">advance fax loan no payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/payday-loan-store-chicago.html\">payday loan store chicago<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/default-on-payday-loan.html\">default on payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/canadian-payday-loan.html\">canadian loan online payday,canadian payday loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/approval-guaranteed-loan-payday.html\">approval guaranteed loan payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/loan-payday.html\">payday cash loan,loan payday,payday advance loan<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/cash-fast-loan-payday.html\">cash fast loan payday<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/chicago-loan-payday-store.html\">chicago in loan payday store,chicago loan payday store<\/a><a href=\"http:\/\/privateinvestigator101.info\/wp-content\/1\/advance-cash-loan-payday.html\">advance cash loan payday today,payday payday loan cash advance loan,advance cash loan payday<\/a><\/u> a bit of sed and grep magic I had a file which contained merely the relevant inserts. It&#8217;s still running slow, but it&#8217;s running a lot quicker than previously.<\/p>\n<p>I know postgres&#8217;s dump functions tends to prefer COPY rather than INSERT because it is so much quicker, but that&#8217;s the point at which my familiarity with the COPY command ends, and I certainly didn&#8217;t want to spend the time to mangle the SQL I have into a couple of COPY commands.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>During the initial import of data to the {{post id=&#8221;jabbermetanetnz-has-moved&#8221; text=&#8221;new jabber.meta.net.nz server&#8221;}}, I noticed that the roster information failed to be migrated if I was using an ODBC backend &#8211; it would just fail entirely. I didn&#8217;t think too much of it until I tried to restart ejabberd, and noticed that it spent around [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13,6],"tags":[],"_links":{"self":[{"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/posts\/59"}],"collection":[{"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/comments?post=59"}],"version-history":[{"count":0,"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/posts\/59\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/media?parent=59"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/categories?post=59"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.meta.net.nz\/~daniel\/blog\/wp-json\/wp\/v2\/tags?post=59"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}