So 2 weeks ago I sadly moved my main server from FreeBSD 8.1 to a temporary home on an Ubuntu 11.04 VPS.  I say sadly because I’ve been using BSD for 10 years and in terms of stability and security I understand that it can’t be beat.. I also think I’m sad to see it go because having a tuned and efficient BSD webserver comes with some street cred,  it’s less mainstream than all these linux servers so I got to feel a little 1337 and a little different than the rest of the bunch.  Unfortuantely I didn’t document the move in it’s entirety, although there were certainly a few posts about postfix which proved to be the biggest challenge as it was my first mail server setup, and on an unfamiliar server environment to boot.  Luckily enough I’ve been a long time Ubuntu home user, although I sport Debian, so it wasn’t all greek to me.

The reason I moved to the Ubuntu VPS was that my main server was being shut down, well in reality the entire Datacenter was being shut down, and I had 24 hours notice to move.  Nice eh?  Luckily enough that worked out to 72 hours which was enough time to setup a BIND DNS, also a first for me, and make sure we didn’t have any major downtime.  We were just without email for 3 days while postfix decided to kick my ass.  all of this so that the few hundred sites on that server had a place to live while the new dedicated box I ordered was being shipped to it’s new home in Seattle.  Pretty stoked about that because I’m going from an old Quad CPU Xeon server with 2GB of ram to a dual CPU, quad core(8 cores in total) newer Xeon with 8GB of ram and twice the hard drive(and even more importantly a hard drive with all it’s sectors in tact – goodbye fschk!).  Intel® Xeon® Processor E5420 (12M Cache, 2.50 GHz, 1333 MHz FSB) to be exact!  So I got word today that the new Debian server is setup just the way I wanted it(which took a few tries) and so I’m setting about to migrate off this VPS(which I’ve already killed twice in 1 week) onto a much more stable and dedicated colo box that I can play with as rough as I want

I figured that I would document the entire process as it’s bound to be a hell of a lot less eventful and painful now that we’re moving between Ubuntu and Debian, two distros that are about as similar as you can get – but different enough in their own right.

Why Debian:

Well this is a question worthy of an entire blog post but I’ll give you my opinion to be taken as that and only that.  First off Debian is more stable and that stability comes at the expense of not being on the cutting edge in terms of kernel releases.  People will argue that I could just stick to the Ubuntu LTS but I don’t know if I could trust myself with that holding back  If this wasn’t such a critical production server, responsible for 100′s of websites and dozens of businesses than I might not care as much but knowing that Debian functions on the philosophy of only releasing new updates when they are 110% good and ready and not a bloody minute before really re-assures me that a server of this size, diversity and importance is going to be as stable as possible.

Secondly, I recently switched off Ubuntu on my home machines onto Debian 6.  This is part of my solitary boycott of Ubuntu now ever since 11.10 and the release of Unity was shoved down our throats.  Ubuntu has taken the path of appealing to the lowest common demoninator with this move and that comes at the price of alienating the hard core linux users that put it on the map in the fist place.  So philosophically I refuse to use Ubuntu straight out but much to my surprise, when I switched onto Debian, I noticed gigantic performance increases in almost every area.   I read through a ton of reviews of all linux distros before making my choice and was full ready to go to Fedora, Arch or CentOS instead of going to Debian which is so similar to Ubuntu and at the end of it all of the reviews said the same thing.. Pound for pound Debian is the best linux distro in there, it’s leaner, it’s meaner and as I’ve learned it allows us hardcore users to customize our experience out the wazoo without facing the wrath of Canonical.   Like I was saying Ubuntu is much like the windows of the linux world.. As far as linux goes it’s designed to appeal to as many users as possible and this causes it’s fair share of bloat with the default install.. Debian comes stripped down and if you want extras(hell rsync isn’t even installed by default) you need to put them on.. The end result is a much more streamlined and efficient operating system, which is exactly what I want for my server.

The last reason is compatibility.  I would hands down still be with FreeBSD right now if it wasn’t for compatibility issues.. I prefer the more draconian approach BSD takes to things, it’s simpler, less flashy and just makes sense to me after 10 years of working with it.  I like going into my ports tree and doing a make install instead of apt-get install.  I like having everything in /usr/local/etc/apache22 instead of /etc/apache2.  Or I should say I’m used to it because using Ubuntu for the past week has made me udnerstand why they do it the way they do and the benefits to that(but still honestly does every single apache module need it’s own .conf file?  come on, just make a section in the httpd.conf for modules).  Anyways, there are some really really cool server applications that I just can’t run on BSD and trust you me I’ve tried.  The main one is Big Blue Button.  Probably the single coolest server application I’ve ever seen that will enable me to take my collaborative teams to a whole new level of connectivity.  Unfortunately it’s made for Ubuntu and fortunately enough for me Debian is close enough for it to work pretty flawlessly.  There is a BSD port but I cannot even begin to describe the nightmare of getting that to work properly.  Oh wait, I just remembered that I already did describe what it took to get it to work here: http://groups.google.com/group/bigbluebutton-users/browse_thread/thread/27f89c10aae707c2/0a5ad6064ec95aa1

So all in all my need for comparability, stability and performance coupled with my need to be a little less mainstream than Ubuntu landed me with a fresh new Debian install.

Installation:

So first things first, let’s get the damn OS installed.. Or more specifically lets get a damn tech at the DC to get the damn OS installed  It’s a total barebones installation but I did specify that everything except for  /tmp should be ext4.  For some reason the techs went with ext3 on the first run around which I spanked them for and got them to reinstall it all with ext4.  Now I was running XFS on my BSD box and from what I understand overall that’s a better FS however I am not using the filesystem for anything that would need XFS at all, as amazing as it is.  That and in the last 2 years I chewed through 4 hard drives, not saying it was all XFS but I wanted something new that I haven’t destroyed.  I run ext4 at home and have been for 1-2 years now and have had no issues.  I also researched the hell out of it and for the most part it comes out well on top of ext3 in performance so I went with that.  Now I said everything but /tmp before and for that I went with tmpfs as my filesystem..  Remember, for the past 6 years I’ve been running an ever expanding and demanding empire of websites on an old server with inadequate memory and so i’ve developed this obsession with squeezing every ounce of performance out of the server(my apache res memory size on my tuned box is 29MB versus the 95MB that it is on  one of the shared servers I have some websites).  Somewhere in this obsession I stumbled across an article about running  /tmp on a tmpfs instead of ext4 and the big performance gains you see from that(assuming you have enough ram) and so here I am, going to test it out.  In the end my partition table ended up like this(It’s a 500GB drive):

Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/sda4 ext4 392287828 341664 372019076 1% /
tmpfs tmpfs 4098612 0 4098612 0% /lib/init/rw
udev tmpfs 4093808 140 4093668 1% /dev
tmpfs tmpfs 4098612 0 4098612 0% /dev/shm
/dev/sda2 ext4 28836860 475212 26896816 2% /usr
/dev/sda3 ext4 48061320 318892 45301008 1% /var
tmpfs tmpfs 523356 0 523356 0% /tmp

It has a 12GB Swap, overkill yes I know but i’ve got room to spare.  Anyways, I won’t get into the explanation for why I went this way, it was through a lot of research and I’m sure if yuo want to verify that this is a good or bad setup you can head over to google and do some digging yourself.

Post-Install

So as I mentioned, a stock Debian install comes pretty barebones.  Not even rsync is installed.. So there’s some basic stuff we need to setup to enable us to even install any further packages and to transfer the other server over.   If your techs are a little bit on the lazy side, as it seems mine were, then you’ll notice that when you try your first apt-get install it will find the package and just sit at 0%.  Good times.. A quick trip into /etc/apt/sources.list shows that they left source as the CD-ROM disc, which now wasn’t in the drive.  So you’ll need to uncomment out the internet sources and as I noted in my previous post you are going to need to  put another repo in the sources to grab some pretty important packages.  In the end my sources.list looked like this:

# deb cdrom:[Debian GNU/Linux 6.0.3 _Squeeze_ - Official amd64 DVD Binary-1 20111008-14:36]/ squeeze contrib main

deb http://security.debian.org/ squeeze/updates main contrib
deb-src http://security.debian.org/ squeeze/updates main contrib
deb http://ftp.debian.org/debian/ squeeze-updates main contrib
deb-src http://ftp.debian.org/debian/ squeeze-updates main contrib
deb http://backports.debian.org/debian-backports squeeze-backports main
deb http://http.us.debian.org/debian/ squeeze contrib non-free main

From here I ran apt-get update && apt-get upgrade and ugpraded my kernel to the latest version.  A quick reboot with my fingers crossed and now uname -a shows this:

Linux ded 2.6.32-5-amd64 #1 SMP Thu Nov 3 03:41:26 UTC 2011 x86_64 GNU/Linux

Which again is a testament to how anal the Debian people are about their kernel release as the latest kernel out is .39 or higher I believe.  Now the server is ready to be used.

The first thing I install is rsync as I want to start moving everything over from the vps.  The first thing I’m moving over are all of the user accounts.  Because Debian and Ubuntu use the same hash algorithm it is possible to just copy passwd group and shadow over and it will work for the most part but problems will crop up down the road because of some small differences with group names across the two distros.  So your best bet is to go into your passwd, group and shadow files and find all of the missing user accounts and groups on your production server that haven’t yet been created on the new server.  It’s as simple as copying and pasting them onto the end of the file in question. *NOTE: You should avoid copying over any of the package specific usernames that get created upon package installation.  You’ll run into problems here and there and it’s best just to let the installer do it’s own thing*  Do all of this before hand will make it possible to just move the entire /home directory tree over and maintain all of the existing file ownerships.  Which is exactly what I’m going to get going in the background while I start installing all of the needed packages.

One of my BSD habits that I take with me is simple directory structure.. I absolutely can’t stand seeing /var/www/home/user1/site.com/public-html/ .  Who the hell needs all of that?  I’ve been running servers for 10 years and I have never in my life seen a need to distinguish public-html and private-html, what the hell is private-html anyways?  Are so many people in need of this that it really needs to be the default directory structure?  Hell no, anything in ./site.com/ is what’s going to show up on www.site.com so why not mirror that on the server if possible?  So everything on my servers just goes into /home/ which is symlinked to /usr/home to appease some programs.

So the first thing I’m doing is moving everything over from the old server’s home directory: #rsync -avpl -e “ssh” root@oldserver.com:/home/ /home/   and off she goes!  Transfering the 120 GB over should buy me enough time to setup some of the modules.

After that I’m going to go through the installed packages on the Ubuntu server and start installing them all manually according to what I actually need.  Remember that Ubuntu will install a bit of bloat that you don’t need by default, so make sure you know what everything instead of just installing it all over as you’ll no doubt be loosing some of the lean meanness of Debian that way.

Installing needed packages:

So we need to see what’s installed on the old server and install it on the new one.. There are going to be some obvious ones here like apache2, php5.3, mysql, etc etc but it’s good to make sure you’ve got a full list and don’t miss anything important.  For this we need to use the following command: dpkg –get-selections on on both servers and then compare the list, going through the ubuntu one carefully and making sure we are only moving over the necessary packages and not bringing all of it’s bloat with it.

How much bloat you ask?  Well out of a freak of coincidence the Ubuntu server has EXACTLY double the installed packages as the Debian server.  531 on the Ubuntu box and 265 on the Debian box.  Now given the Ubuntu server is a life and functioning web server and with that come a few installed packages but I can assure you that I have not installed 265 additional packages since I took ownership of that VPS.  So what I do is  I take the output from both servers, copy and paste them into two separate .txt files on my local machine here and run diff to see where the differences lie.  Time to get busy!

 *Word from the wise* :  A lesson I learned from an old sysadmin is that the first thing you should be installing when setting up a new server remotely, via ssh without KVM, is the screen package.  Take it from my experience, there will be a time when your 25 minuets into a big package installation and your connection to the server will die, or something horrible will happen forcing you to clean up and start over.  Save yourself the headache and do everything from within a screen that will ensure your procs will stay running even if you don’t.

Mysql Transfer :

So if you’re like me then you have a good number of databases, users and the like to move over.  Again because the two distros are so similar you can literally just use rsync to pull the contents of /var/lib/mysql off your old server onto the new one.. I just moved /var/lib/mysql to /var/lib/mysql_bak on the new server incase anything goes wrong and I need to reference the old files and then just pulled the mysql from the old server onto the new one.  The only issue here was that I was seeing this error on the initial startup:  ERROR 1045 (28000): Access denied for user ‘debian-sys-maint’@'localhost’ (using password: YES) followed by my inability to stop the server, etc, etc.    Because of how similar they are the /etc/mysql directory was one of the only ones I didn’t overwrite frmo the old server.  I figured if anything is going to have distro specific shit in there it’s going to be mysql, so best to leave the new installs files as they are.. Unfortunately that wasn’t the case and in order to get through this error I just needed to go into the /etc/mysql/debian.cnf file on the old server and copy the ‘password =’ value over.  Fixed everything from there on in.

Bind9:

As with most everything else you can pretty much copy the entire contents of /etc/bind and be fine with all of the configurations moving over.. The only area where you are going to need to do a little surgery is in regards to the IP addresses.  When moving to a new server in a new location chances are that you’re going to be assigned a whole new lot of IP addresses and you’ll need to migrate everything on over to the new ranges.  Now I suppose you could do some clever bash scripting for this with two arrays of IP addresses if you have a few dozen IP’s on the server but if you have 15 or less your probably better of just using some find and sed magic.

As with anytime you are making a ton of changes automatically let’s back up our /etc/bind/zones/ directory and then use the following command as many times as need be to replace all of the old IP’s with all of the new ones.  Keep in mind that you should be inside of /etc/bind/ before running this or it might take a while

find . -exec sed -i ‘s/old.IP.address.here/new.IP.address.here/g’ {} \;

From here it’s probably a good idea to run a cat * | more and just go through all of the files to make sure that everything is looking peachy and that you didn’t miss anything

Apache and Nginx:

Now depending on how you have your server setup you might have to do the same thing that yuo just did with bind except with apache as well.  Running find and replacing all of your old IP’s in apache conf files with new ones.  If you are like me however, apache is just acting as a proxy for nginx and in which case you need to do the same as with bind but with nginx.

Postfix / Dovecot:

I’m knocking on wood and crossing my fingers but this was incredibly painless to move over(now that it’s actually setup).  Just make sure that you carry all your certs over if you’ve setup SSL and that you’ve recreated  /var/spool/postfix/var/run/saslauthd so that everyone’s happy.  I spent a solid week setting this damn email server up the first time and I’m tickled pink to see it seamless transfer over and work right off the bat.  Again you should double check all of the conf files to make extra sure there’s nothing that needs altering and then telnet into 25, 587, 110 or 995 to test everything out

Testing:

So everything is pretty much moved over now but before you go ahead and edit your nameserver info at your registrar to point at the new server you better test the hell out of everything.  This is pretty easily accomplished by editing your /etc/hosts file(or localhost in windows) and putting in a list of domains you want to test and manually specifying what IP your computer should look for them on, essentially bypassing the nameservers.

something like 127.0.0.1 example.com would mean that instead of going to the actual IP for example.com it would look at the 127 IP instead.  This will allow you to test all of the domains you want to make sure are functioning properly.. It will allow you to check that mysql is doing what it needs to do, that php5 is happy and of course apache/nginx are working properly.  I was pleasantly surprised(shocked is a better word) that it all showed up immediately and perfectly as expected.

If you have a mail server setup you should also put your mail server address(ie: mail.example.com) in your hosts file to test that out with whatever email client you are using

I cannot express the true depth of my shock when everything came up working almost immediately.. Remember I initially moved 100+ domains and databases off of a BSD server that was having all the nameserver and email stuff managed externally onto an Ubuntu server where I was taking care of all of the nameserver and email stuff internally.  It was hell to say the least, mostly because I had to learn the nuances of the new OS and I had to re-setup everything by hand(or well with the help of some bash scripts I wrote) in terms of the domains, the email addresses, the dns entries, apache, nginx, etc etc.  This time it was as simple as copying everything over, double checking a few things and what took me 7-8 days of strife and stress now took me about 3-4 hours of work with everything basically working exactly as it should have and the troubles I did encounter were straight forward and easily solvable.  I could get used to this

So there you have it, I didn’t bother to document a lot of the setup of the individual components as there are all sorts of great guides out there written by people far more intelligent than I about that

Tags: apache, bind, debian, dns. rsync, dovecot, migrate, migration, move, mysql, nginx, postfix, server, ubuntu, web