A recent discussion/argument I had on Reddit got me thinking about the cost of solutions we put in.

In an ideal world everything would have full redundancy, and the customer would never have any downtime. Everything would always be up-to-date and keeping it so would require restarting. The reality is very different unfortunately.

This potentially rambling post was inspired by someone accusing me of having “a horrible idea” because I suggested someone put pfsense on an Atom PC as a VPN router for a small office. He then proceeded to expain to me how you should always buy an expensive black box from a vendor (he didn’t say black box if I am honest, I am interpreting), how you have to always have support on absolutely everything. I called ‘bullshit’ and the whole thing went round in circles a bit until we both realised that were actually singing from the same song sheet, but from different ends of the room.

What am I doing today? Documentation that is what. I am writing a document on how to do this. To any Linux user it is a very simple process and I could just give them a link to my own website.

I am not writing this for a technical audience though. The people who are going to perform this work will be the ‘Level 1 operatives’. This translates roughly to “anyone we can find on the street corners of some Far East city”. If I tell them to press the red button labelled “press me” and it turns out to be orange, they will stop. I cannot assume the ability to edit a file in Vi. How can you work around this, well you need to make everything a copy and paste operation. This is easily done in Bash thanks to IO redirection and of course Sed.

This came up today where I needed to give secure file transfer to customers. To complicate things I had to use an out-of-the-box RHEL6 system. The obvious answer was to use SSH and limit those users to SFTP only. Locking them into a chroot was not a requirement, but it seemed like a good idea to me. I found plenty of docs that got 80% of the way, or took a shortcut, but this should be complete.

I have now got new challenges and am designing much bigger systems. Whereas before I would have take one of these: {% img center /images/p2000-g3-sff.png 250 400 %} Plugged it into a pair of these: {% img center /images/silkworm.jpg 250 450 %} And finally plugged in a pair of these: {% img center /images/dl380g7.jpg 200 400 %}

Now I do not actually do the plugging in - that is not part of the documented process. Also, the system I am designing is on the other side of the world. However I now take one of these, a couple of these and add a couple of racks of these. Finally it all plugs into a couple of these.

How many times have you installed/updated a bit of software and read the line “Please take a back up” or something to that effect? 99 times out of a hundred, you will just continue and ignore it.

Today I had a reminder of why it is import to do so. I did a routine plug-in upgrade on our Jira installation (Customware Salesforce connector for those who want to know). I have done this several times, I had tested it in our Dev installation I was 100% confident it would work as expected. However, I actually decided to take a backup anyway.

This based on my last post where I documented building a Highly Available NFS/NAS server.

There is not a huge amount that needs to be done in order to add iSCSI functionality as well.

Add a file called /etc/drbd/iscsi.res containing:

resource iscsi {
    on nfs1 {
        device /dev/drbd1;
        disk   /dev/vdc;
        meta-disk internal;
        address   10.0.0.1:7789;
    }
    on nfs2 {
        device /dev/drbd1;
        disk   /dev/vdc;
        meta-disk internal;
        address   10.0.0.2:7789;
    }
}

This differs from the previous resource in 2 ways. Obviously it using a different physical disk. Also the port number of the address is incremented; each resource has to have its own port to communicate on.

Take 2 Centos Servers (nfs1 and nfs2 will do nicely) and install ELrepo and EPEL on them both:

yum install \
    https://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm \
    https://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm --nogpgcheck

Each of them should ideally have 2 NICS, with the secondary ones just used for DRBD sync purposes. We’ll give these the address 10.0.0.1/32 and 10.0.0.2/32.

I am also assuming that you have disabled the firewall and SELinux – I do not recommend that for production, but for testing it is fine.