serverascode.com rss feed

Where to find vagrant boxes

2013-04-25T00:00:00-07:00

(won’t find any vagrant boxes in the yellowhead brewery in downtown edmonton, just beer)

25 April – 2013 – Edmonton!

Where to find vagrant boxes

This is just a quick post on a couple of places I know to find vagrant boxes.

Ubuntu cloud images

Today ubuntu 13.04, aka raring ringtail, was released But did you know that ubuntu actually provides vagrant specific boxes, ones that are built every day? They sure do!

http://cloud-images.ubuntu.com/vagrant/raring/current/

So it’s quite simple to try out raring just by using vagrant and ubuntu’s cloud images.


$ mkdir raring; cd raring
$ vagrant init
# Edit Vagrantfile and add the below
$ grep box Vagrantfile | grep -v "#"
  config.vm.box = "raring"
  config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/raring/current/raring-server-cloudimg-amd64-vagrant-disk1.box"

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
SNIP!

#
# Now we can ssh into the box
#

$ vagrant ssh
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)
SNIP!
vagrant@vagrant-ubuntu-raring-64:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.04
DISTRIB_CODENAME=raring
DISTRIB_DESCRIPTION="Ubuntu 13.04"

Nice. That was easy.

Vagrantbox.es

Most vagrant users will know about this site, but I add it here for completeness.

http://www.vagrantbox.es/

Obviously for testing/development using these images is just fine, but most shops will want to build their own production images. I think.

Make your own

Veewee is a good way to automate image creation. I’ve used it quite a bit, but that was a few months ago. It can take a bit of work to get it up and running.

Puppet labs also publishes some information on creating vagrant boxes, as well as several pre-built boxes.

Fedora

Unfortunately I can’t seem to find official fedora or redhat vagrant boxes, which is too bad.

But, fedora is working on it!

http://fedoraproject.org/wiki/Features/FirstClassCloudImages

I’ll try to update this page as I find more resources. Please feel free to comment with suggestions. :)

Thanks,
Curtis.

More over committing with kvm

2013-04-12T00:00:00-07:00

(very similar to one of my graphs below…coincidence?)

12 April – 2013 – Edmonton!

More over committing with kvm

Previously I wrote about overcommiting with kvm.

In this post I’m still doing the exact same thing, but now I’m keep track of load and iops.

Basic environment

What we have is:

a single Dell C6220 node
- 32 threads
- 128GB of memory
- two Intel 520 SSDs in a stripe
ubuntu precise64 cloud image
qcow2 image snapshots
open vSsitch
dnsmasq providing ip addresses
ansible for running stress tests

What we’re going to do is boot 300 2GB instances based off the same image.

Get setup

First thing we do is reboot, just to start fresh. It doesn’t really matter, but I like to start with no swap being used, just to show exactly what happens when we boot hundreds of vms.

First set ksm to do more scanning faster.


root# echo "20000" > /sys/kernel/mm/ksm/pages_to_scan
root# echo "20" > /sys/kernel/mm/ksm/sleep_millisecs

Next—re-setup vswitch, as I have a kind of rigged-up configuration going, which needs some care after a reboot, so obviously this is just for testing, not production. :)


$ sudo ifconfig br-int
$ sudo ifconfig br-int up
$ sudo ifconfig br-int 192.168.100.10 netmask 255.255.255.0

That ip is what dnsmasq is set to listen on.

I was running into an error where some taps existed already, and the boot script would start failing. I’m not sure why, and I just ended up deleting all the ports in the switch for that particular bridge, br-int.


#
# Show/count all the ports
# 

$ sudo ovs-vsctl list-ports br-int | wc -l
300

#
# Delete some ports
#

$ sudo for i in $(seq 1 300); do ovs-vsctl del-port br-int tap$i; done
# This takes a while...

Now restart to make sure dnsmasq is listening on 102.168.100.10.


$ sudo service dnsmasq stop  
 * Stopping DNS forwarder and DHCP server dnsmasq
 [ OK ] 

#
# Now with the right br-int config
#

$ sudo dnsmasq start

Next—time to start hundreds of vms!

Start virtual machines

Ok, now that we have networking (hopefully) all set up, we’re going to boot 300 vms, 10 seconds apart.

Thankfully these are linux vms so they don’t really cause a boot storm, unlike windows 7.

If I booted 30 (note: 30, not 300) windows 7 vms the server’s load would get so high that the system would grind to a halt. I know because I’ve tried it. Even though this is an ubuntu cloud image, which hopefully is specifically setup to use less iops, and knowing that is perhaps an unfair advantage, I still have no problem saying that windows images use more resources than linux images.

Start instances!


root# ./kvm_ubuntu_openvswitch.sh

After that completes there are ~300 vms running.


$ ps ax | grep "kvm -drive" | wc -l
301

We can run tests on those vms.

NOTE: I haven’t taken the time to tell dnsmasq to send dhcp information for more than a /24, so we only have 240 vms with an ip address.


$ wc -l /var/lib/misc/dnsmasq.leases 
240 /var/lib/misc/dnsmasq.leases

So when I run the stress test with ansible, we can only run it across 240 vms, even though 300 are running, it’s just that 60 of them don’t have ips.

Stress test

In my previous post a commenter suggested running stress, so that’s what I’m doing.

Using ansible, I’ll run this stress command:


shell stress --cpu 1 --io 1 --vm 1 --vm-bytes 1024M --timeout 10s

across 20 vms at a time, running over all the vms that are reported by an inventory script that looks at the ips in the dnsmasq.leases file.

Here’s an example of running ansible’s ping module across all those hosts.


 $ ansible all -c ssh -i ./inventory.py -m ping -u ubuntu
 SNIP!
 192.168.100.98 | success >> {
    "changed": false, 
    "ping": "pong"
}

192.168.100.99 | success >> {
    "changed": false, 
    "ping": "pong"
}

192.168.100.97 | success >> {
    "changed": false, 
    "ping": "pong"
}

Note that the ips aren’t in order, so .97 is the last host in this run. Suffice it to say that all 240 hosts “ponged” back. :)

Here’s the simple ansible playbook I’ll be running. These vms don’t have access to the internet, and don’t have stress installed, so I’m just copying over the package and installing it “manually”, and then running the stress command.


$ cat load.yml 
---
- hosts: all
  user: ubuntu
  sudo: yes
  tasks:
  - name: check if stress is already installed
    action: shell which stress
    register: stress_installed
    ignore_errors: True
  - name: copy stress deb to server
    action: copy src=files/stress_1.0.1-1build1_amd64.deb \
    dest=/tmp/stress_1.0.1-1build1_amd64.deb
    only_if: ${stress_installed.rc} > 0
  - name: install stress
    action: shell dpkg -i /tmp/stress_1.0.1-1build1_amd64.deb
    only_if: ${stress_installed.rc} > 0
  - name: run stress
    action: shell stress --cpu 1 --io 1 --vm 1 --vm-bytes 1024M --timeout 10s

Let’s run it across 20 vms at a time and see what happens.


$ ansible-playbook -u ubuntu -c ssh -i ./inventory.py -f 20 ./load.yml

PLAY [all] ********************* 

GATHERING FACTS ********************* 
ok: [192.168.100.110]
ok: [192.168.100.113]
ok: [192.168.100.117]
SNIP!    
192.168.100.97                 : ok=3    changed=2    unreachable=0    failed=0    
192.168.100.98                 : ok=3    changed=2    unreachable=0    failed=0    
192.168.100.99                 : ok=3    changed=2    unreachable=0    failed=0  

# Done!

Ansible can be fun. :)

Graphs

Below are a rather poor set of graphs. Forgive me as I’m a newbie with gnuplot.

As soon as the load starts going up, that is when the test starts, and as soon as it’s on its way down, that’s when it ends. :)

First run:

Second run:

Last load run:

So with three runs we see a load of about 30, where a load of 32 would be Ok with me, given we have 32 threads in this server.

Also, let’s watch some iops.

I gathered iops data using iostat.

Ooof, that’s a misleading graph, isn’t it? An increase of one iop is absolutely nothing. A rounding error perhaps.

So—the iops don’t change during the test run, I guess because stress isn’t running any io test, even though we are running with —io 1. That said, I’m not sure what an io setting of 1 with stress does, something to look into. Perhaps running some tests with fio is something to do in the future.

But that graph sure looks like something’s happening, even though the iops only increase by one. I probably shouldn’t include that graph here, but part of what I’m doing is learning about how to display the results of a performance test.

Conclusion

First off, I’m not a scientist, didn’t take statistics, etc. So I’m not sure what kind of conclusions can be made here. All I can say for sure is that fours runs of the stress command across 20 virtual machines in parallel, over a total of 240 vms all running on the same KVM-based host, seems to bring load up to somewhere around 30 to 35, which is acceptable to me.

Mostly this has generated more questions—such as what exactly is stress doing? How do we know when the vms are too unresponsive? What kind of overcommitting numbers do we want? What would happen if we used fio instead of —io 1 with stress? Do red lines in a graph make things seem worse? :)

As usual, if anyone has any suggestions, questions, or critiques let me know in the comments!

Thanks,
Curtis.

Vagrant and vmware

2013-04-11T00:00:00-07:00

(close up of an old map with many pin holes in edmonton, can barely even read it! Look, Barrhead too!)

11 April – 2013 – Edmonton!

Vagrant and vmware

After IRC, vagrant is probably my most important development tool, mostly because I like to use and investigate openstack, which means using a lot of virtual machines.

Recently Hashicorp released Vagrant 1.1 which introduces the idea of providers. Previously vagrant only supported virtualbox, but now, with 1.1, plugins can be written to support almost any virtualization system that has a command line or API interface of some sort.

For example:

VMWare Fusion (note that this is a paid plugin)
AWS (github repo)
RackSpace
OpenStack (based on the rackspace plugin)

The provider I’m going to focus on here is vmware fusion.

vmware_fusion

One of the things I’ve learned about using brand new technologies is that they often don’t work and/or don’t have any documentation, which frankly are about the same thing to me. That sounds like a kind of grumpy thing to say, but I’m kind of grumpy today. :)

Regardless, I went ahead and bought vmware fusion (which is cheap, BTW, at $49) and also the vagrant vmware_fusion plugin (which is $79).

I think this is the first time that I’ve encountered a plugin that was more expensive than the actual application it was plugging into, but I can understand the pricing because Fusion is probably under-valued, or at least under-priced. Plus the $79 goes towards the development of vagrant, which I use a lot.

Recently I deployed packstack via vagrant and virtualbox, and I wanted to do the same with vmware_fusion, but I ran into a few problems, which I’m going to spend the rest of the post detailing.

NOTE: I should say that nothing here is the vmware_fusion plugins fault. I’m not blaming the plugin at all. Rather just detailing some of the pain points I’ve encountered, which will no doubt disappear as more people use vmware fusion and vagrant together, and as I get my act together. I’ll try to update this post as I find out new information. :)

Routes collide!

I have both vmware fusion and virtualbox installed on my macbook retina. Unfortunately, virtualbox has an iron grip on its networks.

VirtualBox hangs on to its network devices (“vboxnet”) for dear life. I haven’t figured out yet how to actually get rid of them except restarting your computer. — Mitchell Hashimoto

If you encounter the below error, either change subnets (perhaps in virtualbox, perhaps in the vagrantfile, not sure) or reboot.


$ vagrant up apis --provider=vmware_fusion
Bringing machine 'apis' up with 'vmware_fusion' provider...

[apis] Verifying vmnet devices are healthy...
The VMware network device 'vmnet2' can't be started because
its routes collide with another device: 'vboxnet'. Please

either fix the settings of the VMware network device or stop the
colliding device. Your machine can't be started while VMware
networking is broken.

Again, not vmware_fusion’s fault, but still a pain. I can’t simply un-install virtualbox…yet.

vmx settings

Often we want to change the settings in the virtual machine, settings such as memory, number of cpus, etc.

Unfortunately vmx is an undocumented format.

VMX is an undocumented format. You’ll have to google, unfortunately. :) — Mitchell Hashimoto

But at the very least here is how to set memory:


config.vm.provider :vmware_fusion do |p|
  p.vmx['memsize'] = '2048'
end

As more people use vmware_fusion there will be better documentation on vmx settings.

Centos6 box

While Hashicorp has conveniently provided a base precise64 box for vagrant, there isn’t an official centos box. I have previously tried to create a centos6 box for vagrant, but haven’t had much luck, and that was with vagrant < 1.1 and there is even less documentation on the process now.

Then I noticed that vagrantbox.es (which is a very handy site!) has a centos6 box for vmware_fusion, so I grabbed that:

CentOS 6.4 x86_64 Minimal VMware Fusion

Unfortunately it doesn’t seem to work when multiple interfaces are specified in the vagrantfile, so that doesn’t help me much on my quest to run packstack in vmware_fusion. If anyone knows of a good centos6 box, or notices that I’m doing something wrong, please let me know!

Here’s the networking part of the config:


config.vm.network :private_network, ip: "172.10.0.200", :netmask => "255.255.0.0"
config.vm.network :private_network, ip: "10.10.0.200", :netmask => "255.255.0.0"

Let’s boot it:


$ vagrant up --provider=vmware_fusion
Bringing machine 'default' up with 'vmware_fusion' provider...
[default] Cloning Fusion VM: 'centos65fusion'. This can take some time...
[default] Verifying vmnet devices are healthy...
[default] Preparing network adapters...
[default] Starting the VMware VM...
[default] Waiting for the VM to finish booting...
[default] The machine is booted and ready!
[default] Forwarding ports...
[default] -- 22 => 2222
[default] Configuring network adapters within the VM...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

/sbin/ifup eth1 2> /dev/null

Ooops, shouldn’t be seeing the failed command.

What’s the networking like?


$ vagrant ssh
[vagrant@vagrant-centos-6 ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:24:6A:AD  
          inet addr:192.168.134.146  Bcast:192.168.134.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe24:6aad/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:432 errors:426 dropped:0 overruns:0 frame:0
          TX packets:293 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:48690 (47.5 KiB)  TX bytes:38046 (37.1 KiB)
          Interrupt:19 Base address:0x2024 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Nope that’s not what I wanted at all.

Ok, now let’s use the exact same vagrantfile but with the offical vmware_fusion ubuntu box.


config.vm.box = "precise64"

Vagrant up!



#
# Destroy the old one
# 

$ vagrant destroy
[default] Stopping the VMware VM...
[default] Deleting the VM...

#
# Edit the vagrantfile to use precise64 basebox
#

$ vi Vagrantfile

#
# Boot it
# 

$ vagrant up --provider=vmware_fusion
Bringing machine 'default' up with 'vmware_fusion' provider...
[default] Cloning Fusion VM: 'precise64'. This can take some time...
[default] Verifying vmnet devices are healthy...
[default] Preparing network adapters...
[default] Starting the VMware VM...
[default] Waiting for the VM to finish booting...
[default] The machine is booted and ready!
[default] Forwarding ports...
[default] -- 22 => 2222
[default] Configuring network adapters within the VM...
[default] Enabling and configuring shared folders...
[default] -- vagrant-root: /Users/curtis/working/vagrant/grizzly

#
# SSH into the box...
# 

$ vagrant ssh
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Thu Jan 31 13:48:53 2013
vagrant@precise64:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0c:29:29:dc:aa  
          inet addr:192.168.134.139  Bcast:192.168.134.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe29:dcaa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:375 errors:367 dropped:0 overruns:0 frame:0
          TX packets:241 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41473 (41.4 KB)  TX bytes:33220 (33.2 KB)
          Interrupt:18 Base address:0x2024 

eth1      Link encap:Ethernet  HWaddr 00:0c:29:29:dc:b4  
          inet addr:172.10.0.200  Bcast:172.10.255.255  Mask:255.255.0.0
          inet6 addr: fe80::20c:29ff:fe29:dcb4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:1 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:209 (209.0 B)  TX bytes:468 (468.0 B)
          Interrupt:16 Base address:0x20a4 

eth2      Link encap:Ethernet  HWaddr 00:0c:29:29:dc:be  
          inet addr:10.10.0.200  Bcast:10.10.255.255  Mask:255.255.0.0
          inet6 addr: fe80::20c:29ff:fe29:dcbe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:1 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:209 (209.0 B)  TX bytes:468 (468.0 B)
          Interrupt:17 Base address:0x2424 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

That’s what I expect to see.

Example vagrantfiles

UPDATE (April 18th, 2013): This Vagrantfile now doesn’t seem to work with Vagrant 1.2

I have been searching for good examples of vagrantfiles that use vmware_fusion.

So far I’ve just found this one:

https://github.com/uksysadmin/OpenStackCookBook-1/blob/master/Vagrantfile

But I will keep an eye out for other examples.

UPDATE (April 18th, 2013): New network problem

This is a new one…now this can’t be virtualbox’s fault.


$ vagrant up --provider=vmware_fusion
Bringing machine 'percona0' up with 'vmware_fusion' provider...
Bringing machine 'percona1' up with 'vmware_fusion' provider...
Bringing machine 'percona2' up with 'vmware_fusion' provider...
Bringing machine 'haproxy0' up with 'vmware_fusion' provider...
Bringing machine 'haproxy1' up with 'vmware_fusion' provider...
[percona0] Cloning Fusion VM: 'precise64'. This can take some time...
[percona0] Verifying vmnet devices are healthy...
The VMware network device 'vmnet1' can't be started because
its routes collide with another device: 'vmnet13'. Please
either fix the settings of the VMware network device or stop the
colliding device. Your machine can't be started while VMware
networking is broken.

Instructions from Mitchell, edit /Library/Preferences/VMware\ Fusion/networking and:

Get rid of all the lines in that file except the ones that start with “answer VNET_1_” or “answer VNET_8_”. We want to keep those, as they’re the default networks that ship with Fusion. After that, open VMware Fusion.app, then run these commands in a separate terminal:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli —stop
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli —configure
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli —start

Then run

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli —status

And tell me the output. Should only have the vmnet1/vmnet8 devices. After
THAT you shoudl be good to go again.

VMware networking is an absolute nightmare. – Mitchell

It sucks that it’s an edge case, but I still hope that there is some code added to help in situations like this. Thanks to Mitchell for responding on the mailing list, as now I can continue on with other problems. :)

Conclusion

I love vagrant, but am having a heck of a time with vmware_fusion and centos6. Having said that, I KNOW that things are going to get better as I learn and as more people start using vagrant and vmware_fusion.

Hats off to Mitchell for creating a great development tool, one that I use every day!

Thanks,
Curtis.

Vagrant and openstack

2013-04-11T00:00:00-07:00

(it’s ok, we know the universe is a hologram anyways)

11 April – 2013 – Edmonton!

Vagrant and openstack

Earlier I wrote a post on using vmware fusion and vagrant.

Now I’m going to use vagrant and the vmware_fusion plugin to create a precise64 virtual machine, in which I will install devstack, and then I will use the vagrant and openstack plugin to boot a cirros vm inside the devstack vm. Meta…inception…whatever you want to call it. :)

NOTE: Make sure your precise64 vm has more than the default memory of 512—I set mine to 2048. A bit more memory might be nice too, if you’ve got it available.

NOTE: Here is a great post to follow on using devstack and grizzly and quantum, much of which I am reusing here.

Why?

There is no spoon.

Install devstack

Devstack is a really useful development environment for openstack. If you want to try out the new features in openstack grizzly, this is an easy way.


#
# Create the vm
#

$ vagrant up --provider=vmware_fusion
Bringing machine 'default' up with 'vmware_fusion' provider...
SNIP!

#
# Login to the vm
#

$ vagrant ssh
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)
 * Documentation:  https://help.ubuntu.com/
Last login: Thu Apr 11 10:14:43 2013 from 192.168.134.1
vagrant@precise64:~$ sudo apt-get update
Ign http://security.ubuntu.com precise-security InRelease
Ign http://us.archive.ubuntu.com precise InRelease
Ign http://us.archive.ubuntu.com precise-updates InRelease
SNIP!
$ sudo apt-get install git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  git-man libcurl3-gnutls liberror-perl librtmp0 rsync
SNIP!
$ git clone git://github.com/openstack-dev/devstack.git
Cloning into 'devstack'...

#
# Setup devstack with a localrc file
# 

vagrant@precise64:~$ cd devstack
vagrant@precise64:~/devstack$ cat localrc
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,n-cauth, \
horizon,mysql,rabbit,sysstat,cinder,c-api,c-vol,c-sch,n-cond,quantum,q-svc, \
q-agt,q-dhcp,q-l3,q-meta,q-lbaas,n-novnc,n-xvnc,q-lbaas
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=password
SERVICE_PASSWORD=password
ADMIN_PASSWORD=password

#
# Run stack.sh
# 

vagrant@precise64:~/devstack$ ./stack.sh
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ImportError: No module named netaddr
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ImportError: No module named netaddr
SNIP!

# That error doesn't look good...oh well let's continue...

# hit enter a few times
# Go for a walk, get a coffee, do some vacuuming...

Horizon is now available at http://192.168.134.139/
Keystone is serving at http://192.168.134.139:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: ed5cb213364bb0fd15a9
This is your host ip: 192.168.134.139
stack.sh completed in 694 seconds.

Now that devstack seems to have completed the install, check and see if basic openstack commands are working.


# 
# Source the user, password file generated by devstack
#

vagrant@precise64:~/devstack$ source openrc 

#
# And lets see what's running and is available
#

vagrant@precise64:~/devstack$ nova list
+----+------+--------+----------+
| ID | Name | Status | Networks |
+----+------+--------+----------+
+----+------+--------+----------+
vagrant@precise64:~/devstack$ nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID                                   | Name                            | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| 73f320dd-5769-4ec2-a0e7-e44979070e8c | cirros-0.3.1-x86_64-uec         | ACTIVE |        |
| 4af449b1-a70b-4857-93ea-9690bc5db779 | cirros-0.3.1-x86_64-uec-kernel  | ACTIVE |        |
| 017e58df-27bc-4bb4-89d3-f133760a3f0e | cirros-0.3.1-x86_64-uec-ramdisk | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+

#
# Oooh, we have quantum too!
#

vagrant@precise64:~$ quantum net-list
+--------------------------------------+---------+--------------------------------------------------+
| id                                   | name    | subnets                                          |
+--------------------------------------+---------+--------------------------------------------------+
| 5a39203e-3d83-4d47-a75e-9ec98f5ed595 | private | dae29b88-1562-42e4-8e30-0ecce7b40f47 10.0.0.0/24 |
| a608d79d-ace8-4335-81c3-3490393d7700 | public  | cc058059-b342-41d9-8c68-98d6feedcfbd             |
+--------------------------------------+---------+--------------------------------------------------+
vagrant@precise64:~$ quantum subnet-list
+--------------------------------------+------+-------------+--------------------------------------------+
| id                                   | name | cidr        | allocation_pools                           |
+--------------------------------------+------+-------------+--------------------------------------------+
| dae29b88-1562-42e4-8e30-0ecce7b40f47 |      | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
+--------------------------------------+------+-------------+--------------------------------------------+

Cool beans.

Using vagrant with openstack

First, get the vagrant-openstack plugin.


$ vagrant plugin install vagrant-openstack
Installing the 'vagrant-openstack' plugin. This can take a few minutes...
Installed the plugin 'vagrant-openstack (0.0.2)'!
$ vagrant plugin list
vagrant-openstack (0.0.2)
vagrant-vmware-fusion (0.4.2)

Before we get too far, let’s create a keypair in devstack.


vagrant@precise64:~$ source ~/devstack/openrc 
vagrant@precise64:~$ nova keypair-add --pub-key ~/.ssh/authorized_keys vagrant
vagrant@precise64:~$ nova keypair-list
+---------+-------------------------------------------------+
| Name    | Fingerprint                                     |
+---------+-------------------------------------------------+
| vagrant | dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6 |
+---------+-------------------------------------------------+

I’m going to create a new local directory to work with vagrant out of.


$ cd ~/working/vagrant
$ mkdir vagrant-openstack
$ cd vagrant-openstack
$ vagrant init
$ vi Vagrantfile
# Add config information...

We need to insert some information into the vagrantfile for openstack.

First get the image ID. Devstack automatically adds an image, but each time devstack is run the ID will be different.


vagrant@precise64:~$ nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID                                   | Name                            | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| 0cf481ad-482e-441c-b8a6-49e792ae0dfb | cirros-0.3.1-x86_64-uec         | ACTIVE |        |
| 2630cd9e-c375-49d0-81bd-ffbfc638e752 | cirros-0.3.1-x86_64-uec-kernel  | ACTIVE |        |
| 7375ddbc-51c7-4492-bd2b-de30f10210db | cirros-0.3.1-x86_64-uec-ramdisk | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+

In this example we want the 0cf481ad-482e-441c-b8a6-49e792ae0dfb image ID.

Also, we probably want to add a smaller flavor for the cirros image. By default the smallest flavor uses 512MB of ram.


#
# Default flavors
#

vagrant@precise64:~$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| 1  | m1.tiny   | 512       | 0    | 0         |      | 1     | 1.0         | True      | {}          |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      | {}          |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      | {}          |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      | {}          |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      | {}          |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+

#
# Add a smaller flavor
#  

vagrant@precise64:~$ nova-manage flavor create --name=m1.teeny --memory=64 \
--cpu=1 --root_gb=0 --ephemeral_gb=0 --flavor=6 --swap=0 --is_public yes
2013-04-11 11:36:08    DEBUG [nova.openstack.common.lockutils] Got semaphore \
"dbapi_backend" for method "__get_backend"...
m1.teeny created

#
# Now we have a 6th flavor!
# 

vagrant@precise64:~$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| 1  | m1.tiny   | 512       | 0    | 0         |      | 1     | 1.0         | True      | {}          |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      | {}          |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      | {}          |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      | {}          |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      | {}          |
| 6  | m1.teeny  | 64        | 0    | 0         |      | 1     | 1.0         | True      | {}          |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+

Flavor 6 is what we’ll use.

Next, check the OS vars in devstack to see what to put into the vagrantfile:


vagrant@precise64:~$ env | grep "^OS"
OS_PASSWORD=password
OS_AUTH_URL=http://192.168.134.139:5000/v2.0
OS_USERNAME=demo
OS_TENANT_NAME=demo
OS_CACERT=/opt/stack/data/CA/int-ca/ca-chain.pem
OS_NO_CACHE=1

Now with all that information we can fill out the vagrantfile. Mine looks like this:


$ cat Vagrantfile 
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "base"
  
  config.vm.provider :openstack do |os|

    os.url = "http://192.168.134.139:5000/v2.0"
    os.tenant = "demo"
    os.user = "demo"
    os.password = "password"

    os.flavor = "6"
    os.keypair = "vagrant"
    os.image = "0cf481ad-482e-441c-b8a6-49e792ae0dfb"

    # Not sure why but I feel like calling this vm whitney
    os.name = "whitney"

    os.ssh_username = "cirros"
    os.ssh_private_key = "~/.ssh/id_dsa.pub"
  end

end

Using vagrant, boot the vm.


#
# Boot it
# 

$ vagrant up --provider=openstack
Bringing machine 'default' up with 'openstack' provider...
[default] New VM created 419e5940-e068-42a4-bb28-68ad72f85d8a => whitney

#
# Check status
#

$ vagrant status
Current machine states:

default                  running (openstack)

The nova instance is running. To stop this machine, you can run
`vagrant halt`. To destroy the machine, you can run `vagrant destroy`.

Let’s see what’s happening in devstack.


#
# What does virsh know?
# 

vagrant@precise64:~$ sudo virsh list
 Id Name                 State
----------------------------------
  1 instance-00000001    running

#
# And lets ask openstack... 
#

vagrant@precise64:~$ source ~/devstack/openrc 
vagrant@precise64:~$ nova list
+--------------------------------------+---------+--------+------------------+
| ID                                   | Name    | Status | Networks         |
+--------------------------------------+---------+--------+------------------+
| 419e5940-e068-42a4-bb28-68ad72f85d8a | whitney | ACTIVE | private=10.0.0.3 |
+--------------------------------------+---------+--------+------------------+
vagrant@precise64:~$ nova show 419e5940-e068-42a4-bb28-68ad72f85d8a
+-----------------------------+----------------------------------------------------------------+
| Property                    | Value                                                          |
+-----------------------------+----------------------------------------------------------------+
| status                      | ACTIVE                                                         |
| updated                     | 2013-04-11T18:40:43Z                                           |
| OS-EXT-STS:task_state       | None                                                           |
| private network             | 10.0.0.3                                                       |
| key_name                    | vagrant                                                        |
| image                       | cirros-0.3.1-x86_64-uec (0cf481ad-482e-441c-b8a6-49e792ae0dfb) |
| hostId                      | cbfc5a689eaff0c72de8f66161efb06270322d48baf6d9120f612c42       |
| OS-EXT-STS:vm_state         | active                                                         |
| flavor                      | m1.teeny (6)                                                   |
| id                          | 419e5940-e068-42a4-bb28-68ad72f85d8a                           |
| security_groups             | [{u'name': u'default'}]                                        |
| user_id                     | 26c0f9a23e9c44f6b660557122119171                               |
| name                        | whitney                                                        |
| created                     | 2013-04-11T18:40:31Z                                           |
| tenant_id                   | bb54c65c4aba482f8f6d363e0730df95                               |
| OS-DCF:diskConfig           | MANUAL                                                         |
| metadata                    | {}                                                             |
| accessIPv4                  |                                                                |
| accessIPv6                  |                                                                |
| progress                    | 0                                                              |
| OS-EXT-STS:power_state      | 1                                                              |
| OS-EXT-AZ:availability_zone | nova                                                           |
| config_drive                |                                                                |
+-----------------------------+----------------------------------------------------------------+

Nice.

Now, unless we give this vm a “public ip” we won’t be able to ssh in without hopping into the devstack host first.

But first…one. More. Step.

By default, with devstack, it seems the default security group is pretty restrictive. So we need to add a couple rules.


#
# Default secgroup rules
#

vagrant@precise64:~$ nova secgroup-list-rules default
+-------------+-----------+---------+----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+----------+--------------+
|             | -1        | -1      |          | default      |
|             | -1        | -1      |          | default      |
+-------------+-----------+---------+----------+--------------+

#
# Add ping
# 

vagrant@precise64:~$ nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

#
# Add ssh
# 

vagrant@precise64:~$ nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

#
# New secgroup rules
#

vagrant@precise64:~$ nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
|             | -1        | -1      |           | default      |
|             | -1        | -1      |           | default      |
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

#
# And now we should be able to ping and ssh in to whitney
# 

vagrant@precise64:~$ ping -c 1 -w 1 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_req=1 ttl=63 time=72.2 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 72.261/72.261/72.261/0.000 ms
vagrant@precise64:~$ ssh cirros@10.0.0.3
cirros@10.0.0.3's password: # enter "cubswin:)"
$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux

NOTE: I’m skipping the part about using the authorized_keys file cirros sets up for itself based on the keypair specified. But you can ssh into the cirros instance without a password if everything is setup right, ssh -A, ssh-agent, etc.

Conclusion

And that concludes our brief look at booting a vm inside of vm, using:

macbook retina
vagrant
vmware fusion
vagrant-openstack
devstack
cirros
openstack grizzly

Everything works!

We can even delete the vm we just created:


$ vagrant destroy
[default] Deleting the instance...

Check in with openstack…


#
# vm gone!
# 

vagrant@precise64:~$ nova list
+----+------+--------+----------+
| ID | Name | Status | Networks |
+----+------+--------+----------+
+----+------+--------+----------+

Thanks,
Curtis.

Thoughts on "no reliable cloud"

2013-04-10T00:00:00-07:00

(my pishell server—maybe this could be part of a reliable cloud)

10 April – 2013 – Edmonton!

Thoughts on “no reliable cloud”

Recently Hendrik Volkmer put up a blog post entitled There will be no reliable cloud. Part of it was based on a presentation I watched at the last OpenStack summit (wish I was going to the Portland summit, but alas is not to be).

The Cloud Scaling presentation was one I enjoyed and considered thought provoking. I wrote a few notes on that presentation last year.

No reliable cloud

Here’s a quote from the top of the first post:

Stop wasting your time trying to [find a reliable cloud]. Stop wasting your time (and money) trying to build one. If you find a service provider that claims that they have it: Maybe question their understanding of cloud – and business.

I put that there to remind me of the point of the series of posts, and because it essentially defines the attention grabbing headline. :)

tl;dr

My thoughts on these posts come down to this:

He’s mostly talking web-scale applications
A single zone will not be reliable
But still have to make zones as reasonably reliable as possible (where’s the line?)
We should design reliable applications on top of unreliable zones (but how?)
Contain failure!
HA pairs are probably not the direction to go in to gain reliability
Clustering software often brings in complexity that can destroy reliability gains
Stateless systems are a lot more fun :)
Keep the stateful part of an application or system small

Thinking about reliability in a cloud, especially an OpenStack cloud, is an interesting thought experiment. Fortunately, the OpenStack cloud I help to run, which is the back-end for a single application, is actually mostly stateless—except for machine images, the OpenStack database, and the application database. Not a lot of stateful information, except those darn windows images that are many tens of times the size of a standard Linux cloud image.

Notes from the part one post

For a short post it sure goes over a lot of information and links!

HA pairs
- HA pairs fail catastrophically
- HA pairs don’t scale
- Classic HA example: NFS + DRBD and clustering, such as Pacemaker…then problems?
- HA pairs often end up cheating CAP theorem
- Cluster software causes more system outages than hardware failures of software bugs (this I can attend to having used clustered LVM)
Distributed systems
- Eg. Percona Xtradb Cluster
Availability vs reliability
- HA systems that need to go down for maintenance are a joke

How to build a reliable cloud
- A cloud is a distributed system
- Use the stateless (from Cloud Scaling presentation) approach for stateless parts
- Distributed data stores for the stateful parts (eg. distributed mysql, distributed file systems such as ceph)
- But the distributed stateful part is often what fails (eg. EBS in Amazon)
Notes from blog post comments (notably Randy Bias of Cloud Scaling)
- On OpenStack
  - Move to MySQL Cluster with the NBDEngine running 2-4 mysql instances, and load balancing across them
  - Or perhaps OpenStack will get rid of the RDBMS and replace with K/V store
  - Even with 1000s of nodes, metadata use is still low in OpenStack, could be put in memory and persist data using any appropriate back-end
- No point in having highly redundant hardware for stateless services
- Build reliable applications on unreliable clouds

Ok, now on to part two.

Notes from part two

The second post builds on the basic information provided in the first.

Complexity + Scale => Reduced Reliability + Increased Chance of catastrophic failures

Complexity
- Complex system fail catastrophically
Failure domains
- OpenStack example
  - Single controller, single cloud (or zone)
  - HA setup — two controllers in an HA mode of some kind
  - Single controller, multiple cloud (or multiple zones)
- A single zone is unreliable
- If both HA nodes fail, still unreliable, and HA is more complex
- Two zones is two failure domains, which is more reliable than a single HA-enabled zone
- (But of course you should make each zone as reliable as possible)
Reliability engineering (aka math)
- Reliability engineering matters except when it doesn’t
- “The higher the number of dependent components => the lower the overall availability and the bigger the impact of failure”
- In a cloud with many nodes, adding the ability for live migration will actually decrease reliability, because all nodes are now tied together
- Many reliability calculations come from mechanical engineering, which is much different than software engineering
- Many complex systems fail by cascading, failure starts small and grows big, until it engulfs the entire system
- General approach is to make failure local and contained
- Partial failure is desirable
Business side
- Software reliability is cheaper
- Most web scale applications consist of a large stateless part and a small stateful piece
- It does not make business sense to provide a super-reliable cloud
A single compute node or even zone will never be reliable
Best not to consider virtual machines, such as those in EC2, as servers

NOTE: There will eventually be a part three post, but as of this writing it’s not up yet.

Conclusion

To me, it boils down to building reliable applications on unreliable clouds, which I think is what a lot of people are doing, and is what seems to come out every time AWS fails.

The first issue that pops into my mind though is RDBMS systems, and how to replicate data between zones, which is often a network concern. Actually, replicating any data between zones could be a problem, which is why, I’m guessing, that he’s (perhaps) suggesting to keep stateful pieces small.

First look at PackStack

2013-03-13T00:00:00-07:00

(Yup—an instagram picture of my garish orange single speed summer bike. But it’s a summer bike!)

13 March – 2013 – Edmonton!

First look at PackStack

I am a big fan, and user, of OpenStack. I also really like the Ansible configuration management and orchestration system. In fact, I use Ansible to deploy OpenStack, and all kinds of other things as well.

Recently on the Ansible mailing list the lead developer (and now CTO of Ansibleworks) suggested that it was time to bring together everyone who is working, or wants to work with, both Ansible and OpenStack.

One of the suggestions was to follow what PackStack has done—it’s based on puppet—and port it over to Ansible. (NOTE: I’m not even sure that’s the right git repo, things are moving so fast!)

While I had heard of PackStack before, I had never used it, so I decided it was time I took a look so that I can perhaps help with the OpenStack + Ansible project. Frankly, it was on my list of technology to check out because I always need to have some virtual machines running OpenStack, and it would be nice if there was an easy way to quickly build a multi-host OpenStack install (especially if I would like to contribute code back to the community at some point).

Also—I’m sure Ansible will soon be one of Vagrants supported deployment systems, and when that happens it will be very easy to deploy OpenStack with Vagrant.

So I spent a couple hours creating an Ansible playbook that would simply fire off the PackStack command with a generic answer file. So I haven’t ported anything from PackStack to Ansible—I’m simply using Ansible and Vagrant to create an environment for PackStack to do it’s work.

RedHat/CentOS

One big note—PackStack currently only supports RedHat/CentOS. I’m using CentOS 6.

Host organization

That repository also contains a Vagrant file, and an Ansible hosts file. I have four virtual machines making up a small OpenStack cluster:


$ cat ansible_hosts 
[openstack]
apis ansible_ssh_host=192.168.100.130
scheduler ansible_ssh_host=192.168.100.131
compute01 ansible_ssh_host=192.168.100.132
compute02 ansible_ssh_host=192.168.100.133

And all of those IPs and names are reflected in the PackStack.yml, files/packstack.cfg, and Vagrantfile files. I don’t know if they are the most descriptive names, but this is how I’ve currently organized it.


$ grep "100\.13" Vagrantfile 
apis_config.vm.network :hostonly, "192.168.100.130" # nic3
scheduler_config.vm.network :hostonly, "192.168.100.131" # nic3
compute01_config.vm.network :hostonly, "192.168.100.132" # nic3
compute02_config.vm.network :hostonly, "192.168.100.133" # nic3

The apis host is going to run horizon (not that I need it) and the front-facing APIs, and the scheduler will run the mysql server and nova-scheduler…and some cinder related services as well (which likely need more investigation as I have really only have experience thus far with OpenStack Essex—but at least I can experiment with other versions of OpenStack now, thanks to PackStack).

For example, here is what nova services are running where:


[root@apis packstack(keystone_admin)]$ nova-manage service list
Binary           Host       Zone  Status     State Updated_At
nova-scheduler   scheduler  nova  enabled    :-)   2013-03-13 21:15:11
nova-cert        apis       nova  enabled    :-)   2013-03-13 21:15:15
nova-consoleauth apis       nova  enabled    :-)   2013-03-13 21:15:15
nova-compute     compute01  nova  enabled    :-)   2013-03-13 21:15:13
nova-compute     compute02  nova  enabled    :-)   2013-03-13 21:15:12

Smiley faces are good. :-)

Deploying PackStack

Deploying this specific example only requires a couple of commands (I’m running OSX, and using Virtualbox).

First, tell Vagrant to build the servers.

NOTE: You could have IP collisions if you have other virtual machines running—the IPs used here are hard-coded.


$ vagrant up
[apis] Importing base box 'centos6'...
[apis] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.

Guest Additions Version: 4.1.6
VirtualBox Version: 4.2.6
[apis] Matching MAC address for NAT networking...
[apis] Clearing any previously set forwarded ports...
[apis] Fixed port collision for 22 => 2222. Now on port 2200.
SNIP!

Once those are built you should have four vms running:


$ vagrant status
Current VM states:

apis                     running
scheduler                running
compute01                running
compute02                running

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

Once that’s done, you can simply run the packstack.yml playbook.


$ ansible-playbook -k -u root PackStack.yml 
SSH password: #enter the vagrant password of 'vagrant', just have to do this once

PLAY [openstack] ********************* 

GATHERING FACTS ********************* 
ok: [compute01]
ok: [scheduler]
ok: [compute02]
ok: [apis]

SNIP! #Tons of stuff happens here, mostly done by PackStack

TASK: [run PackStack] ********************* 
skipping: [compute01]
skipping: [compute02]
skipping: [scheduler]
changed: [apis]

PLAY RECAP ********************* 
apis                           : ok=13   changed=11   unreachable=0    failed=0    
compute01                      : ok=7    changed=5    unreachable=0    failed=0    
compute02                      : ok=7    changed=5    unreachable=0    failed=0    
scheduler                      : ok=7    changed=5    unreachable=0    failed=0

When it completes, there should be a multi-server OpenStack cluster running!


$ ssh root@192.168.100.130
root@192.168.100.130's password: #vagrant password again
Last login: Wed Mar 13 16:34:03 2013 from 192.168.100.1            
[root@apis ~]# source keystonerc_admin 
[root@apis ~(keystone_admin)]$ nova list
# nothing will appear here, but at least you should get no error messages

Because the Ansible playbook downloaded and installed the Cirros image, nova image-list should give some output:


[root@apis ~(keystone_admin)]$ nova image-list
+--------------------------------------+--------+--------+--------+
| ID                                   | Name   | Status | Server |
+--------------------------------------+--------+--------+--------+
| 84eebd30-953f-4ffe-b9f9-afb7099f1e75 | cirros | ACTIVE |        |
+--------------------------------------+--------+--------+--------+

At any rate, it’s a good start.

Once Vagrant supports Ansible and VMWare Fusion, it will be crazy easy to create a working OpenStack cluster on my laptop.

Acknowledgments

Nothing I ever do is novel, and this blog post is no different. I based this work off the following: Installing a 4 node Fedora 18 OpenStack Folsom cluster with PackStack.

Issues/Caveats/Questions

I had at least one problem getting this going…PackStack doesn’t seem to want to install puppet on CentOS 6. I had to make sure the Ansible playbook setup the EPEL repository so that the puppet RPMs were available.
AFAIK, Virtualbox doesn’t support nested virtualization, so you won’t be able to boot any Inception styled OpenStack instances, ie. vms within vms. Though, again AFAIK, VMWare Fusion on OSX does supported nested virtualization, which is why I’m excited about Vagrant supporting Fusion. I’d prefer to use KVM, but I need to run OSX for work.
Don’t think Cinder is working.
It can take a good 20 to 30 minutes for this entire build process to complete, mostly because there are a ton of puppet modules to download.
I’m not even sure what version of OpenStack this installs…something to look into. Might have to jump to a more recent version of Fedora to get something like OpenStack Grizzly.


[root@apis ~(keystone_admin)]$ yum info openstack-nova-common
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: www.muug.mb.ca
 * epel: www.muug.mb.ca
 * extras: mirror.its.sfu.ca
 * updates: www.muug.mb.ca
Installed Packages
Name       : openstack-nova-common
Arch       : noarch
Version    : 2012.2.2
Release    : 1.el6
Size       : 115 k
Repo       : installed
From repo  : epel
Summary    : Components common to all OpenStack Nova services
URL        : http://openstack.org/projects/compute/
License    : ASL 2.0
Description: OpenStack Compute (codename Nova) is open source software designed to
           : provision and manage large networks of virtual machines, creating a
           : redundant and scalable cloud computing platform. It gives you the
           : software, control panels, and APIs required to orchestrate a cloud,
           : including running instances, managing networks, and controlling access
           : through users and projects. OpenStack Compute strives to be both
           : hardware and hypervisor agnostic, currently supporting a variety of
           : standard hardware configurations and seven major hypervisors.
           : 
           : This package contains scripts, config and dependencies shared
           : between all the OpenStack nova services.

Software defined networking, Openvswitch, and Ubuntu 12.04

2013-02-21T00:00:00-08:00

(sunny with snow as usual)

21 February – 2013 – Edmonton!

Software defined networking, Openvswitch, and Ubuntu 12.04

Recently I’ve been testing over committing on KVM. Once I started running hundreds of virtual machines (vms) on a single node, I realized that in order to get them to do anything I have to access them over the network to run something like Ansible playbooks designed to test load.

In order to provide networking resources to the vms, I decided to take a look at Openvswitch and what it takes to get it up and running with KVM and Ubuntu 12.04/Precise.

Software defined networking

Like cloud and big-data, software defined networking (SDN) is a loaded term. But, like those terms, I feel I need to at least try to get a grasp of what it means.

“A good working definition of SDN is the separation of the data and control functions of today’s routers and other layer two networking infrastructure with a well-defined programming interface between the two.” — Via Arstechnica

SDN is a big part of OpenStack as well. Starting with the Folsom release, networking was split out into it’s own *as-a-Service capability called Quantum, whereas previously it was a sub-component of Nova. So given I’m a big fan, and user, of OpenStack, it’s important for me to get a good grasp of SDN.

Openflow is also an important technology in SDN that requires some research time.

But, having said all that, basically I’m just going to install and use Openvswitch on a single compute node. :)

Building on other’s work

I followed these blog posts on configuring Openvswitch on Ubuntu 12.04/Precise:

I’m not really doing anything new here—though I hope to at some point… :)

Installation

I put together an ansible playbook to install Openvswitch in Ubuntu 12.04. There is no easy, direct way (that I’m aware of) to install Openvswitch in Precise…unfortunately I just can’t do apt-get install openvswitch and have everything work like magic. I guess building the module is the only unusual thing, and this will disappear in future versions of Ubuntu—perhaps it’s already not necessary in 12.10, not sure, haven’t looked it up.

I’m not going to directly cut and paste my ansible playbook into this post, but suffice it to say that most dependencies can be installed via apt-get, but there is one step required to build and module.


# Install these packages:
#   - openvswitch-datapath-source 
#   - bridge-utils
#   - module-assistant  
#   - openvswitch-brcompat
#   - openvswitch-common
#   - openvswitch-switch
#   - linux-headers-3.2.0-23-generic
#   - linux-headers-generic-pae
# Then build the module:
$ module-assistant auto-install openvswitch-datapath

Next we set BRCOMPAT=yes in /etc/default/openvswitch-switch and restart openvswitch-switch.

Pox

Pox, among other things, is an Openflow controller.

“At its core, it’s a platform for the rapid development and prototyping of network control software using Python. Meaning, at a very basic level, it’s one of a growing number of frameworks…for helping you write an OpenFlow controller.” — Via Pox website

Hey, it’s Python, it’s Openflow…what else do I need. Sign me up. :)


$ cd /usr/local/src
$ git clone http://github.com/noxrepo/pox
$ zdaemon -p 'python /usr/local/src/pox/pox.py \
--no-cli forwarding.l2_learning' -d start

And now Pox should be listening on port 6633:


$ netstat -ant | grep 6633
tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN     
$ sudo lsof -i | grep 6633
python   34150   root    3u  IPv4 39211055      0t0  TCP *:6633 (LISTEN)

More information about pox can be found on the Openflow site.

Configure the bridge

Now we can add a bridge to the openv switch.


$ ovs-vsctl add-br br-int

And then configure br-int (or whatever you’ve called the bridge), and I’m using the eth2 interface in this example.


$ ovs-vsctl add-port br-int eth2; ifconfig eth2 0; ifconfig br-int <IPv4 Address> \
netmask 255.255.255.0

Let’s tell Openvswitch to use the Pox controller that is running.


$ ovs-vsctl set-controller br-int tcp:<IPv4 Address>:6633

Finally we need interface up and down scripts.

ifdown:


$ cat /sbin/ovs-ifdown
#!/bin/sh 
switch='br-int' 
/sbin/ifconfig $1 0.0.0.0 down 
ovs-vsctl del-port ${switch} $1

ifup:


$ cat /sbin/ovs-ifup
#!/bin/sh switch='br-int' 
/sbin/ifconfig $1 0.0.0.0 up
 ovs-vsctl add-port ${switch} $1

Now we can boot some vms!

Booting a vm

I’m booting vms via a script, and part of the kvm command line options is the network tap, which uses the ifup/ifdown scripts:


  -net tap,script=/sbin/ovs-ifup,downscript=/sbin/ovs-ifdown

And, here is a running instance:


$ ps ax | grep "kvm -drive" | head -1
  9831 ?        Sl    21:01 kvm -drive if=virtio,file=/mnt/intel/1.img -m 2048 \
  -boot a -net nic,macaddr=52:54:00:a1:c0:fd \
  -net tap,script=/sbin/ovs-ifup,downscript=/sbin/ovs-ifdown -nographic -vnc :1 \
   -chardev file,id=charserial0,path=/mnt/intel/1.console.log \
   -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 \
   -device isa-serial,chardev=charserial1,id=serial1 \
   -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Right now there’s about 300 vms running on this single compute node.


$ ps ax  |grep "kvm -drive" | wc -l
301

Fun stuff. :)

What now?

Well, I’ve achieved my goal of getting Openvswitch up and running to enable networking between vms on a single compute node.


ubuntu@ubuntu:~$ ifconfig eth0 | grep 192
          inet addr:192.168.100.111  Bcast:192.168.100.255  Mask:255.255.255.0
ubuntu@ubuntu:~$ ping -c 1 192.168.100.23
PING 192.168.100.23 (192.168.100.23) 56(84) bytes of data.
64 bytes from 192.168.100.23: icmp_req=1 ttl=64 time=0.452 ms

--- 192.168.100.23 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.452/0.452/0.452/0.000 ms

I think my next step will be to work in multiple virtual machines to see if I can do some of the interesting and useful things that Openflow is capable of, and to find out how I can work with the Pox system to learn more about SDN.

Another important thing to do is to get a test environment of OpenStack Folsom (or Grizzly) up and running to see how Quantum utilizes Openflow and SDN.

I’m hoping to spend the next few days learning about Pox. Wish me luck. :)

Thanks,
Curtis.

Over committing with KVM

2013-02-20T00:00:00-08:00

(Still winter in Edmonton, will have to blog a lot more in the summer)

20 February – 2013 – Edmonton!

Over committing with KVM

It’s quite possible to over commit resources with the KVM hypervisor.

I should say first that most of the work I’ve been doing around over committing in KVM is based on a project I am working in where the virtual machines are stateless. This makes over committing easier because if we run out of resources on a compute node and this causes a vm to crash, it’s not the end of the world—the end-user can just restart their session.

To me over committing means having virtual machines running on a node where the total CPU, main memory (ie. ram), and disk that is available to the virtual machines is much larger than the actual physical resources available on the node. Hopefully five or ten times larger.

Hardware

I am running these tests on a single Dell C6220 node. It has 32 cores (including hyperthreading), 128GB of main memory, and SSD drives.


test_host:~$ cat /proc/cpuinfo | grep processor | wc -l
32

The SSD drive types we are testing with are:

2x Standard Dell 300GB SSD
2x Samsung 830 512GB
2x Intel 520 480GB

Each of them is in a stripe/RAID0 configuration, so we have three striped devices, md2 (Samsung) and the poorly named md126 (Intel) and md127 (Dell).


test_host:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
[raid10] 
md0 : active raid1 sdb1[1] sda1[0]
      307136 blocks [2/2] [UU]
      
md2 : active raid0 sdb5[1] sda5[0]
      882155520 blocks super 1.2 512k chunks
      
md1 : active raid1 sdb2[1] sda2[0]
      41910144 blocks super 1.2 [2/2] [UU]
      
md126 : active raid0 sdd[1] sdc[0]
      937702400 blocks super 1.2 512k chunks
      
md127 : active raid0 sdf[1] sde[0]
      586072064 blocks super 1.2 512k chunks
      
unused devices: <none>

The striped Samsungs are really fast—84K IOPS from the stripe!

Memory over committing: KSM (not KVM)

KSM, or Kernel Samepage Merging, is a memory de-duplication feature that is present in most Linux systems that support KVM. I wrote a bit about it in a previous blog post. Basically if you up the scanning rate it will de-duplicate more memory. This means if I’m running 300 Ubuntu Precise images there is a lot of memory that can be de-duplicated.

CPU over committing

I don’t have a lot of information on how over committing CPU works in KVM, but there are some best practices documented by a couple organizations, which I have listed below. This is an area I need to do more research in.

IBM says:
- Target system use at max 80%—ie. each vm shouldn’t be using 100% of their CPU, and in fact should max out at 80%
- Allocate minimum VCPUs per vm—ie. if the vm doesn’t need four CPUs, rather say, only one, then just give it one
RedHat says:
- Virtualized CPUs are over committed best when each guest only has a single VCPU. The Linux scheduler is very efficient with this type of load. KVM should safely support guests with loads under 100% at a ratio of five VCPUs. Over committing single VCPU guests is not an issue.

Notice that RedHat is saying a 5:1 ratio for CPU over committing.

Disk space over committing

This is straight forward: image snapshots. qcow2 images can be created as snapshots of a backing file.


test_host:/mnt/intel$ file 263.img 
263.img: QEMU QCOW Image (v2), has backing file 
(path /mnt/intel/precise-server-cloudimg-amd64-disk1.qcow2), 2147483648 bytes

This is also how OpenStack handles images. It puts one base image on the compute node and each instance based on that same image is backed by a qcow2 snapshot. I suppose we could do the same with LVM, if desired, using snapshots.

Can’t over commit IOPS

The part that is difficult in terms of running hundreds of vms on a single node is the amount of IOPS each vm takes up. In my experiments the Ubuntu cloud image doesn’t really do anything at all in terms of IOPS—maybe one or two IOPS per image when they are idle. So if I’m running 300 images, they don’t even use 300 IOPS when idle. Even during boot they hardly do anything. (Of course when they are working they could be using a lot of IOPS.)

But, if I boot 100 Windows 7 images five minutes apart, I need 5000 IOPS.

Yup: 5000 IOPS.

Six SATA drives in RAID10 are going to barely provide 400 IOPS let alone 5000. So, SSDs, or at least some kind of faster storage, are required. Pretty much any SSD is capable of 5000 IOPS.

IMHO, it’s a lot more efficient to run the Ubuntu cloud image than a standard Windows image.

Load

Right now I have 300 Ubuntu images running, each being given 2048MB of memory.


test_host:~$ ps ax | grep "kvm -drive" | wc -l
300

And load and memory usage are just fine. A bit of swapping, but that’s Ok.


top - 10:58:13 up 8 days, 23:55,  2 users,  load average: 1.35, 1.29, 1.92
Tasks: 845 total,   2 running, 843 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  3.1%sy,  0.0%ni, 95.9%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  131997760k total, 122223236k used,  9774524k free,   142448k buffers
Swap: 33554424k total,   242172k used, 33312252k free, 42323472k cached
SNIP!

They have been running for a few days, and are not using much in terms of IOPS as they are completely idle. Below is the output from iostat. All the vms are running off qcow2 images on the Intel 520 based stripe, md126.


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.67    0.00    3.08    0.05    0.00   96.20

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              16.40         0.00         0.17          0          1
sdb              16.40         0.00         0.17          0          1
sdc              44.70         0.00         0.16          0          1
sde               0.00         0.00         0.00          0          0
sdd              44.20         0.00         0.15          0          1
sdf               0.00         0.00         0.00          0          0
md127             0.00         0.00         0.00          0          0
md126            60.10         0.00         0.31          0          3
md1              35.20         0.00         0.17          0          1
md2               0.00         0.00         0.00          0          0
md0               0.00         0.00         0.00          0          0
dm-0              0.00         0.00         0.00          0          0
nb0               0.00         0.00         0.00          0          0

It would be great to have IOPS usage on a per-process basis, but I couldn’t find a tool that would do that. The best I could do is iostat with IOPS per storage device.

Running tests

Each vm gets its IPs from dnsmasq running on an Openvswitch bridge (topic of a future blog post). So we have 300 tap devices.


test_host:~/performance_testing$ sudo ovs-vsctl show
SNIP!
        Port "tap166"
            Interface "tap166"
        Port "tap0"
            Interface "tap0"
    ovs_version: "1.4.0+build0"

test_host:~/performance_testing$ ifconfig | grep tap | wc -l
300

I like using the Ansible configuration management system. I’ve written a custom inventory script that pulls the IPs out of the dnsmasq lease file.


test_host:~/performance_testing$ ansible all -c ssh -i ./inventory.py \
-m ping -u ubuntu
SNIP 298 hosts!
192.168.100.98 | success >> {
    "changed": false, 
    "ping": "pong"
}

192.168.100.99 | success >> {
    "changed": false, 
    "ping": "pong"
}

Then, using that inventory and Ansible I can run whatever load tests I want across however many virtual instances I want. For example I could run fio tests across 10% of the vms and watch the IOPS usage on the compute node.

If you have any questions, criticisms or concerns, please let me know by posting in the comments. :)

Thanks,
Curtis.

Canadian OpenStack Users Group - CanStack!

2013-02-20T00:00:00-08:00

20 February – 2013 – Edmonton!

Canadian OpenStack Users Group – CanStack!

Recently I have started helping out the Canadian OpenStack Users Group. With the support of my employer, I’ve put up a website called CanStack and a blog for that site as well. And, of course, the obligatory twitter account. There is even a github repository for the blog posts, so contributing a post is only a git commit away! :)

The fun thing about the website and blog is that the website is hosted in a beta Canadian OpenStack-based cloud, and the blog is hosted in Amazon S3. I am a big proponent of object storage systems such as S3 and OpenStack Swift, so it was a good experience to get a static blog up on S3.

I use jekyll to generate the blog, just like I do this site, though serverascode.com is hosted by github. Thanks github!

My hope with the CanStack website and blog is that they will help to find more members to bring together in our monthly meetings, and that hopefully we can provide some useful information about OpenStack as well.

I just put up a blog post on how we are using OpenStack in a virtual classroom project, complete with what server and network hardware we are using, and even what data center!

So, if you are interested in what Canadians are doing with OpenStack, canstack.ca might be a good place to start.

As always, comments, suggestions, and criticisms are welcome. :)

Thanks,
Curtis.

tcpflow

2013-02-08T00:00:00-08:00

8 February – 2013 – Edmonton!

(winter on the UofA campus, nice sunny day though)

tcpflow

This is just a quick little post on tcpflow.

I’ve been using graphite as a a graphing solution for system statistics. I’ve also been using carbon-relay-ng to relay packets from different OpenStack projects to a central graphite server.

However, I’ve been having an issue (and am still having an issue actually) with concatenated packets arriving at the graphite server, despite the fact that the clients aren’t sending them.

In order to troubleshoot this I eventually started running tcpflow because it gave me cleaner output than tcpdump did…at least by default anyways. I’m sure that tcpdump can give me pretty much any output I need, but I liked the symplicity of tcpflow, for example:


[root@stats ~]# tcpflow -c port 2003
tcpflow[11791]: listening on eth0
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.user 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.nice 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.sys 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.wait 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.idle 99 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.irq 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.soft 0 1360364000
010.000.000.001.39216-010.000.000.006.02003: debbuild.cputotals.steal 0 1360364000
SNIP!

vs.


[root@stats ~]# tcpdump -nnvvXSs 1514 -i eth0 port 2003
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes
22:59:32.650673 IP (tos 0x0, ttl 64, id 41032, offset 0, flags [DF], 
proto TCP (6), length 103)
    10.0.0.1.39216 > 10.0.0.6.2003: Flags [P.], cksum 0xab18 (correct), 
    seq 1882211360:1882211411, ack 2847074948, win 115, options 
    [nop,nop,TS val 804219404 ecr 2428141312], length 51
	0x0000:  4500 0067 a048 4000 4006 8642 0a00 0001  E..g.H@.@..B....
	0x0010:  0a00 0006 9930 07d3 7030 4420 a9b2 ea84  .....0..p0D.....
	0x0020:  8018 0073 ab18 0000 0101 080a 2fef 6a0c  ...s......../.j.
	0x0030:  90ba 7f00 636f 6c6c 6563 7464 3031 2d63  ....collectd01-c
	0x0040:  6c69 656e 742d 7465 7374 2e74 6573 742e  lient-test.test.
	0x0050:  6669 7273 7420 2033 3530 3420 3133 3630  first..3504.1360
	0x0060:  3336 3433 3733 0a                        364373.
22:59:32.650750 IP (tos 0x0, ttl 64, id 53955, offset 0, flags [DF], 
proto TCP (6), length 52)
    10.0.0.6.2003 > 10.0.0.1.39216: Flags [.], cksum 0xc56c (correct), 
    seq 2847074948, ack 1882211411, win 501, options [nop,nop,TS val 
    2428143328 ecr 804219404], length 0
	0x0000:  4500 0034 d2c3 4000 4006 53fa 0a00 0006  E..4..@.@.S.....
	0x0010:  0a00 0001 07d3 9930 a9b2 ea84 7030 4453  .......0....p0DS
	0x0020:  8010 01f5 c56c 0000 0101 080a 90ba 86e0  .....l..........
	0x0030:  2fef 6a0c                                /.j.
SNIP!

One thing to note is the -c switch on tcpflow, it means print to the console instead of files in the local directory.

I would love to hear about other uses of tcpflow or ways to alter the output of tcpdump, so please let me know of anything interesting by commenting. :)

Thanks,
Curtis.

Converting VMWare Windows images to OpenStack with virt-v2v

2012-11-26T00:00:00-08:00

26 November – 2012 – Edmonton!

(some robot)

Converting VMWare Windows images to OpenStack with virt-v2v

As I’ve mentioned in previous posts, I’m currently working on a project that uses Apache VCL to provide university students with the ability to remotely login to a virtual machine and use specialized software to complete their classwork—a virtual computer lab if you will.

Because we are moving off our current backend to OpenStack, we need to convert Windows images from VMWare to something that will work in OpenStack/KVM.

Note about Windows…

Let me note that I am not a “windows guy”—I haven’t really used windows in the last 10 years. Because I have usually been employed as a Unix/Linux admin, I’ve always either ran OpenBSD, Linux, or OSX on my desktop, and literally never had to work with Windows. Until now of course. ;)

Moving the backend to OpenStack

Currently we are in the process of moving the backend of our VCL system from a VMWare ESXi based cluster to OpenStack (Essex to be precise.) There are a lot of reasons for this move, and I won’t get into them here.

Unfortunately you can’t just take an ESXi image and drop it into OpenStack and have it work. Certainly we can actually import the image into glance, as vmdk images are supported, but the OS on the image will not have the virtio drivers for disk and network installed, drivers that OpenStack uses by default.

Neither is it as easy as just installing the drivers into the OS image while it’s running in ESXi. While I believe it’s possible to do this in Windows Server 2008, I don’t believe you can install drivers into Windows 7 without having the actual “hardware” accessible to the image (please correct me if I’m wrong—I’d love to hear that I am), not without some registry and other hacks outside of my purview.

This means we needed to find a way to convert the images from ones that work in VMWare ESXi, to ones that will work in OpenStack + KVM, preferably a way that doesn’t require me to learn anything about the Windows registry. ;)

virt-v2v to the rescue

Fortunately RedHat provides a system called virt-v2v. This allows the cross-conversion of images for several different types of hypervisors.

The main things I found out about RedHat and virt-v2v are:

Some of the packages are not available in CentOS, or at least I couldn’t find them, so you need a RedHat license. If you are converting images from VMWare ESXi to OpenStack, then you probably have enough money in the project to buy a RedHat license. ;)
It doesn’t work in a virtual machine—it expects hardware virtualization, so you need a hardware server. You don’t need much, just enough to support hardware virtualization with KVM.
Currently, and this might change, you have to download the image from the ESXi server each time you try a conversion, so if it takes a long time to download, then the conversion will also take a long time to finish.

I’m sure most of the above could be changed with a bit of Perl hacking, but as far as I can tell, without changing any of the RedHat code you do need to meet those requirements.

Note—RedHat has pretty good documentation on the subject, a few quick google searches will tell you as much as I know. :)

Installing virt-v2v

First, as I’ve mentioned, you need a hardware server with RedHat Enterprise 6.x on it.


[root@localhost ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.3 (Santiago)

It needs to be registered with the RedHat Network and have the “RHEL Server Supplementary” and “RHEL V2VWIN” channels enabled, as per the below image.


[root@localhost ~]# yum repolist
Loaded plugins: product-id, rhnplugin, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
repo id       repo name     status
rhel-x86_64-server-6 Red Hat Enterprise Linux Server (v. 6 for 64-bit x86_64) 8,712
rhel-x86_64-server-supplementary-6 RHEL Server Supplementary (v. 6 64-bit x86_64) 311
rhel-x86_64-server-v2vwin-6 RHEL V2VWIN (v. 6 for 64-bit x86_64) 2
repolist: 9,025

Once those are configured, install the following packages.


[root@localhost ~]# yum install virt-v2v virtio-win libguestfs-winsupport libvirt

And then reboot.

Once the server has rebooted, start libvirtd if it isn’t already.


[root@localhost ~]# service libvirtd start

Next we need to add a storage pool to libvirt. In this example, just because of the way RHEL defaulted my temporary install, we have a huge /home/images directory in which to put the converted images.


[root@localhost ~]# df -h /home
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_home
                      5.4T  235G  4.9T   5% /home
[root@localhost ~]# cat pool.xml 
  <pool type="dir">
        <name>virtimages</name>
        <target>
          <path>/home/images</path>
        </target>
      </pool>

Using that pool.xml file, we’ll configure the pool.


[root@localhost ~]# virsh pool-create --file pool.xml
[root@localhost ~]# virsh pool-list
Name                 State      Autostart 
-----------------------------------------
virtimages           active     no

Finally, we configure a .netrc file, obviously entering the proper ESXi host, login, and password.


[root@localhost ~]# cat .netrc
machine <esxi host> login <login> password <password>

Now we can run virt-v2v!


[root@localhost images]# virt-v2v -ic esx://<esxi host>/?no_verify=1 -o libvirt
 -os virtimages <exsi vm name>
<esxi vm name>: 100% [========================]D 0h51m16s
virt-v2v: WARNING: No mapping found for bridge interface public in config file. 
The converted guest may not start until its network interface is updated.
virt-v2v: <esxi vm name> configured with virtio drivers.

I don’t worry about the “No mapping” message.

Bring that image into OpenStack

Now that we have an image in the libvirt pool we configured, it’s time to:

Convert the raw/vmdk image to qcow2 using “qemu-img”.
Import the converted qcow2 image into glance.
Boot the image once and manually login to finish off virt-v2v’s “firstboot” scripts. This means you need an admin login to the image.
“Set” the hardware, make any changes to the instance such as updating and the like.
Shut down the instance.
Create a new OpenStack image from that instance using nova image-create.
Then delete the original image, and instance booted from it, if you want. It’s not much use except for archival purposes.
Boot a new instance from the new image, the one created with nova image-create, just to make sure it all works out Ok.

At this point you should, hopefully, have a working Windows image, one that was converted from VMWare ESXi!

Thanks,
Curtis.

KSM and KVM

2012-11-11T00:00:00-08:00

KSM and KVM

11 November – 2012 – Edmonton!

(Winter in Edmonton. It’s not that cold…yet.)

UPDATE: I did some more research and have a better idea of what pages_saved and pages_saving means. So we are saving quite a bit of memory!

KSM and KVM

I recently found out about the ksm technology that is in Ubuntu 12.04 + kvm by default. ksm is a memory de-duplication process. As far as I understand it, ksm can allow virtual machines (actually any application, not just virtualization) to share memory pages—it finds all duplicated memory pages and merges them, thereby saving memory in some situations.

One of the projects I am working on is a classroom as a service, or virtual classrooms. Students can login to a web gui and request a reservation to a virtual machine image which they can then access with a RDP client.

In this project all of the images are based on—unfortunately—Windows 7. One would think that if we are running many similar Windows 7 images ksm could do a lot of de-duplication.

I have been doing a few experiments in my spare time to see if ksm can help to over-commit memory. If I can I’d rather be able to run 400 virtual machines than 200. If we can over-commit on memory 1:2 or 1:4 there could be substantial cost savings for the project.

The Test

I have a basic Windows 7 image in qcow2 format.


root@ksm_test:/mnt/ksm-test$ file win7-base.qcow2
win7-base.qcow2: QEMU QCOW Image (v2), 21474836480 bytes

I am going to run 30 Windows 7 images with four gigs of ram and two virtual cpus each, based off a qcow2 snapshot from the original backing image.

The server I am running this test on is a Dell c6220 with 32 HT cores and 128 gigs of main memory.

ASIDE: /mnt/ksm-test is an xfs file system. I found that this test on a ext4 based filesystem used considerably more IOPs than xfs because the jdb2 process was doing a lot of journaling. There are likely some settings I should be using with ext4 to get better performance, but instead I just hopped over to xfs and haven’t gone back to ext4 yet.

This is the little script I use to boot the vms:


root@ksm_test:~$ cat test_ksm.sh 
#!/bin/bash

# How much memory to boot with
MEM=4048
BACKING_DIR=/mnt/ksm-test
BACKING_FILE=win7-base.qcow2
SLEEP=60

pushd $BACKING_DIR

for i in {1..30}; do
	echo "====> Starting a new instance..."
	# Remove the old backing file
	rm -f win7-$i.qcow2

	# Create a new backing file that is a qcow2 snapshot of the original file
	qemu-img create -f qcow2 -b $BACKING_FILE win7-$i.qcow2

	# Actually start the intstance
	/usr/bin/kvm \
	-M pc-1.0 \
	-smp 2,sockets=2,cores=1,threads=1 \
	-enable-kvm \
	-m $MEM \
	-drive file=win7-$i.qcow2,if=virtio \
	-boot d \
	-net nic,model=virtio \
	-net user \
	-nographic \
	-vnc :$i \
	-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
	-daemonize

	# Let's just sleep for a few seconds...
	echo "====> Sleeping for $SLEEP..."
	sleep $SLEEP 
done

popd

exit 0

After that script runs we have 30 kvm Win7 instances running:


root@ksm_test:/mnt/ksm-test$ ps ax  |grep "bin\/kvm" | wc -l
30

For the first while things are a little crazy on the host because 30 Windows 7 vms just booted in 30 minutes. After a few hours, or rather overnight, the vms settle down quite a bit to just doing a few IOPs each.

As far as what these vms are doing—I login to a couple every once and a while just to make sure they are up, but otherwise they are doing nothing but whatever they do by default.

The Defaults

ksm is enabled in Ubuntu by default when using kvm. However, the defaults are fairly conservative:


root@ksm_host:~$ cat /sys/kernel/mm/ksm/pages_to_scan 
100
root@ksm_host:~$ cat /sys/kernel/mm/ksm/sleep_millisecs 
200

ksm will scan 100 pages, sleep for 200 milliseconds and then scan 100 more, and so on. But with millions of pages it will take a long, long time to scan all of them.

I set the pages_to_scan to 20000 and sleep_millisecs to 20—I’m guessing these are pretty aggressive settings.


root@ksm_host:~$  echo "20000" > /sys/kernel/mm/ksm/pages_to_scan
root@ksm_host:~$  echo "20" > /sys/kernel/mm/ksm/sleep_millisecs

The Results

I suppose saying “results” sounds scientific. :)

The reality is that I’m really just cutting and pasting the ksm information that has been recorded after several days of running 30 Windows 7 virtual machines that should all be very close in terms of memory use.

From the ksm.txt file:


A high ratio of pages_sharing to pages_shared indicates good sharing,
but a high ratio of pages_unshared to pages_sharing indicates wasted 
effort. pages_volatile embraces several different kinds of activity, 
but a high proportion there would also indicate poor use of madvise 
MADV_MERGEABLE.

And the results of ksm after a few days of running 30 vms…


root@ksm_test:~$ for i in `ls -1 /sys/kernel/mm/ksm`; \
do echo "===> $i"; cat /sys/kernel/mm/ksm/$i;  done
===> full_scans
5417
===> pages_shared
443355
===> pages_sharing
26704343
===> pages_to_scan
20000
===> pages_unshared
3164064
===> pages_volatile
183552
===> run
1
===> sleep_millisecs
20

So, if I understand these numbers correctly, pages_shared is the amound of memory ksk is actually using. Thus in this example, ksm is using 1.7GB of memory:


root@ksm_test:~$ getconf PAGESIZE
4096

So if we have 443355 pages shared, ksm is using this many bytes:


root@ksm_test:~$ echo "443355 * 4096" | bc
1815982080

which is about 1.7GB.

However, saved memory, ie. pages_sharing, is quite high! So this is good. :)


root@ksm_test:~$ echo $((26704343*`getconf PAGE_SIZE`/1024/1024/1024)) GB
101 GB

I’m not sure how this equates to all the memory being used on the machine, but as far as ksm is concerned it’s saving us about 100 gigs. Nice.


root@ksm_test:~$ free
             total       used       free     shared    buffers     cached
Mem:     131997772  131556896     440876          0     108448  108129628
-/+ buffers/cache:   23318820  108678952
Swap:     41943032     302836   41640196

Update: Now that I have a better understanding of what ksm is doing and what it’s numbers mean, using a modified version of this script we can see some interesting results, though in the below example I am running 60 2 gig 1 vcpu instances instead of 30 4 gig 2 vcpu instances like the rest of this post:


root@ksm_test:~$ ./ksm_stat.sh 
Shared memory is 2071 MB
Saved memory is 95514 MB
Unshared memory is 21336 MB
Volatile memory is 2549 MB
Shared pages usage ratio is 46.11
Unshared pages usage ratio is .22

So thanks for reading, and if you have any suggestions as to what I might be doing incorrectly, be it settings or my math or my general assumptions about ksm :), please let me know in the comments.

Thanks,
Curtis.

OpenStack 2012 Summit Day #4

2012-10-18T00:00:00-07:00

OpenStack 2012 Summit Day #4

18 October – 2012 – San Diego

Security Track!

Having been a security administrator at a medium sized University; having taking fairly extensive education in Information Security; having attended security conferences like CanSec West and H.O.P.E I am happy that all of Thursday has a dedicated security track. Unfortunately I can only attend a couple of sessions, but at least I can do that much. :)

Creating an OpenStack Security Group

Bryan D. Payne from Neubula and Robert Clark from HP talk about the need for a OpenStack Security Group. The OSSG is “hiring”, ie. need volunteers especially security engineers, technical writers, and security experts that operate OpenStack clouds.

Computer Security

Design for security from the start
Understand your threats
Understand your goals
Pervasive security culture (not just “that paranoid guy has it under control”)

Security Challenges for OpenStack

Security as an afterthought
Security as silos
Security by non-experts

There was some mention of the Cloud Security Alliance which Bryan is a co-lead on a workgroup for and that the CSA is a more oriented to high-level theory whereas the OSSG will need to be directed at applied security.

Delivering Secure OpenStack IaaS for SaaS Products

“Everyone has a great plan until they get punched in the mouth”—Mike Tyson

Andrew Hay, is the Chief Evangelist at CloudPassage, Inc. where he serves as the public face of the company and its cloud server security product portfolio.

Three places to “put security” in OpenStack

Quantum
- Coolest thing about Quantum is the ability to inject 3rd party products
Keystone
Nova
- Security groups/firewalling with iptables
- VLANS
- Initial configuration of nova

Issues in Cloud Security

Host security
Security of images
- Network-based security is only so good in multitenant cloud
The ultimate target is the endpoint—so secure it
Cloud servers are more exposed
De-provisioning of servers releases public IPs that might have the wrong firewall rules

He then went over some basic host security concepts, quite a few of which I disagree with, or at least he wasn’t able to really go into depth with what he actually meant by them.

Generally speaking I think host security has been reduced mostly because in OpenStack most people use images that have already been provided to them and don’t run any specific configuration management (ie. chef/puppet/ansible) “hardening” processes.

Nor do people usually build their own OpenStack images, which I think is an important skill to have. Though if more OS vendors provided base OpenStack images things would be a little easier. As far as I know, at this time, only Ubuntu publishes OpenStack compatible cloud images.

SDSC

Unfortunately, or fortunately, depending on your perspective, that was all the OpenStack security talks I was able to attend on Thursday because I had to go to a meeting at the San Digeo Super Computing Center, or the SDSC, and wasn’t able to come back to the summit.

I was excited to go to find out more about what they are doing, especially because they have one of the largest OpenStack Swift installations in the world. I think about five petabytes worth.

They were also very excieted about their new supercomputer Gordon, which has a ton of SSD storage (ie. flash—get it…gordon + flash).

Also they had some old tape systems they were retiring. Tape is not dead yet though!

(old tape silo being retired; sad to see it go)

OpenStack 2012 Summit Day #3

2012-10-17T00:00:00-07:00

OpenStack 2012 Summit Day #3

17 October – 2012 – San Diego

Keynotes

HP says something enterprise business-like. Not too sure. 100s of billions of “spend” available. Capex → Opex. Many, many slides with business pictographs.

Troy Toman from Rackspace talks about how proud they are of OpenStack. Brags about how Rackspace’s contribution percentage has actually reduced from 54% in Essex and 30% in Folsom and how that is a good thing, which it is. :)

Notes from Rackspace’s production usage of OpenStack:

They have Quantum and Melange in production
Glance is backed by Swift
They have deployed the Cells code; at least three cells in each regon
Three regions worldwide
All of the “control” pieces, eg. API servers, are running on a private OpenStack cloud; an internal infrastructure called inova
Continuous delivery model for building and deploying to cloud—they pull from trunk at least once a day, and do this in under and hour, and have been deploying this once a week or so.
Fail fast fix fast
Coming soon: block storage and network products
Releasing PHP and Java cloud SDKs

Cisco Webex talks about what they do. Lots of open source technologies…puppet, salt, cobbler, etc to run a private cloud so they have “one throat to choke”.

Service Resiliency Doesn’t Always mean “HA” or “Cluster”

The guys from CloudScaling talk about redundancy patterns. This was actually pretty fascinating, and looks like they have taken care of a lot of scale-out redundancy, specifically not “HA-mmer” pairs, all the way up to the DB level, at which part they fall back to MMR MySQL.

“HA” pairs are not the only type of redundancy
- Airplanes require seven catastrophic failures to fall out of the sky, whereas with HA pairs you just need one layer to fail
- Create small failure domains that don’t propagate = scale up
Alternative redundancy patterns
Redundancy patterns in Open Source Cloud Software

CloudScaling have come up with a methodology for “HA” they call Service Distribution.

Service Distribution

Resilient
Stateless
Scale-out
Implementation
- OSPF (quagga), Anycast (zebra), Load-balancing proxy (pound)
Perfect for site resiliency
- Traditional HA pairs don’t support this
Use ZeroMQ (peer-to-peer) vs RabittMQ and CloudScaling contributed the code back to OpenStack for ZeroMQ
Making data centers look like ISP backbones

CirrOS, cloud-init: the future of cloud guests

I think the dev sessions are really about starting conversations that will be finished elsewhere, and if you aren’t fairly deep into the topic there’s not a lot of ways to get your head into the session. I’ve felt the pain of the lack of Cloud-init in Redhat. I’m surprised that there is just one person creating Cirros.

Topics Covered

Cirros
Cloud-init
Config drive 2
Execute code from metadata in various ways

Adding OpenVZ support to Nova

Rackspace has code that will eventually make it’s way into trunk for using OpenVZ in OpenStack. The session leader, Devananda van der Veen, has created a way to build an OpenVZ kernel on Ubuntu. Rackspace uses OpenVZ to run their cloud database system because they get better performance than using something like KVM for virtualization.

Most of the discussion was about how to get the OpenVZ driver into the OpenStack continuous integration system.

Cloud Hosted Desktops and OpenStack

Ken Ringdahl, Vice President Engineering, Desktone discusses Desktop as a Service. I’m interested in this topic not because I’m interested in the topic, but because the project I’m currently working on is essentially desktop in the cloud via Apache VCL and I’m constantly thinking about how to get off VCL and away from it’s 40k lines of perl, and just use OpenStack with a thin layer on top of it. The reality is that OpenStack has 95% of the code required to implement a service similar to VCL.

So Ken goes through his slides. Mentions VDI and that VDI is hard! Capex, buy more hardware, can’t fit it in their data center, SAN administrators, etc. VDI is tough. But DaaS is not VDI.

The point of the session is that DaaS is a three billion dollar market and is a new business area that OpenStack can access. Further he talks about the fact that running a desktop resource manager/broker on top of OpenStack can create operational and other cost savings. It also allows and organization like Desktone to focus on user experience versus running infrastructure.

Desktops are Different

Storage is key
- Random IO
- Write intensive IO
- Scale: 500 servers is a lot, 500 desktops is not
Optimize OpenStack for DaaS
- HA for desktops is different from servers
The future of VDI is non-persistence…nothing stored in VMs
# of VMs under management requires a different approach
- Eg. VM and guest state changes
Monitoring is critically important
DaaS encompasses all parts of OpenStack

OpenStack 2012 Summit Day #2

2012-10-16T00:00:00-07:00

OpenStack 2012 Summit Day #2

16 October – 2012 – San Diego

(Another picture of the hotel…can you tell I just got Instagram?)

Keynotes

The keynotes mostly focused on the increased sized of the OpenStack community and the creation of the foundation that now runs OpenStack. Except Shuttleworth’s in which he used Juju to live upgrade from Essex to Folsom which turned some heads. It’s the first time I’ve heard him speak, and he seemed exactly like the kind of person that can sit in-between business and open source. Chris Kemp was good, and took a couple of shots at VMWare which I always think is fun. ;)

High Availability

Florian Haas of Hastexo presented an update on high-availability with OpenStack. The company I currently work for has, in the past, contracted Florian/Hastexo to consult on OpenStack deployments and I like the things that he says. He’s the go to person for HA + OpenStack.

He talked at length about Pacemaker which is a cluster resource manager. Interestingly he notes that it runs air traffic control systems. Also he mentions that it is extremely friendly to 3rd party functionality via plugins/resource agents,

He also went through the OpenStack High Availability Guide which he was largely responsible for creating, and mentioned that it is in source control and thus people can contribute patches.

The slides for this talk are online.

Frameworks and APIs for Advanced Service Insertion

This was a developer session on the future methodologies of inserting things like firewalls, load balancers and such into Quantum. It was interesting to hear some of the thoughts on the topologies of Quantum and how OpenStack is trying not just to replicate existing networking technologies, but create something new. It was nice to hear the word router with an Italian accent. OpenStack is a global project.

Lunch

Not much to say, on this. There was tortilla soup though. :)

The Future of Infrastructure Automation

This session was not what I thought it was going to be. I didn’t realize it was part of the strategy track until I was sitting down listening, and so before knowing that I figured it would be about what comes after puppet/chef/etc…which it was not.

I don’t have a lot to say about this session, other than the speaker liked webhooks and push notifications, that it would be nice to for a guest in OpenStack to be able to write metadata instead of just reading, and that some organizations are working on this functionality. I definitely would like to be able to write metadata from guests so that was good to hear.

Surviving your first check-in: An engineers guide to contributing to OpenStack

This was a great session on lessons learned by an engineer (specifically not a developer), Collin McNamera, who wanted to contribute code to OpenStack. To get his code into OpenStack he had a ratio of 100:1 in terms of time spent figuring out how to contribute and waiting for answers vs actually coding. That was for his first contribution so the ratio is better now, but it was a tough slog at first.

Lessons learned:

Join, or start, a local meetup group
Make sure that according to your employer you can contribute code to OpenSource projects
Execute your OpenStack CLA (contributor license agreement)
Setup your dev environment, probably using virtual machines
- Devstack
- With devstack you can point it to a different OpenStack git repo if you want
- Use vm snapshotting to help you keep using your dev environment
Configure git with your name, email, etc and this will save time in the future. Do it right first!
Install git-review
Clone a project, or use the directories that came with Devstack
Add your ssh key to OpenStack Code Review
Create a topic branch
Change code
Test code: run_tests.sh
Commit changes – make sure to put a bug id or blueprint # in the first line
Then give back to the community by teaching others! :)

While it’s disappointing to hear how hard it is to contribute code, Collin was a great speaker is obviously passionate about doing good open source work and contributing back to the community.

Swift Project Update

As I mentioned in my previous post I am a big fan of object storage, so Swift is an important project to me. :)

New features in the last six months
- Folsom version of Swift is 1.7.4
- Unique-as-possible – Allows more flexible growth, and more harddrives as hand-off nodes
- Deep statsd integration – Enables notifications that things like graphite can graph
- SSD optimization – It turns out that in large clusters the account and container servers can become limited by IOPs.
- Versioned writes – This feature is “almost” there
- Moved swift client into it’s own project

Code golf for the last six months
- 37 people have contributed
- 20 have provided their first patch
- Three new core developers
- 170 total commits
- 17 is the most files touched by a single commit (statsd integration)
- 3466 is the most lines removed in a single patch (moving swift-client out to a new project)

What’s next?
- Global clusters – ie. Geographic replication, will hopefully be in Grizzly
- CORS Support – Better integrate with the browser security model
- Optimize concurrent reads

(Nebula logo projected on the pool at the club)

Parties

This night I went to both the Rackspace and Nebula parties, though I came back to the hotel pretty early. At the Rackspace party I learned that Miramar, the location of the flight school in Top Gun, is only a few minutes away and actually watched some military jets maneuvering out over the ocean. Surprised no one sang You’ve Lost That Loving Feeling. ;)

ipmitool and BIOS Access

2012-10-16T00:00:00-07:00

ipmitool and BIOS Access

16 October – 2012 – San Diego

(BIOS access via the terminal)

Accessing the BIOS in a terminal with ipmitool

When I put my systems administrator hat on one of the things that bothers me is getting remote access to the BIOS. I love to use the command line, and any time I have to fire up a GUI of some kind to do work I get an icky feeling.

So when I had to remotely access some Dell C6220s to turn on virtualization, which is off by default in these Dell servers, I didn’t want to have to go through the rigamarole of getting the right OS configuration that would allow me to remotely access the console via a web-based Java GUI run out of Firefox. Even writing that sentence takes too long. So I thought I would try out serial over lan (SOL) access.

And it works!


server2 $ ipmitool -I lanplus -H server1-ipmi -U root -P password sol activate
[SOL Session operational.  Use ~? for help]

Ubuntu 12.04.1 LTS server1 ttyS1

server1 login:

If we reboot the server we can see all the usual boot screens and can access them all without having to fire up the Java based console, instead I can just ssh into a second server that has access to the ipmi network and connect with ipmitool.

Here are some of the key mappings that will help to use ipmitool. The main thing to note is how to exit from the SOL session, and to do that you basically hit ENTER the a tilde, then a period. (For some reason this will usually log me not only out of the SOL session, but also the ssh session. Something to look into.)


server2 $ ipmitool -I lanplus -H 10.10.10.10 -U username -P password sol activate
[SOL Session operational.  Use ~? for help]
Here are some useful commands:
        KEY MAPPING FOR CONSOLE REDIRECTION:
         Use the <ESC><0> key sequence for <F10>
    	Use the <ESC><!> key sequence for <F11>
    	Use the <ESC><@> key sequence for <F12>
        Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>
    	Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>
    	Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>
    	Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>
        Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter
    	key, and X is the upper case of that key
        Use the <ESC><R><ESC><r><ESC><R> key sequence for <Ctrl><Alt><Del>
Help commands from ~?:
Supported escape sequences:
    	~.  - terminate connection
    	~^Z - suspend ipmitool
    	~^X - suspend ipmitool, but don't restore tty on restart
    	~B  - send break
    	~?  - this message
    	~~  - send the escape character by typing it twice
    	(Note that escapes are only recognized immediately after newline.)

There are all kinds of things one can do with ipmitool other than terminal based BIOS access, and I won’t list all of them here, but one nice thing is the ability to set the boot device, for example setting the boot device to disk and doing so for all future reboots.


server2 $ ipmitool -H 10.10.10.10 -U root -P password chassis bootdev disk \
options=persistent

Here are the kernel configs I used to tell Ubuntu to use serial. Note that I added the “\” in the text below, so you can’t just cut and paste it; needs to be all on one line.


pxeboot-server:/tftp/pxelinux.cfg# cat default 
# D-I config version 2.0
DEFAULT server
prompt 0
timeout 1
# serial console
console 0
serial 0 115200 0

LABEL server
kernel ubuntu-installer/amd64/linux
append ramdisk_size=14984 locale=en_US keyboard-configuration/layoutcode=us \
console-keymaps-at/keymap=us locale=en_US console-setup/layoutcode=en_US \
netcfg/wireless_wep= netcfg/choose_interface=eth0 netcfg/get_hostname=c01-07 \
url=http://10.10.10.10/node.preseed vga=normal \
initrd=ubuntu-installer/amd64/initrd.gz -- \
console=ttyS1,115200 earlyprint=serial,ttyS1,115200

Finally I’ll show a screenshot of a text install happening, and I am watching that via text-based SOL access.

So give SOL a shot, and let me know if I’ve made any mistakes in this post, or if there are other interesting things that can be done with ipmitool. :)

OpenStack 2012 Summit Day #1

2012-10-15T00:00:00-07:00

OpenStack 2012 Summit Day #1

15 October – 2012 – San Diego

OpenStack 2012 Summit

This week I’m at the OpenStack 2012 Summit. It’s by far the biggest conference I’ve been to. Usually I go to conferences on security, such as H.O.P.E or CanSec or even smaller library related conferences. Given that OpenStack is one of the fastest growing open source projects of all time, it’s no surprise to find out that the conference has grown from 75 people a couple years ago, to about 1300 this year, up from 700 only a year ago. It’s massive and it’s growing.

tl;dr

Need to get into Ceph
Database as a service too
Upgrading is a big issue in OpenStack
San Diego is nice
OpenStack is kinda a big deal

Opening “keynote”

Not much to say here, some summit housekeeping with trippy techno rave music to wake me up. :)

Open Compute

The first session I attended was by Cole Crawford of the Open Compute Foundation. Basically this was an overview of the history of things like the 19" rack (it came from waaaaay back from the train system) and how we need to change our standards to allow things like interoperability and other good stuff.

I was hoping to find out how a small organization—like the one that I work for—could be involved in Open Compute, specifically how we could access similar hardware as Facebook and others are using. I didn’t quite get that out of the presentation, but hopefully in the future I will be able to get us into Open Compute hardware in some fashion or another.

At the very least I signed up for the mailing list. :)

Intercloud Object Storage: Colony

The Colony session was lead by Shigetoshi Yokoyama of the Japan National Institute of Informatics and there is a copy of the slides online. Colony is described as “federated swifts” for intercloud object storage services.

Having worked in a world class library that was interested in running petabytes of storage, I’m enthralled with object storage. It’s an important storage paradigm, and surprisingly, for reasons unknown, one that doesn’t seem to get a lot of attention in Canada. Perhaps it’s because startups and other organizations just use S3 and haven’t run into any privacy issues as of yet. Speaking of Amazon S3: it has a trillion+ objects stored now.

I think that in situations where we need to store a lot of replicated data—such as what a library or researcher would like to, or rather should like to store—object storage such as swift is a great way to go. (Though that said, perhaps systems like Amazon’s Glacier make more sense for those use cases, but maybe not.) I was at a presentation by the then CIO of the University of Alberta who figured there was three to five petabytes of research data on campus that needs to be preserved. That’s a lot of storage, especially when factoring in replication, and it really needs to happen at some point—and it can’t just be a huge tape system.

Further, considering that libraries and researchers (ie. their respective Universities and organizations), are supposed to work together it would make sense to be able to have some kind of interoperability between object storage clouds, private or otherwise, even if it’s just for some geographic separation.

Certainly Colonly is going to be a project to keep an eye on as object storage evolves and we try to do it across data centers and organizations.

Operating your OpenStack Private Cloud

The next session I attended was Operating your OpenStack Private Cloud, which is good because that is one of the things that I do. I manage a small, eight node, OpenStack cluster that will eventually backend a Apache VCL setup for virtual classrooms. So a very small private cloud, but I’ve still hit a few “pain points” that larger installations do.

A few points the speaker made:

Monitoring: statsd, graphite, etc
Operations tools don’t exist for OpenStack (yet)—we need better ops tools so he wrote Pulsar
- Would be nice to see an operations dashboard (perhaps in horizon)
- Eg. can’t get a list of all instances on a node and their IP addresses
- Some tools need hostnames and some need ids…why
- DSH for Ubuntu, PSH for Redhat?
- Don’t forget your bashfu; still working with Linux :)
Database backups
- Holland
Performance and scale considerations
- Block storage solution of some kind (cinder, or cinder + some vendor)
- Local disk – raw image is only slightly faster than qcow2, but qcow2 may make better business sense
- IO will degrade on local disks when glance copies images between machines
Scheduling
- CFQ made the most sense
- You have the power to change all this! You have the power to change your scheduler! Benchmark workloads and plan accordingly.
Lots of things you can do with glance performance-wise
- Image caching
- If you’re using qcow2 it doesn’t have to move the whole image (something to check into)
Don’t want swift to become unbalanced
They use chef for automated deployment
- controller in 15 minutes
- compute in two
Day to day tasks
- Made up of mostly dealing with new issues
- Eg. resizing
- Hardware failures…still have all this hardware
Don’t forget—it’s not (always) you, it’s a bug
Nice thing about a private cloud is you can pick the architecture that matches your requirements
OpenStack/We need to provide an upgrade path (I hear this 20 times today)
Metrics
- Load average is actually quite telling
- How long does an API call take?

Lunch!

Lunch was Ok. There are so many people here it’s tough to move around. And given OpenStack gave out giant bags, everyone (like me) has their own personal giant bag, plus the one they got from OpenStack (though I gave mine back). Giant bags galore!

Then I had a beer at the bar downstairs:

Database as a Service

This session is specifically about RedDwarf. RedDwarf is taking MySQL and treating it like a “first class citizen” in OpenStack. It’s a managed MySQL database service.

I believe that SQL databases should be separated out from compute. That said, there are many people that would disagree with me. There’s a theory out there that storage and compute should be together on the same nodes, and if that’s the case then I assume so would database. Right now I would prefer to separate out storage, compute, and database. Maybe it doesn’t fit perfectly with the cloud paradigm, but sometimes you have to apply technology not just theorize about it. SQL isn’t going anywhere, and neither is its unique workload.

HP and Rackspace have slightly different implementations of RedDwarf, and both of their services are in production. Each are trying to keep their system compatible with OpenStack, and have committed themselves to using the same CLI: python-reddwarfclient. Further, they are both using OpenStack-common and Swift for securely storing database snapshots.

The team has a desire to bring RedDwarf forward, and they want to involve more community usage and get it into incubation in OpenStack and at that point a lot of doors open, such as a GUI in Horizon.

You can get a test install using the below commands, and it will “fake” a nova backend.


$ git clone https://github.com/hub-cap/reddwarf_lite
$ ./reddwarf_lite/bin/start_server.sh

There was also a lot of discussion as to the fact that RedDwarf could be used to build Other Things as a Service and also the fact that they are running it on top of OpenVZ versus another virtualization hypervisor and did so for performance reasons (up to 30% better was a number mentioned).

Extending OpenStack for Fun and Profit: Creating New Functionality with the Enhancements API

This session is about having two sides—one side is OpenStack, and the other side is “your innovation”. It’s about getting OpenStack and your innovation to talk to one another; about getting your innovation to market and make money.

The speaker, Tim Smith, is the co-founder of GridCentric which has essentially implemented fork() for virtual machines. The main idea is spin up replica instances quickly and efficiently. They are reinventing boot, and in order to do that they need to extend Nova. GridCentric’s code for their extension is up on github and would be a good example to work from. (I’ve looked at GridCentric’s offerings before in relation to virtual classrooms—ie. not having a bootstorm when a bunch of students startup instances for a class. With something like GridCentric’s technology you don’t have to boot the instance.)

Nova is a framework providing:

A standardized API
Messaging/RPC
Database-backed ORM

Nova can be extended by:

API extensions
Custom “services”
Nova CLI extensions
Dashboard extensions

Tim suggests that there are a few business challenges such as the fast evolution of OpenStack but generally seems positive about extending OpenStack. Certainly GridCentric has based some of their business on OpenStack.

A commenter noted that there are actually pre-action and post-action boot hooks in Folsom which may help to extend Nova.

Storing VMs with Cinder and Ceph RBD

Considering how interested I am in storage, this project, Ceph, is one of the most important I know of. So when Josh from Inktank talks about Ceph, I’m here to listen. As a note, this room is standing room only for this session!

Ceph is designed for scalability—it has no single point of failure, and you don’t have to be locked into a single hardware vendor. It’s software based and self-managing. There is also the ability to run custom functions on the storage node—such as create a thumbnail from an image file.

CRUSH is basically what sets Ceph apart from other storage systems. It’s a pseudo-random placement algorithm and it ensures even distribution across the cluster and has a rules-based system. Basically it avoids look up tables, which eventually kill distributed storage. Further, unlike RAID the entire cluster can recover in parallel.

RBD, the Rados Block Device, is the part of Ceph used to access block storage. KVM has a native driver for RBD. They are thin-provisioned, striped across nodes, and support snapshots and cloning. With RBD you can spin up thousands of virtual machines with the same base image.

Why use block storage in OpenStack?

Persistent
Not tied to a single host—decouple storage from compute
Enables live migration

Now with Folsom, Cinder has learned how to talk to Glance, which can respond with Images—ie. boot from volume. But still involves a large copy from Glance. However, images can be stored in RBD, and have been able to for some time! So we can skip the copy and just use RBD directly.

That said, it isn’t quite perfect yet. There are enhancements being made in Grizzly. Eg. is not in Horizon, but is in the API and CLI.

NOTES:

The Ceph file system is not recommended for production, but everything else is, such as RBD. If you don’t use the Ceph file system you don’t need metadata servers.
According to Josh, in terms of running OSDs on the same node as the hypervisor, as long as you have enough memory you should be Ok. Everything is running in userspace. This runs a bit counterintuitive to me, and also is contrary to one of the positive points made about block storage—decoupling. But it’s a valid use case.

From Folsom to Grizzy: A DevOps Upgrade Pattern

This was the last session of the day, and I was quite tired, so apologies. Suffice it to say: Even imagining upgrading OpenStack is difficult and requires a lot of bullet points.

Misc Notes from the Day

There’s free sunshine and stout outside on the balcony
Probability of data loss in Swift (this is not to say it’s high, rather an analysis)
Infiniband doesn’t have a lot of traction in OpenStack AFAIK, need to look into this
San Diego has basically the same weather all the time
Japan’s summers are very hot and humid

My OpenBSD Lab

2012-06-13T00:00:00-07:00

My OpenBSD Lab

13 June – 2012 – Edmonton

Above you can see my little OpenBSD lab. For some reason my workplace has several different kinds of small form factor servers, such as the big silver Netcom box, a couple of Soekris boxes, and I can’t even tell what make/model the little blue boxes are. They are all essentially small, fanless servers with at least two ethernet ports. Since they weren’t being used for anything and I needed to setup a few test installs of OpenBSD, I went to work. :)

The “lab” is comprised of two carped firewalls (the blue boxes; note the red cable between them is a cross over cable for pfsync traffic), a couple of cheap desktop switches, and the square silver thing is an OpenBSD bridge that goes in between the two switches. The carped firewalls are connected to another part of my test network, which eventually leads out to the Internet. I also have a couple of OpenBSD virtual machines running on my RHEL 5 xen dom0. One of them provides an OpenBSD pxe boot solution to install OpenBSD onto systems like the Soekris that can’t boot from USB.

When testing I plug my laptop into the internal switch (the green cable), so that in order to get out to the Internet it has to go over the bridge and through the firewalls. “Over the Bridge and through the Firewall”…sounds like a book Hemingway would have written if he was a Unix security administrator. ;)

Using this little lab I can test out all kinds of interesting OpenBSD functionality such as packet filtering with pf, virtual IP address failover with carp, bridging, and network authentication using authpf, along with anything else that I need to work on or try out in the OpenBSD world, or just try to stay familiar with OpenBSD in general.

Bridge up!


$ ifconfig bridge0 up

36 hot swappable hard-drive bay Supermicro server specs

2012-06-07T00:00:00-07:00

36 hot swappable hard-drive bay Supermicro server specs

7 June – 2012 – Edmonton

The above is the front of four Supermicro 36-bay chassis servers racked but not completely filled with drives.

This is the back of the server. As you can see, there are 12 hot swappable slots in the back. The motherboard has room for seven low profile boards of varying PCIe speeds.

Sorry for the poor picture quality—I don’t get paid enough to have a phone with a good camera. ;)

This is output from the bottom server that has four drives lit up:


# ls /dev/sd?
/dev/sda  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf

so with the two internal hard-drives that makes six, which is what we see above. If the server was filled up with drives, there would be 38.

First things First: The Specs

Why not get this out of the way? That’s what you’re here for, isn’t it? :) The below would be for one server:

Item	Part or Part #	Quantity
4U 36 bay Chassis	SM SC847E16-R1400LPB	1
iPass cable	SM CBL-0281L	1
iPass cable	SM 0108L-02	1
Internal HD Brackets	SM MCP-220-84701-0N	2
Motherboard	SM X8DT6-F	1
CPU	Xeon E5645 (6 cores)	2
Heat Sink/CPU Fan	Dynatron G666	2
Kingston 24GB Pack	SM KVR1066D3Q8R7SK3/24G	2
SATA power splitter	SM CBL-0082L	1

The total cost for the above is around $4000.

What you get is one 4U, dual CPU server (24 cores counting hyperthreading), with 48GB of RAM, the MB supports up to 196GB I believe, and room for 36 hot swappable drives, PLUS two internal OS drives that are actually bracketed deep inside the chassis, ie. difficult to get out.

Note: I’ve done my best to make sure the above is correct, and I do have exactly that hardware, but don’t base your business on those specs without doing some testing. Order one server and see how it goes before ordering a dozen+.

Note: You would need to find a local vendor to put the pieces together, or you could do it yourself—but I don’t recommend that. I’d put 5% of the total cost aside for professional assembly. We were lucky enough that our vendor was willing to put them together for free—at least on the first order. :)

(Terrible image…but you can kind of see the two internal hard-drives.)

So with those two internal slots, plus 24 bays in the front, and 12 in the back is a total of 38 3.5" slots, 36 of them hot swappable. Or, put another way, if using 3TB SATA drives…114TB of raw storage. (Also I think you can fit four 2.5" drives internally instead of two 3.5" drives. But don’t quote me on that.)

Bulk storage costs

The amount of data an organization has to store never seems to go down, only up up UP!—probably at least 50% per year. High-end, “enterprise storage”, can cost between $50K-$100K per terabyte, where K is thousands.

I’m not kidding. It’s really that much, and there are few organizations that can afford to have their storage increase by 50% per year at that cost. Everywhere I’ve worked has started out with a large Enterprise SAN and quickly realized they can’t afford it.

Of course, there are reasons why enterprise storage costs so much. It’s not easy to design and build a large, production, high-performance SAN, and in most cases trying to replicate one with commodity hardware is not a good idea, and could cost people their jobs. Not a nice thought, but it’s true.

But in some specialized cases, such as bulk data storage, not using an Enterprise SAN might be an option. However, unless you work at Facebook, or another company that somehow has access to hardware based on Open Compute’s Open Vault, what do you do for bulk storage hardware?

One possibility is the above 36-bay Supermicro chassis based server. I know there are people out there using them. Lots of them.

Now you can afford test infrastructure

One note I feel important to make is that when you purchase cost effective hardware there is often money on the table to buy more hardware for test. I always like to have test hardware running so that I don’t have to worry about “trying something out” in production. When running production systems I never want to be afraid to make a change, and the way to do that is to have test infrastructure to experiment and…test with.

If your enterprise hardware is so expensive that test infrastructure is completely out of the question, then I wonder how easy it will be to try things out; to make changes without fear of brining down the production system?

To me there is massive value in test hardware.

Also, test hardware can be moved from test to production in case of failure.

Support

Obviously if you spec your own hardware, versus say buying from a tier one vendor, support means that if it breaks you pull a new part off the shelf or out of test and replace it in production yourself.

Further, tier one vendors will usually certify their hardware for particular operating systems and applications. You won’t have that if you spec your own. But you certainly can start small (and inexpensively) and test out your specs/OS/applications, then go bigger when you know it all works well together.

Personally, I don’t like spending 3 hours on the phone with the tier one support, going through generic call scripts with someone who doesn’t care about my problem in the least, and then having the local support tech either be a “cell phone with hands” or cancel four times in a row before simply replacing the whole motherboard because they have no idea what is actually wrong. I suppose that sounds kind of negative, but those are real anecdotes. :) PS. We have lots of tier one vendor servers…

Bonus

The Java IPMI remote console gui works great on Linux! Some tier one vendors don’t…and they make you pay extra to actually use the remote console. (cough HP cough)

That said…don’t look at how they store passwords in the ssh interface. Just don’t.

Conclusion

I feel that in specialized applications systems like this supermicro server can be very effective. Especially if it leaves money on the table for test infrastructure. I’d rather have twelve (nine production, three in test) of these boxes than two four-socket tier one servers.

I recommend using 5% of the budget for parts to go on the shelf, and 5%-25% for test.

If you have any questions/concerns, please comment.

Thanks,
Curtis.

Installing IBM high-iops FusionIO Cards in Redhat/Centos 6

2012-05-23T00:00:00-07:00

Installing IBM high-iops FusionIO Cards in Redhat/Centos 6

23 May – 2012 – Edmonton

In a previous post I had described how I deployed a IBM branded FusionIO drive on Redhat Enterprise 5.

I am now running that same card on CentOS 6, and am using the new version (2.2.3) of IBM’s version of the driver.

Actually I think there is a new-new version (3) of the driver now out for some people. I’m not sure if IBM has put out this driver or not for their high-iops cards.

CentOS version:


[root@srv ~]# cat /etc/redhat-release 
CentOS release 6.2 (Final)

The kernel I am running is stock RHEL 6:


[root@srv ~]# uname -a
Linux example.com 2.6.32-220.17.1.el6.x86_64 #1 \
SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux

This is what I see in terms of PCI devices for the FusionIO cards:


[root@srv ~]# lspci | grep -i fusion
8f:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)
90:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)

So the card is physically installed in the server, but the driver has not been loaded, so they are not usable at this point. Also should note that one 640GB cards actually looks like 2x 320GB devices to the OS.

First, we download the zip file containing the RPMs from IBM.

Warning: These drivers are for the IBM version of the FusionIO cards. If you are not running the IBM version you probably need different drivers and RPMs.


# wget ftp://download2.boulder.ibm.com/ecc/sar/CMA/XSA/ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip
SNIP!

Inside that zip are several RPMs:


[root@srv tmp]# mkdir fio
[root@srv tmp]# cd fio/
[root@srv fio]# unzip ../ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip 
Archive:  ../ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip
  inflating: rhel6/fio-common-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-firmware-highiops-101583.6-1.0.noarch.rpm  
  inflating: rhel6/fio-snmp-agentx-1.1.1.5-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-util-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/high_iops-gui-2.3.1.1874-1.1.noarch.rpm  
  inflating: rhel6/iomemory-vsl-2.2.3.66-1.0.el6.el6.src.rpm  
  inflating: rhel6/iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.el6.el6.x86_64.rpm  
  inflating: rhel6/libfio-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/libfusionjni-1.1.1.5-1.0.el6.x86_64.rpm

So far when I’ve been running these servers I haven’t installed all of those RPMs, only a subset.

So lets install those RPMs:


[root@srv rhel6]# yum localinstall --nogpg \
 fio-common-2.2.3.66-1.0.el6.x86_64.rpm \
 libfio-2.2.3.66-1.0.el6.x86_64.rpm fio-util-2.2.3.66-1.0.el6.x86_64.rpm \
 fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm \
 fio-firmware-highiops-101583.6-1.0.noarch.rpm \
 iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.el6.el6.x86_64.rpm
SNIP!
Transaction Test Succeeded
Running Transaction
  Installing     : fio-util-2.2.3.66-1.0.el6.x86_64                   1/6 
  Installing     : fio-common-2.2.3.66-1.0.el6.x86_64                 2/6 
  Installing     : iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.e   3/6 
  Installing     : libfio-2.2.3.66-1.0.el6.x86_64                     4/6 
  Installing     : fio-sysvinit-2.2.3.66-1.0.el6.x86_64               5/6 
  Installing     : fio-firmware-highiops-101583.6-1.0.noarch          6/6 

Installed:
  fio-common.x86_64 0:2.2.3.66-1.0.el6                                    
  fio-firmware-highiops.noarch 0:101583.6-1.0                             
  fio-sysvinit.x86_64 0:2.2.3.66-1.0.el6                                  
  fio-util.x86_64 0:2.2.3.66-1.0.el6                                      
  iomemory-vsl-2.6.32-71.el6.x86_64.x86_64 0:2.2.3.66-1.0.el6.el6         
  libfio.x86_64 0:2.2.3.66-1.0.el6

As you can see the sysvinit RPM contains a couple of init.d files.


[root@srv rhel6]# rpm -qpl fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm 
/etc/init.d/iomemory-vsl
/etc/sysconfig/iomemory-vsl

Let’s chkconfig this on permanently.


[root@srv rhel6]# chkconfig iomemory-vsl on

We also need to enable iomemory-vsl in /etc/sysconfig/iomemory-vsl.


[root@srv init.d]# cd /etc/sysconfig
[root@srv sysconfig]# grep ENABLED iomemory-vsl 
# If ENABLED is not set (non-zero) then iomemory-vsl init script will not be
#ENABLED=1
[root@srv sysconfig]# vi iomemory-vsl 
[root@srv sysconfig]# grep ENABLED iomemory-vsl 
# If ENABLED is not set (non-zero) then iomemory-vsl init script will not be
ENABLED=1
[root@srv sysconfig]#

And we can start or restart iomemory-vsl:


[root@srv sysconfig]# service iomemory-vsl restart
Stopping iomemory-vsl: 
Unloading module iomemory-vsl
                                                           [FAILED]
Starting iomemory-vsl: 
Loading module iomemory-vsl
Attaching: [                    ] (  0%) /Attaching:
[                    
Attaching: [====================] (100%) \
fioa
Attaching: [====================] (100%)
fiob
                                                           [  OK  ]

At this point I’m going to reboot the server as well, just to make sure everything is going to get loaded if the server spontaneously restarts, which they have been known to do. ;)


[root@srv sysconfig]# reboot

Now after the reboot there are a couple more block storage devices on this server:


[root@srv ~]# ls /dev/fio?
/dev/fioa  /dev/fiob

We want to create a lvm physical volume (pv) on that block device:


[root@srv ~]# pvcreate /dev/fioa
  Device /dev/fioa not found (or ignored by filtering).

Ooops. Error message. What went wrong? Well, the “or ignored by filtering” is where to start looking. This FusionIO knowledge base entry (which you have to login to see, how annoying is that) shows that we need to add an entry to the lvm.conf on the server:


Locate and edit the /etc/lvm/lvm.conf configuration file.
Add an entry similar to the following to that file:
types = [ "fio", 16 ]

That is precisely what I will do.


[root@srv lvm]# grep types lvm.conf
    # List of pairs of additional acceptable block device types found 
    # types = [ "fd", 16 ]
    types = [ "fio", 16 ]

And now:


# let's see if the types were loaded
[root@srv ~]# lvm dumpconfig | grep types
  	types=["fio", 16]
[root@srv ~]# pvcreate /dev/fioa
  Physical volume "/dev/fioa" successfully created
[root@srv ~]# pvcreate /dev/fiob
  Physical volume "/dev/fiob" successfully created

And create a volume group and add the pvs to it.


[root@srv ~]# vgcreate hiops /dev/fioa
  Volume group "hiops" successfully created
[root@srv ~]# vgextend hiops /dev/fiob
  Volume group "hiops" successfully extended
[root@srv ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree  
  hiops    2   0   0 wz--n- 504.91g 504.91g
  system   1   9   0 wz--n-  58.56g  36.66g
  vm       1  11   2 wz--n-   1.31t 228.09g

I should note at this point that there is only 504g in the hiops volume group when there should be about 600g.

Previously, using the fio-format command, I had formatted these drives to only 80% capacity. But that was on another server, and I’m not sure it’s really necessary to do that unless you are looking for extreme performance or perhaps additional reliability.

I believe that in some cases with SSD, PCIe or otherwise, it’s not a bad idea to use less than 100% of the drive. That said, if you are looking to max out these drives performance-wise, I’d suggest talking to your vendor rather than just listening to me. :)

(AFAIK, these cards can actually take an external power source to increase performance even more. But we don’t use that functionality.)

So I’m going to reformat these drives to 100% usage. Just for fun. Why not get back that 100g because the performance/endurance at 100% is going to be fine for our usage.

Note: Brand new drives won’t have to be formatted. I’m only doing this because I had formatted the drives when they were in the other server.

Warning: Reformatting will obviously delete any data on these drives!


# first detach the /dev/fioa
[root@srv ~]# fio-detach /dev/fct0
Detaching: [====================] (100%) -
[root@srv ~]# fio-format -s 100% /dev/fct0
Creating a device of size 322.55GBytes (300.40GiBytes).
  Using block (sector) size of 512 bytes.

WARNING: Formatting will destroy any existing data on the device!
Do you wish to continue [y/n]? y
Formatting: [====================] (100%) \
Formatting: [====================] (100%)
Format successful.
# then attach...
[root@srv ~]# fio-attach /dev/fct0
Attaching: [====================] (100%) -
fioa

And we can add that device back with pvcreate and then we should see a larger drive:


[root@srv ~]# pvcreate /dev/fioa
  Physical volume "/dev/fioa" successfully created
[root@srv ~]# pvs /dev/fioa
  PV         VG    Fmt  Attr PSize   PFree  
  /dev/fioa  hiops lvm2 a-   300.40g 300.40g

I reformatted the other side of the drive back to 100% as well. (With new drives this shouldn’t be necessary.)

And the fio-status now is:


[root@srv ~]# fio-status

Found 2 ioDrives in this system with 1 ioDrive Duo
Fusion-io driver version: 2.2.3 build 66

Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	External Power: NOT connected
	PCIE Power limit threshold: 24.75W
	Sufficient power available: Unknown
	Connected ioDimm modules:
	  fct0:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	  fct1:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX

fct0	Attached as 'fioa' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	Alt PN:68Y7382
	Located in slot 0 Upper of ioDrive Duo SN:XXXXX
	PCI:8f:00.0
	Firmware v5.0.6, rev 101583
	322.55 GBytes block device size, 396 GBytes physical device size
	Sufficient power available: Unknown
	Internal temperature: avg 50.2 degC, max 51.2 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10.00%

fct1	Attached as 'fiob' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	Alt PN:68Y7382
	Located in slot 1 Lower of ioDrive Duo SN:XXXXX
	PCI:90:00.0
	Firmware v5.0.6, rev 101583
	322.55 GBytes block device size, 396 GBytes physical device size
	Sufficient power available: Unknown
	Internal temperature: avg 46.3 degC, max 46.8 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10.00%

Finally we can create a logical volume (lv) to use.


[root@srv ~]# vgs hiops
  VG    #PV #LV #SN Attr   VSize   VFree  
  hiops   1   0   0 wz--n- 300.40g 300.40g
[root@srv ~]# lvcreate -n test -L10.0G /dev/hiops
  Logical volume "test" created

If you have any corrections or other comments, please let me know!

Thanks,
Curtis.

Bottle, Elixir, Bootstrap and Datatables - Instant Admin Backend

2012-05-15T00:00:00-07:00

Bottle, Elixir, Bootstrap and Datatables – Instant Admin Backend

15 May – 2012 – Edmonton

UPDATE: I’ve recently stopped using Elixir and have moved to using straight SQLAlchemy.

Recently I have been working on a web-based administrative backend, and have found doing so unusually easy, mostly because of the combination of Bottle, a python micro-framework, Elixir, a wrapper over top of SQLAlchemy (which is itself an SQL toolkit and ORM), Twitter’s bootstrap, scaffolding for websites, and finally Datatables, which enables advanced interactions with HTML tables.

The only difficulty I had, and it was slight, was integrating bootstrap and datatables together, but this post helped out quite a bit.

I would certainly suggest to anyone looking to create an administrative backend, or any web application really, to look into these four technologies, as it would take very little time to create a minimum viable product, or a demo, using a combination of Bottle, Elixir, Bootstrap, and Datatables.

In future posts I will present an example application.

OCZ Z-Drive R4 Installation and Performance

2012-05-08T00:00:00-07:00

OCZ Z-Drive R4 Installation and Performance

8 May – 2012 – Edmonton

In a previous post I mentioned how we had purchased 11 300GB OCZ Z-Drive R4 PCIe-SSD cards. (Please note that this was a special case purchase—the cards didn’t meet any specific requirements we had other than that they were easily available, PCIe-SSD, and low profile.)

We bought the low profile version because these drives are going into the Supermicro SC847E16-R1400LPB chassis (the subject of future posts), which have room for seven low profile cards. I believe the full height zdrive R4s are faster, so this is a compromise.

Installation

Each of our servers is going to get one zdrive. I placed them in a x8 slot.

Once the OS is up and installed (these cards are not bootable, ie. the OS can’t be installed onto the cards) the proprietary kernel module needs to be loaded.

There is an installation guide.

I’m running Centos 6.2 on these servers.


# cat /etc/redhat-release 
CentOS release 6.2 (Final)
# uname -a
Linux ocz_server 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

I’m using the:


Red Hat Enterprise Linux 6.x, CentOS 6.x 64-bit	1.0.0.1480	Mar 2, 2012

version of the driver.

When that tar file is downloaded and unzipped all there is inside is the ocz10xx.ko kernel module.


# wget http://www.oczenterprise.com/files/drivers/OCZ%20RHEL-Centos_6.x_64-Bit_r1480.tar.gz 
--2012-05-08 21:40:23--  http://www.oczenterprise.com/files/drivers/OCZ%20RHEL-Centos_6.x_64-Bit_r1480.tar.gz
Resolving www.oczenterprise.com... 74.52.187.58
Connecting to www.oczenterprise.com|74.52.187.58|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4072553 (3.9M) [application/x-gzip]
Saving to: “OCZ RHEL-Centos_6.x_64-Bit_r1480.tar.gz”

100%[======================================>] 4,072,553   1.03M/s   in 4.0s    

2012-05-08 21:40:27 (991 KB/s) - “OCZ RHEL-Centos_6.x_64-Bit_r1480.tar.gz” saved [4072553/4072553]

# tar zxvf OCZ\ RHEL-Centos_6.x_64-Bit_r1480.tar.gz 
ocz10xx.ko

which can be loaded by:


# insmod ocz10xx.ko
# lsmod | grep ocz
ocz10xx               479350  1

When that module is loaded the following is reported to dmesg:


ocz10xx: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ocz10xx module is older than RHEL 6.2 ... applying fixups
  alloc irq_desc for 26 on node -1
  alloc kstat_irqs on node -1
ocz10xx 0000:07:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
ocz10xx 0000:07:00.0: setting latency timer to 64
OCZ Storage Controller is found, using IRQ 26, driver version 2.0.0.1480.
OCZ Linux driver ocz10xx, driver version 2.0.0.1480.
OCZ DRIVE LEVEL=OCZ_FAST, STATE=ONLINE
scsi5 : OCZ Storage Controller
scsi 5:0:126:0: Direct-Access     ATA      OCZ Z-DRIVE R4 C 2.15 PQ: 0 ANSI: 5
sd 5:0:126:0: Attached scsi generic sg8 type 0
sd 5:0:126:0: [sdg] 586135549 512-byte logical blocks: (300 GB/279 GiB)
sd 5:0:126:0: [sdg] Write Protect is off
sd 5:0:126:0: [sdg] Mode Sense: 41 00 00 00
sd 5:0:126:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdg: unknown partition table
sd 5:0:126:0: [sdg] Attached SCSI disk

and I now have a /dev/sdg to use.

Loading the kernel module at boot

First, let me say that I don’t have a lot of experience with kernel modules. I’m hoping that if I’ve made a mistake that someone will alert me in the comments. Or perhaps I missed where this is documented by OCZ.

Running insmod is great for the first time one tries out the zdrive, but what happens after a reboot?

Usually kernel modules go in /lib/modules/`uname -r` but this module doesn’t seem to be tied to a particular kernel version. While I could put it in that directory, each time I get a new kernel I’d have to move it. This would not be good for maintainability. Assuming the module works with all 6.x kernels—which is what the OCZ drivers page suggests—it should be Ok to put this module in a more permanent location.

What I did was build and RPM with three files:


# rpm -qf /etc/depmod.d/ocz-zdrive-r4.conf 
ocz-zdrive-r4-r1480-2.el6.x86_64
# rpm -qf /etc/modprobe.d/ocz-zdrive-r4.conf 
ocz-zdrive-r4-r1480-2.el6.x86_64
# rpm -qf /usr/share/ocz-zdrive-r4/module/ocz10xx.ko 
ocz-zdrive-r4-r1480-2.el6.x86_64

The .conf files contain:


# cat /etc/depmod.d/ocz-zdrive-r4.conf 
search /usr/share/ocz-zdrive-r4/module
# cat /etc/modprobe.d/ocz-zdrive-r4.conf 
alias ocz10xx ocz

which will ensure that the ocz10xx.ko module is loaded with all the other kernel modules, so that you can put file systems on the zdrive into fstab and have them mounted at boot:


# uptime
 23:05:24 up 15 min,  2 users,  load average: 0.00, 0.02, 0.00
# lsmod | grep ocz
ocz10xx               479350  1 
# mount | grep ocz
/dev/mapper/ocz-test on /mnt/ocz-xfs-test type xfs (rw)
# cat /etc/fstab | grep ocz
/dev/mapper/ocz-test /mnt/ocz-xfs-test          xfs    defaults        1

Please let me know if there is something wrong with the methodology. :)

Performance testing

As I’ve said before, good performance testing is hard to do. All I can really do at this point is run the same tests that FusionIO (GAH! Behind a support login now! Bad FusionIO, bad!) suggests running on their drives.

WARNING: The write tests will destroy data on the drive!

NOTE: A little bird told me that you need to run the write tests first, otherwise the flash drive—if it’s empty—may (depending on the vendor, perhaps) know it’s empty and return zeroes and you’ll be testing RAM instead of the card.

Write bandwidth test


# fio --filename=/dev/sdg --direct=1 --rw=randwrite --bs=1m \
--size=5G --numjobs=4 --runtime=10 --group_reporting --name=file1
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.7
Starting 4 processes
Jobs: 4 (f=4): [wwww] [100.0% done] [0K/1021M /s] [0 /974  iops] [eta 00m:00s]
file1: (groupid=0, jobs=4): err= 0: pid=2444
  write: io=9281.0MB, bw=948572KB/s, iops=926 , runt= 10019msec
    clat (usec): min=679 , max=81086 , avg=4111.26, stdev=4974.85
     lat (usec): min=848 , max=81251 , avg=4313.21, stdev=4974.67
    clat percentiles (usec):
     |  1.00th=[ 1704],  5.00th=[ 1928], 10.00th=[ 2064], 20.00th=[ 2672],
     | 30.00th=[ 2736], 40.00th=[ 2800], 50.00th=[ 2960], 60.00th=[ 3408],
     | 70.00th=[ 3568], 80.00th=[ 3760], 90.00th=[ 4960], 95.00th=[ 9664],
     | 99.00th=[35584], 99.50th=[36608], 99.90th=[38144], 99.95th=[38656],
     | 99.99th=[81408]
    bw (KB/s)  : min=139912, max=273338, per=25.14%, avg=238498.74, stdev=28729.62
    lat (usec) : 750=0.09%, 1000=0.03%
    lat (msec) : 2=8.60%, 4=76.87%, 10=9.69%, 20=2.80%, 50=1.90%
    lat (msec) : 100=0.03%
  cpu          : usr=4.53%, sys=2.98%, ctx=9287, majf=0, minf=120
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9281/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9281.0MB, aggrb=948572KB/s, minb=948572KB/s, maxb=948572KB/s, mint=10019msec, maxt=10019msec

Disk stats (read/write):
  sdg: ios=83/18421, merge=578/0, ticks=13/69730, in_queue=69712, util=99.21%

Read IOPS test


# fio --filename=/dev/sdg --direct=1 --rw=randread --bs=4k \
--size=5G --numjobs=64 --runtime=10 --group_reporting --name=file1
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 64 processes
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Jobs: 64 (f=64): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr]
 [100.0% done] [374.1M/0K /s] [91.4K/0  iops] [eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=2465
  read : io=3589.4MB, bw=367442KB/s, iops=91860 , runt= 10003msec
    clat (usec): min=100 , max=283036 , avg=693.42, stdev=1539.29
     lat (usec): min=101 , max=283036 , avg=693.61, stdev=1539.29
    clat percentiles (usec):
     |  1.00th=[  262],  5.00th=[  378], 10.00th=[  438], 20.00th=[  506],
     | 30.00th=[  556], 40.00th=[  604], 50.00th=[  652], 60.00th=[  700],
     | 70.00th=[  756], 80.00th=[  828], 90.00th=[  948], 95.00th=[ 1064],
     | 99.00th=[ 1400], 99.50th=[ 1592], 99.90th=[ 2288], 99.95th=[ 2832],
     | 99.99th=[56064]
    bw (KB/s)  : min=  816, max= 7032, per=1.55%, avg=5706.40, stdev=599.67
    lat (usec) : 250=0.81%, 500=17.90%, 750=50.29%, 1000=23.77%
    lat (msec) : 2=7.05%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=0.56%, sys=6.49%, ctx=919527, majf=0, minf=2240
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=918880/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3589.4MB, aggrb=367441KB/s, minb=367441KB/s, maxb=367441KB/s, mint=10003msec, maxt=10003msec

Disk stats (read/write):
  sdg: ios=914696/0, merge=0/0, ticks=612829/0, in_queue=607382, util=98.84%

Read bandwidth test


# fio --filename=/dev/sdg --direct=1 --rw=randread --bs=1m --size=5G \
 --numjobs=4 --runtime=10 --group_reporting --name=file1
file1: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.7
Starting 4 processes
Jobs: 4 (f=4): [rrrr] [100.0% done] [1599M/0K /s] [1524 /0  iops] [eta 00m:00s]
file1: (groupid=0, jobs=4): err= 0: pid=2543
  read : io=16475MB, bw=1647.2MB/s, iops=1647 , runt= 10002msec
    clat (usec): min=828 , max=79515 , avg=2423.79, stdev=1154.98
     lat (usec): min=828 , max=79515 , avg=2424.04, stdev=1154.98
    clat percentiles (usec):
     |  1.00th=[ 1528],  5.00th=[ 1768], 10.00th=[ 1912], 20.00th=[ 2064],
     | 30.00th=[ 2160], 40.00th=[ 2256], 50.00th=[ 2320], 60.00th=[ 2416],
     | 70.00th=[ 2544], 80.00th=[ 2736], 90.00th=[ 2992], 95.00th=[ 3280],
     | 99.00th=[ 3856], 99.50th=[ 4128], 99.90th=[ 6176], 99.95th=[13120],
     | 99.99th=[78336]
    bw (KB/s)  : min=369211, max=526336, per=25.10%, avg=423322.49, stdev=33254.52
    lat (usec) : 1000=0.18%
    lat (msec) : 2=14.73%, 4=84.40%, 10=0.63%, 20=0.04%, 100=0.02%
  cpu          : usr=0.19%, sys=5.89%, ctx=16488, majf=0, minf=1151
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=16475/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=16475MB, aggrb=1647.2MB/s, minb=1647.2MB/s, maxb=1647.2MB/s, mint=10002msec, maxt=10002msec

Disk stats (read/write):
  sdg: ios=32621/0, merge=0/0, ticks=71360/0, in_queue=71316, util=99.09%

Write IOPS test


# fio --filename=/dev/sdg --direct=1 --rw=randwrite --bs=4k --size=5G \
--numjobs=64 --runtime=10 --group_reporting --name=file
file: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
file: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 64 processes
Jobs: 64 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 
 64 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwJobs: 64
 (f=64): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 
[100.0% done] [0K/408.8M /s] [0 /99.8K iops] [eta 00m:00s]
file: (groupid=0, jobs=64): err= 0: pid=2556
  write: io=3670.1MB, bw=374777KB/s, iops=93694 , runt= 10030msec
    clat (usec): min=40 , max=302579 , avg=677.08, stdev=1765.33
     lat (usec): min=40 , max=302580 , avg=678.03, stdev=1765.34
    clat percentiles (usec):
     |  1.00th=[  117],  5.00th=[  390], 10.00th=[  450], 20.00th=[  506],
     | 30.00th=[  548], 40.00th=[  580], 50.00th=[  620], 60.00th=[  652],
     | 70.00th=[  692], 80.00th=[  748], 90.00th=[  820], 95.00th=[  892],
     | 99.00th=[ 1064], 99.50th=[ 1144], 99.90th=[31616], 99.95th=[32640],
     | 99.99th=[33536]
    bw (KB/s)  : min= 2208, max= 9448, per=1.56%, avg=5834.54, stdev=562.51
    lat (usec) : 50=0.25%, 100=0.58%, 250=1.49%, 500=16.44%, 750=62.05%
    lat (usec) : 1000=16.98%
    lat (msec) : 2=1.91%, 4=0.06%, 10=0.09%, 20=0.02%, 50=0.11%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=0.68%, sys=6.54%, ctx=942753, majf=0, minf=2070
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=939753/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3670.1MB, aggrb=374776KB/s, minb=374776KB/s, maxb=374776KB/s, mint=10030msec, maxt=10030msec

Disk stats (read/write):
  sdg: ios=609/926539, merge=2759/0, ticks=337/599297, in_queue=594561, util=99.08%

Conclusion

From a cursory look these drives seem to perform well. At least when they are brand new. :) We’ll see how they perform over time.

If anyone would like to see specific tests, please let me know.

Thanks,
Curtis.

What 11 OCZ Z-Drive R4 Cards Look Like

2012-04-13T00:00:00-07:00

What 11 OCZ Z-Drive R4 Cards Look Like

13 April – 2012 – Edmonton

…in one big box.

Recently we picked up 11 300GB OCZ Z-Drive PCIe SSD cards to put in some new servers we bought. Don’t ask me to explain what and why because this was somewhat of an unusual purchase, but suffice it to say we’re going to be running a lot of fast VMs on top of this storage. I would have liked to get the full height cards, but because of the servers they are going in we have to use the half-height version.

Look for some performance and configuration posts in the future, along with updates on using FusionIO on RHEL 6 as I’ve updated the server those are installed in to RHEL 6.

Deploying Ruby-on-Rails applications using RPM packaging

2012-01-17T00:00:00-08:00

Deploying Ruby-on-Rails applications using RPM packaging

17 January – 2012 – Edmonton

It’s been a long time between posts but the time has come!

In this post I hope to take a good look at one way to deploy a working ruby on rails (RoR) application by packaging it in an RPM.

In this example all of the gems the application requires are downloaded and built/compiled at the same time the RPM is, and thus the RPM contains all the required gems (100+ in this example). The best way to deploy an application, in my opinion, would be to standardize on a set of gems that is available at the OS level—so the RPM would not contain any gems, rather would require the general OS level gems.

Unfortunately, for many reasons, which I won’t get into, that is just not possible for me at this time. Maybe in the future when all gems can easily be built into RPMs, and also when internal developers can agree on a set of gems. Someday…

Environment

We’re deploying to a specific RHEL6 server environment.

Ruby version

We’ll be deploying the RoR application to Redhat Enterprise 6 (RHEL6) virtual machine which has, and likely always will have, ruby 1.8.7 (with backported security patches of course!).


[root@RoR-TEST ~]# ruby -v
ruby 1.8.7 (2010-06-23 patchlevel 299) [x86_64-linux]
[root@RoR-TEST ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.1 (Santiago)

This will likely be a problem in the future, as it seems that Rails 3.2 will be the last version that supports ruby 1.8.X (where X seems to be 7+ as 1.8.6 is specifially not supported). At some point the dev team may want to go to a Rails version that will not run on Ruby 1.8.7.

Apache and passenger

We’ll also be deploying the RoR app using apache and passenger.

Requirements

A few things are required to build and deploy an RPM.

The application code in some kind of version control system and hopefully that VCS supports tagging…svn, mercurial, and git all support tags.
A build server that is the same as OS and arch as the production server being deployed to. In this case, RHEL6 and X86_64.
- A spec file for the application.
- This build server needs bundle and gem available in the binary PATH because currently the example spec file needs it to be there.
- A working rpmbuild environment, configured as appropriate.
A test server to test the RPM deployment, ie. a place to actually install the RPM into.

The spec file

Building a RPM requires, among other things, a spec file. This file is the heart of a RoR RPM deployment.

I have put an example spec file up on github to peruse and abuse. Again, it’s not going to work out of the box, but it’s a good example, or will be at some point. :)

The build portion of the spec file is what is interesting in terms of deploying a RoR app with RPM.

Prior to the build section the code has been pulled out of a git repository into a local build directory by the rpmbuild process.

In the build section, which I’m cutting and pasting examples out of, we are going to cd into that checked out repository and use bundle to compile and install all the gems into ./vendor/bundle.


%build
pushd %{name}

# Install all required gems into ./vendor/bundle using the handy bundle commmand
bundle install --deployment

Once that has completed, which could be quite a long process depending on the number and complexity of the gems required, we remove the assets and recompile them.



# Compile assets, this only has to be done once AFAIK, so in the RPM is fine
rm -rf ./public/assets/*
bundle exec rake assets:precompile

Then we need to also build bundler into the RPM as well, which requires a smidge of trickery:



# For some reason bundler doesn't install itself, this is probably right,
# but I guess it expects bundler to be on the server being deployed to
# already. But the rails-helloworld app crashes on passenger looking for
# bundler, so it would seem to me to be required. So, I used gem to install
# bundler after bundle deployment. :) And the app then works under passenger.

PWD=`pwd`
cat > gemrc <<EOGEMRC
gemhome: $PWD/vendor/bundle/ruby/1.8
gempath:
- $PWD/vendor/bundle/ruby/1.8
EOGEMRC
        #gem --source %{gem_source} --config-file ./gemrc install bundler
        gem --config-file ./gemrc install bundler
# Don't need the gemrc any more...
rm ./gemrc

Finally, it seems that some of the gems have a funny location for ruby set, which we need to change because the rpmbuild process will mark that as a requirement. This issue may be fixed now.



# Some of the files in here have /usr/local/bin/ruby set as the bang
# but that won't work, and makes the rpmbuild process add /usr/local/bin/ruby
# to the dependencies. So I'm changing that here. Either way it prob won't
# work. But at least this rids us of the dependencie that we can never meet.
for f in `grep -ril "\/usr\/local\/bin\/ruby" ./vendor`; do
         sed -i "s|/usr/local/bin/ruby|/usr/bin/ruby|g" $f
         head -1 $f
done

popd

Basically, three major things happen in the build section:

Use the handy bundler application to install all the required gems
Also install bundler itself
Work around other issues as found

Once that is done, we have a nice spec file that can be built and then installed!

rpmbuild

Now we build our RPM. In this example I’m building a RoR application called special_collections. rhel6b is my RHEL6 build server/environment.


[curtis@rhel6b SPECS]$ rpmbuild -ba special_collections.spec 
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.J1hbLc
+ umask 022
+ cd /home/curtis/rpmbuild/BUILD
+ rm -rf ./special_collections
+ git clone https://code.example.com/git/special_collections
Initialized empty Git repository in /home/curtis/rpmbuild/BUILD/special_collections/.git/
SNIP!
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/curtis/rpmbuild/BUILDROOT/special_collections-0.1.4-1.el6.ualib.x86_64
Wrote: /home/curtis/rpmbuild/SRPMS/special_collections-0.1.4-1.el6.ualib.src.rpm
Wrote: /home/curtis/rpmbuild/RPMS/x86_64/special_collections-0.1.4-1.el6.ualib.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.VOkPMU
+ umask 022
+ cd /home/curtis/rpmbuild/BUILD
+ rm -rf /home/curtis/rpmbuild/BUILDROOT/special_collections-0.1.4-1.el6.ualib.x86_64
+ exit 0

NOTES:

The above rpmbuild could take a long time depending on the number of gems that the application requires. It’s important to rembember that in this process all the gems are being downloaded from rubygems.org and then also compiled on the build server, each and every time the rpm is built. So it’s slow. There are some things I’m looking at doing to reduce the time it takes to build the RPM, but that’s where it is right now. Maybe someone will read this blog and give me some comments on what I can be doing better!
The resulting RPM is quite large…in this case about 80MB compressed. This is because it has 100+ gems in it.

Installing the RPM on a brand new server

I have a brand new server all ready for this ruby application to be deployed. It’s a minimal install.


[root@RoR-TEST ~]# rpm -qa | grep -i "apache\|ruby\|passenger"
[root@RoR-TEST ~]# 
# Nothing! No ruby, passenger, or apache currently installed.
[root@RoR-TEST ~]# rpm -qa | wc -l
293
# And only 293 RPMs!

Normally I install a RPM from a custom yum repository, but in this example I will use yum localinstall so I copy the RPM from the build server to the new server.

Note that I have several 3rd party repositories configured on this server, including epel, rpmforge, and the passenger repository. Obviously one has to trust a 3rd party repository to use it. Configuring yum priorities might be a good idea as well to try to avoid unwanted collisions.

So, to install:


[root@RoR-TEST tmp]# yum localinstall special_collections-0.1.4-1.el6.ualib.x86_64.rpm 
SNIP!
 rubygem-passenger-native-libs  x86_64  1:3.0.11-1.el6_1.8.7.352   passenger                                       29 k
 rubygem-rack                   noarch  1:1.1.0-2.el6              epel                                           446 k
 rubygem-rake                   noarch  0.8.7-2.1.el6              optional                                       403 k
 rubygems                       noarch  1.3.7-1.el6                optional                                       206 k
 sgml-common                    noarch  0.6.3-32.el6               base                                            43 k

Transaction Summary
========================================================================================================================
Install      73 Package(s)

Total size: 234 M
Total download size: 74 M
Installed size: 413 M
Is this ok [y/N]: y
SNIP!
  rubygem-daemon_controller.noarch 0:0.2.6-1.el6                   rubygem-fastthread.x86_64 0:1.0.7-2.el6             
  rubygem-passenger.x86_64 1:3.0.11-1.el6                          rubygem-passenger-native.x86_64 1:3.0.11-1.el6      
  rubygem-passenger-native-libs.x86_64 1:3.0.11-1.el6_1.8.7.352    rubygem-rack.noarch 1:1.1.0-2.el6                   
  rubygem-rake.noarch 0:0.8.7-2.1.el6                              rubygems.noarch 0:1.3.7-1.el6                       
  sgml-common.noarch 0:0.6.3-32.el6                               

Complete!

Configure the application

Currently the RPM will create a directory in /etc/ that contains the database.yml file for the rails app:


[root@RoR-TEST special_collections]# pwd
/etc/railsapps/special_collections
[root@RoR-TEST special_collections]# ls
database.yml

Edit that to set the proper database information.

Configure apache

Now that apache has been installed because it is required by the custom RPM it needs to be configured.

First let’s make sure it’ll start on a reboot. Don’t want to have to login on the weekend three months from now after a spontaneous reboot now do we? :)


[root@RoR-TEST yum.repos.d]# chkconfig httpd on

Now to setup the apache rails environment for this particular application. Note that in this case, we’re doing one RoR app per virtual host. It’s just easier for me because there are some variables that need to be set in the virtual host config file.

I also always configure a /etc/httpd/conf.d/vhost.d directory for virtual host files, and tell httpd to check there for *.conf files.


[root@RoR-TEST vhost.d]# grep vhost.d /etc/httpd/conf/httpd.conf 
Include conf.d/vhost.d/*.conf

The vhost config file looks like this:


[root@RoR-TEST vhost.d]# cat specialcollections.example.com.conf 
<VirtualHost *:80>
   ServerName specialcollections.example.com
   DocumentRoot /usr/share/railsapps/special_collections/public

   # Because of the way we're deploying rails apps, ie. by using bundler during the rpm
   # build process to install all the required gems into $RAILSAPP/$NAME/vendor/bundle/ruby/1.8
   # this has to be set here. Otherwise the app will not have the required gems to run.
   SetEnv GEM_HOME /usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8/
   <Directory /usr/share/railsapps/special_collections/public>
        Options -MultiViews
    </Directory>
</VirtualHost>

Startup apache:


[root@RoR-TEST vhost.d]# service httpd configtest
[root@RoR-TEST vhost.d]# service httpd start

Done with apache.

Rake

Now to configue the initial database.

First, the paths need to be setup. I create a file called special_collectionsrc that has path information setup. Note that this rc file is someting I created specifically for this application because each rails app will have it’s own paths and gems. Then, when wanting to use rake with the specific application that file is sourced to ensure the correct rake and other gems are used.


[root@RoR-TEST ~]# which rake
/usr/bin/rake
# oops not the right one!
[root@RoR-TEST ~]# which bundle
/usr/bin/which: no bundle in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
# oops isn't on the path!
[root@RoR-TEST ~]# cat special_collectionsrc 
#!/bin/bash
export GEM_HOME=/usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8
PATH=/usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8/bin:$PATH
export RAILS_ENV=production

Once that file is sourced, we should be able to find rake on the path:


[root@RoR-TEST ~]# source special_collectionsrc 
[root@RoR-TEST ~]# which rake
/usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8/bin/rake
[root@RoR-TEST ~]# which bundle
/usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8/bin/bundle

cd to /usr/share/railsapps/special_collections/ and load the db:


[root@RoR-TEST special_collections]# rake db:load
/usr/share/railsapps/special_collections/vendor/bundle/ruby/1.8/gems/curb-0.7.16/lib/curb_core.so: warning: already initialized constant CURL_SSLVERSION_DEFAULT
-- create_table("collections", {:force=>true})
   -> 0.4194s
-- create_table("gallery_images", {:force=>true})
   -> 0.0040s
-- initialize_schema_migrations_table()
   -> 0.0077s
-- assume_migrated_upto_version(20111104163654, ["db/migrate"])
   -> 0.0048s

Whenever working with this particular RoR app the rc file should be sourced.

Done raking.

That’s…it

At this point the rails app should be available at the virtual host URL that was configured in the vhost. :)

While it’s a long process to get that intial spec file and rpmbuild working, once it’s done the application can be deployed in a few minutes, and now the developers can simply worry about commiting and tagging code, and let the sysadmin deal with deploying the actual application in a replicable manner. Of course there will be some back and forth, new gems might not compile, etc, but the general structure is in place. Further, the deployment is quite automatable—a new tag could mean a new RPM build and deployment to test.

Feel free to comment below with any corrections/issues, and thanks,
Curtis.

Quickly building command line apps to create files from templates in python

2011-10-13T00:00:00-07:00

Quickly building command line apps to create files from templates in python

13 October – 2011 – Edmonton

First, let me say that I am not an expert in Python. :) That said, I have been working on a script for a while now that creates a kickstart file from a command line application that uses:

a configuration file to get defaults,
command line options to add or override options, and
cheetah template files.

I think I finally have a good system for doing this, and could see it being useful to others in terms of writing their own quick application to generate some kind of text file from a command line application using the above three points, and what I have done is commited a set of skeleton files so that you could do this yourself very easily.

Dependencies

cli-template-generator was written to run on RHEL5 which means python 2.4.3.

It also requires python-argparse and python-cheetah if you are on RHEL5.

I think it will work in later python versions.

Using the cli-template-generator

First, clone the cli-template-generator repository.


$ git clone git@github.com:curtisgithub/cli-template-generator.git
Cloning into cli-template-generator...
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 8 (delta 0), reused 8 (delta 0)
Receiving objects: 100% (8/8), done.
$ cd cli-template-generator/
$ ls
README  skeleton.conf  skeleton.py  skeleton.tpl

There are three files that work together to create a text file from a template: skeleton.{conf,py,tpl}.

The tpl file contains the Cheetah template information:


$ cat skeleton.tpl 
Hello $hello

Which means the hello variable will be replaced with the string it’s set to in either the config file or the command line argument.

The conf file holds the default configuration option(s).


$ cat skeleton.conf 
[default]
#
# Add default configuration options in this file
# Eg.
# Key:	Value

hello:	World!

By default, the skeleton.conf file sets the variable hello to the string World!. So if we run skeleton.py we will see:


$ ./skeleton.py 
Hello World!

Also by default we have one command line configuration option, --hello, that we can set to whatever we want. We can see what options are available using --help.


$ ./skeleton.py --help
usage: skeleton.py [-h] [-c CONFIGFILE] [--hello HELLO]

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIGFILE, --config-file CONFIGFILE
                        Use a different config file than ./skeleton.conf
  --hello HELLO         Who are you saying hello to?

So if we run skeleton.py with the --hello option we should get different results.


$ ./skeleton.py --hello Curtis!
Hello Curtis!

So, as you can see, it should be fairly easy to copy the skeleton files, replace some of the default locations for the files, add some configuration options to the configuration file and also to the parser in the py file, and edit the template so that it uses your new variables.

Bam! You have a custom cli text file generator! Given the amount of text files in Unix/Linux, there could be a lot of good uses for this.

If you see any mistakes or things that could be improved, please feel free to contribute back via github.

Thanks,
Curtis.

Getting the number of commits in mercurial, git, and svn

2011-09-28T00:00:00-07:00

Getting the number of commits in mercurial, git, and svn

28 September – 2011 – Edmonton

This is a short post on one way to get the number of commits in hg, git, and svn.

One of the things that I wanted to graph with Cacti is the number of commits that happen in our git, mercurial, and svn repositories. Yup we use all three.

I know that the number of commits isn’t the best metric in terms of figuring out how much our repos are being used, but it’s certainly one of the numbers to look at, and it’s easy to start with. I’m aware of things like churn in hg, but haven’t looked into them fully. Obviously one could make one large commit, or many smaller ones. I prefer many smaller ones, but that’s just me. Basically I’m saying I’ll add more metrics later.

In order to graph the number of commits, I need to find the number of commits.

At this point I’m most interested in the number of commits that happened in the last 24 hours, one day ago, or yesterday, which I’m aware are not necessarily all the same thing. ;) I’ll run the cronjob that checks commits just after midnight, so the numbers should be kinda accurate.


$ cd some_hg_repo
$ hg log --template '{rev}:{node|short}\n'  --date -1 | wc -l

git


$ cd some_git_repo
$ git log --since="24 hours ago" | grep "^commit" | wc -l

svn


$ cd some_svn_repo
# and SVN_REPO=`pwd` or something like that
# Where YESTERDAY=`date --date yesterday +\{\%Y-\%m-\%d\}`
$ svn log -q -r $YESTERDAY file:///$SVN_REPO | grep "^r" | wc -l

Note that with svn, if there hasn’t been a commit in since YESTERDAY it will return that last commit before that—could be two days ago or more—so unless there are no commits, the number of commits will be at least one, which may not be what you are expecting.

Thanks,
Curtis

OpenBSD pf and set limit states

2011-09-12T00:00:00-07:00

OpenBSD pf and set limit states

12 September – 2011 – Edmonton

So you have a OpenBSD firewall. Actually you have at least two because you are doing carp for high availability (not load balancing but HA), right?

Awesome! It’s fun isn’t it? I suppose I have to admit it’s more fun testing it in a lab environment than in production. :)

One thing I noticed when doing a bit of non-scientific load testing on a pair of small carped firewalls is that in OpenBSD the size of the state table is limited to 10000 entries by default. I would imagine that most people won’t run into the limit, but I was surprised at how easy it was to hit 10000 sessions using something like siege.

Using a client laptop—an older core 2 duo with 4 gigs of ram and a 60 gig SSD drive (a Lenovo T61 specifically) on one side of the firewall and a virtualized web server with 512MB of RAM and one CPU on the other—I was able to hit the state limit in a couple of seconds with a command such as:


laptop$ siege -b -c 40 -r 100 http://testserver/testpage

where the resulting test page looks like:


<html>
<body>
hi there
</body>
</html>

When the siege command is run I watch the state tables on the OpenBSD firewalls with a command such as:


openbsd_fw{1,2}#  while true; do pfctl -s info; sleep 1; done

With nothing happening the result of that command looks about like this:


openbsd_fw1$ pfctl -s info
Status: Enabled for 0 days 00:22:05              Debug: err

State Table                          Total             Rate
  current entries                       38               
  searches                          215881          162.9/s
  inserts                            30364           22.9/s
  removals                           46342           35.0/s
Counters
  match                              30804           23.2/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                362            0.3/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s

We can see that the current limit is 10000:


openbsd_fw1$ pfctl -sm     
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

So lets fire up that siege command and see what happens by watching the current entries on the firewall that has the master carp IP. (Note that with pfsync all the states will be transferred to the backup firewall as well, but for simplicity let’s focus on the master.)


openbsd_fw1$ ifconfig | grep -i master
        carp: MASTER carpdev fxp2 vhid 1 advbase 1 advskew 0
        status: master
        carp: MASTER carpdev fxp0 vhid 2 advbase 1 advskew 0
        status: master
openbsd_fw1$ while true; do pfctl -s info | grep "current entries"; sleep 1; done
  current entries                       17               
  current entries                       15               
  current entries                       13               
  current entries                       12               
  current entries                       12 
# siege starts up here              
  current entries                     4820               
  current entries                    10000               
  current entries                    10000               
  current entries                    10000               
  current entries                    10000               
  current entries                    10000 
^C

So you can see it only takes a couple seconds to hit that limit.

Let’s up it to 200000 and see what happens.


openbsd_fw1$ grep "set limit" /etc/pf.conf
set limit states 200000
openbsd_fw1$ pfctl -nf /etc/pf.conf
openbsd_fw1$ pfctl -f /etc/pf.conf
openbsd_fw1$ pfctl -sm
states        hard limit   200000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

Run the siege command on the client:


openbsd_fw1$ while true; do pfctl -s info | grep "current entries"; sleep 1; done 
  current entries                       40               
  current entries                       40               
  current entries                       40               
  current entries                       40  
# siege starts up here             
  current entries                      560               
  current entries                     6686               
  current entries                    12480               
  current entries                    17728               
  current entries                    23060               
  current entries                    27116               
  current entries                    28332               
  current entries                    29498               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
  current entries                    29884               
^C

And looks like we max out the siege command now at about 30k sessions. Nice!

To conclude, this was just a quick look at session limits on OpenBSD. If you’re running a pf firewall it may be something to consider looking at to make sure you’re not hitting the limit which would reduce the effectiveness of your firewall.

Note that I haven’t shown any memory usage from the firewall, but the small boxes have 512MB of RAM and even at 200K sessions the memory usage only went up very slightly so I don’t think it’s constrained for memory reasons.

Thanks,
Curtis.

Cacti, Better Cacti Graphs, and SSH Original Command

2011-09-12T00:00:00-07:00

Cacti, Better Cacti Graphs, and SSH Original Command

12 September – 2011 – Edmonton

So you’re at the point where you want to monitor your servers performance. Actually let’s take it a step further and you want to do this with Cacti. Actually let’s take another step and say that you’re going to use Cacti and Better Cacti Graphs (BCG)…and ssh.

We’re getting pretty specific here aren’t we?

So you setup Cacti, RPM up old BCG, and you configure ssh logins for BCG so that it can grab stats off the client system.

But, and here’s the kicker—you want to make sure that you limit what the cacti user can do with a ssh login. (Possibly because your security analyst wants it to be “more secure” and yet has never actually used ssh.)

Well, there is fairly little known functionality built into OpenSSH that will allow you to lock down what the user can do with a ssh key based login. The best thing to do at this point to learn more about this functionality would be to google SSH_ORIGINAL_COMMAND. Sorry, can’t help but use lmgtfy.com, no offense intended. :) It’s just fun to say.

The point is that there are many (little known) options that can be put in a authorized_keys file to limit what the user can do with they login with that key.

eg. The beginning of the key that I put in the cacti user’s authorized_keys file on the client server:


command="/usr/share/cacti-ssh-auth/ssh_commands_check.sh",from="SOME_IP_ADDRESS",
no-port-forwarding,no-X11-forwarding ssh-dss SNIP_REST_OF_KEY!

What this setting does is:

Only allows the user to run the script that comes after command, meaning they can only run ssh_commands_check.sh, and it runs by default.
Only allow logins from SOME_IP_ADDRESS (eg. 10.0.4.30 or something), ie. the monitoring server where cacti is installed. Authentication via IP addresses isn’t the best idea, but why not.
Disable port forwarding and X11 forwarding for the session, always.

The contents of the ssh_commands_check.sh script look like this:


#!/bin/sh

# When using $SSH_ORIGINAL_COMMAND to watch what commands we get out
# of better-cacti-graphs, this is what we see:
# cat /proc/diskstats
# cat /proc/stat
# wget -U Cacti/1.0 -q -O - -T 5 "http://localhost/server-status?auto"
# uptime
# free -ob

case "$SSH_ORIGINAL_COMMAND" in
        'cat /proc/diskstats')
                cat /proc/diskstats
                ;;
        'wget -U Cacti/1.0 -q -O - -T 5 http://localhost/server-status?auto')
                wget -U Cacti/1.0 -q -O - -T 5 "http://localhost/server-status?auto"
                ;;
        'uptime')
                uptime
                ;;
        'free -ob')
                free -ob
                ;;
        *)
                # Then essentially do nothing b/c only the above
                # commands are allowed to run. :)
                # I don't really want to echo the actual command
                # until I can find some way to escape anything 
                # malicious. For another day!
                logger -i "$0 ERROR: disallowed command attempted"
                ;;
esac

Great you say. But how did you find out exactly what commands cacti is trying to run? Well SSH_ORIGINAL_COMMAND to the rescue!

I used something like:


command="echo $SSH_ORIGINAL_COMMAND >> /var/tmp/ssh_check_cmd.txt; $SSH_ORIGINAL_COMMAND",
from="SOME_IP_ADDRESS",no-port-forwarding,no-X11-forwarding ssh-dss SNIP_REST_OF_KEY!

and then tail -f /var/tmp/ssh_check_cmd.txt and saw this:


client_server$ tail -f ssh_orig_cmd.txt 
cat /proc/diskstats
cat /proc/stat
wget -U Cacti/1.0 -q -O - -T 5 "http://localhost/server-status?auto"
uptime
free -ob

so we can be fairly sure that those are the commands the cacti monitor server is asking the client to run.

NOTE that I would suggest checking what commands the ssh cacti scripts are running and not blindly using the ones I put above because things could have changed since I put up this post. And in fact could be completely wrong. It will take a bit of time to figure this out.

Also you can review the code for BCG’s use of ssh to find out exactly what commands are running, but note that bash/ect might interpret things differently, so it’s best to check with SSH_ORIGINAL_COMMAND. :)

Thanks and good luck,
Curtis.

Packaging code is about sharing

2011-08-18T00:00:00-07:00

Packaging code is about sharing

18 August – 2011 – Edmonton

I have been packaging code into RPMs for internal use for about five years now. I’m not going to say I’m an expert at it—I’m still learning—but I’m getting better at it and I think that’s a good thing.

Recently I packaged a piece of software that didn’t have an existing RPM (AFAIK). I was then able to share that RPM, and installation instructions, with a partner institution so that they could easily install the software as well—in fact with one command. :)

From one perspective it’s obvious that packaging software is about sharing. But often I spend so much time just getting the RPM built, which can mean the gruelling process of pulling requirements out of developers who think packaging is…not important or even intrusive, that I forget how it’s not just about easing sysadmin maintenance of servers; that it’s about being able to share systems and software with peers.

I often wonder where some Linux users think packages come from. Certainly not a big white stork in the middle of the night. Dedicated volunteers (and I don’t mean me) are building thousands of packages every day! So hat tip to all those volunteers. Licensing is important, but so is packaging. :)

Booting partitions bigger than 2TB on a HP DL160 G6 with RHEL5

2011-08-09T00:00:00-07:00

Booting partitions bigger than 2TB on a HP DL160 G6 with RHEL5

9 August – 2011 – Edmonton

Yesterday I was working on a HP DL160 G6 server. Originally it had two 160GB hard-drives, but of course, I wanted more storage…a lot more. :) So we ordered four 2TB drives to put in it. Then I realized the backplane would only support two drives, so I had to order a backplane that can support four drives.

Once all the parts arrived I replaced the backplane and put the four drives in. It was fairly simple actually. Then when I booted the server with the new backplane and disks the P410 RAID card noticed the new drives, and suggested configuring RAID 1+0, a suggestion I accepted. That leaves me with about 4TB usable.

By default, the system creates one large drive of 4TB, which Redhat Enterprise 5 sees as /dev/sda.

However, mbr, the default partition type on RHEL5, cannot boot partitions larger than 2TB. So, after the first install via kickstart I was missing 2TB. Not cool! Well, actually it’s fine, the computer did what it was told, but I wanted to use the rest.

The solution? It was actually fairly easy. Maybe too easy. But it’s working.

Because, it seems, the HP BIOS in this server supports UEFI/EFI/GPT/whatever, in the %pre section of the kickstart we can create a gpt partition.


%pre
# B/c sda on this server is 4.0TB we need to try to use gpt instead of mdr.
/usr/sbin/parted -s /dev/sda mklabel gpt

Also make sure that if you have a clearpart command in your kickstart to comment it out or delete it.


# Removing b/c of the parted in %pre
#clearpart --drives=sda --all --initlabel

Then run your kickstart as usual and hopefully your system will boot with whatever partitions you configured. In my case, I created two partitions, one 60GB for the system and the rest for virtual machines, and placed logical volumes over top:


4TB_RAID10_SERVER$ pvs
  PV         VG     Fmt  Attr PSize  PFree 
  /dev/sda2  system lvm2 a-   58.56g 36.66g
  /dev/sda3  vm     lvm2 a-    3.58t  3.58t

And running parted we can see that it is indeed a gpt layout:


4TB_RAID10_SERVER$ parted /dev/sda print
Model: HP LOGICAL VOLUME (scsi)
Disk /dev/sda: 4001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  525MB   524MB   ext4               boot
 2      525MB   63.4GB  62.9GB                     lvm
 3      63.4GB  4001GB  3937GB                     lvm

So now I can create that rsync backup server I’ve always wanted, and I have up to 3.58TB to store the backups on. Good times and I’m glad it all worked out. Now if only they were 3TB drives… ;)

ksplice bought out by Oracle, RHEL desupported

2011-07-21T00:00:00-07:00

ksplice bought out by Oracle, RHEL desupported

21 July – 2011 – Edmonton

I am extremely saddened to see that ksplice has been bought out by Oracle and will no longer be supporting Redhat Enterprise. I can only hope that the Linux community can find a way to provide this service outside of the limited confines of Oracle Corp.

Basic infrastructure to support production linux servers

2011-07-20T00:00:00-07:00

Basic infrastructure to support production linux servers

20 July – 2011 – Edmonton

Every IT group providing Linux servers will require some infrastructure services. What I mean by that is that there are services that Linux sysadmins need that help them to run their servers in a efficient, scalable way.

Production services need support from infrastructure services.

This post lists a basic collection of services that a Linux sysadmin might require. For me, this would be the minimum services required to run a group of Linux servers, be it 10 or 1000.

Notes:

I mention specific solutions, but there are many different ways to obtain the same basic infrastructure.
Most of what I discuss below is geared towards Redhat servers, but is also completely applicable to any Linux disto that has a packaging system, which is pretty much all of them.

1. Write documentation with mediawiki

One of the most important things a sysadmin does is document what they did so that they can take a relaxing, rejuvenating vacation. By this I mean that they have documented their systems so that the other admins taking over can use it to fix things, to understand what is installed, where it is, what it does, who owns it, ect, when the primary sysadmin is unavailable because he/she is in downtown Tokyo and is having a hard time connecting to the local wireless because they can’t read Japanese.

Documentation is also important so you can remember what you did six months ago to get iscsi working, or how to configure a xen dom0 to use a bridge that comes from a vlan, or that command to dd a logical volume from one server to another over ssh without having to figure it out all over again. I suppose you could blog about it too. :)

Regardless of what is being documented, usually the system I will use is a Mediawiki instance. Mediawiki supports searching (also full text searching if you use sphinx and the sphinx search plugin so that you can search in pre tags too), file uploads, categories, and many, many other features, especially via the diverse plugin/extension community.

When people don’t want to use mediawiki I ask them what they want to do, how they want to work, and inevitably mediawiki can do it.

2. Serve packages using mrepo

Most Linux distros use some form of package management to install software. For example, in Debian/Ubuntu a deb file and in Redhat it’s an rpm.

These packages, somewhat akin to an advanced zip file, contain all the files, requirements, metadata, ect, for a particular piece of software or service. The ./configure; make; make install dance is only done on the build servers.

In most of the environments I work in, we strive to package all code. This means that all software and applications are installed on a server in a package. Configuration can be done by hand, or via a centralized configuration management system, but the code is actually installed via a package. This goes for Perl CPAN modules too. :) EVERYTHING.

The package server provides a central spot that the servers obtain packages from, eg. a mrepo server.

The packages server will do three major things:

Download RPM packages from external repositories, eg. the official Redhat repositories, EPEL, RPMForge, ect.
Also it will allow us to store our own custom packages.
This means our servers don’t go to the internet to get updates, they go to the packages server.

Serve those packages to all of our Linux servers.

Allow for the ability to “freeze” the repositories so that we can install software updates in test environments, test them, and then install the exact same version of the packages/software in production, thus being a somewhat more sure that everything is going to work OK after the update.

3. Centralize syslogging with rsyslog

There should be a central syslog server somewhere in your infrastucture, and all production servers should send syslog packets to that server.

This will allow for a central spot for gathering syslog messages from all servers. We can then run scripts/processes to analyze these logs looking for issues.

Also, should a server get hacked the first thing malicious users usually do is (try) to delete logs. But, if the logs have been sent to a central log server then they (probably) can’t do that.

I usually replace, when possible, syslog with rsyslog and at minimum use TCP delivery instead of UDP.

4. Centralize root email with sendmail or postfix and dovecot

There are many scripts on servers that will send email to root if something breaks, eg. cron or logwatch. So all of these emails should be sent to a central email address that sysadmins have access to.

I’m not a huge fan of what is essentially logging over email, but it’s nearly impossible to avoid.

Currently I configure each server’s root alias to email a central address which is delivered to a Maildir and serve that up over IMAPS with dovecot. Dovecot is pretty great.

(Note that while sendmail is default on RHEL5, postfix is default on RHEL6!)

5. Serve kickstarts over http

The kickstart server is simply a plain http server that serves kickstart files, and in conjunction with the packaging server allows for rapid, repeatable installation of Redhat/CentOS. Or you could do it over nfs, or pop it on the USB key. I always use a web server to serve up the kickstarts.

ks=http://example.com/ks/newserver.ks if you know what I mean. :)

6. Centralize configuration management

This is a central server that can be used to access all the Linux/Unix servers via ssh and to try automate with custom scripts or things like pdsh, chef, puppet, fabric, ect.

Also stores a copy of every Linux servers RPM database so that if a server gets hacked you can check what files have changed, if any, on the hacked server. (Though that should be in backups too.)

7. Build packages with rpmbuild

To create custom RPM packages a build server for each OS and architecture is required. So if you run RHEL5 and RHEL 6 on x86_64 then you’ll need two build servers, one for each OS and arch.

This is where rpmbuild -ba some.spec will be run to build a RPM. Then the RPM will be copied to the packaging server.

8. Login from anywhere securely with ssh

(Of course with ssh!)

Linux/Unix admins should be able to access a ssh gateway from the Internet to be able to access servers and possibly their workstations or just the central management server.

Only public key authentication would be allowed to this server (meaning no password based authentication) which makes it very secure in terms of auth.

Some workplaces will want to put this behind a commercial VPN. Try to avoid this…IMHO ssh is one of the, if not the, most secure network applications on the planet—certainly better than some million SLOC ssl “vpn”.

PS. Did you know you can create a ssh-based vpn with something like sshuttle? If you did sysadmin 2pts for you! :)

9. Monitor uptime with nagios

Essentially a Nagios server, or similar, that monitors production services and servers and will let you know when they go down. I have also used hobbit, which is apparently called Xymon now.

(But they won’t go down, right? In fact, the monitoring system will go down more than the production serivces, won’t it. :) I’ve always thought that was the hard part of uptime monitoring.)

10. Manage code with version control systems

Every IT workplace should have a central code repository. Sure, hg and git are distributed, but I still think it’s nice to have a central location. Github seems to be successful at centralizing distributed revision control, so I think centralizing a local git or hg instance (with hgweb.cgi for example) will work for most workplaces as well. :)

I have a github account and have used svn and hg at work. Use whatever you want—just do it now!

11. Backup with rdiff-backup

This would be a server used to rsync snapshot-type backups over ssh to disk for rapid restores. I like rdiff-backup

It would likely compliment that huge Netbackup, or Commvault (calmvault?), or other backup system you have that doesn’t work half the time and requires the installation of a gigantic root-running tarball or monolithic 600MB RPM which doubles the size of your 500MB minimal Redhat OS install. Good times in commercial backup land.

12. Test with jmeter

If you have dev, test, production environments then you should also automate testing. This would be the service that helps you do that, running Jmeter and/or Selenium for example; do everything from one location, perhaps when a RPM/package is built then it’s automatically tested from here. Wouldn’t that be nice!?

13. Monitor performance with cacti

Everyone wants performance. Or do they? How does one know how much performance one needs? You have to do performance testing, and you have to monitor performance. You can’t just buy performance, you have to put some work in.

That said, for monitoring performance I usually use either Nagios or something like Cacti using snmp with some custom scripts.

14. Secure remote bios level access via a remote KVM or IPMI

By this I mean the secure network you attach your remote KVM to the hardware dom0 servers, and/or to the IPMI interfaces that most teir 1 servers come with (not that I endorse only tier 1 vendors) so that you don’t have to hang out in that cold, loud, unfriendly server room.

Using fusion-io drives on Redhat Enterprise 5

2011-06-27T00:00:00-07:00

Using fusion-io drives on Redhat Enterprise 5

27 June – 2011 – Edmonton

Update: Please note that this post is getting a bit old. Currently I am running these IBM FusionIO drives on RHEL 6. I’ll be posting about that and a few other PCIe-SSD subjects in the next short while. – 24 Apr 2012

FusionIO IODrive Overview

So at work we have a rather large IBM x3850 x5 server. It has 4 sockets each with six cores and hyperthreading (not that I’m necessarily a fan of hyperthreading—really I haven’t done enough research to make up my mind) which ends up with RHEL5 seeing 48 CPUS.


$ cat /proc/cpuinfo | grep proc | wc -l
48

Fun.

But the important part of this post is that this server also has three 640GB fusion-io drives which I have installed and configured as a volume group called fio


$ ls /dev/fio
fio/  fioa  fiob  fioc  fiod  fioe  fiof  
$ vgs fio
  VG   #PV #LV #SN Attr   VSize VFree
  fio    6   4   0 wz--n- 1.76T 1.08T

and where the fio[a,b,c,d,e,f] are the drives, with each 640 gig card actually appearing as 2 320 gig disks.


$ dmesg  |grep -i "found device"
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:89:00.0: Found device 0000:89:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:8a:00.0: Found device 0000:8a:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:93:00.0: Found device 0000:93:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:94:00.0: Found device 0000:94:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:98:00.0: Found device 0000:98:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:99:00.0: Found device 0000:99:00.0

Resources

The most important resource for using these FusionIO drives is the official knowledge base which has several articles specifically for linux. I would suggest reading all of them. :)

Install

Once the cards were put into the server, which is somewhat harrowing given their individual cost, and the server was booted, the software drivers that were downloaded from the IBM website were installed. This server runs RHEL5


$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

as that RHEL version that is what IBM supports for drivers.


$ rpm -i iodrive-driver-1.2.7.5-1.0_2.6.18_164.el5.x86_64.rpm \
iodrive-firmware-1.2.7.6.43246-1.0.noarch.rpm \
iodrive-jni-1.2.7.5-1.0.x86_64.rpm \
iodrive-snmp-1.2.7.5-1.0.x86_64.rpm \
iodrive-util-1.2.7.5-1.0.x86_64.rpm \

Currently I am using the drivers as they were downloaded, which means using a specific matching kernel to match. The drivers do come with a source RPM so that you can rebuild them for your latest kernel, but I have opted not to do that yet. So install the matching kernel


$ yum install kernel-2.6.18-164.el5

and reboot.

However, I am also using the amazing ksplice service to ensure that depsite the fact that I am using a rather old kernel to match the FusionIO drivers that the kernel is still up to date in terms of security issues:


$ uptrack-uname -r
2.6.18-238.12.1.el5
$ uname -r
2.6.18-164.el5

The uptrack-uname -r command asks uptrack what security equivalent version of the kernel is. Great stuff that kslplice.

Once the drivers are installed we can load the modules


$ modprobe fio-driver

and now we can see the drives


$ ls /dev/fio*
fioa  fiob  fioc  fiod  fioe  fiof

and at this point we can configure the drives.

Worker processes

Once the drivers are installed there is a /etc/init.d/iodrive startup script. One of the things this script does is startup some worker processes which I believe are used to move data around the FusionIO drives to ensure their performance and longevity.


$ chkconfig --list iodrive
iodrive 0:off	1:on	2:on	3:on	4:on	5:on	6:off


$ ps ax | grep worker
 5271 ?        S<   1169:51 [fct0-worker]
 5588 ?        S<   1168:07 [fct1-worker]
 5593 ?        S<   359:01 [fct2-worker]
 5598 ?        R<   206:02 [fct3-worker]
 5603 ?        S<   203:15 [fct4-worker]
 5608 ?        S<   203:12 [fct5-worker]
20921 pts/2    S+     0:00 grep worker

These processes will take up some CPU time. Frankly, because there are 48 CPUs in this server, using up one to run these workers is OK. But it was a little confusing at first seeing all this activity—one worker process for each card.

Configuration

Given that we are going to manage the FusionIO drives via LVM, we will need to configure LVM to allow it. See this knowledge base article.


$ grep fio /etc/lvm/lvm.conf
    types = [ "fio", 16 ]

Then add each /dev/fio* drive as a phyical volume and then add them to a volume group.


$ pvs | grep fio
  /dev/fioa  fio    lvm2 a-   300.31G 320.00M
  /dev/fiob  fio    lvm2 a-   300.31G 100.31G
  /dev/fioc  fio    lvm2 a-   300.31G 100.31G
  /dev/fiod  fio    lvm2 a-   300.31G 300.31G
  /dev/fioe  fio    lvm2 a-   300.31G 300.31G
  /dev/fiof  fio    lvm2 a-   300.31G 300.31G
$ vgs fio
  VG   #PV #LV #SN Attr   VSize VFree
  fio    6   4   0 wz--n- 1.76T 1.08T

fio-status

Useful way to check the status of the FusionIO drives.


$ fio-status

Found 6 ioDrives in this system with 3 ioDrive Duos
Fusion-io driver version: 1.2.7.5

Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:59518
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct0:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77479
	  fct1:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77478

fct0	Attached as 'fioa' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77479
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:59518
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 56.6 degC, max 59.6 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct1	Attached as 'fiob' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77478
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:59518
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 61.0 degC, max 63.0 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%


Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:59507
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct2:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77143
	  fct3:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77144

fct2	Attached as 'fioc' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77143
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:59507
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 62.0 degC, max 65.5 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct3	Attached as 'fiod' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77144
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:59507
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 64.0 degC, max 66.4 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%


Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:100366
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct4:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77344
	  fct5:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77345

fct4	Attached as 'fioe' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77344
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:100366
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 68.9 degC, max 71.9 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct5	Attached as 'fiof' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77345
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:100366
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 63.0 degC, max 66.0 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

XFS

Prior to finding out about the official knowledge base, I had decided to purchase a subscription from Redhat for the XFS file system. Then, upon reading this kb article, I found that they heavily recommend XFS as the file system to run on top of a FusionIO drive


XFS is currently the recommended filesystem. It can achieve up to 3x 
the performance of a tuned ext2/ext3 solution. At this time, there is 
no know additional tuning for running XFS in a single- or multi-ioDrive 
configuration

so that is the file system we use.


$ mount | grep fio
/dev/mapper/fio-vault1 on /var/lib/vault1 type xfs (rw)

Mounting drives after a reboot

I’ll admit I hadn’t thought of this during the initial installation. After a few days we moved the server to a new location which thus required a power down and restart.

While the server was restarting, and I was standing in the cold, loud server room because the new room didn’t have any networking for IPMI (which is not good), I noticed it took a very long time to get past the udev portion of the boot, and in fact the FusionIO drives failed to mount from fstab. Of course there is a logical reason for that—read about it here.

Because we are using the 1.2 driver, I followed the straight forward instructions here.

Performance testing

Performance testing is hard. Maybe it’s just me. But testing superdisk like these FusionIO drives on a server with 48 CPUS and 64 gigs of main memory is not easy. Again I will admit I took a shot at benchmarking the FusionIO disk having not read the kb. I messed around with Bonnie++, io-whatever, but nothing quite came out right, partially because I didn’t put a lot of time into it, and because the server has so much memory that it makes it hard to beat the cache (I did try to reduce the memory the OS could see via kernel configuration, but didn’t have a lot of luck with that).

Finally I read this kb article which suggested using the fio utility (which I don’t believe is a utility put out by FusionIO, rather just aptly named).

The fio tool is not in the RHEL repositories but it is in rpmforge/repoforge.


$ cd /var/tmp
$ wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
$ rpm -Uvh rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
$ yum repolist | grep forge
rpmforge                           RHEL 5Server - RPMforge.net - enabled: 10,636
$ yum search fio | grep -i benchmark
fio.x86_64 : I/O benchmark and stress/hardware verification tool

Here are a couple of example runs. Please note that at this point I do not know much about fio. Benchmarking disk is a highly technical thing to do, and getting tests right would take a lot of research and consideration, which I have not done.

It seems that the fio benchmark utility suports direct=1 which means use non-buffered-io, thereby skipping memory cacheing and going straight to the disk.


$ cat fio-randwrite.fio 
[randwrite[

direct=1
rw=randwrite 
bs=1m 
size=5G 
numjobs=4 
runtime=10 
group_reporting 
directory=/mnt/fio-test-xfs
$ fio fio-randwrite.fio 
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 1.55
Starting 4 processes
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 4 (f=4): [wwww] [100.0% done] [0K/522.8M /s] [0 /510  iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=28487
  write: io=4556.0MB, bw=466161KB/s, iops=455 , runt= 10008msec
    clat (msec): min=1 , max=1692 , avg= 9.83, stdev=22.04
     lat (msec): min=1 , max=1692 , avg= 9.84, stdev=22.04
    bw (KB/s) : min=  559, max=264126, per=24.79%, avg=115540.55, stdev=20377.90
  cpu          : usr=0.10%, sys=14.85%, ctx=59071, majf=0, minf=92
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/4556/0, short=0/0/0

     lat (msec): 2=0.53%, 4=1.27%, 10=97.17%, 20=0.59%, 50=0.09%
     lat (msec): 100=0.18%, 250=0.15%, 2000=0.02%

Run status group 0 (all jobs):
  WRITE: io=4556.0MB, aggrb=466161KB/s, minb=477349KB/s, maxb=477349KB/s,
  mint=10008msec, maxt=10008msec

Disk stats (read/write):
  dm-11: ios=0/158802, merge=0/0, ticks=0/55956241, in_queue=55915327, 
  util=66.05%, aggrios=0/159667, aggrmerge=0/0, aggrticks=0/55932489,
  aggrin_queue=55785218, aggrutil=65.96%
    fioc: ios=0/159667, merge=0/0, ticks=0/55932489, in_queue=55785218, 
    util=65.96%

And then a similar test using RAID10 SAS disk formated ext3.


$ cat fio-randwrite.fio 
[randwrite[

direct=1
rw=randwrite 
bs=1m 
size=5G 
numjobs=4 
runtime=10 
group_reporting 
directory=/mnt/sas-test
$ fio fio-randwrite.fio 
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 1.55
Starting 4 processes
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 4 (f=4): [wwww] [1200.0% done] [0K/0K /s] [0 /0  iops] [eta
 1158050441d:07h:00m:05sJobs: 4 (f=4): [wwww] [inf% done] [0K/0K /s] 
[0 /0  iops] [eta 1158050441d:07h:00m:04s]  Jobs: 4 (f=4): [wwww] 
[1300.0% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07h:00m:04sJobs: 
4 (f=4): [wwww] [inf% done] [0K/0K /s] [0 /0  iops] 
[eta 1158050441d:07h:00m:03s]  Jobs: 1 (f=1): [___w] [66.1% done] 
[0K/0K /s] [0 /0  iops] [eta 00m:19s]               
randwrite: (groupid=0, jobs=4): err= 0: pid=28586
  write: io=4096.0KB, bw=112369 B/s, iops=0 , runt= 37326msec
    clat (usec): min=12140K, max=37183K, avg=32696578.04, stdev= 0.00
     lat (usec): min=12140K, max=37183K, avg=32696579.88, stdev= 0.00
    bw (KB/s) : min=   27, max=   83, per=31.61%, avg=34.46, stdev= 0.00
  cpu          : usr=0.00%, sys=51.90%, ctx=9598, majf=0, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/4/0, short=0/0/0

     lat (msec): >=2000=100.00%

Run status group 0 (all jobs):
  WRITE: io=4096KB, aggrb=109KB/s, minb=112KB/s, maxb=112KB/s, 
  mint=37326msec, maxt=37326msec

Disk stats (read/write):
  dm-12: ios=128/4721384, merge=0/0, ticks=5582/602531980, in_queue=602926524,
  util=97.85%, aggrios=129/87424, aggrmerge=0/4634618, aggrticks=5631/10828734,
  aggrin_queue=10826088, aggrutil=98.01%
    sdb: ios=129/87424, merge=0/4634618, ticks=5631/10828734, in_queue=10826088,
    util=98.01%

That’s a pretty big difference: io=4556.0MB for the FusionIO drives versus io=4096.0KB for the SAS RAID10. I’m going to have to look into this more! :)

Thanks.

PS. I found this list of device bandwidths interesting.

Installing chef on Centos 5

2011-05-11T00:00:00-07:00

Installing chef on Centos 5

11 May 2011 – Edmonton

Mirroring the FrameOS RPM repository

Installing Chef is pretty easy given that FrameOS has created all of the RPMs for us. See this blog post for more information.

I mirror their repository on a local, centralized server using mrepo. Below is an example of my configuration.


[root@repos etc]# cd /etc/mrepo.conf.d/
[root@repos mrepo.conf.d]# cat chef.conf 
[chef]
# FRAMEOS builds RPMS for chefhere: 
# http://blog.frameos.org/2011/04/14/announcing-rbel-frameos-org/
#
# This might also be a good repo to use:
# - http://download.elff.bravenet.com/5/x86_64/
name = FrameOS rbel Chef RPMs $release ($arch)
release = 5
#arch = x86_64 i386
arch = x86_64
metadata = repomd repoview yum

### Additional repositories
chef = http://rbel.frameos.org/stable/el$release/$arch/

This repo is then available to my servers at http://repos/mrepo/chef-x86_64/RPMS.chef/.

Installing chef-server

Then I created a CentOS 5 virtual machine called chef-server. I enabled the repo I mention above on that server, and then ran:


[root@chef-server yum.repos.d]# yum install rubygem-chef-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * extras: ftp.telus.net
 * updates: repos
Setting up Install Process
Resolving Dependencies
--> Running transaction check
SNIP!

I added these iptables rules to /etc/sysconfig/iptables.


# Chef
# -- web interface
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 4040 -j ACCEPT
# -- chef-server
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 4000 -j ACCEPT
# -- amqp server
-A RH-Firewall-1-INPUT -m state --state NEW -m multiport -p tcp --dport 5672,4369,50229 -j ACCEPT
# -- search indexes (solr)
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT
# data store (couchdb)
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5984 -j ACCEPT

And then ran the setup script (after taking a look at what it does :) ).


[root@chef-server sbin]# setup-chef-server.sh
Checking RabbitMQ...
RabbitMQ not running. Starting...
Starting rabbitmq-server: SUCCESS
rabbitmq-server.
Configuring RabbitMQ default Chef user...

Starting CouchDB...

Starting couchdb:                                          [  OK  ]
Enabling Chef Services...

Starting Chef Services...

Starting chef-server:                                      [  OK  ]
Starting chef-server-webui:                                [  OK  ]
Starting chef-solr:                                        [  OK  ]
Starting chef-expander:                                    [  OK  ]

At which point I had a chef-server running, to which I can add clients and nodes.

Installing chef-client

Installing chef-client is also easily done with the provided rpms. I created another CentOS 5 virtual machine, called chef-client. (Actually I created many of them. It’s fun once you get it automated. :) )


[root@chef-client ~]# yum install chef-client

(Note that this may not be the preferred way to bootstrap a chef-client, but it has been working for me.)

Then create a client.rb file in /etc/chef, where chef-server.example.com is the fqdn of your chef-server and is accessible on port 4000 from your chef-client.


[root@chef-client2 chef]# cat client.rb 
log_level        :info
    log_location     STDOUT
    chef_server_url  'http://chef-server.example.com:4000'

Next, copy the validation.pem file from the chef-server to /etc/chef on the chef-client, likely using scp (or, do it when the server is built in a kickstart file :) ).


[root@chef-client2 chef]# ls
client.rb  validation.pem

Then start chef-client.


[root@chef-client2 chef]# service chef-client start

But in the /var/log/chef/client.log you will see an error that says client.pem is not present. This is good—chef-client will create the client.pem file.


# Logfile created on [Date] 1 by logger.rb/22285
 INFO: Daemonizing..
 INFO: Forked, in 1762. Priveleges: 0 0
 INFO: *** Chef 0.10.0 ***
 INFO: Client key /etc/chef/client.pem is not present - registering
 WARN: Failed to read the private key /etc/chef/validation.pem: #<Errno::ENOENT: No such file or directory - /etc/chef/validation.pem>
 ERROR: Chef::Exceptions::PrivateKeyMissing: I cannot read /etc/chef/validation.pem, which you told me to use to sign requests!
 FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
 ERROR: Sleeping for 1800 seconds before trying again
 FATAL: SIGTERM received, stopping

Now, restart chef-client so that new client.pem file can be used in conjuncation with the validation.pem file to register the node/client with the chef-server.


[root@chef-client2 chef]# service chef-client restart

As long as the client.pem is there, the validation.pem is there, and the networking is OK, you should connect:


 INFO: *** Chef 0.10.0 ***
 INFO: Run List is []
 INFO: Run List expands to []
 INFO: Starting Chef Run for chef-client2.example.com
 INFO: Loading cookbooks []
 WARN: Node chef-client2.example.com has an empty run list.
 INFO: Chef Run complete in 6.815418 seconds
 INFO: Running report handlers
 INFO: Report handlers complete
 FATAL: SIGTERM received, stopping
 INFO: Daemonizing..
 INFO: Forked, in 2032. Priveleges: 0 0

And now the client should appear in the node and client lists. (Note that I have not detailed how to add a user/client to the chef system, you’ll have to do that to use knife.)


[someuser@chef-server ~]$ knife node list
  chef-client2.example.com
SNIP!
[someuser@chef-server ~]$ knife client list
  chef-client2.example.com
SNIP!

(The SNIP!s mean I’ve removed some items for brevity.)

Finally, run chkconfig chef-client on on the chef-client to ensure the service starts at boot.

Installing chef-client from a kickstart file

When building new vms I install chef-client from a kickstart file. This is also easily done!

The first important option in the kickstart file is the repo option.


repo --name=chef --baseurl=http://your_repo_server/mrepo/chef-x86_64/RPMS.chef/

and then, in the %packages section simply add:


rubygem-chef

which will be installed from the chef repo configured in the repo option.

Also, enable the service:


services --enabled chef-client

Finally, in the %post section I add the below. Note the [PASTE THE CONTENTS OF YOUR VALIDATION.PEM HERE!!!] portion—that means put the results of cat /etc/chef/validation.pem there, not that actual phrase. :)


%post
# chef-client

if [ ! -e /etc/chef ]; then
        mkdir /etc/chef
fi

cat > /etc/chef/client.rb << EOCLRB
log_level        :info
    log_location     STDOUT
    chef_server_url  'http://chef-server.example.com:4000'
EOCLRB
chmod 600 /etc/chef/client.rb

cat > /etc/chef/validation.pem << EOVALPEM
[PASTE THE CONTENTS OF YOUR VALIDATION.PEM HERE!!!]
EOVALPEM
chmod 600 /etc/chef/validation.pem

When the server built from this kickstart boots chef-client will startup. However, it will fail the first time it starts up because the client.pem had to be generated. But, the next time it starts up it will connect to the chef-server and register. If you want it to register right away, then ssh into the server and run service chef-client restart and it should register.

Installing Jekyll on Ubuntu 10.04

2011-05-09T00:00:00-07:00

Installing Jekyll on Ubuntu 10.04

9 May 2011 – Edmonton

Another small post…given that I’ve recently changed jobs and thus have new workstation(s) to install and configure as I like them, and that I recently purchased a used Lenovo T61 laptop (which BTW is running very well on Ubuntu 10.04/Lucid, perhaps fodder for another post) I’ve repeated a several software installations lately on Lucid, including getting jekyll running locally so I can review blog posts before I send them up to github to run serverascode.com.

When you run gem install jekyll on Lucid, you will recieve this error message:


$ grep -i release /etc/lsb-release 
DISTRIB_RELEASE=10.04
$ gem install jekyll
ERROR:  Error installing jekyll:
	liquid requires RubyGems version >= 1.3.7

Lucid comes with gem 1.3.5 which is not the version that Jekyll’s gem requires. When searching for the error message I found this post which describes one way of getting jekyll running on lucid, which is to install the gem package from Ubuntu 10.10. Now, obviously installing a package from what essentially is a different version of Ubuntu isn’t usually a recommended way to go, it’s certainly a quick and easy one (duh! :) ). I downloaded the rubygems1.8_1.3.7-2 package and installed it. Then I was able to run gem install jekyll and then run jekyll:


$ jekyll --server --auto
Configuration from /curtisgithub.github.com/_config.yml
Auto-regenerating enabled: /curtisgithub.github.com -> curtisgithub.github.com/_site
[2011-04-20 13:20:25] regeneration: 12 files changed
[2011-04-20 13:20:25] INFO  WEBrick 1.3.1
[2011-04-20 13:20:25] INFO  ruby 1.8.7 (2010-01-10) [x86_64-linux]
[2011-04-20 13:20:30] INFO  WEBrick::HTTPServer#start: pid=7082 port=4000
SNIP!

It remains to be seen if I’ll run into issues with having the gem from Ubuntu 10.10 running on Ubuntu 10.04. I’ll update this post if I do. :)

Using LVM hosttags

2011-04-13T00:00:00-07:00

Using LVM hosttags

13 April 2011 – Edmonton

This is a somewhat minor post, but I thought it would be worthwhile to take a peek at using LVM hosttags to manage dom0 access to logical volumes on top of SAN LUNs because there doesn’t seem to be a lot of documentation on using hosttags online. Perhaps that’s because no one is doing it this way.

While it’s not my favorite way of managing SAN disks across servers (I like clvmd but it brings considerable complexity), hosttags are certainly one way to do it. Hosttags are a relatively simple method, and better than LVM filters IMHO. The point of using LVM hosttags is to ensure that only one server is ever writing to a SAN LUN at a time (unless you have something like GFS or similar in use, which is managing that process of multiple writers, and in that case what are you doing here? :) ).

In this example we have three Redhat Enterprise 5.x dom0 servers connected to a large—and expensive—storage area network. We’ll call them vmhost1, vmhost2, and vmhost3. The LUNs are provided to the vmhosts via the SANs configuration software. The vmhosts see them as regular disk, but they aren’t regular disk because each of the servers can see them, where see means read and write. (Note that I may be using terms incorrectly, but by SAN LUN I mean the slice of SAN disk provided to the server over fibre channel.)


[root@vmhost1 ~]# multipath -l | grep 00000002
mpath3 (877880e80144455000001445500000002) dm-12 HITACHI,OPEN-V*4
[root@vmhost2 ~]# multipath -l | grep 00000002
mpath3 (877880e80144455000001445500000002) dm-11 HITACHI,OPEN-V*4
[root@vmhost3 ~]# multipath -l | grep 00000002
mpath3 (877880e80144455000001445500000002) dm-11 HITACHI,OPEN-V*4

As is shown in the above output, each of the hosts can see the same SAN LUN: 877880e80144455000001445500000002 which in each case is also called mpath3. But I don’t really care about what names the disk is given because I run LVM on top of those disks.


[root@vmhost2 ~]# pvs | grep mpath3
  /dev/mapper/mpath3 some_volume_group LVM2 a-    96.62G  16.62G

So, we have a SAN LUN, a physical volume (PV) made from that LUN, and a volume group (VG) called some_volume_group created from that PV.

So it sort of looks like this in terms of hierarchy:

Fibre Channel SAN LUN
- multipathd
  - PV
    - VG
      - Logical volume (LV) which has hosttags assigned

Using hosttags

First we make sure we have hosttags configured on each of the vmhosts.


[root@vhmost2 lvm]# pwd
/etc/lvm
[root@vmohost2 lvm]# grep hosttags lvm.conf
tags { hosttags = 1 }

Obviously because I run using LVM hosttags in production, I already have hosttags configured and being used to control vmhost access to LVs.


[root@vmohost2 lvm]# uname -n
xmhost2.example.com
[root@vmhost2 lvm]# lvdisplay @`uname -n` | grep "LV Name"
SNIP!
  lv Name                /dev/some_logical_volume/test

If I want to create a LV on a vmhost, and that VG is configured to use hosttags, then it has to be done properly. This is an example that will fail because the host does not have permission, ie. the LV is not available because of the lack of a hosttag attribute on the LV that names the uname -n host specifically.


[root@vmhost2 ~]# lvcreate -n test2 -L10.0G /dev/some_volume_group
# Will fail out with error message

But this next command will work, because we are saying create a LV and assign a hosttag to it which is the same as the hostname that is creating it. The vmhost can’t create a LV if it’s not made available via a hosttag.


[root@vmhost2 ~]# uname -a
vmhost2.example.com
[root@vmhost2 ~]# lvcreate --addtag @vmhost2.example.com -n test2 \
-L10.0G /dev/some_volume_group

Now, on vmhost1 we can see that the LV appears in the list, but it is not available:


[root@vmhost1 ~]# lvs some_volume_group | grep test2
  test2            some_volume_group -wi--- 10.00G

But it’s available on vmhost2, where it was created, and where it has a hosttag attribute of vmhost2.example.com.


[root@vmhost2 ~]# lvs some_volume_group | grep test2 
  test2            some_volume_group -wi-a- 10.00G

And, if you use lvdisplay @`uname -n` you can see what LVs have the servers uname -n tag:


[root@vmhost2 ~]# lvdisplay @vmhost2.example.com | grep "LV Name"
  LV Name                /dev/some_volume_group/test2

whereas vmhost1 does not see that LV:


[root@vmhost1 ~]# lvdisplay @vmhost1.example.com | grep "LV Name"
# Nothing returns, as expected

So, while this sounds complicated, it really isn’t. Essentially for a LV to be available to a server that can see the same LUNs as other servers, in terms of LVM, it must have its hosttag added to the LVs metadata. Otherwise, it’s not available and can’t be used on that host.

While all three hosts can see the LV, it’s only available to those vmhosts that have had their hostname added to the specific LVs tags. It adds complexity to the vmhost setup and use, but it’s better to do this than to end up having two virtual machines writing to the same LV. Other options include using filtering in LVM, or even going all the way and using clvmd, which I have done, but that’s another story… :)

What is serverascode.com?

2011-04-11T00:00:00-07:00

What is serverascode.com?

11 April 2011 – Edmonton

serverascode.com has two major functions: 1) to document how I am working through the process of learning to treat servers as code, and 2) to be my systems administration resume.

Starting with #2, I believe that in my line of work—Unix/Linux systems administration—in todays “market”, a two page traditional resume will not get me the job I’m looking for. The only thing that will is a good set of open source contributions; that if servers are code, then much of what I do as a systems administrator should be available on-line in a source code repository. As I write this post, I only have a few lines of code available at github but as time goes on that will increase.

As far as #1, I love to read about the latest way systems administrator are working; how they are being successful leveraging technology to manage large numbers of often diverse, highly interconnected systems while still ensuring they are highly operational and secure. I read sites like Hacker News and other blogs that describe a sort of “new age” methodology to systems administration; so called “devops” or “agile sysadmin” or just plain 2010+ systems administration. However, I have not had an opportunity to apply those concepts, and I would like to use this blog, and services like github, to begin treating a server as code.

For better or for worse, systems administration has changed greatly—and will continue to change—especially with the recent trend in virtualization. I believe at the core of this change is the concept of treating server(s) as code. My feeling is that in 2010+ we don’t admin a server, we code it. From procurement to deployment to maintenance to decommissioning—it’s all code now.

On top of every Linux distribution, which can perhaps now be a called a OS framework, comes configuration conventions, and a packaging system. Then, over top of the framework, we add a centralized management server instance (perhaps the only server I should be logging into) which runs configuration management software, such as chef, puppet, and others, which control installation and configuration of applications, alerting things such as change management, and other systems I have not yet determined.

Suffice it to say that I am looking forward to working towards treating servers as code and documenting that process in blog posts and, hopefully more-so, as code!

Resume and Contact

2011-04-10T00:00:00-07:00

Resume and Contact

Updated 20 October – 2012 – Edmonton

(Taken at the 2012 OpenStack Summit)

Contact

Name: Curtis
Email: curtis@serverascode.com
Twitter: @serverascode
Github: curtisgithub (I know…not the best account name)
Hacker News: serverascode
Stack Exchange: curtis

Goals

I would like to continue working with Linux, OpenBSD, OpenStack, Python, Ansible and other great Open Source software (and hardware) projects, especially projects related to devops, security, the “cloud,” performance, monitoring, and modern storage systems such as object storage and those using solid state devices. I’m also interested in increasing my knowledge of networking, especially seeing how powerful projects like Quantum will be in OpenStack. I would be very interested in working on a project developing tools related to OpenStack and security.

My dream job would be to work at a great company. Sounds simple, but what defines a great compay? As far as I’m concerned, it’s one that’s fast moving, yet thoughtful, and willing to fail fast. A company that accomplishes a lot in a short amount of time, and that doesn’t mean 100 hour weeks for its employees. I’d like to be part of a team that creates great products and services, open source or not.

I would like to work in an organization that fosters diversity in its staff and works hard to maintain that diversity, not just in terms of gender and ethnicity, but also what staff do outside their work life.

I’d also like to work for a company with a good sense of humor. I don’t think sense of humor gets enough attention these days, though I admit that is a difficult requirement for an entire organization to meet.

Finally, I would like to work in an environment where I can learn from my peers, which likely means going to a place where everyone is smarter than I am, and where they can help bring me up to their level. If I’m lucky I can give them something in return, and hopefully contribute to the open source community as well.

(And since we’re talking dream jobs, it would be nice to bring my dog to work every once in a while.)

Background

Most of my career has been spent in Systems Administration. I have recently branched out to working as a “Cloud Developer” mostly around the OpenStack ecosystem. While that title likely has a short lifespan, I do believe it’s currently accurate, especially when one considers the somewhat recently initiated devops movement.

I previously worked for a world-class library at a large Canadian university that wanted to archive and store digital artifacts (ie. our culture) for 500 years. That meant I learned a lot of about all kinds of modern storage systems—and I’m not just talking about your standard head-based fibre SAN, because that model just doesn’t work for all situations. We needed checksumming filesystems.

While working at the library I was lucky enough to travel to San Francisco and meet the people behind the Internet Archive (an organization that has eight or nine petabytes of storage) find out what they were doing, and bring those concepts back and work on implementing some of them. I also worked with projects as diverse as a Canada-wide L2 networking to sync iRODS instances. Nothing like pinging a private network that is thousands of kilometres away. :)

Prior to working in a library (which I can assure you is quite an experience, walking through the stacks to get to your office is amazing), I worked as a Systems Administrator at a large Canadian Online and and Distance Education university on several research projects as well as with scaling educational information systems such as Moodle. We used Redhat Enterprise and Xen and KVM based virtualization quite heavily. In this position I was able to work on several performance testing projects, such as pushing millions and millions of messages through Zimbra using Jmeter, as well as package many systems into RPM.

I gained the bulk of my security experience—and my understanding that “siloing” is not the best way to create security—at the same university where I was the Security Systems Administrator. I worked on firewalls (of course), intrusion detection, and layered security. There I began the process of moving the university away from a single monolithic commercial firewall to multiple firewalls based on OpenBSD.

Skills

Professional Unix/Linux systems administrator
Operating private clouds based on OpenStack
Virtualization with Xen, KVM, VirtualBox
Linux Operating System Packaging (eg. RPM)
Experience with large, modern storage systems
Information security expert (CISSP, GCUX)¹
Scripting in python, bash
Life-long learning

Projects

Lead developer on an open source virtual lab (ie. DaaS) based on OpenStack² called The Labinski!
Deployment and operation of a private cloud based on OpenStack
Installed and performance tested multiple vendor PCIe-SSD cards
Development of a one petabyte+ storage system
Deployed iRODS across a Canarie Lightpath (ie. across Canada)
Deployed live migration capable virtual machine host cluster via XEN/KVM and SAN (ie. clustered LVM)
Deployed OpenBSD/pf as main firewall for large organization
Packaged and deployed systems such as Moodle, DSpace in production

Education

2 Year After-degree in Information Security, 2001
Bachelor of Education, 2000

Experience

Cloud Developer – Educational Services, Startups – Non-profit Research Organization – June 2012 to Present
Senior Unix Systems and Storage Administrator – Digital Preservation – Top 20 University Library – April 2011 to June 2012
Unix Systems Administrator – Production Services and Research – Large Online and Distance University – Oct 2007 to April 2011
IT Services – Various – July 2005 – Sept 2007
Systems Administrator – Banking – Private Company – May 2004 to June 2005
Security Systems Administrator – Large Online and Distance University – Dec 2001 to June 2004

¹ I have allowed these certifications to expire

² Still an early beta stage project