logo

Installing IBM high-iops FusionIO Cards in Redhat/Centos 6

In a previous post I had described how I deployed a IBM branded FusionIO drive on Redhat Enterprise 5.

I am now running that same card on CentOS 6, and am using the new version (2.2.3) of IBM’s version of the driver.

Actually I think there is a new-new version (3) of the driver now out for some people. I’m not sure if IBM has put out this driver or not for their high-iops cards.

CentOS version:

[root@srv ~]# cat /etc/redhat-release 
CentOS release 6.2 (Final)

The kernel I am running is stock RHEL 6:

[root@srv ~]# uname -a
Linux example.com 2.6.32-220.17.1.el6.x86_64 #1 \
SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux

This is what I see in terms of PCI devices for the FusionIO cards:

[root@srv ~]# lspci | grep -i fusion
8f:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)
90:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)

So the card is physically installed in the server, but the driver has not been loaded, so they are not usable at this point. Also should note that one 640GB cards actually looks like 2x 320GB devices to the OS.

First, we download the zip file containing the RPMs from IBM.

%{color:red}Warning:% These drivers are for the IBM version of the FusionIO cards. If you are not running the IBM version you probably need different drivers and RPMs.

# wget ftp://download2.boulder.ibm.com/ecc/sar/CMA/XSA/ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip
SNIP!

Inside that zip are several RPMs:

[root@srv tmp]# mkdir fio
[root@srv tmp]# cd fio/
[root@srv fio]# unzip ../ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip 
Archive:  ../ibm_dd_highiop_ssd-2.2.3_rhel6_x86-64.zip
  inflating: rhel6/fio-common-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-firmware-highiops-101583.6-1.0.noarch.rpm  
  inflating: rhel6/fio-snmp-agentx-1.1.1.5-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/fio-util-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/high_iops-gui-2.3.1.1874-1.1.noarch.rpm  
  inflating: rhel6/iomemory-vsl-2.2.3.66-1.0.el6.el6.src.rpm  
  inflating: rhel6/iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.el6.el6.x86_64.rpm  
  inflating: rhel6/libfio-2.2.3.66-1.0.el6.x86_64.rpm  
  inflating: rhel6/libfusionjni-1.1.1.5-1.0.el6.x86_64.rpm 

So far when I’ve been running these servers I haven’t installed all of those RPMs, only a subset.

So lets install those RPMs:

[root@srv rhel6]# yum localinstall --nogpg \
 fio-common-2.2.3.66-1.0.el6.x86_64.rpm \
 libfio-2.2.3.66-1.0.el6.x86_64.rpm fio-util-2.2.3.66-1.0.el6.x86_64.rpm \
 fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm \
 fio-firmware-highiops-101583.6-1.0.noarch.rpm \
 iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.el6.el6.x86_64.rpm
SNIP!
Transaction Test Succeeded
Running Transaction
  Installing     : fio-util-2.2.3.66-1.0.el6.x86_64                   1/6 
  Installing     : fio-common-2.2.3.66-1.0.el6.x86_64                 2/6 
  Installing     : iomemory-vsl-2.6.32-71.el6.x86_64-2.2.3.66-1.0.e   3/6 
  Installing     : libfio-2.2.3.66-1.0.el6.x86_64                     4/6 
  Installing     : fio-sysvinit-2.2.3.66-1.0.el6.x86_64               5/6 
  Installing     : fio-firmware-highiops-101583.6-1.0.noarch          6/6 

Installed:
  fio-common.x86_64 0:2.2.3.66-1.0.el6                                    
  fio-firmware-highiops.noarch 0:101583.6-1.0                             
  fio-sysvinit.x86_64 0:2.2.3.66-1.0.el6                                  
  fio-util.x86_64 0:2.2.3.66-1.0.el6                                      
  iomemory-vsl-2.6.32-71.el6.x86_64.x86_64 0:2.2.3.66-1.0.el6.el6         
  libfio.x86_64 0:2.2.3.66-1.0.el6                                        


As you can see the sysvinit RPM contains a couple of init.d files.

[root@srv rhel6]# rpm -qpl fio-sysvinit-2.2.3.66-1.0.el6.x86_64.rpm 
/etc/init.d/iomemory-vsl
/etc/sysconfig/iomemory-vsl

Let’s chkconfig this on permanently.

[root@srv rhel6]# chkconfig iomemory-vsl on

We also need to enable iomemory-vsl in /etc/sysconfig/iomemory-vsl.

[root@srv init.d]# cd /etc/sysconfig
[root@srv sysconfig]# grep ENABLED iomemory-vsl 
# If ENABLED is not set (non-zero) then iomemory-vsl init script will not be
#ENABLED=1
[root@srv sysconfig]# vi iomemory-vsl 
[root@srv sysconfig]# grep ENABLED iomemory-vsl 
# If ENABLED is not set (non-zero) then iomemory-vsl init script will not be
ENABLED=1
[root@srv sysconfig]#

And we can start or restart iomemory-vsl:

[root@srv sysconfig]# service iomemory-vsl restart
Stopping iomemory-vsl: 
Unloading module iomemory-vsl
                                                           [FAILED]
Starting iomemory-vsl: 
Loading module iomemory-vsl
Attaching: [                    ] (  0%) /Attaching:
[                    
Attaching: [====================] (100%) \
fioa
Attaching: [====================] (100%)
fiob
                                                           [  OK  ]

At this point I’m going to reboot the server as well, just to make sure everything is going to get loaded if the server spontaneously restarts, which they have been known to do. ;)

[root@srv sysconfig]# reboot

Now after the reboot there are a couple more block storage devices on this server:

[root@srv ~]# ls /dev/fio?
/dev/fioa  /dev/fiob

We want to create a lvm physical volume (pv) on that block device:

[root@srv ~]# pvcreate /dev/fioa
  Device /dev/fioa not found (or ignored by filtering).

Ooops. Error message. What went wrong? Well, the “or ignored by filtering” is where to start looking. This FusionIO knowledge base entry (which you have to login to see, how annoying is that) shows that we need to add an entry to the lvm.conf on the server:

Locate and edit the /etc/lvm/lvm.conf configuration file.
Add an entry similar to the following to that file:
types = [ "fio", 16 ]

That is precisely what I will do.

[root@srv lvm]# grep types lvm.conf
    # List of pairs of additional acceptable block device types found 
    # types = [ "fd", 16 ]
    types = [ "fio", 16 ]

And now:

# let's see if the types were loaded
[root@srv ~]# lvm dumpconfig | grep types
  	types=["fio", 16]
[root@srv ~]# pvcreate /dev/fioa
  Physical volume "/dev/fioa" successfully created
[root@srv ~]# pvcreate /dev/fiob
  Physical volume "/dev/fiob" successfully created

And create a volume group and add the pvs to it.

[root@srv ~]# vgcreate hiops /dev/fioa
  Volume group "hiops" successfully created
[root@srv ~]# vgextend hiops /dev/fiob
  Volume group "hiops" successfully extended
[root@srv ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree  
  hiops    2   0   0 wz--n- 504.91g 504.91g
  system   1   9   0 wz--n-  58.56g  36.66g
  vm       1  11   2 wz--n-   1.31t 228.09g

I should note at this point that there is only 504g in the hiops volume group when there should be about 600g.

Previously, using the fio-format command, I had formatted these drives to only 80% capacity. But that was on another server, and I’m not sure it’s really necessary to do that unless you are looking for extreme performance or perhaps additional reliability.

I believe that in some cases with SSD, PCIe or otherwise, it’s not a bad idea to use less than 100% of the drive. That said, if you are looking to max out these drives performance-wise, I’d suggest talking to your vendor rather than just listening to me. :)

(AFAIK, these cards can actually take an external power source to increase performance even more. But we don’t use that functionality.)

So I’m going to reformat these drives to 100% usage. Just for fun. Why not get back that 100g because the performance/endurance at 100% is going to be fine for our usage.

%{color:blue}Note:% Brand new drives won’t have to be formatted. I’m only doing this because I had formatted the drives when they were in the other server.

%{color:red}Warning:% Reformatting will obviously delete any data on these drives!

# first detach the /dev/fioa
[root@srv ~]# fio-detach /dev/fct0
Detaching: [====================] (100%) -
[root@srv ~]# fio-format -s 100% /dev/fct0
Creating a device of size 322.55GBytes (300.40GiBytes).
  Using block (sector) size of 512 bytes.

WARNING: Formatting will destroy any existing data on the device!
Do you wish to continue [y/n]? y
Formatting: [====================] (100%) \
Formatting: [====================] (100%)
Format successful.
# then attach...
[root@srv ~]# fio-attach /dev/fct0
Attaching: [====================] (100%) -
fioa

And we can add that device back with pvcreate and then we should see a larger drive:

[root@srv ~]# pvcreate /dev/fioa
  Physical volume "/dev/fioa" successfully created
[root@srv ~]# pvs /dev/fioa
  PV         VG    Fmt  Attr PSize   PFree  
  /dev/fioa  hiops lvm2 a-   300.40g 300.40g

I reformatted the other side of the drive back to 100% as well. (With new drives this shouldn’t be necessary.)

And the fio-status now is:

[root@srv ~]# fio-status

Found 2 ioDrives in this system with 1 ioDrive Duo
Fusion-io driver version: 2.2.3 build 66

Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	External Power: NOT connected
	PCIE Power limit threshold: 24.75W
	Sufficient power available: Unknown
	Connected ioDimm modules:
	  fct0:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	  fct1:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX

fct0	Attached as 'fioa' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	Alt PN:68Y7382
	Located in slot 0 Upper of ioDrive Duo SN:XXXXX
	PCI:8f:00.0
	Firmware v5.0.6, rev 101583
	322.55 GBytes block device size, 396 GBytes physical device size
	Sufficient power available: Unknown
	Internal temperature: avg 50.2 degC, max 51.2 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10.00%

fct1	Attached as 'fiob' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:XXXXX
	Alt PN:68Y7382
	Located in slot 1 Lower of ioDrive Duo SN:XXXXX
	PCI:90:00.0
	Firmware v5.0.6, rev 101583
	322.55 GBytes block device size, 396 GBytes physical device size
	Sufficient power available: Unknown
	Internal temperature: avg 46.3 degC, max 46.8 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10.00%



Finally we can create a logical volume (lv) to use.

[root@srv ~]# vgs hiops
  VG    #PV #LV #SN Attr   VSize   VFree  
  hiops   1   0   0 wz--n- 300.40g 300.40g
[root@srv ~]# lvcreate -n test -L10.0G /dev/hiops
  Logical volume "test" created

If you have any corrections or other comments, please let me know!