logo

Using fusion-io drives on Redhat Enterprise 5

%{color:red}Update:% Please note that this post is getting a bit old. Currently I am running these IBM FusionIO drives on RHEL 6. I’ll be posting about that and a few other PCIe-SSD subjects in the next short while. - 24 Apr 2012

FusionIO IODrive Overview

So at work we have a rather large IBM x3850 x5 server. It has 4 sockets each with six cores and hyperthreading (not that I’m necessarily a fan of hyperthreading–really I haven’t done enough research to make up my mind) which ends up with RHEL5 seeing 48 CPUS.

$ cat /proc/cpuinfo | grep proc | wc -l
48

Fun.

But the important part of this post is that this server also has three 640GB fusion-io drives which I have installed and configured as a volume group called fio

$ ls /dev/fio
fio/  fioa  fiob  fioc  fiod  fioe  fiof  
$ vgs fio
  VG   #PV #LV #SN Attr   VSize VFree
  fio    6   4   0 wz--n- 1.76T 1.08T

and where the fio[a,b,c,d,e,f] are the drives, with each 640 gig card actually appearing as 2 320 gig disks.

$ dmesg  |grep -i "found device"
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:89:00.0: Found device 0000:89:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:8a:00.0: Found device 0000:8a:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:93:00.0: Found device 0000:93:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:94:00.0: Found device 0000:94:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:98:00.0: Found device 0000:98:00.0
fioinf IBM 640GB High IOPS MD Class PCIe Adapter 0000:99:00.0: Found device 0000:99:00.0

Resources

The most important resource for using these FusionIO drives is the official knowledge base which has several articles specifically for linux. I would suggest reading all of them. :)

Install

Once the cards were put into the server, which is somewhat harrowing given their individual cost, and the server was booted, the software drivers that were downloaded from the IBM website were installed. This server runs RHEL5

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

as that RHEL version that is what IBM supports for drivers.

$ rpm -i iodrive-driver-1.2.7.5-1.0_2.6.18_164.el5.x86_64.rpm \
iodrive-firmware-1.2.7.6.43246-1.0.noarch.rpm \
iodrive-jni-1.2.7.5-1.0.x86_64.rpm \
iodrive-snmp-1.2.7.5-1.0.x86_64.rpm \
iodrive-util-1.2.7.5-1.0.x86_64.rpm \

Currently I am using the drivers as they were downloaded, which means using a specific matching kernel to match. The drivers do come with a source RPM so that you can rebuild them for your latest kernel, but I have opted not to do that yet. So install the matching kernel

$ yum install kernel-2.6.18-164.el5

and reboot.

However, I am also using the amazing ksplice service to ensure that depsite the fact that I am using a rather old kernel to match the FusionIO drivers that the kernel is still up to date in terms of security issues:

$ uptrack-uname -r
2.6.18-238.12.1.el5
$ uname -r
2.6.18-164.el5

The @uptrack-uname -r@ command asks uptrack what security equivalent version of the kernel is. Great stuff that kslplice.

Once the drivers are installed we can load the modules

$ modprobe fio-driver

and now we can see the drives

$ ls /dev/fio*
fioa  fiob  fioc  fiod  fioe  fiof 

and at this point we can configure the drives.

Worker processes

Once the drivers are installed there is a /etc/init.d/iodrive startup script. One of the things this script does is startup some worker processes which I believe are used to move data around the FusionIO drives to ensure their performance and longevity.

$ chkconfig --list iodrive
iodrive 0:off	1:on	2:on	3:on	4:on	5:on	6:off


$ ps ax | grep worker
 5271 ?        S<   1169:51 [fct0-worker]
 5588 ?        S<   1168:07 [fct1-worker]
 5593 ?        S<   359:01 [fct2-worker]
 5598 ?        R<   206:02 [fct3-worker]
 5603 ?        S<   203:15 [fct4-worker]
 5608 ?        S<   203:12 [fct5-worker]
20921 pts/2    S+     0:00 grep worker

These processes will take up some CPU time. Frankly, because there are 48 CPUs in this server, using up one to run these workers is OK. But it was a little confusing at first seeing all this activity–one worker process for each card.

Configuration

Given that we are going to manage the FusionIO drives via LVM, we will need to configure LVM to allow it. See this knowledge base article.

$ grep fio /etc/lvm/lvm.conf
    types = [ "fio", 16 ]

Then add each /dev/fio* drive as a phyical volume and then add them to a volume group.

$ pvs | grep fio
  /dev/fioa  fio    lvm2 a-   300.31G 320.00M
  /dev/fiob  fio    lvm2 a-   300.31G 100.31G
  /dev/fioc  fio    lvm2 a-   300.31G 100.31G
  /dev/fiod  fio    lvm2 a-   300.31G 300.31G
  /dev/fioe  fio    lvm2 a-   300.31G 300.31G
  /dev/fiof  fio    lvm2 a-   300.31G 300.31G
$ vgs fio
  VG   #PV #LV #SN Attr   VSize VFree
  fio    6   4   0 wz--n- 1.76T 1.08T


fio-status

Useful way to check the status of the FusionIO drives.

$ fio-status

Found 6 ioDrives in this system with 3 ioDrive Duos
Fusion-io driver version: 1.2.7.5

Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:59518
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct0:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77479
	  fct1:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77478

fct0	Attached as 'fioa' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77479
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:59518
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 56.6 degC, max 59.6 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct1	Attached as 'fiob' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77478
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:59518
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 61.0 degC, max 63.0 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%


Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:59507
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct2:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77143
	  fct3:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77144

fct2	Attached as 'fioc' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77143
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:59507
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 62.0 degC, max 65.5 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct3	Attached as 'fiod' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77144
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:59507
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 64.0 degC, max 66.4 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%


Adapter: ioDrive Duo
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:100366
	PCIE Power limit threshold: 24.75W
	Connected ioDimm modules:
	  fct4:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77344
	  fct5:	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77345

fct4	Attached as 'fioe' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77344
	Alt PN:68Y7382
	Located in 0 Upper slot of ioDrive Duo SN:100366
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 68.9 degC, max 71.9 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%

fct5	Attached as 'fiof' (block device)
	IBM 640GB High IOPS MD Class PCIe Adapter, Product Number:68Y7381 SN:77345
	Alt PN:68Y7382
	Located in 1 Lower slot of ioDrive Duo SN:100366
	Firmware v43246
	322.46 GBytes block device size, 396 GBytes physical device size
	Internal temperature: avg 63.0 degC, max 66.0 degC
	Media status: Healthy; Reserves: 100.00%, warn at 10%



XFS

Prior to finding out about the official knowledge base, I had decided to purchase a subscription from Redhat for the XFS file system. Then, upon reading this kb article, I found that they heavily recommend XFS as the file system to run on top of a FusionIO drive

XFS is currently the recommended filesystem. It can achieve up to 3x 
the performance of a tuned ext2/ext3 solution. At this time, there is 
no know additional tuning for running XFS in a single- or multi-ioDrive 
configuration 

so that is the file system we use.

$ mount | grep fio
/dev/mapper/fio-vault1 on /var/lib/vault1 type xfs (rw)


Mounting drives after a reboot

I’ll admit I hadn’t thought of this during the initial installation. After a few days we moved the server to a new location which thus required a power down and restart.

While the server was restarting, and I was standing in the cold, loud server room because the new room didn’t have any networking for IPMI (which is not good), I noticed it took a very long time to get past the udev portion of the boot, and in fact the FusionIO drives failed to mount from fstab. Of course there is a logical reason for that–read about it here.

Because we are using the 1.2 driver, I followed the straight forward instructions here.

Performance testing

Performance testing is hard. Maybe it’s just me. But testing superdisk like these FusionIO drives on a server with 48 CPUS and 64 gigs of main memory is not easy. Again I will admit I took a shot at benchmarking the FusionIO disk having not read the kb. I messed around with Bonnie++, io-whatever, but nothing quite came out right, partially because I didn’t put a lot of time into it, and because the server has so much memory that it makes it hard to beat the cache (I did try to reduce the memory the OS could see via kernel configuration, but didn’t have a lot of luck with that).

Finally I read this kb article which suggested using the fio utility (which I don’t believe is a utility put out by FusionIO, rather just aptly named).

The fio tool is not in the RHEL repositories but it is in rpmforge/repoforge.

$ cd /var/tmp
$ wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
$ rpm -Uvh rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
$ yum repolist | grep forge
rpmforge                           RHEL 5Server - RPMforge.net - enabled: 10,636
$ yum search fio | grep -i benchmark
fio.x86_64 : I/O benchmark and stress/hardware verification tool


Here are a couple of example runs. Please note that at this point I do not know much about fio. Benchmarking disk is a highly technical thing to do, and getting tests right would take a lot of research and consideration, which I have not done.

It seems that the fio benchmark utility suports direct=1 which means use non-buffered-io, thereby skipping memory cacheing and going straight to the disk.

$ cat fio-randwrite.fio 
[randwrite[

direct=1
rw=randwrite 
bs=1m 
size=5G 
numjobs=4 
runtime=10 
group_reporting 
directory=/mnt/fio-test-xfs
$ fio fio-randwrite.fio 
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 1.55
Starting 4 processes
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 4 (f=4): [wwww] [100.0% done] [0K/522.8M /s] [0 /510  iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=28487
  write: io=4556.0MB, bw=466161KB/s, iops=455 , runt= 10008msec
    clat (msec): min=1 , max=1692 , avg= 9.83, stdev=22.04
     lat (msec): min=1 , max=1692 , avg= 9.84, stdev=22.04
    bw (KB/s) : min=  559, max=264126, per=24.79%, avg=115540.55, stdev=20377.90
  cpu          : usr=0.10%, sys=14.85%, ctx=59071, majf=0, minf=92
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/4556/0, short=0/0/0

     lat (msec): 2=0.53%, 4=1.27%, 10=97.17%, 20=0.59%, 50=0.09%
     lat (msec): 100=0.18%, 250=0.15%, 2000=0.02%

Run status group 0 (all jobs):
  WRITE: io=4556.0MB, aggrb=466161KB/s, minb=477349KB/s, maxb=477349KB/s,
  mint=10008msec, maxt=10008msec

Disk stats (read/write):
  dm-11: ios=0/158802, merge=0/0, ticks=0/55956241, in_queue=55915327, 
  util=66.05%, aggrios=0/159667, aggrmerge=0/0, aggrticks=0/55932489,
  aggrin_queue=55785218, aggrutil=65.96%
    fioc: ios=0/159667, merge=0/0, ticks=0/55932489, in_queue=55785218, 
    util=65.96%

And then a similar test using RAID10 SAS disk formated ext3.

$ cat fio-randwrite.fio 
[randwrite[

direct=1
rw=randwrite 
bs=1m 
size=5G 
numjobs=4 
runtime=10 
group_reporting 
directory=/mnt/sas-test
$ fio fio-randwrite.fio 
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
randwrite: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 1.55
Starting 4 processes
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
randwrite: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 4 (f=4): [wwww] [1200.0% done] [0K/0K /s] [0 /0  iops] [eta
 1158050441d:07h:00m:05sJobs: 4 (f=4): [wwww] [inf% done] [0K/0K /s] 
[0 /0  iops] [eta 1158050441d:07h:00m:04s]  Jobs: 4 (f=4): [wwww] 
[1300.0% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07h:00m:04sJobs: 
4 (f=4): [wwww] [inf% done] [0K/0K /s] [0 /0  iops] 
[eta 1158050441d:07h:00m:03s]  Jobs: 1 (f=1): [___w] [66.1% done] 
[0K/0K /s] [0 /0  iops] [eta 00m:19s]               
randwrite: (groupid=0, jobs=4): err= 0: pid=28586
  write: io=4096.0KB, bw=112369 B/s, iops=0 , runt= 37326msec
    clat (usec): min=12140K, max=37183K, avg=32696578.04, stdev= 0.00
     lat (usec): min=12140K, max=37183K, avg=32696579.88, stdev= 0.00
    bw (KB/s) : min=   27, max=   83, per=31.61%, avg=34.46, stdev= 0.00
  cpu          : usr=0.00%, sys=51.90%, ctx=9598, majf=0, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/4/0, short=0/0/0

     lat (msec): >=2000=100.00%

Run status group 0 (all jobs):
  WRITE: io=4096KB, aggrb=109KB/s, minb=112KB/s, maxb=112KB/s, 
  mint=37326msec, maxt=37326msec

Disk stats (read/write):
  dm-12: ios=128/4721384, merge=0/0, ticks=5582/602531980, in_queue=602926524,
  util=97.85%, aggrios=129/87424, aggrmerge=0/4634618, aggrticks=5631/10828734,
  aggrin_queue=10826088, aggrutil=98.01%
    sdb: ios=129/87424, merge=0/4634618, ticks=5631/10828734, in_queue=10826088,
    util=98.01%

That’s a pretty big difference: io=4556.0MB for the FusionIO drives versus io=4096.0KB for the SAS RAID10. I’m going to have to look into this more! :)

PS. I found this list of device bandwidths interesting.