logo

KSM and KVM

UPDATE: I did some more research and have a better idea of what pages_saved and pages_saving means. So we are saving quite a bit of memory!

I recently found out about the ksm technology that is in Ubuntu 12.04 + kvm by default. ksm is a memory de-duplication process. As far as I understand it, ksm can allow virtual machines (actually any application, not just virtualization) to share memory pages–it finds all duplicated memory pages and merges them, thereby saving memory in some situations.

One of the projects I am working on is a classroom as a service, or virtual classrooms. Students can login to a web gui and request a reservation to a virtual machine image which they can then access with a RDP client.

In this project all of the images are based on–unfortunately–Windows 7. One would think that if we are running many similar Windows 7 images ksm could do a lot of de-duplication.

I have been doing a few experiments in my spare time to see if ksm can help to over-commit memory. If I can I’d rather be able to run 400 virtual machines than 200. If we can over-commit on memory 1:2 or 1:4 there could be substantial cost savings for the project.

The Test

I have a basic Windows 7 image in qcow2 format.

root@ksm_test:/mnt/ksm-test$ file win7-base.qcow2
win7-base.qcow2: QEMU QCOW Image (v2), 21474836480 bytes

I am going to run 30 Windows 7 images with four gigs of ram and two virtual cpus each, based off a qcow2 snapshot from the original backing image.

The server I am running this test on is a Dell c6220 with 32 HT cores and 128 gigs of main memory.

ASIDE: /mnt/ksm-test is an xfs file system. I found that this test on a ext4 based filesystem used considerably more IOPs than xfs because the jdb2 process was doing a lot of journaling. There are likely some settings I should be using with ext4 to get better performance, but instead I just hopped over to xfs and haven’t gone back to ext4 yet.

This is the little script I use to boot the vms:

root@ksm_test:~$ cat test_ksm.sh 
#!/bin/bash

# How much memory to boot with
MEM=4048
BACKING_DIR=/mnt/ksm-test
BACKING_FILE=win7-base.qcow2
SLEEP=60

pushd $BACKING_DIR

for i in {1..30}; do
	echo "====> Starting a new instance..."
	# Remove the old backing file
	rm -f win7-$i.qcow2

	# Create a new backing file that is a qcow2 snapshot of the original file
	qemu-img create -f qcow2 -b $BACKING_FILE win7-$i.qcow2

	# Actually start the intstance
	/usr/bin/kvm \
	-M pc-1.0 \
	-smp 2,sockets=2,cores=1,threads=1 \
	-enable-kvm \
	-m $MEM \
	-drive file=win7-$i.qcow2,if=virtio \
	-boot d \
	-net nic,model=virtio \
	-net user \
	-nographic \
	-vnc :$i \
	-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
	-daemonize

	# Let's just sleep for a few seconds...
	echo "====> Sleeping for $SLEEP..."
	sleep $SLEEP 
done

popd

exit 0

After that script runs we have 30 kvm Win7 instances running:

root@ksm_test:/mnt/ksm-test$ ps ax  |grep "bin\/kvm" | wc -l
30

For the first while things are a little crazy on the host because 30 Windows 7 vms just booted in 30 minutes. After a few hours, or rather overnight, the vms settle down quite a bit to just doing a few IOPs each.

As far as what these vms are doing–I login to a couple every once and a while just to make sure they are up, but otherwise they are doing nothing but whatever they do by default.

The Defaults

ksm is enabled in Ubuntu by default when using kvm. However, the defaults are fairly conservative:

root@ksm_host:~$ cat /sys/kernel/mm/ksm/pages_to_scan 
100
root@ksm_host:~$ cat /sys/kernel/mm/ksm/sleep_millisecs 
200

ksm will scan 100 pages, sleep for 200 milliseconds and then scan 100 more, and so on. But with millions of pages it will take a long, long time to scan all of them.

I set the pages_to_scan to 20000 and sleep_millisecs to 20–I’m guessing these are pretty aggressive settings.

root@ksm_host:~$  echo "20000" > /sys/kernel/mm/ksm/pages_to_scan
root@ksm_host:~$  echo "20" > /sys/kernel/mm/ksm/sleep_millisecs

The Results

I suppose saying “results” sounds scientific. :)

The reality is that I’m really just cutting and pasting the ksm information that has been recorded after several days of running 30 Windows 7 virtual machines that should all be very close in terms of memory use.

From the ksm.txt file:

A high ratio of pages_sharing to pages_shared indicates good sharing,
but a high ratio of pages_unshared to pages_sharing indicates wasted 
effort. pages_volatile embraces several different kinds of activity, 
but a high proportion there would also indicate poor use of madvise 
MADV_MERGEABLE.

And the results of ksm after a few days of running 30 vms…

</pre>

root@ksm_test:~$ for i in `ls -1 /sys/kernel/mm/ksm`; \
do echo "===> $i"; cat /sys/kernel/mm/ksm/$i;  done
===> full_scans
5417
===> pages_shared
443355
===> pages_sharing
26704343
===> pages_to_scan
20000
===> pages_unshared
3164064
===> pages_volatile
183552
===> run
1
===> sleep_millisecs
20

So, if I understand these numbers correctly, pages_shared is the amound of memory ksk is actually using. Thus in this example, ksm is using 1.7GB of memory:

root@ksm_test:~$ getconf PAGESIZE
4096

So if we have 443355 pages shared, ksm is using this many bytes:

root@ksm_test:~$ echo "443355 * 4096" | bc
1815982080

which is about 1.7GB.

However, saved memory, ie. pages_sharing, is quite high! So this is good. :)

root@ksm_test:~$ echo $((26704343*`getconf PAGE_SIZE`/1024/1024/1024)) GB
101 GB

I’m not sure how this equates to all the memory being used on the machine, but as far as ksm is concerned it’s saving us about 100 gigs. Nice.

root@ksm_test:~$ free
             total       used       free     shared    buffers     cached
Mem:     131997772  131556896     440876          0     108448  108129628
-/+ buffers/cache:   23318820  108678952
Swap:     41943032     302836   41640196

Update: Now that I have a better understanding of what ksm is doing and what it’s numbers mean, using a modified version of this script we can see some interesting results, though in the below example I am running 60 2 gig 1 vcpu instances instead of 30 4 gig 2 vcpu instances like the rest of this post:

root@ksm_test:~$ ./ksm_stat.sh 
Shared memory is 2071 MB
Saved memory is 95514 MB
Unshared memory is 21336 MB
Volatile memory is 2549 MB
Shared pages usage ratio is 46.11
Unshared pages usage ratio is .22

So thanks for reading, and if you have any suggestions as to what I might be doing incorrectly, be it settings or my math or my general assumptions about ksm :), please let me know in the comments.