Hey, I wrote a “part 2” to this article, you may want to check it out!

Hello,

recently FreeBSD10 has come out and one of the most intresting new features was the introduction of bhyve, a “type 2 hypervisor” that allow you to easily create a Virtual Machine inside of a FreeBSD Host.

As with every new technology, it is yet very rough, but the first “driving” experience was very good. Recently we had a new project starting, some new hardware still unused and in general I’m not very fond of VMWare so we decided to do a comparison between VMWare and bhyve to understand what would be the real performance downfall of using a new technology.

We didn’t know exactly what to test so we decided to create an “empty” FreeBSD 10.0 (amd64) machine with the latest ports infrastructure (portsnap fetch && portsnap extract) and tested how much time would be required to compile bash from ports. Bash has many dependencies, like gettext, perl, various stuff, so it seemed the right compromise between CPU and HD use, thus a correct benchmark of a machine doing “general use”.

So, we had two exactly identical DELL R620 servers configured like this.

• 2CPUs – “Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz”
• 128GB RAM
• 2 HD ST9250610NS (SATA 7200 RPM) with native Mirroring (PERC H710 Mini)

I know the drives are slow, but our plan is to use and external iSCSI SAN, thus we didn’t buy fast drives. Yet, for the test the slowness is not really relevant, as long they are the same on both platforms.

The compiling command was just:

cd /usr/ports/shells/bash/
time make -DBATCH install clean

The distfiles were already available on the local network trough our proxy.

Please keep in mind that we are comparing a brand new technology (bhyve) with a well tested and well tuned one (VMWare) with a development effort going on since almost 10 years!

First test. Defining the base value.

Our first test was compiling Bash on a single machine to define the best scenario available. We run the same test in two different scenarios: single CPU and dual CPU. On VMWare there’s an (apparent) difference between 2 CPUs and 2 Cores thus we run the test twice to check the differences. The seconds you see reported is the amount of seconds reported by “time”.

1 VM – 1 CPU – 2 GB RAM

The performances in this scenario were pretty close between the two systems.

• bhyve took 332 seconds
• VMWare took 313 seconds

The difference is about 6%.

1 VM – 2 CPUs – 2GB RAM

Performances were pretty close in this scenario as well.

• bhyve took 508 seconds
• VMWare in “2 CPUs 1 Core” mode took 422 seconds
• VMWare in “1 CPU 2 Cores” mode took 426 seconds

The difference is about 20%

What I couldn’t really understand (but that’s something not related to bhyve or VMWare) is how a multiprocessor machine is slower than a singleprocessor machine in doing the compilation… any idea?

Second Test. Let’s push it to the limits.

Our second test had the purpose of understanding how the system would react when pushed to the limit, in a scenario where we had “a lot” of machine working at the same time. We wanted to see real “multitasking” performances of the Virtualization engine, let’s say. We came out with the number “20” because we wanted to have some kind of CPU overcommitment: 20VMs on physical 16CPUs.

So we did 20 copies of our original clean machine (the same we copied for previous test), we turned on all the 20 machines and we did run the build command on all the machines at the same time using a cluster shell. Results were pretty funny, let’s see them in details (the seconds you see is a average of all the values from the different machines).

20 VM – 1 CPU – 2GB Ram

Performances were still pretty close even if some difference started to appear.

• bhyve took 673 seconds
• VMWare took 549 seconds

The difference is about 22%.

Note: the time is about twice of the “single machine single cpu” test but just an hundred few seconds more than the “single machine multiple cpus”.

20 VM – 2 CPUs – 2GB RAM

Here something terrible happened. We couldn’t get the bhyve machines to finish compiling. We waited a bunch of time, but after 5000 seconds we just gave up. None of the machines had finished compiling yet but we just stopped the test. I don’t know why this happened. It was working, it was just dead slow. So here’s the detail:

• bhyve was stopped after 5000 seconds
• VMWare took 980 seconds

The difference would be 410%, but since bhyve machine didn’t finish is a totally useless value.

Conclusions

Our final conclusion is that we are very pleased with bhyve performances! The amount of work that has been done is great and seeing that such a young software at his first public release has performances so close to a multi-awarded software was a definitely pleasing experience!

What came out is that we’ll probably adopt bhyve as one of the Virtualization Platform we officially use in our company, obviously it’s not yet ready to be the only one, either for the Linux VMs support still not working perfectly and the lack of VGA Emulation support (thus no Windows VMs, yet) – but since we have just 1 Windows Server and just about 5 Linux servers versus more than 30 FreeBSD servers, that’s not a big stop for us.

If you have any question on the data (and maybe any suggestion with the problem we had with 20 VMs with 2CPUs) feel free to add a comment.

Big thanks to Patrizio Ottaviani for running all the tests and providing me the results to come up with the final report and the graphs.