Saturday, November 11, 2017

The end of my Firefox story?

My history of using web browsers is not short, but not vast. I've started using Linux around 2000 IIRC, and my first DE was KDE and my first browser was Konqueror.

Here's how it looked (screenshot from Wikipedia):

As KDE was pretty resource heavy, and also building from source took a lot of time, I had started looking for a replacement. After trying out various DEs and WMs eventually I had moved to fluxbox and then to openbox a couple of years later. As I switched away from KDE, there was no reason to use Konqueror either, so I switched to Mozilla.

Back then it looked like this (screenshot from testingeducation.org):

Even at that time Mozilla was not on a light side. Also, in additional to browser, it included Mail and IRC client and some other stuff I forgot. Not quite unix-ish "Do One Thing and Do It Well", right?

Fortunately, in 2002 the first light-weight version of Mozilla became available; additional software like IRC and mail clients was ripped out, it was just a browser. Initially it was called Phoenix, then Firebird and finally Firefox. I was happy to see a project like that and switched to it pretty early (most probably while it was still called Firebird).

Fast forward to 2007 (or 2008). Co-worker showed me Vimperator. It seemed awkward initially, but after a little while I didn't want to switch back. Man, it get really addictive once you get used to it.

Eventually this firefox/vimperator combination became even more valuable to me. I spend probably 90% (or even more) of time either in a terminal or in a browser. Also, from time to time I use different computers with different OSes: FreeBSD, macOS, Windows, Linux. And firefox/vimperator allows to rapidly setup my development environment as it's available for all these OSes, and I just need to copy my tiny .vimepratorrc and that's it. Additionally, this convenient vi-like key binding system allows me not to care about OS-specific shortcuts (like switching/closing tabs, etc).

BTW, I recommend to take a look at MobaXterm if you're looking for a handy terminal emulator for Windows.

November, 2017. Situation with vimperator is not good. Firefox is switching to the new multiprocess architecture called e10s. Firefox 57 drops non-WebExtensions-based addons, with vimperaror being one of them. This is sad. I'm still using Firefox ESR while it's supported, but that won't last that long (image from https://www.mozilla.org/en-US/firefox/organizations/faq/):

Time to looks for alternatives. There ain't many.

As I'd love to keep my environment consistent across all OSes I use, I need cross-platform support. This doesn't work for most of the stand-alone browser projects like vimb and surf. However, qutebrowser does seem to provide binaries for many OSes. There are some obvious drawbacks of using tiny stand-alone browsers like that though, like no extensions (unless it supports some compatibility layer) and probably different level of stability on different OSes (depending on what core developers actively use), but I think it's worth checking out.

Another interesting option is Tridactyl which aims to provide WebExtensions based vimperator replacement. It looks like it's in beta stage right now, development is fairly active. Really hope that it'll be successful.

Sunday, August 6, 2017

Creating OpenBSD guest with libvirt/bhyve

I've been asked a couple of times how to setup OpenBSD guest with libvirt/bhyve. It's fairly straight-forward, but minimal libvirt version required for that is 3.5.0 because it's the first version that allows to specify vgaconf settings.

First, download the installation image: https://ftp.openbsd.org/pub/OpenBSD/6.1/amd64/install61.fs and create a disk image:

truncate -s 2G openbsd61.raw

Create an initial XML (pay attention to the vgaconf='off' bit):

Now with virsh do:

virsh # define /path/to/openbsd61.xml
Domain openbsd61 defined from /path/to/openbsd61.xml

virsh # start openbsd61
Domain openbsd61 started

virsh # vncdisplay openbsd61                                                                                                                                                                                       
127.0.0.1:0                                                                                                                                                                                                        

Using VNC client connect to the guest at the specified port:

vncviewer :0

This gives a boot loader prompt. For some reason, it doesn't set root device properly, so I have to do:

set device hd1a
boot

Then there'll be a number of installer questions (where I usually accept defaults) until it asks what disk should be used as a root device. It's important to check this carefully and choose the right one (you can find it by its size for example), otherwise it might just try to override the install image.

When the installation finishes, shutdown the guest, go back to virsh and edit the domain xml:

virsh # edit openbsd61

And remove the disk block corresponding to install61.fs. And then start domain again:

virsh # start openbsd61   
Domain openbsd61 started  

Now you should be able to connect to the guest via VNC or via SSH.

PS My sample xml uses the autoport feature that will be available only in libvirt 3.7.0. If you're using an earlier version, specify the VNC port explicitly:

<graphics type='vnc' port='5904'>

Sunday, May 21, 2017

Profiling OpenStack Horizon

OpenStack Horizon is a web based interface for managing OpenStack clouds. Just like many other OpenStack components, it's implemented in Python and uses the Django framework.

Sometimes you might notice that Horizon is sluggish and slow and it's not obvious what causes this. Natural thing to do in this case is to profile it to see problematic points.

There are at least two possible ways to profile that:

  • Use osprofiler, an "official" OpenStack Profiler,
  • Use standard Python profiling facilities like cProfile.

I recommend using the latter and I'll elaborate on that a little later.

Preparation

I think it's a good idea to profile Horizon in the same environment where you run your stage Horizon, i.e. pick one of the nodes where you have Horizon installed already (e.g. configured to be served using Apache) and bring your own instance of Horizon with the same configuration, but on a different port and using Django's built in http server.

For that purpose, you can copy over Horizon sources somewhere to your home directory (either an appropriate branch from git, or maybe grab sources from the package you use, that depends on your processes), copy over the existing config files (e.g. local_settings.py) and from top level source directory of horizon run:

~/horizon$ ./manage.py runserver 0.0.0.0:8000

This way your brand new Horizon instance will be available on port 8000. Also, you might want to check iptables configuration that it actually allows connections to this port.

Note: We're not using virtualenv because as we're running this on a Horizon node, we have all the dependencies installed via system packages.

Once this instance becomes accessible, we can start actual debugging.

Profiling

There's a wonderful Django Extensions package that makes Django profiling very easy. That could be installed via packages, e.g. on Ubuntu you can do:

#  apt-get install python-django-extensions

It could be installed using pip too in case your distro misses this package:

# pip install django-extensions

After installing the package, it's required to include it to the Horizon's INSTALLED_APPS list. While in the local copy of Horizon, open openstack_dashboard/settings.py, find INSTALLED_APPS and add 'django_extensions' to this list. The resulting setting should look something like this:

INSTALLED_APPS = [
    'openstack_dashboard',
    'sbr_reports',
    'sbr_log_processing',
    'django.contrib.contenttypes',
    'django.contrib.auth',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.humanize',
    'django_pyscss',
    'openstack_dashboard.django_pyscss_fix',
    'compressor',
    'horizon',
    'openstack_auth',
    'django_extensions', #  for runprofileserver
]

Now it's possible to run the profile server. Terminate the old 'runserver' that was used to test basic configuration, create a directory for the profiling data and start the profiling server:

$ mkdir ~/horizon_profile_data_001
$ ./manage.py runprofileserver --use-cprofile --prof-path=/home/user/horizon_profile_data_001 0.0.0.0:8000

Note: looks like --prof-path doesn't decode "~", so have to pass the full path.

Now Horizon is running in profiling mode, so we can open it up in the browser and navigate to the problematic pages, for example "Admin -> System -> Networks".

Analyzing Results

If everything went file, in ~/horizon_profile_data_001 you'll see something like this:

~/horizon_profile_data_001$ ls |grep networks
admin.networks.001278ms.1495361937.prof
~/horizon_profile_data_001$

Now we can use the pstats module to analyze the results:

python -m pstats admin.networks.001278ms.1495361937.prof

This opens an interactive interface for exploring profiling data. If you're not familiar with that, a good start would be to do:

sort cumtime
stats 40

Here's how it looks for me:

Here you can get an idea how much time was spent in template rendering, Neutron client, etc.

Why not osprofiler

  • osprofiler only traces calls that were decorated with the profiler decorator (example). This means that if some function was not decorated, either intentionally or by mistake, you will not get details about it,
  • It uses JSON-formatted results, but I wasn't able to find a schema or documentation on it,
  • Its CLI is non-intuitive. Specifically, it provides osprofiler trace show command (as per docs), but there's no osprofiler trace list or other command to list traces,
  • It requires to have Mongo installation and therefore is harder to configure,

Also, the documentation stats that osprofiler is different from cProfile in a way that it skips random Python calls, however, it's not hard to strip those from the .prof file (but why would you need to do that though?), and it doesn't look like an advantage to me. Maybe it makes more sense for profiling components like Nova or Neutron though.

Further Reading

Saturday, April 1, 2017

Update on libvirt bhyve driver: UEFI support

New libvirt version 3.2.0 will be released really soon (maybe even before I publish this post), and I'm happy to announce that it brings UEFI support for the bhyve driver.

Specifically, starting with this version it's possible:

  • Point domains to the UEFI boot ROM to boot UEFI guests (instead of using grub-bhyve(1) for example)
  • Connect to guests through VNC (e.g. using vncviewer from net/tigervnc)
  • Use an USB tablet input device

I've updated the official documentation with description of these features and an example, scroll to Example config (Linux UEFI guest, VNC, tablet).

Gotchas: as you might now, bhyve's fbuf device accepts the vga option (e.g. vga=off etc), and different guest OSes might require different settings to make things work. It's not possible to change this using libvirt right now, so the default value (which is vga=io) is used, hence some guests will not work.

Kudos to Fabian Freyer for doing the majority of that work as a part of GSoC 2016.

Sunday, December 4, 2016

Building libvirt from git on macOS

Update 1 (Dec 20th, 2016): a patched version of Apple's rpcgen was committed to homebrew, please use this one instead. Proper command for it is:

./configure ac_cv_path_RPCGEN=$(brew --prefix rpcgen)/bin/rpcgen ...

Update 2 (Dec 21th, 2016): now it's possible to install libvirt's current git version using homebrew (thanks to Justin Clift!) with:

brew install --HEAD libvirt

Considering all that, this blog post doesn't make much sense anymore, just keeping that for sake of history.


Libvirt is available on macOS via homebrew, however, building a git version (rather than from a release tarball) might be a little tricky.

Recently, I've fixed some minor issues concerning building on macOS, so here is a little guide on building that on macOS.

The first thing needed is a relatively fresh rpcgen, as the one that comes with macOS is too old. I've created a homebrew formula for rpcgen from FreeBSD, you can grab it here: https://gist.github.com/novel/0d74cdbc7b71f60640a42b52c9cc1459 and install it. Please consult homebrew docs if you're not sure how to do that.

Now it's just a matter to point it to the proper rpcgen and build it as usual:

$ ./configure ac_cv_path_RPCGEN=/usr/local/bin/bsdrpcgen --without-sasl
$ make
$ make check

There are some tests failing, need to figure out why, but at least test suite is working.

PS The --without-sasl flag is not really necessary, it's just me being lazy to find out why it failed to find proper headers/libs for me at that time.

Sunday, November 13, 2016

Playing around with valgrind in libvirt

Valgrind is a well known memory debugging tool. It doesn't seem to officially support FreeBSD, but, luckily, it's available in FreeBSD ports.

Using it is not hard (at least on a basic level), but requires some patience and attention.

Fortunately, libvirt comes provides an easy interface for adding unit tests, and it could be used for running with valgrind as well. Obviously, tests should cover code paths we want to test, but it's still much more convenient than triggering all the stuff manually.

Individual tests could be executed like this:

VIR_TEST_RANGE=22 VIR_TEST_DEBUG=1 ./tests/bhyvexml2argvtest

bhyvexml2argvtest is a test that checks that the bhyve driver converts XML to a proper set of actual bhyve(8) commands. VIR_TEST_RANGE tests to run only test #22 from the suite, and VIR_TEST_DEBUG=1 makes it produce more verbose output.

Adding valgrind into the game is fairly simple:

VIR_TEST_RANGE=22 VIR_TEST_DEBUG=1 valgrind --trace-children=yes --tool=memcheck --leak-check=full   ./tests/bhyvexml2argvtest

Note: I also configured libvirt with '-O0 -g' in CFLAGS for debugging.

It produces a bunch of information about leaks, for example:

==28853== 128 bytes in 1 blocks are definitely lost in loss record 134 of 173
==28853==    at 0x4C246C0: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==28853==    by 0x7D906D8: vasprintf_l (in /lib/libc.so.7)
==28853==    by 0x512876F: virVasprintfInternal (virstring.c:481)
==28853==    by 0x512886D: virAsprintfInternal (virstring.c:502)
==28853==    by 0x40D9A9: testCompareXMLToArgvHelper (bhyvexml2argvtest.c:109)
==28853==    by 0x40DFD8: virTestRun (testutils.c:180)
==28853==    by 0x40DD50: mymain (bhyvexml2argvtest.c:177)
==28853==    by 0x40FA77: virTestMain (testutils.c:992)
==28853==    by 0x40DE0E: main (bhyvexml2argvtest.c:193)
==28853== 
==28853== 128 bytes in 1 blocks are definitely lost in loss record 135 of 173
==28853==    at 0x4C246C0: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==28853==    by 0x7D906D8: vasprintf_l (in /lib/libc.so.7)
==28853==    by 0x512876F: virVasprintfInternal (virstring.c:481)
==28853==    by 0x512886D: virAsprintfInternal (virstring.c:502)
==28853==    by 0x40D9F5: testCompareXMLToArgvHelper (bhyvexml2argvtest.c:111)
==28853==    by 0x40DFD8: virTestRun (testutils.c:180)
==28853==    by 0x40DD50: mymain (bhyvexml2argvtest.c:177)
==28853==    by 0x40FA77: virTestMain (testutils.c:992)
==28853==    by 0x40DE0E: main (bhyvexml2argvtest.c:193)

Moving from bottom to top of these traces, it's fair to assume that the issue is somewhere in testCompareXMLToArgvHelper. It looks like this:

 96 
 97 static int
 98 testCompareXMLToArgvHelper(const void *data)
 99 {
100     int ret = -1;
101     const struct testInfo *info = data;
102     char *xml = NULL;
103     char *args = NULL, *ldargs = NULL, *dmargs = NULL;
104 
105     if (virAsprintf(&xml, "%s/bhyvexml2argvdata/bhyvexml2argv-%s.xml",
106                     abs_srcdir, info->name) < 0 ||
107         virAsprintf(&args, "%s/bhyvexml2argvdata/bhyvexml2argv-%s.args",
108                     abs_srcdir, info->name) < 0 ||
109         virAsprintf(&ldargs, "%s/bhyvexml2argvdata/bhyvexml2argv-%s.ldargs",
110                     abs_srcdir, info->name) < 0 ||
111         virAsprintf(&dmargs, "%s/bhyvexml2argvdata/bhyvexml2argv-%s.devmap",
112                     abs_srcdir, info->name) < 0)
113         goto cleanup;
114 
115     ret = testCompareXMLToArgvFiles(xml, args, ldargs, dmargs, info->flags);
116 
117  cleanup:
118     VIR_FREE(xml);
119     VIR_FREE(args);
120     return ret;
121 }

We can see that we call virAsnprintf a few times to populate xml, args, ldargs and dmargs, but later, in the cleanup section, we only free xml and args, leaking ldargs and dmargs, just like valgrind reported.

Now trying to fix that by adding two VIR_FREE calls for leaked variables and try again.

After running the test again, this warning is gone (yay!), so moving to the next one:

==29877== 296 (200 direct, 96 indirect) bytes in 1 blocks are definitely lost in loss record 146 of 171
==29877==    at 0x4C25C90: calloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==29877==    by 0x50C47D8: virAllocVar (viralloc.c:560)
==29877==    by 0x510D3A1: virObjectNew (virobject.c:193)
==29877==    by 0x510D4FC: virObjectLockableNew (virobject.c:219)
==29877==    by 0x51EBFC1: virGetConnect (datatypes.c:127)
==29877==    by 0x40D680: testCompareXMLToArgvFiles (bhyvexml2argvtest.c:34)
==29877==    by 0x40DA1B: testCompareXMLToArgvHelper (bhyvexml2argvtest.c:115)
==29877==    by 0x40DFF0: virTestRun (testutils.c:180)
==29877==    by 0x40DD68: mymain (bhyvexml2argvtest.c:179)
==29877==    by 0x40FA8F: virTestMain (testutils.c:992)
==29877==    by 0x40DE26: main (bhyvexml2argvtest.c:195)
==29877== 

And indeed, it looks like connect object is leaking here:

 22 static int testCompareXMLToArgvFiles(const char *xml,
 23                                      const char *cmdline,
 24                                      const char *ldcmdline,
 25                                      const char *dmcmdline,
 26                                      unsigned int flags)
 27 {
 28     char *actualargv = NULL, *actualld = NULL, *actualdm = NULL;
 29     virDomainDefPtr vmdef = NULL;
 30     virCommandPtr cmd = NULL, ldcmd = NULL;
 31     virConnectPtr conn;
 32     int ret = -1;
 33 
 34     if (!(conn = virGetConnect()))
 35         goto out;
  --- skip ---
 82  out:
 83     VIR_FREE(actualargv);
 84     VIR_FREE(actualld);
 85     VIR_FREE(actualdm);
 86     virCommandFree(cmd);
 87     virCommandFree(ldcmd);
 88     virDomainDefFree(vmdef);
 89     return ret;
 90 }

So adding virObjectUnref call for the connection object and trying again. And after this there's only one warning left:

==30532== 7 bytes in 1 blocks are definitely lost in loss record 6 of 169
==30532==    at 0x4C246C0: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==30532==    by 0x7D361DF: strdup (in /lib/libc.so.7)
==30532==    by 0x5128C71: virStrdup (virstring.c:714)
==30532==    by 0x410B8F: bhyveBuildNetArgStr (bhyve_command.c:94)
==30532==    by 0x411410: virBhyveProcessBuildBhyveCmd (bhyve_command.c:305)
==30532==    by 0x40D6FF: testCompareXMLToArgvFiles (bhyvexml2argvtest.c:46)
==30532==    by 0x40DA27: testCompareXMLToArgvHelper (bhyvexml2argvtest.c:116)
==30532==    by 0x40DFFC: virTestRun (testutils.c:180)
==30532==    by 0x40DD74: mymain (bhyvexml2argvtest.c:180)
==30532==    by 0x40FA9B: virTestMain (testutils.c:992)
==30532==    by 0x40DE32: main (bhyvexml2argvtest.c:196)

This one is a little harder. Reading the code of bhyveBuildNetArgStr, it's not exactly obvious where it leaks, because net->ifname was freed in every code path.

 46 static int
 47 bhyveBuildNetArgStr(virConnectPtr conn,
 48                     const virDomainDef *def,
 49                     virDomainNetDefPtr net,
 50                     virCommandPtr cmd,
 51                     bool dryRun)
 52 {
 53     char macaddr[VIR_MAC_STRING_BUFLEN];
 54     char *realifname = NULL;
 55     char *brname = NULL;
 56     char *nic_model = NULL;
 57     int ret = -1;
 58     virDomainNetType actualType = virDomainNetGetActualType(net);
 59 
 60     if (STREQ(net->model, "virtio")) {
 61         if (VIR_STRDUP(nic_model, "virtio-net") < 0)
 62             return -1;
 63     } else if (STREQ(net->model, "e1000")) {
 64         if ((bhyveDriverGetCaps(conn) & BHYVE_CAP_NET_E1000) != 0) {
 65             if (VIR_STRDUP(nic_model, "e1000") < 0)
 66                 return -1;
 67         } else {
 68             virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
 69                            _("NIC model 'e1000' is not supported "
 70                              "by given bhyve binary"));
 71             return -1;
 72         }
 73     } else {
 74         virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
 75                        _("NIC model '%s' is not supported"),
 76                        net->model);
 77         return -1;
 78     }
 79 
 80     if (actualType == VIR_DOMAIN_NET_TYPE_BRIDGE) {
 81         if (VIR_STRDUP(brname, virDomainNetGetActualBridgeName(net)) < 0)
 82             goto out;
 83     } else {
 84         virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
 85                        _("Network type %d is not supported"),
 86                        virDomainNetGetActualType(net));
 87         goto out;
 88     }
 89 
 90     if (!net->ifname ||
 91         STRPREFIX(net->ifname, VIR_NET_GENERATED_PREFIX) ||
 92         strchr(net->ifname, '%')) {
 93         VIR_FREE(net->ifname);
 94         if (VIR_STRDUP(net->ifname, VIR_NET_GENERATED_PREFIX "%d") < 0)
 95             goto out;
 96     }
 97 
 98     if (!dryRun) {
 99         if (virNetDevTapCreateInBridgePort(brname, &net->ifname, &net->mac,
100                                            def->uuid, NULL, NULL, 0,
101                                            virDomainNetGetActualVirtPortProfile(net),
102                                            virDomainNetGetActualVlan(net),
103                                            VIR_NETDEV_TAP_CREATE_IFUP | VIR_NETDEV_TAP_CREATE_PERSIST) < 0)
104             goto out;
105 
106         realifname = virNetDevTapGetRealDeviceName(net->ifname);
107 
108         if (realifname == NULL)
109             goto out;
110 
111         VIR_DEBUG("%s -> %s", net->ifname, realifname);
112         /* hack on top of other hack: we need to set
113          * interface to 'UP' again after re-opening to find its
114          * name
115          */
116         if (virNetDevSetOnline(net->ifname, true) != 0)
117             goto out;
118     } else {
119         if (VIR_STRDUP(realifname, "tap0") < 0)
120             goto out;
121     }
122 
123 
124     virCommandAddArg(cmd, "-s");
125     virCommandAddArgFormat(cmd, "%d:0,%s,%s,mac=%s",
126                            net->info.addr.pci.slot, nic_model,
127                            realifname, virMacAddrFormat(&net->mac, macaddr));
128 
129     ret = 0;
130  out:
131     VIR_FREE(net->ifname);
132     VIR_FREE(realifname);
133     VIR_FREE(brname);
134     VIR_FREE(nic_model);
135 
136     return ret;
137 }

Also, I had an assumption (wrong!) that I was running with dryRun == true in the test. Anyway, when things go in unexpected ways, it's good to run lldb:

(lldb) breakpoint set --file bhyve_command.c --line 94
Breakpoint 1: where = bhyvexml2argvtest`bhyveBuildNetArgStr + 731 at bhyve_command.c:94, address = 0x0000000000410b3d

Then a couple of next's and we can set a watchpoint and see what happens:

(lldb) print net->ifname
(char *) $1 = 0x0000000807343678 "vnet%d"
(lldb) watchpoint set var net->ifname
Watchpoint created: Watchpoint 1: addr = 0x80727f108 size = 8 state = enabled type = w
    declare @ '/home/novel/code/libvirt/src/bhyve/bhyve_command.c:49'
    watchpoint spec = 'net->ifname'
    new value: 0x0000000807343678
(lldb) c
Process 53109 resuming

Watchpoint 1 hit:
old value: 0x0000000807343678
new value: 0x0000000000000000
Process 53109 stopped
* thread #1: tid = 100513, 0x0000000800ccadb6 libvirt.so.0`virStrdup(dest=0x000000080727f108, src="vnet0", report=true, domcode=57, filename="bhyvexml2argvmock.c", funcname="virNetDevTapCreateInBridgePort", linenr=32) + 43 at virstring.c:712, stop reason = watchpoint 1
    frame #0: 0x0000000800ccadb6 libvirt.so.0`virStrdup(dest=0x000000080727f108, src="vnet0", report=true, domcode=57, filename="bhyvexml2argvmock.c", funcname="virNetDevTapCreateInBridgePort", linenr=32) + 43 at virstring.c:712
   709            size_t linenr)
   710  {
   711      *dest = NULL;
-> 712      if (!src)
   713          return 0;
   714      if (!(*dest = strdup(src))) {
   715          if (report)
(lldb) bt
* thread #1: tid = 100513, 0x0000000800ccadb6 libvirt.so.0`virStrdup(dest=0x000000080727f108, src="vnet0", report=true, domcode=57, filename="bhyvexml2argvmock.c", funcname="virNetDevTapCreateInBridgePort", linenr=32) + 43 at virstring.c:712, stop reason = watchpoint 1
  * frame #0: 0x0000000800ccadb6 libvirt.so.0`virStrdup(dest=0x000000080727f108, src="vnet0", report=true, domcode=57, filename="bhyvexml2argvmock.c", funcname="virNetDevTapCreateInBridgePort", linenr=32) + 43 at virstring.c:712
    frame #1: 0x0000000800862168 bhyvexml2argvmock.so`virNetDevTapCreateInBridgePort(brname="virbr0", ifname=0x000000080727f108, macaddr=0x000000080727f004, vmuuid="\x04\x11㮰P+874\a\b", tunpath=0x0000000000000000, tapfd=0x0000000000000000, tapfdSize=0, virtPortProfile=0x0000000000000000, virtVlan=0x0000000000000000, fakeflags=9) + 83 at bhyvexml2argvmock.c:32
    frame #2: 0x0000000000410bf3 bhyvexml2argvtest`bhyveBuildNetArgStr(conn=0x000000080725e000, def=0x000000080727c000, net=0x000000080727f000, cmd=0x0000000807245280, dryRun=false) + 913 at bhyve_command.c:99
    frame #3: 0x00000000004113eb bhyvexml2argvtest`virBhyveProcessBuildBhyveCmd(conn=0x000000080725e000, def=0x000000080727c000, dryRun=false) + 610 at bhyve_command.c:304
    frame #4: 0x000000000040d70a bhyvexml2argvtest`testCompareXMLToArgvFiles(xml="/home/novel/code/libvirt/tests/bhyvexml2argvdata/bhyvexml2argv-net-e1000.xml", cmdline="/home/novel/code/libvirt/tests/bhyvexml2argvdata/bhyvexml2argv-net-e1000.args", ldcmdline="/home/novel/code/libvirt/tests/bhyvexml2argvdata/bhyvexml2argv-net-e1000.ldargs", dmcmdline="/home/novel/code/libvirt/tests/bhyvexml2argvdata/bhyvexml2argv-net-e1000.devmap", flags=0) + 225 at bhyvexml2argvtest.c:48
    frame #5: 0x000000000040da26 bhyvexml2argvtest`testCompareXMLToArgvHelper(data=0x000000000063e460) + 405 at bhyvexml2argvtest.c:117
    frame #6: 0x000000000040dfe3 bhyvexml2argvtest`virTestRun(title="BHYVE XML-2-ARGV net-e1000", body=(bhyvexml2argvtest`testCompareXMLToArgvHelper at bhyvexml2argvtest.c:101), data=0x000000000063e460) + 245 at testutils.c:180
    frame #7: 0x000000000040dd5b bhyvexml2argvtest`mymain + 789 at bhyvexml2argvtest.c:179
    frame #8: 0x000000000040fa82 bhyvexml2argvtest`virTestMain(argc=1, argv=0x00007fffffffe6e8, func=(bhyvexml2argvtest`mymain at bhyvexml2argvtest.c:127)) + 1675 at testutils.c:992
    frame #9: 0x000000000040de19 bhyvexml2argvtest`main(argc=1, argv=0x00007fffffffe6e8) + 50 at bhyvexml2argvtest.c:195
    frame #10: 0x000000000040d44f bhyvexml2argvtest`_start + 383
(lldb) 

But virStrdup is not really helpful, so going to the previous frame to see the logic behind that:

(lldb) fr s 1
frame #1: 0x0000000800862168 bhyvexml2argvmock.so`virNetDevTapCreateInBridgePort(brname="virbr0", ifname=0x000000080727f108, macaddr=0x000000080727f004, vmuuid="\x04\x11㮰P+874\a\b", tunpath=0x0000000000000000, tapfd=0x0000000000000000, tapfdSize=0, virtPortProfile=0x0000000000000000, virtVlan=0x0000000000000000, fakeflags=9) + 83 at bhyvexml2argvmock.c:32
   29                                      virNetDevVlanPtr virtVlan ATTRIBUTE_UNUSED,
   30                                      unsigned int fakeflags ATTRIBUTE_UNUSED)
   31   {
-> 32       if (VIR_STRDUP(*ifname, "vnet0") < 0)
   33           return -1;
   34       return 0;
   35   }
(lldb) 

Finally, things are getting clear: bhyveBuildNetArgStr was calling virNetDevTapCreateInBridgePort that is mocked for tests. And the mocked version does not free ifname before calling strdup on it, so it's never freed again. The rest of the fix is trivial, problem solved.

All in all, this appears like an interesting and useful exercise. This is probably too detailed, but hopefully it'll be useful for somebody who's just starting the journey of searching for memory leaks (like myself).

Friday, November 4, 2016

bhyve vs VirtualBox benchmarking, part2

"There are in order of increasing severity: lies, damn lies, statistics, and computer benchmarks",
man 8 diskinfo

I got some feedback to my previous post about benchmarking of bhyve and VirtualBox:

So I decided to do some more tests and include e1000 for networking tests and try different drivers and image types for I/O tests.

Networking test

This is a little more extensive test than the one in my previous blog post, now it includes e1000 and virtio-net for both VirtualBox and bhyve. Setting for the test is still the same: iperf and bridged network mode. Commands remain the same, on VM side I run:

iperf -s

On the host, I run:

iperf -c $vmip

And calculate the average of 8 runs. Results:

And actual values (in Gbits/sec) are:

VirtualBoxbhyve
e10001.428751.57375
virtio-net0.432752.4

The most shocking part here is that in VirtualBox e1000 is more than 3x times faster than virtio-net. This seems a little strange, esp. considering that e1000 performance in bhyve is almost the same, but virtio-net in bhyve is approx. 1.5x times faster than e1000 (that's probably a very huge difference too, but at least it's expected virtio to be faster).

I/O testing

I decided to check things suggested in the tweet above and started with disk configuration. I've converted my image to the "fixed size" type image like this:

VBoxManage clonehd uefi_fbsd_20gigs.vdi uefi_fbsd_20gigs_fixed.vdi --variant Fixed

Then I conducted tests for the following configurations:

  • VirtualBox + fixed size image
  • VirtualBox + dynamic size image
  • bhyve + virtio-blk
  • bhyve + ahci-hd

The same image was used for all tests. Fixed size image was produced using the command specified above, raw image for bhyve was created from the VirtualBox image using the qemu-img(1) tool.

I started with bonnie++ test at results surprised me, to put it softly:

Here, we can see that bhyve with virtio-blk shows the best write speed (that is expected), but then shows the worst rewrite speed (which is a little surprising, but the gap is minimal) and shows the worst read speed (more than 2 times slower than VirtualBox; extremely surprising). After that I decided to take a few days break and then to try some different ways of benchmarking.

So, for read performance I used the diskinfo(8) tool this way:

diskinfo -tv ada0 | grep middle

and use the average of 16 runs. For writing, I used dd(1):

dd bs=1M count=2048 if=/dev/zero of=test conv=sync; sync; rm test

and also use the average of 16 runs. I got the following results:

And the numbers (all values are kbytes/sec):

vbox (fixed size img)vbox (dynamic img)bhyve (ahci-hd)bhyve (virtio-blk)
diskinfo1232397152877912960552647685
dd113737135088113889115924

Frankly, this didn't help to figure out state of things, because these results are kind of opposite to what bonnie++ showed: bhyve with virtio-blk shows very high read speed as demonstrated by diskinfo, 1.7x faster than VirtualBox. On the other hand, the numbers that diskinfo is showing are crazy: 2647 mbytes/sec. This feels more like RAM transfer rates, so it looks like some sort of caching is involved here.

As for the dd(1) test part, bhyve with virtio-blk is 16.5% slower than VirtualBox using dynmanic size image.

Conclusion

  • For VirtualBox, if choosing between e1000 and virtio-net, e1000 definitely provides better performance over virtio-net, at least on FreeBSD hosts with FreeBSD guests
  • virtio-net in bhyve is approx. 1.5x faster than VirtualBox with e1000
  • I'll refrain from comments on I/O tests.

Further Reading