python-like kwargs in C++

When dealing with functions that have lots of optional parameters, or at least for which resonable defaults are readily available, it’s often a bit of a frustration in C++. Generally defaults are specified during a function declaration as in:

double BinarySearch(std::function<double(double> fn,
                    int max_depth = 16, double epsilon = 1e-9,
                    double lower_bound = 0, double upper_bound = 100);

And then if we call BinarySearch with only one parameter then the call will use the default values for the rest. But what if I want to specify custom bound, and use the defaults for the other parameters? Admittedly, this is a somewhat contrived example since bounds are less likely to be optional then the others, and we could reorder them better going from most-likely-to-be-specified to least, but it’s easy to see how something more flexible would be desirable.

Consider then the following two code snippets. Which is more readable?

First snippet:

double solution = BinarySearch(fn, 0, 100);

Second snippet:

double solution = BinarySearch(fn, lower_bound = 0, upper_bound = 100);

I really like the way that optional arguments work in python with kwargs. I’d love to have that same kind of functionality in C++. kwargs.h implements one mechanism of achieving this.

How does it work

kwargs takes advantage of variadic templates in C++ to build up a single data structure which contains all of the optional parameters (I’ll call this a “parameter pack”). Each optional parameter of type T is stored in a structure of type Arg<tag,T> where tag is a unique numeric identifier associated with a particular optional argument key.

The parameter pack data structure derives from Arg<tag,T> for each (tag,T) pair that shows up in the list of optional arguments.

Overloading of the assignment (=) operator gives us an opportunity for building the (tag,T) pairs within the parameter list of the function call.

Get it

See the source and doxygen on github.

Example

I’ll conclude with an example usage:

#include <iostream>
#include "kwargs.h"
 
// these are tags which will uniquely identify the arguments in a parameter
// pack
enum Keys {
  c_tag,
  d_tag
};
 
// global symbols used as keys in list of kwargs
kw::Key<c_tag> c_key;
kw::Key<d_tag> d_key;
 
// a function taking kwargs parameter pack
template <typename... Args>
void foo(int a, int b, Args... kwargs) {
  // first, we construct the parameter pack from the parameter pack
  kw::ParamPack<Args...> params(kwargs...);
 
  std::cout << "foo:\n--------"
            << "\na: " << a
            << "\nb: " << b
  // We can attempt to retrieve a key while providing a default fallback value.
  // If c_key is in kwargs then this will return the value associated with
  // that key, and will have the correct type. Note that the type of the default
  // parameter in this case is const char*.
            << "\nc: " << kw::Get(params,c_key,"null");
  // We can also do stuff conditionally based on whether or not arg exists in
  // the param pack. We still need to provide a default value, since we need to
  // know the return type of the Get function when the key is not in kwargs.
  // In this case, the default value wont ever be used at runtime.
  if( kw::ContainsTag<c_tag,Args...>::result ) {
    std::cout << "\nd: " << kw::Get(params,d_key,0);
  }
 
  std::cout << "\n\n";
}
 
int main( int argc, char** argv )
{
  foo(1, 2);
  foo(1, 2, c_key=3);
  foo(1, 2, c_key=3, d_key=4);
  foo(1, 2, d_key=4);
}

No Comments

Full Disk Encryption with USB master key

When I decided to go with full disk encryption on my machines, I had a pretty hard time figuring out exactly what to do. Not only were there tools and commands to learn, but there was quite a bit of design involved in the process. In this post, I will describe the setup I chose, and the scripts I use to make things a bit more convenient.

Partition Design

partition size label file system mount point notes
sda1 200MiB boot ext4 /boot
sda2 20GiB luks/ext4 /
sda3 218GiB luks/ext4 /home
sda4 6GiB Extended Partition
sda5 4096MiB recovery ext4 / Minimal Ubuntu for recovery
sda6 1900MiB luks/swap

Creating the partition layout

After deciding on a partition scheme, the next step was to create the partitions and format them. I created a 14.04 USB installer and booted the machine in the “Try Ubuntu” mode.

This part is easy. I used gparted to create the partition layout above. For the three encrypted partitions (marked luks/… above) I left them “unformatted”.

Formatting the encrypted partitions

Instructions for the next step came mostly from the Arch wiki. The process involves first formatting the encrypted partitions with a Linux Unified Key Setup (LUKS) header. The luks header contains the AES master key as well as key information for up to eight passphrases (note, in this context a passphrase may be a file of arbitrary data) allowing access to that master key.

Initially, I chose to lock down the root and home partitions using a keyfile passphrase, and later added a password as a passphrase, so I will go through that process.

First, we need to create a keyfile for each partition. I used /dev/random to source random bytes (rather than /dev/urandom which is faster but pseudorandom). Knowing that /dev/random sources user interface event arrival times for randomness I made sure to do lots of typing while I was reading off a key. I chose to use a keysize of 512bytes. I initially accomplished this with

$ cat /dev/random > root.key

while watching in another terminal with ls -l until the file was over 512 bytes. Then I used

$ truncate -s 512 root.key

to reduce the size to 512. After doing this a few times I created a small script to provide saner feedback on how much data had been read from /dev/random

#!/usr/bin/python
# key_progress.py
import sys
 
max = 512
bytes = 0
while( bytes < max ):
  data = sys.stdin.read(1)
  bytes += len(data)
  fraction = bytes/float(max)
  digits = int(10*fraction)
  bars = '='*digits
  spaces = ' '*(10-digits)
  percent = 100*fraction
  text = "\r[%s%s] : %04.2f%%" % (bars,spaces,percent)
  sys.stdout.write(text)
  sys.stdout.flush()
sys.stdout.write('\n')

Usage is like

$ cat /dev/random | tee -a root.key | key_progress.py

Next, I set up the root partition. As described in the wiki, we create the luks header with

# cryptsetup --cipher=aes-xts-plain64 --key-file=~/root.key --key-size=512 --hash=sha51 --iter-time=5000 --use-random luksFormat /dev/sda2

Please research the different cipher options but at the time of this writing my research indicates that aes-xts is generally the best option for a general-use large block device (hard drive).

Note that one may omit the --key-file=/path/to/file flag to specify a password instead of a keyfile, but I chose to start with a keyfile and add a password later.

Once the block device has a proper luks header we can expose the encrypted block device as an unencrypted block device with the following command.

# cryptsetup luksOpen /dev/sda2 root

This will expose the unencrypted blocks of /dev/sda2 in /dev/mapper/root as if it were a raw block device. We can then create an ext4 filesystem on that encrypted device via

# mkfs.ext4 /dev/mapper/root

I repeated the same steps for the home partition:

# cat /dev/random | tee -a home.key | key_progress.py
# cryptsetup --cipher=aes-xts-plain64 --key-file=~/home.key --key-size=512 --hash=sha51 --iter-time=5000 --use-random luksFormat /dev/sda3
# cryptsetup luksOpen /dev/sda3 home
# mkfs.ext4 /dev/mapper/home

Installing Ubuntu 14.04

The installation process is another of the easy steps. Just launch “install ubuntu” from the desktop icon. When it asks where to install choose “do something else” and specify the partitions manually.

device type mount point format?
/dev/sda1 ext4 /boot No
/dev/mapper/root ext4 / No
/dev/mapper/home ext4 /home No
/dev/sda6 swap

The rest of the installation is business as usual

Post-install work

After installing, and before rebooting we need to do some work on the new system.

First, let’s setup an encrypted swap. If the ubuntu live-usb system happens to be using our formatted swap space then we need to turn off the swap. Then we run the following

# swapoff -a
# cryptsetup -d /dev/random create cryptswap /dev/sda6
# mkswap -f /dev/mapper/cryptswap -v1

At this point I realized that I had no way to provide the keyfile for unlocking the root partition at boot so I added a password to encrypted root partition.

# cryptsetup --key-file=/root.key luksAddKey /dev/sda2

Now we need to setup the new systems cryptab. Start by mounting the filesystem for the new installation.

# cd /mnt
# mkdir root
# mount /dev/mapper/root root
# mount /dev/mapper/home root/home
# mount /dev/sda1 root/boot

Now we edit /etc/cryptab to setup the encrypted volumes. My crypttab looks like the following:

# root filesystem
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 none luks

# home filesystem
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

You can find the UUID of the relevant partitions by running blkid as root.

Note that I’ve placed home.key in /root so that once the root partition is decrypted the home partition can be decrypted without requiring a second input of the password.

Now that the cryptab is setup, we need to rebuild the initramfs for boot. We do this by chroot-ing into the new system and running update-initramfs -u

# mount -o bind /proc root/proc
# mount -o bind /dev root/dev
# mount -o bind /dev/pts root/dev/pts
# mount -o bind /sys root/sys
# chroot root
# update-initramfs -u

And that’s it. Reboot now and ubuntu will ask you for a password to expose the root partition before mounting it. You may notice a warning about /home not being mounted and given options to wait, skip, or do something else. Just give it a few more seconds. I guess the timeout built in for mounting /home isn’t long enough to also deal with decrypting it.

USB Master Key

The process of entering the password every boot is troublesome and, to be honest, my ability to recall secure passwords is not to be trusted. In order to address this shortcoming, I decided to create a USB stick whose sole purpose was to act like a password. I do not consider this insecure because, should I lose the USB key, I can revoke the encryption passphrase from the LUKS header and make it no longer usable on my system.

I considered several options for how to go about creating the USB key. I considered a separate boot initramfs with grub installed on the key to actually boot from the key. I also considered storing the keyfile on a filesystem and searching removable filesystems at boot, or trying to identify the disk by partition label or UUID. Ultimately, I decided instead to use an unformatted partition of the USB key to store the keyfile.

The following can be performed on target system, however I suggest creating a backup of the boot/initramfs-...-generic file. If you mess anything up you can drop to grub (press shift 5 times at boot) and alter the grub boot command to use the backup initramfs to get back into a useable system.

First I created the keyfile as usual:

$ cat /dev/random | tee -a keymaster_root.key | key_progress.py
$ truncate -s 512 keymaster_root.key

Then I partitioned the USB key with an unformatted partition of 1MiB and the rest as LUKS/ext4. The second partition is somewhat irrelevant for this discussion except to show that the USB key is still useful for storage.

Now copy the keyfile to the unformatted space, and add the key to an unused slot of the LUKS header for the root partition

# dd if=keymaster_root.key of=/dev/sdb1 bs=1 
# cryptsetup luksAddKey /dev/sdb1 keymaster_root.key

Now we find the unique name for the USB stick, particularly it’s first partition so we can identify it when present:

$ ls -l /dev/disk/by-id | grep sdb1

In my case the USB key is given the path /dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1411416161600078-0:0-part1. I then edited my crypttab as follows:

# /etc/crypttab
# root
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 /dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1411416161600078-0:0-part1 luks,keyscript=/sbin/usb_keymaster

# home
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/nadie_chain_root_home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

The keyscript is a script which dm-crypt runs to get the passphrase. It expects the passphrase on stdout. I used the script from the stack overflow link above but took the initial parts of the script from the blog linked above (which makes the script work with plymouth).

The script attempts to read from the raw device provided in ${CRYPTTAB_KEY} (which is the third column in crypttab). If the device file does not exist after 5 seconds, it drops to asking for a password.

#!/bin/sh
#/sbin/usb_keymaster
 
# define counter-intuitive shell logic values (based on /bin/true & /bin/false)
# NB. use FALSE only to *set* something to false, but don't test for
# equality, because a program might return any non-zero on error
TRUE=0
FALSE=1
 
# set DEBUG=$TRUE to display debug messages, DEBUG=$FALSE to be quiet
DEBUG=$TRUE
 
# default path to key-file on the USB/MMC disk
KEYFILE=".keyfile"
 
# maximum time to sleep waiting for devices to become ready before
# asking for passphrase
MAX_SECONDS=2
 
# is plymouth available? default false
PLYMOUTH=$FALSE
if [ -x /bin/plymouth ] && plymouth --ping; then
    PLYMOUTH=$TRUE
fi
 
# is usplash available? default false
USPLASH=$FALSE
# test for outfifo from Ubuntu Hardy cryptroot script, the second test
# alone proves not completely reliable.
if [ -p /dev/.initramfs/usplash_outfifo -a -x /sbin/usplash_write ]; then
    # use innocuous command to determine if usplash is running
    # usplash_write will return exit-code 1 if usplash isn't running
    # need to set a flag to tell usplash_write to report no usplash
    FAIL_NO_USPLASH=1
    # enable verbose messages (required to display messages if kernel boot option "quiet" is enabled
    /sbin/usplash_write "VERBOSE on"
    if [ $? -eq $TRUE ]; then
        # usplash is running
        USPLASH=$TRUE
        /sbin/usplash_write "CLEAR"
    fi
fi
 
# is stty available? default false
STTY=$FALSE
STTYCMD=false
# check for stty executable
if [ -x /bin/stty ]; then
    STTY=$TRUE
    STTYCMD=/bin/stty
elif [ `(busybox stty >/dev/null 2>&1; echo $?)` -eq $TRUE ]; then
    STTY=$TRUE
    STTYCMD="busybox stty"
fi
 
# print message to usplash or stderr
# usage: msg <command> "message" [switch]
# command: TEXT | STATUS | SUCCESS | FAILURE | CLEAR (see 'man usplash_write' for all commands)
# switch : switch used for echo to stderr (ignored for usplash)
# when using usplash the command will cause "message" to be
# printed according to the usplash <command> definition.
# using the switch -n will allow echo to write multiple messages
# to the same line
msg ()
{
    if [ $# -gt 0 ]; then
        # handle multi-line messages
        echo $2 | while read LINE; do
            if [ $PLYMOUTH -eq $TRUE ]; then
                # use plymouth
                plymouth message --text="$LINE"      
            elif [ $USPLASH -eq $TRUE ]; then
                # use usplash
                /sbin/usplash_write "$1 $LINE"      
            else
                # use stderr for all messages
                echo $3 "$2" >&2
            fi
        done
    fi
}
 
dbg ()
{
    if [ $DEBUG -eq $TRUE ]; then
        msg "$@"
    fi
}
 
# read password from console or with usplash
# usage: readpass "prompt"
readpass ()
{
    if [ $# -gt 0 ]; then
        if [ $PLYMOUTH -eq $TRUE ]; then
            PASS="$(plymouth ask-for-password --prompt "$1")"
        elif [ $USPLASH -eq $TRUE ]; then
            usplash_write "INPUTQUIET $1"
            PASS="$(cat /dev/.initramfs/usplash_outfifo)"
        else
            [ $STTY -ne $TRUE ] && msg TEXT "WARNING stty not found, password will be visible"
            /lib/cryptsetup/askpass "$1"
            # echo -n "$1" >&2
            # $STTYCMD -echo
            # read -r PASS </dev/console >/dev/null
            # [ $STTY -eq $TRUE ] && echo >&2
            # $STTYCMD echo
        fi
    fi
    echo -n "$PASS"
}
 
dbg STATUS "Executing usb_keymaster ..."
 
# flag tracking key-file availability
OPENED=$FALSE
 
for TRY in 1 2 3 4 5
do
  if ! [ -e "${CRYPTTAB_KEY}" ]; then 
    dbg TEXT "Waiting for USB stick to be recognized [${TRY}]"
    sleep 1
  fi
done # for TRY
if [ -e "${CRYPTTAB_KEY}" ]; then
  dd if="${CRYPTTAB_KEY}" bs=1 skip=0 count=512 2>/dev/nul
  OPENED=$TRUE
fi
 
 
# clear existing usplash text and status messages
[ $USPLASH -eq $TRUE ] && msg STATUS "                               " && msg CLEAR ""
 
if [ $OPENED -ne $TRUE ]; then
  # dbg TEXT "Failed to find suitable USB/MMC key-file ..."
  readpass "$(printf "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: ")"
else
  # dbg TEXT "Success loading key-file from $SFS ($LABEL)"
  # msg TEXT "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)"
  msg "Unlocking ${CRYPTTAB_SOURCE} (${CRYPTTAB_NAME}) from USB key"
fi
 
#
[ $USPLASH -eq $TRUE ] && /sbin/usplash_write "VERBOSE default"

Be sure to chmod +x /sbin/usb_keymaster to make the script executable.

In order to get this script into the initramfs we need to add the following hook file:

#!/bin/sh
#/etc/initramfs-tools/hooks/usb_keymaster
 
PREREQ=""
 
prereqs() {
  echo "$PREREQ"
}
 
case "$1" in
  prereqs)
    prereqs
    exit 0
  ;;
esac
 
. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions
 
copy_exec /sbin/usb_keymaster /sbin

Lastly, we need to ensure that the USB drivers are loaded at boot time (this may not be necessary with 14.04). Edit /etc/initramfs-tools/modules.

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
uhci_hcd
ehci_hcd
usb_storage

Lastly update the initramfs with

update-initramfs -u

No Comments

Skype in 64bit Ubuntu 13.04

I’ve had some trouble getting skype to work in ubuntu 13.04. The first problem arose after switching to the nvidia binary graphics drivers. Skype kept trying to load the 64bit libraries instead of the 32bit compatability ones. This can be repaired by changing the linker’s search path. The command to launch skype then is:

   ~$ LD_LIBRARY_PATH=/usr/lib32 skype

The second problem was that skype could not find pulse audio, so it was trying to look up hardware devices using ALSA (or so I’ve guessed by stdout). The solution is to install libpulse0:i386

   ~$ sudo apt-get install libpulse0:i386

After restarting skype my microphone and audio devices are all listed as “pulse audio” instead of having a dropdown for various options.

Also, as of skype 4.2.0.11 screen sharing appears to now be available in the linux version… finally.

No Comments