Martin

Parsing command line options

Posted on March 21, 2019March 28, 2021 by Martin

In case you haven’t noticed, I love Boost 😛 so I’m going to introduce you to its Program Options library. It is a command line options parser; it can also read options from an INI file, and it works with STL containers. It supports showing a help message, setting default values for missing options, allows multiple instances of the same option, etc. For complete set of features I refer you to the documentation. This post will be a short introduction.

Let’s start with some code that defines what options our program will accept:

desc.add_options()
	("help", "produce help message")
	("int,i", value<int>(&v)->default_value(42), "int value")
	("float,f", value<float>(&f)->default_value(3.141f), "float value")
	("string,s", value<string>(&s)->default_value("Vorbrodt"), "string value")
	("int_list,a", value<vector<int>>(&vi), "list of int values")
	("string_list,b", value<vector<string>>(&vs), "list of string values")
;

desc.add_options()

("help", "produce help message")

("int,i", value<int>(&v)->default_value(42), "int value")

("float,f", value<float>(&f)->default_value(3.141f), "float value")

("string,s", value<string>(&s)->default_value("Vorbrodt"), "string value")

("int_list,a", value<vector<int>>(&vi), "list of int values")

("string_list,b", value<vector<string>>(&vs), "list of string values")

;

The first option invoked with --help will display a friendly description, like this:

Allowed options:
–help produce help message
-i [ –int ] arg (=42) int value
-f [ –float ] arg (=3.14100003) float value
-s [ –string ] arg (=Vorbrodt) string value
-a [ –int_list ] arg list of int values
-b [ –string_list ] arg list of string values
Options help message.

Next is an integer value option; the first string "int,i" means that you can specify it as either --int or -i on the command line. When specified its value will be pulled into variable v, and if not specified the default value will be 42. Next two options for float and string behave in the exact same way. The parser will throw an exception if you specify those options more than once.
Next are list options: they allow you to specify the same option multiple times and are returned through the vi and vs variables which are std::vector’s of int and string.

Here is the program invocation and the output it produces:

./bin/options -i 1 -f 3.141 -s “Martin” -a 10 -a 11 -a 12 -b “Vorbrodt’s” -b “Blog”
Int value was set to 1
Float value was set to 3.141
String value was set to “Martin”
List of ints value was set to 10
List of ints value was set to 11
List of ints value was set to 12
List of strings value was set to “Vorbrodt’s”
List of strings value was set to “Blog”
Program invocation and output.

Complete source code of the program below.

options.cpp:

#include <iostream>
#include <vector>
#include <string>
#include <boost/program_options.hpp>

using namespace std;
using namespace boost::program_options;

int main(int argc, char** argv)
{
	int v{};
	float f{};
	string s{};
	vector<int> vi{};
	vector<string> vs{};

	options_description desc("Allowed options");
	desc.add_options()
		("help", "produce help message")
		("int,i", value<int>(&v)->default_value(42), "int value")
		("float,f", value<float>(&f)->default_value(3.141f), "float value")
		("string,s", value<string>(&s)->default_value("Vorbrodt"), "string value")
		("int_list,a", value<vector<int>>(&vi), "list of int values")
		("string_list,b", value<vector<string>>(&vs), "list of string values");

	variables_map vm{};
	store(parse_command_line(argc, argv, desc), vm);
	notify(vm);

	if (vm.size() == 0 || vm.count("help"))
	{
		cout << desc << "\n";
		return 1;
	}

	if(vm.count("int")) cout << "Int value was set to " << v << endl;
	if(vm.count("float")) cout << "Float value was set to " << f << endl;
	if(vm.count("string")) cout << "String value was set to \"" << s << "\"" << endl;
	if(vm.count("int_list"))
		for(auto it : vi)
			cout << "List of ints value was set to " << it << endl;
	if(vm.count("string_list"))
		for(auto& it : vs)
			cout << "List of strings value was set to \"" << it << "\"" << endl;

	return 1;
}

#include <iostream>

#include <vector>

#include <string>

#include <boost/program_options.hpp>

using namespace std;

using namespace boost::program_options;

int main(int argc, char** argv)

{

int v{};

float f{};

string s{};

vector<int> vi{};

vector<string> vs{};

options_description desc("Allowed options");

desc.add_options()

("help", "produce help message")

("int,i", value<int>(&v)->default_value(42), "int value")

("float,f", value<float>(&f)->default_value(3.141f), "float value")

("string,s", value<string>(&s)->default_value("Vorbrodt"), "string value")

("int_list,a", value<vector<int>>(&vi), "list of int values")

("string_list,b", value<vector<string>>(&vs), "list of string values");

variables_map vm{};

store(parse_command_line(argc, argv, desc), vm);

notify(vm);

if (vm.size() == 0 || vm.count("help"))

{

cout << desc << "\n";

return 1;

}

if(vm.count("int")) cout << "Int value was set to " << v << endl;

if(vm.count("float")) cout << "Float value was set to " << f << endl;

if(vm.count("string")) cout << "String value was set to \"" << s << "\"" << endl;

if(vm.count("int_list"))

for(auto it : vi)

cout << "List of ints value was set to " << it << endl;

if(vm.count("string_list"))

for(auto& it : vs)

cout << "List of strings value was set to \"" << it << "\"" << endl;

return 1;

}

ANSI escape codes

Posted on March 21, 2019April 11, 2021 by Martin

ANSI escape codes are a way to do more than just plain text in the terminal (be it Windows cmd.exe or UNIX xterm). A picture is worth a thousand words so here’s what I was able to do with them:

All of the text appearance manipulation and coloring was done using a tiny library I wrote yesterday and a very intuitive C++ syntax. Here’s the code responsible for the screenshot above:

colors.cpp:

#include <iostream>
#include "ansi_escape_codes.hpp"

using namespace std;
using namespace ansi_escape_codes;

int main(int argc, char** argv)
{
	cout << bold << "BOLD" << reset << endl;
	cout << faint << "FAINT" << reset << endl;
	cout << italic << "ITALIC" << reset << endl;
	cout << underline << "UNDERLINE" << reset << endl;
	cout << slow_blink << "SLOW BLINK" << reset << endl;
	cout << inverse << "REVERSE" << reset << endl;
	cout << endl;

	for(int n = 0; n <= 255; ++n)
		cout << color_n(n) << "X";
	cout << reset << endl << endl;

	for(int n = 0; n <= 255; ++n)
		cout << color_bg_n(n) << "X";
	cout << reset << endl << endl;

	for(int r = 0; r <= 255; r+=32)
		for(int g = 0; g <= 255; g+=32)
			for(int b = 0; b <= 255; b+=32)
				cout << color_rgb(r, g, b) << "X";
	cout << reset <<endl << endl;

	for(int r = 0; r <= 255; r+=32)
		for(int g = 0; g <= 255; g+=32)
			for(int b = 0; b <= 255; b+=32)
				cout << color_bg_rgb(r, g, b) << "X";
	cout << reset <<endl << endl;

	return 1;
}

#include <iostream>

#include "ansi_escape_codes.hpp"

using namespace std;

using namespace ansi_escape_codes;

int main(int argc, char** argv)

{

cout << bold << "BOLD" << reset << endl;

cout << faint << "FAINT" << reset << endl;

cout << italic << "ITALIC" << reset << endl;

cout << underline << "UNDERLINE" << reset << endl;

cout << slow_blink << "SLOW BLINK" << reset << endl;

cout << inverse << "REVERSE" << reset << endl;

cout << endl;

for(int n = 0; n <= 255; ++n)

cout << color_n(n) << "X";

cout << reset << endl << endl;

for(int n = 0; n <= 255; ++n)

cout << color_bg_n(n) << "X";

cout << reset << endl << endl;

for(int r = 0; r <= 255; r+=32)

for(int g = 0; g <= 255; g+=32)

for(int b = 0; b <= 255; b+=32)

cout << color_rgb(r, g, b) << "X";

cout << reset <<endl << endl;

for(int r = 0; r <= 255; r+=32)

for(int g = 0; g <= 255; g+=32)

for(int b = 0; b <= 255; b+=32)

cout << color_bg_rgb(r, g, b) << "X";

cout << reset <<endl << endl;

return 1;

}

Pretty easy to follow I hope 🙂 The library defines a bunch of stream manipulators that inject the appropriate escape sequences. For example, cout << bold << "BOLD"; will print out, you guessed it, bolded text. color_n picks from the 8 bit color table. color_rgb let’s you define a 24 bit truecolor. The _bg_ version is for selecting the background color. Here’s the complete source code for the library, hope you will find it useful!

P.S. The color_rgb does not appear to work on Mac OS terminal. So far all the codes work only on Linux; haven’t tested on Windows 🙂

ansi_escape_code.hpp:

#pragma once

#include <ostream>

namespace ansi_escape_codes {

#define ENTRY(name, code) \
	static inline struct name##_opcode {} name; \
	inline std::ostream& operator << (std::ostream& os, const name##_opcode&) \
	{ os << "\x1b[" #code "m"; return os; }

ENTRY(reset, 0)
ENTRY(bold, 1)
ENTRY(faint, 2)
ENTRY(italic, 3)
ENTRY(underline, 4)
ENTRY(slow_blink, 5)
ENTRY(rapid_blink, 6)
ENTRY(inverse, 7)
ENTRY(conceal, 8)
ENTRY(crossed_out, 9)
ENTRY(default_font, 10)
ENTRY(alt_font_1, 11)
ENTRY(alt_font_2, 12)
ENTRY(alt_font_3, 13)
ENTRY(alt_font_4, 14)
ENTRY(alt_font_5, 15)
ENTRY(alt_font_6, 16)
ENTRY(alt_font_7, 17)
ENTRY(alt_font_8, 18)
ENTRY(alt_font_9, 19)
ENTRY(fraktur, 20)
ENTRY(doubly_underline, 21)
ENTRY(normal, 22)
ENTRY(not_italic_not_raktur, 23)
ENTRY(underline_off, 24)
ENTRY(blink_off, 25)
ENTRY(inverse_off, 27)
ENTRY(reveal, 28)
ENTRY(not_crossed_out, 29)
ENTRY(default_foreground_color, 39)
ENTRY(default_background_color, 49)
ENTRY(framed, 51)
ENTRY(encircled, 52)
ENTRY(overlined, 53)
ENTRY(not_framed_or_encircled, 54)
ENTRY(not_overlined, 55)

#define COLOR_ENTRY(name, code) \
	static inline struct name##_color {} name; \
	inline std::ostream& operator << (std::ostream& os, const name##_color&) \
	{ os << "\x1b[" #code "m"; return os; }

COLOR_ENTRY(black,   30)
COLOR_ENTRY(red,     31)
COLOR_ENTRY(green,   32)
COLOR_ENTRY(yellow,  33)
COLOR_ENTRY(blue,    34)
COLOR_ENTRY(magenta, 35)
COLOR_ENTRY(cyan,    36)
COLOR_ENTRY(white,   37)

COLOR_ENTRY(black_bg,   40)
COLOR_ENTRY(red_bg,     41)
COLOR_ENTRY(green_bg,   42)
COLOR_ENTRY(yellow_bg,  43)
COLOR_ENTRY(blue_bg,    44)
COLOR_ENTRY(magenta_bg, 45)
COLOR_ENTRY(cyan_bg,    46)
COLOR_ENTRY(white_bg,   47)

COLOR_ENTRY(bright_black,   90)
COLOR_ENTRY(bright_red,     91)
COLOR_ENTRY(bright_green,   92)
COLOR_ENTRY(bright_yellow,  93)
COLOR_ENTRY(bright_blue,    94)
COLOR_ENTRY(bright_magenta, 95)
COLOR_ENTRY(bright_cyan,    96)
COLOR_ENTRY(bright_white,   97)

COLOR_ENTRY(bright_black_bg,   100)
COLOR_ENTRY(bright_red_bg,     101)
COLOR_ENTRY(bright_green_bg,   102)
COLOR_ENTRY(bright_yellow_bg,  103)
COLOR_ENTRY(bright_blue_bg,    104)
COLOR_ENTRY(bright_magenta_bg, 105)
COLOR_ENTRY(bright_cyan_bg,    106)
COLOR_ENTRY(bright_white_bg,   107)

struct color_n
{
	color_n(unsigned char n) : m_n(n) {}
	friend std::ostream& operator << (std::ostream& os, const color_n& n);
private:
	unsigned char m_n;
};

struct color_bg_n
{
	color_bg_n(unsigned char n) : m_n(n) {}
	friend std::ostream& operator << (std::ostream& os, const color_bg_n& n);
private:
	unsigned char m_n;
};

inline std::ostream& operator << (std::ostream& os, const color_n& n)
{ os << "\x1b[38;5;" << (int)n.m_n << "m"; return os; }

inline std::ostream& operator << (std::ostream& os, const color_bg_n& n)
{ os << "\x1b[48;5;" << (int)n.m_n << "m"; return os; }

struct color_rgb
{
	color_rgb(unsigned char r, unsigned char g, unsigned char b) : m_r(r), m_g(g), m_b(b) {}
	friend std::ostream& operator << (std::ostream& os, const color_rgb& c);
private:
	unsigned char m_r, m_g, m_b;
};

struct color_bg_rgb
{
	color_bg_rgb(unsigned char r, unsigned char g, unsigned char b) : m_r(r), m_g(g), m_b(b) {}
	friend std::ostream& operator << (std::ostream& os, const color_bg_rgb& c);
private:
	unsigned char m_r, m_g, m_b;
};

inline std::ostream& operator << (std::ostream& os, const color_rgb& c)
{ os << "\x1b[38;2;" << (int)c.m_r << ";" << (int)c.m_g << ";" << (int)c.m_b << "m"; return os; }

inline std::ostream& operator << (std::ostream& os, const color_bg_rgb& c)
{ os << "\x1b[48;2;" << (int)c.m_r << ";" << (int)c.m_g << ";" << (int)c.m_b << "m"; return os; }

}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

#pragma once

#include <ostream>

namespace ansi_escape_codes {

#define ENTRY(name, code) \

static inline struct name##_opcode {} name; \

inline std::ostream& operator << (std::ostream& os, const name##_opcode&) \

{ os << "\x1b[" #code "m"; return os; }

ENTRY(reset, 0)

ENTRY(bold, 1)

ENTRY(faint, 2)

ENTRY(italic, 3)

ENTRY(underline, 4)

ENTRY(slow_blink, 5)

ENTRY(rapid_blink, 6)

ENTRY(inverse, 7)

ENTRY(conceal, 8)

ENTRY(crossed_out, 9)

ENTRY(default_font, 10)

ENTRY(alt_font_1, 11)

ENTRY(alt_font_2, 12)

ENTRY(alt_font_3, 13)

ENTRY(alt_font_4, 14)

ENTRY(alt_font_5, 15)

ENTRY(alt_font_6, 16)

ENTRY(alt_font_7, 17)

ENTRY(alt_font_8, 18)

ENTRY(alt_font_9, 19)

ENTRY(fraktur, 20)

ENTRY(doubly_underline, 21)

ENTRY(normal, 22)

ENTRY(not_italic_not_raktur, 23)

ENTRY(underline_off, 24)

ENTRY(blink_off, 25)

ENTRY(inverse_off, 27)

ENTRY(reveal, 28)

ENTRY(not_crossed_out, 29)

ENTRY(default_foreground_color, 39)

ENTRY(default_background_color, 49)

ENTRY(framed, 51)

ENTRY(encircled, 52)

ENTRY(overlined, 53)

ENTRY(not_framed_or_encircled, 54)

ENTRY(not_overlined, 55)

#define COLOR_ENTRY(name, code) \

static inline struct name##_color {} name; \

inline std::ostream& operator << (std::ostream& os, const name##_color&) \

{ os << "\x1b[" #code "m"; return os; }

COLOR_ENTRY(black, 30)

COLOR_ENTRY(red, 31)

COLOR_ENTRY(green, 32)

COLOR_ENTRY(yellow, 33)

COLOR_ENTRY(blue, 34)

COLOR_ENTRY(magenta, 35)

COLOR_ENTRY(cyan, 36)

COLOR_ENTRY(white, 37)

COLOR_ENTRY(black_bg, 40)

COLOR_ENTRY(red_bg, 41)

COLOR_ENTRY(green_bg, 42)

COLOR_ENTRY(yellow_bg, 43)

COLOR_ENTRY(blue_bg, 44)

COLOR_ENTRY(magenta_bg, 45)

COLOR_ENTRY(cyan_bg, 46)

COLOR_ENTRY(white_bg, 47)

COLOR_ENTRY(bright_black, 90)

COLOR_ENTRY(bright_red, 91)

COLOR_ENTRY(bright_green, 92)

COLOR_ENTRY(bright_yellow, 93)

COLOR_ENTRY(bright_blue, 94)

COLOR_ENTRY(bright_magenta, 95)

COLOR_ENTRY(bright_cyan, 96)

COLOR_ENTRY(bright_white, 97)

COLOR_ENTRY(bright_black_bg, 100)

COLOR_ENTRY(bright_red_bg, 101)

COLOR_ENTRY(bright_green_bg, 102)

COLOR_ENTRY(bright_yellow_bg, 103)

COLOR_ENTRY(bright_blue_bg, 104)

COLOR_ENTRY(bright_magenta_bg, 105)

COLOR_ENTRY(bright_cyan_bg, 106)

COLOR_ENTRY(bright_white_bg, 107)

struct color_n

{

color_n(unsigned char n) : m_n(n) {}

friend std::ostream& operator << (std::ostream& os, const color_n& n);

private:

unsigned char m_n;

};

struct color_bg_n

{

color_bg_n(unsigned char n) : m_n(n) {}

friend std::ostream& operator << (std::ostream& os, const color_bg_n& n);

private:

unsigned char m_n;

};

inline std::ostream& operator << (std::ostream& os, const color_n& n)

{ os << "\x1b[38;5;" << (int)n.m_n << "m"; return os; }

inline std::ostream& operator << (std::ostream& os, const color_bg_n& n)

{ os << "\x1b[48;5;" << (int)n.m_n << "m"; return os; }

struct color_rgb

{

color_rgb(unsigned char r, unsigned char g, unsigned char b) : m_r(r), m_g(g), m_b(b) {}

friend std::ostream& operator << (std::ostream& os, const color_rgb& c);

private:

unsigned char m_r, m_g, m_b;

};

struct color_bg_rgb

{

color_bg_rgb(unsigned char r, unsigned char g, unsigned char b) : m_r(r), m_g(g), m_b(b) {}

friend std::ostream& operator << (std::ostream& os, const color_bg_rgb& c);

private:

unsigned char m_r, m_g, m_b;

};

inline std::ostream& operator << (std::ostream& os, const color_rgb& c)

{ os << "\x1b[38;2;" << (int)c.m_r << ";" << (int)c.m_g << ";" << (int)c.m_b << "m"; return os; }

inline std::ostream& operator << (std::ostream& os, const color_bg_rgb& c)

{ os << "\x1b[48;2;" << (int)c.m_r << ";" << (int)c.m_g << ";" << (int)c.m_b << "m"; return os; }

}

Two-factor authentication

Posted on March 20, 2019March 28, 2021 by Martin

If you don’t know what multi-factor authentication is please read this before continuing. I am going to assume you understand the security concepts mentioned in this post…

In plain English: two-factor authentication is something you know (your password) and something you have (your token). I will focus on the token in this post. Apps like Google Authenticator and Authy generate one-time time-based tokens, or passwords. They generate them by hashing a shared secret combined with the current time. By default, the resulting token changes every 30 seconds giving the user a short window to authenticate to a service.
You can set it up with GitHub for example; in your user security settings you enable 2-factor authentication, GitHub then generates the shared secret for you which you import into the authenticator app. From then on when you login you must provide your password plus the generated token. Because both you and GitHub have access to the shared secret, both can generate the same token at the same time. If the user provided and the GitHub generated tokens match, the authentication succeeds and you’re logged in.

So what is this post about anyways? What I set-out to do today was to generate the one-time tokens programmatically from a C++ program. I wanted to test this by feeding the same shared secret to Authy and see that both my program and Authy generate the same tokens. With this working I, or you the reader, could add two-factor authentication to our applications, which is cool 🙂

Initially I started reading about the algorithm used to generate the tokens: Time-based One-time Password Algorithm. I sure as hell didn’t want to implement all of this from scratch, so I started looking for an OpenSSL implementation. During my search I came across a free (and available on Mac and Linux) framework to do what I wanted: OATH Toolkit. Once I started reading the documentation the rest fell in place very easily. I generated a dummy shared secret: 00112233445566778899 and fed it to Authy as well as my program (Google Authenticator requires it to be base32 encoded).

Below are screenshots of Authy and my program generating the same tokens. And of course the code!

#include <iostream>
#include <thread>
#include <cstdlib>
#include <liboath/oath.h>

using namespace std;

int main(int argc, char** argv)
{
	auto result = oath_init();
	if(result != OATH_OK)
	{
		cerr << oath_strerror(result) << endl;
		return -1;
	}

	const char secret[] = "00112233445566778899";
	const auto time_step = OATH_TOTP_DEFAULT_TIME_STEP_SIZE;
	const auto digits = 6;

	while(true)
	{
		auto t = time(NULL);
		auto left = time_step - (t % time_step);

		char otp[digits + 1] = {};
		result = oath_totp_generate(
			secret,
			sizeof(secret),
			t,
			time_step,
			OATH_TOTP_DEFAULT_START_TIME,
			digits,
			otp);

		if(result != OATH_OK)
		{
			cerr << oath_strerror(result) << endl;
			return -1;
		}

		cout << "OTP: " << otp << " (" << left << ") \r";
		cout.flush();

		this_thread::sleep_for(1s);
	}

	oath_done();

	return 1;
}

#include <iostream>

#include <thread>

#include <cstdlib>

#include <liboath/oath.h>

using namespace std;

int main(int argc, char** argv)

{

auto result = oath_init();

if(result != OATH_OK)

{

cerr << oath_strerror(result) << endl;

return -1;

}

const char secret[] = "00112233445566778899";

const auto time_step = OATH_TOTP_DEFAULT_TIME_STEP_SIZE;

const auto digits = 6;

while(true)

{

auto t = time(NULL);

auto left = time_step - (t % time_step);

char otp[digits + 1] = {};

result = oath_totp_generate(

secret,

sizeof(secret),

time_step,

OATH_TOTP_DEFAULT_START_TIME,

digits,

otp);

if(result != OATH_OK)

{

cerr << oath_strerror(result) << endl;

return -1;

}

cout << "OTP: " << otp << " (" << left << ") \r";

cout.flush();

this_thread::sleep_for(1s);

}

oath_done();

return 1;

}

Plugins: loading code at runtime

Posted on March 19, 2019March 28, 2021 by Martin

On windows we have the .dll files, .so files on Linux, and .dylib files on Mac. They all have one thing in common: they can be loaded at runtime and provide entry points to call into. One example is an online chat client that uses plugins to add support for more protocols (Google Chat, ICQ, IRC, etc). There are plenty more examples but you get the idea: drop a binary in plugins folder and restart the app and you have just added more functionality without having to recompile and redeploy your application.

So how do we do it? We could go the difficult route and use OS specific calls to find and load the plugin file, then do some more platform specific code to extract the entry point and call it. Or, we could make it easy on ourselves and use Boost DLL library 🙂

I am no expert on this library nor do I want to write a complete tutorial on it; the documentation is great. I have just started using it today in fact and wanted to see how easy it would be to get a basic plugin built and loaded by another program. It took all of 20 minutes to come up with a basic structure so I decided to make a short blog post about my experience so far. And so far it looks promising!

I started by creating a dynamic library which exports two function calls: one to get the name of the plugin and another to get its version. Below is all the code needed to create such a plugin with Boost:

#include <boost/dll/alias.hpp>

namespace plugin
{
	const char* plugin_name()
	{
		return "Vorbrodt's 1st Plugin";
	}

	const char* plugin_version()
	{
		return "1.0";
	}
}

BOOST_DLL_ALIAS(plugin::plugin_name, plugin_name);
BOOST_DLL_ALIAS(plugin::plugin_version, plugin_version);

#include <boost/dll/alias.hpp>

namespace plugin

{

const char* plugin_name()

{

return "Vorbrodt's 1st Plugin";

}

const char* plugin_version()

{

return "1.0";

}

BOOST_DLL_ALIAS(plugin::plugin_name, plugin_name);

BOOST_DLL_ALIAS(plugin::plugin_version, plugin_version);

Next I wrote a simple program which accepts the path to the plugin as a command line argument, loads the library, finally extracts and calls the two entry points. Here’s the code:

#include <iostream>
#include <boost/function.hpp>
#include <boost/dll/import.hpp>

using namespace std;
using namespace boost::dll;
using namespace boost::filesystem;

int main(int argc, char** argv)
{
	path lib_path(argv[1]);

	typedef const char* (plugin_name_t)();
	typedef const char* (plugin_version_t)();

	auto plugin_name = import_alias<plugin_name_t>(lib_path, "plugin_name", load_mode::append_decorations);
	auto plugin_version = import_alias<plugin_version_t>(lib_path, "plugin_version", load_mode::append_decorations);

	cout << "Plugin name    : " << plugin_name() << endl;
	cout << "Plugin version : " << plugin_version() << endl;

	return 1;
}

#include <iostream>

#include <boost/function.hpp>

#include <boost/dll/import.hpp>

using namespace std;

using namespace boost::dll;

using namespace boost::filesystem;

int main(int argc, char** argv)

{

path lib_path(argv[1]);

typedef const char* (plugin_name_t)();

typedef const char* (plugin_version_t)();

auto plugin_name = import_alias<plugin_name_t>(lib_path, "plugin_name", load_mode::append_decorations);

auto plugin_version = import_alias<plugin_version_t>(lib_path, "plugin_version", load_mode::append_decorations);

cout << "Plugin name : " << plugin_name() << endl;

cout << "Plugin version : " << plugin_version() << endl;

return 1;

}

Plugin name : Vorbrodt’s 1st Plugin
Plugin version : 1.0
Program output.

It works! You can take it from here and build a cool plugin engine. Oh I forgot to mention, the same code compiles and behaves the same on Windows, Linux, and Mac 🙂

Micro-benchmarks

Posted on March 18, 2019March 28, 2021 by Martin

I was looking for a way to benchmark a piece of code… I came up with 5 libraries that make it very easy 🙂

I’m not going to write a tutorial on how to use each one because I would basically be rewriting their documentation sections. What I will do is show you how to get started. As an example I will write a simple benchmark that tests copy constructor of std::string. But first, the libraries:

Catch2 and Nonius are header only libraries; be aware of long compile times 🙁 Google, Catch2, and Nonius automatically pick the number of runs and iterations for you, which is nice: no guessing how many times you need to run a function you want to benchmark to get a reasonable performance reading.

Google Benchmark:

#include <string>
#include <benchmark/benchmark.h>

using namespace std;

string short_string = "hello";
string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

void BM_StringCopyShort(benchmark::State& state)
{
	for (auto _ : state)
		string copy(short_string);
}
BENCHMARK(BM_StringCopyShort);

void BM_StringCopyLong(benchmark::State& state)
{
	for (auto _ : state)
		string copy(long_string);
}
BENCHMARK(BM_StringCopyLong);

BENCHMARK_MAIN();

#include <string>

#include <benchmark/benchmark.h>

using namespace std;

string short_string = "hello";

string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

void BM_StringCopyShort(benchmark::State& state)

{

for (auto _ : state)

string copy(short_string);

}

BENCHMARK(BM_StringCopyShort);

void BM_StringCopyLong(benchmark::State& state)

{

for (auto _ : state)

string copy(long_string);

}

BENCHMARK(BM_StringCopyLong);

BENCHMARK_MAIN();

Catch2:

#include <string>
#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>

using namespace std;

string short_string = "hello";
string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

TEST_CASE("string ops", "[benchmark]")
{
	BENCHMARK("StringCopyShort")
	{
		string copy(short_string);
	}

	BENCHMARK("StringCopyLong")
	{
		string copy(long_string);
	}
}

#include <string>

#define CATCH_CONFIG_MAIN

#include <catch2/catch.hpp>

using namespace std;

string short_string = "hello";

string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

TEST_CASE("string ops", "[benchmark]")

{

BENCHMARK("StringCopyShort")

{

string copy(short_string);

}

BENCHMARK("StringCopyLong")

{

string copy(long_string);

}

Hayai:

#include <string>
#include <hayai/hayai.hpp>

using namespace std;

string short_string = "hello";
string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

BENCHMARK(StringOps, StringCopyShort, 100, 1'000'000)
{
	string copy(short_string);
}

BENCHMARK(StringOps, StringCopyLong, 100, 100'000)
{
	string copy(long_string);
}

#include <string>

#include <hayai/hayai.hpp>

using namespace std;

string short_string = "hello";

string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

BENCHMARK(StringOps, StringCopyShort, 100, 1'000'000)

{

string copy(short_string);

}

BENCHMARK(StringOps, StringCopyLong, 100, 100'000)

{

string copy(long_string);

}

Celero:

#include <string>
#include <celero/Celero.h>

using namespace std;

string short_string = "hello";
string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

BASELINE(StringCopyShort, Baseline10, 100, 100'000)
{
	string copy(short_string);
}

BENCHMARK(StringCopyShort, Baseline1000, 100, 1'000'000)
{
	string copy(short_string);
}

BASELINE(StringCopyLong, Baseline10, 100, 100'000)
{
	string copy(long_string);
}

BENCHMARK(StringCopyLong, Baseline1000, 100, 1'000'000)
{
	string copy(long_string);
}

CELERO_MAIN;

#include <string>

#include <celero/Celero.h>

using namespace std;

string short_string = "hello";

string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

BASELINE(StringCopyShort, Baseline10, 100, 100'000)

{

string copy(short_string);

}

BENCHMARK(StringCopyShort, Baseline1000, 100, 1'000'000)

{

string copy(short_string);

}

BASELINE(StringCopyLong, Baseline10, 100, 100'000)

{

string copy(long_string);

}

BENCHMARK(StringCopyLong, Baseline1000, 100, 1'000'000)

{

string copy(long_string);

}

CELERO_MAIN;

Nonius:

#include <string>
#define NONIUS_RUNNER
#include <nonius/nonius.h++>
#include <nonius/main.h++>

using namespace std;

string short_string = "hello";
string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

NONIUS_BENCHMARK("StringCopyShort", []
{
	string copy(short_string);
})

NONIUS_BENCHMARK("StringCopyLong", []
{
	string copy(long_string);
})

#include <string>

#define NONIUS_RUNNER

#include <nonius/nonius.h++>

#include <nonius/main.h++>

using namespace std;

string short_string = "hello";

string long_string("0123456789abcdefghijklmnopqrstuvwxyz");

NONIUS_BENCHMARK("StringCopyShort", []

{

string copy(short_string);

})

NONIUS_BENCHMARK("StringCopyLong", []

{

string copy(long_string);

})

SQL database access

Posted on March 18, 2019March 28, 2021 by Martin

When I first set out to make a post about database access from C++ I was going to write about MySQL Connector/C++. But for some reason that didn’t sit well with me. After sleeping on it I realized it didn’t appeal to me because it was 1) limited to only one database backend and 2) too low-level of an API. I wanted to write about a library that supports multiple database backends and abstracts the connection details as much as possible. Ideally I wanted a library that brings you closest to SQL syntax rather than deal in C++ mechanics. So in my quest for cool and portable C++ libraries I decided to keep looking…

And then I came across SOCI – The C++ Database Access Library 🙂 It has everything I was looking for: multiple backend support (DB2, Firebird, MySQL, ODBC, Oracle, PostgresSQL, and SQLite3), and a very natural way of issuing SQL queries thanks to operator overloading and template sorcery. Even their first example is purposely left without comments because it is that easy to read and understand.

Besides a very natural way of talking to a SQL backend what I like most about it is that it allows you, in a non-intrusive way (no existing code change needed), to store and retrieve your custom data structures to and from database tables.

So I installed MySQL server on my Mac using brew package manager and started coding. Within 30 minutes I had a working example that connected to my database server, created a database and a table, inserted rows into the table from my custom Person data structure, counted the rows, retrieved a table entry with a given ID, and finally cleaned up after itself.

The only code that requires explaining is the struct type_conversion<Person>; object. It is SOCI’s mechanism of converting to and from custom data structures and requires 2 methods: from_base which converts from a set of row values to a structure, and to_base which goes the other way. The rest is self explanatory! Here’s how you can get started:

#include <iostream>
#include <string>
#include <exception>
#include <soci/soci.h>
#include <soci/mysql/soci-mysql.h>

using namespace std;
using namespace soci;

struct Person
{
	int ID;
	string FirstName;
	string LastName;
	int DOB;
	string EMail;
};

namespace soci
{
	template<>
	struct type_conversion<Person>
	{
		typedef values base_type;

		static void from_base(const values& v, indicator, Person& p)
		{
			p.ID = v.get<int>("ID");
			p.FirstName = v.get<string>("FirstName");
			p.LastName = v.get<string>("LastName");
			p.DOB = v.get<int>("DOB");
			p.EMail = v.get<string>("EMail");
		}

		static void to_base(const Person& p, values& v, indicator& ind)
		{
			v.set("ID", p.ID);
			v.set("FirstName", p.FirstName);
			v.set("LastName", p.LastName);
			v.set("DOB", p.DOB);
			v.set("EMail", p.EMail);
			ind = i_ok;
		}
	};
}

int main()
{
	try
	{
		session sql(mysql, "host=localhost user=root password=''");

		sql << "CREATE DATABASE blog";
		sql << "USE blog";
		sql << "CREATE TABLE people (ID INT, FirstName TEXT, LastName TEXT, DOB INT, EMail TEXT)";

		Person him{1, "Martin", "Vorbrodt", 19800830, "martin@vorbrodt.blog"};
		Person her{2, "Dorota", "Vorbrodt", 19800127, "dorota@vorbrodt.blog"};
		sql << "INSERT INTO people (ID, FirstName, LastName, DOB, EMail) VALUES (:ID, :FirstName, :LastName, :DOB, :EMail)", use(him);
		sql << "INSERT INTO people (ID, FirstName, LastName, DOB, EMail) VALUES (:ID, :FirstName, :LastName, :DOB, :EMail)", use(her);

		int count{};
		sql << "SELECT COUNT(*) FROM people", into(count);
		cout << "Table 'people' has " << count << " row(s)" << endl;

		Person pOut{};
		sql << "SELECT * FROM people WHERE ID = 1", into(pOut);
		cout << pOut.FirstName << ", " << pOut.LastName << ", " << pOut.DOB << ", " << pOut.EMail << endl;

		sql << "DROP TABLE people";
		sql << "DROP DATABASE blog";
	}
	catch (exception& e)
	{
		cerr << e.what() << endl;
	}
}

#include <iostream>

#include <string>

#include <exception>

#include <soci/soci.h>

#include <soci/mysql/soci-mysql.h>

using namespace std;

using namespace soci;

struct Person

{

int ID;

string FirstName;

string LastName;

int DOB;

string EMail;

};

namespace soci

{

template<>

struct type_conversion<Person>

{

typedef values base_type;

static void from_base(const values& v, indicator, Person& p)

{

p.ID = v.get<int>("ID");

p.FirstName = v.get<string>("FirstName");

p.LastName = v.get<string>("LastName");

p.DOB = v.get<int>("DOB");

p.EMail = v.get<string>("EMail");

}

static void to_base(const Person& p, values& v, indicator& ind)

{

v.set("ID", p.ID);

v.set("FirstName", p.FirstName);

v.set("LastName", p.LastName);

v.set("DOB", p.DOB);

v.set("EMail", p.EMail);

ind = i_ok;

}

};

}

int main()

{

try

{

session sql(mysql, "host=localhost user=root password=''");

sql << "CREATE DATABASE blog";

sql << "USE blog";

sql << "CREATE TABLE people (ID INT, FirstName TEXT, LastName TEXT, DOB INT, EMail TEXT)";

Person him{1, "Martin", "Vorbrodt", 19800830, "[email protected]"};

Person her{2, "Dorota", "Vorbrodt", 19800127, "[email protected]"};

sql << "INSERT INTO people (ID, FirstName, LastName, DOB, EMail) VALUES (:ID, :FirstName, :LastName, :DOB, :EMail)", use(him);

sql << "INSERT INTO people (ID, FirstName, LastName, DOB, EMail) VALUES (:ID, :FirstName, :LastName, :DOB, :EMail)", use(her);

int count{};

sql << "SELECT COUNT(*) FROM people", into(count);

cout << "Table 'people' has " << count << " row(s)" << endl;

Person pOut{};

sql << "SELECT * FROM people WHERE ID = 1", into(pOut);

cout << pOut.FirstName << ", " << pOut.LastName << ", " << pOut.DOB << ", " << pOut.EMail << endl;

sql << "DROP TABLE people";

sql << "DROP DATABASE blog";

}

catch (exception& e)

{

cerr << e.what() << endl;

}

Table ‘people’ has 2 row(s)
Martin, Vorbrodt, 19800830, [email protected]
Program output.

Measuring CPU time

Posted on March 16, 2019March 28, 2021 by Martin

Measuring how long your program ran for is easy with std::chrono, but what if you need details about user and system space time? Easy! Use Boost CPU Timer library 🙂 It’s another simple library from Boost with only few classes: auto_cpu_timer is like a RAII object; it starts the clock in its constructor, stops it in its destructor, and prints the elapsed time to standard output. cpu_timer class is for manual starting and stopping of the clock; it also allows you to retrieve the elapsed times (wall, user, and system) in nanoseconds.

In the below example I create three threads: procUser spends 100% of its time in the user space calling std::sqrt function. procSystem spawns a lot of threads causing transitions into the kernel. And procTimer is just an illustration of cpu_timer usage.

#include <iostream>
#include <thread>
#include <cmath>
#include <cstdlib>
#include <boost/timer/timer.hpp>

using namespace std;
using namespace boost::timer;

int main()
{
	srand((unsigned int)time(NULL));

	auto_cpu_timer program_timer(3);

	auto procUser = [](long work)
	{
		for (long i = 0; i < work; ++i)
			sqrt(123.456L);
	};

	auto procSystem = [](long work)
	{
		for (long i = 0; i < work; ++i)
			thread([](){}).detach();
	};

	auto procTimer = [](long work)
	{
		cpu_timer timer;
		timer.start();

		for(long i = 0; i < work; ++i)
			rand();

		timer.stop();
		cout << "Thread timer:" << timer.format(3);
	};

	thread t1(procUser, 1000000000);
	thread t2(procSystem, 100000);
	thread t3(procTimer, 100000000);

	t1.join();
	t2.join();
	t3.join();

	cout << "Program timer:";

	return 1;
}

#include <iostream>

#include <thread>

#include <cmath>

#include <cstdlib>

#include <boost/timer/timer.hpp>

using namespace std;

using namespace boost::timer;

int main()

{

srand((unsigned int)time(NULL));

auto_cpu_timer program_timer(3);

auto procUser = [](long work)

{

for (long i = 0; i < work; ++i)

sqrt(123.456L);

};

auto procSystem = [](long work)

{

for (long i = 0; i < work; ++i)

thread([](){}).detach();

};

auto procTimer = [](long work)

{

cpu_timer timer;

timer.start();

for(long i = 0; i < work; ++i)

rand();

timer.stop();

cout << "Thread timer:" << timer.format(3);

};

thread t1(procUser, 1000000000);

thread t2(procSystem, 100000);

thread t3(procTimer, 100000000);

t1.join();

t2.join();

t3.join();

cout << "Program timer:";

return 1;

}

Thread timer: 0.750s wall, 1.790s user + 0.850s system = 2.640s CPU (352.1%)
Program timer: 3.171s wall, 5.080s user + 2.980s system = 8.060s CPU (254.2%)
Program output.

Printing stack traces

Posted on March 15, 2019March 15, 2019 by Martin

This one will be short 🙂

If you want to add more diagnostic information to your programs make sure to check out Boost.Stacktrace library. With it you can capture and print current stack traces. It’s especially useful when combined with exception handling; it allows you to know right away where the exception originated from.

#include <iostream>
#include <future>
#define BOOST_STACKTRACE_GNU_SOURCE_NOT_REQUIRED
#include <boost/stacktrace.hpp>

using namespace std;
using namespace boost::stacktrace;

void f1();
void f2();
void f3();

void f1() { f2(); }
void f2() { f3(); }
void f3() { cout << stacktrace() << endl << endl; }

int main(int argc, char** argv)
{
	f1();

	auto f = async(launch::async, []() { cout << stacktrace() << endl << endl; });
	f.get();

	return 1;
}

#include <iostream>

#include <future>

#define BOOST_STACKTRACE_GNU_SOURCE_NOT_REQUIRED

#include <boost/stacktrace.hpp>

using namespace std;

using namespace boost::stacktrace;

void f1();

void f2();

void f3();

void f1() { f2(); }

void f2() { f3(); }

void f3() { cout << stacktrace() << endl << endl; }

int main(int argc, char** argv)

{

f1();

auto f = async(launch::async, []() { cout << stacktrace() << endl << endl; });

f.get();

return 1;

}

0# f3() in /Users/martin/stacktrace
1# f2() in /Users/martin/stacktrace
2# f1() in /Users/martin/stacktrace
3# main in /Users/martin/stacktrace

0# main::$_0::operator()() const in /Users/martin/stacktrace
1# void std::__1::__async_func<main::$_0>::__execute<>(std::__1::__tuple_indices<>) in /Users/martin/stacktrace
2# std::__1::__async_func<main::$_0>::operator()() in /Users/martin/stacktrace
3# std::__1::__async_assoc_state<void, std::__1::__async_func<main::$_0> >::__execute() in /Users/martin/stacktrace
4# void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (std::__1::__async_assoc_state<void, std::__1::__async_func<main::$_0> >::*)(), std::__1::__async_assoc_state<void, std::__1::__async_func<main::$_0> >*> >(void*) in /Users/martin/stacktrace
5# _pthread_body in /usr/lib/system/libsystem_pthread.dylib
6# _pthread_start in /usr/lib/system/libsystem_pthread.dylib
Program output

Generating Unique IDs

Posted on March 14, 2019March 14, 2019 by Martin

I like simple and well written C++ libraries, and I’m especially fond of Boost, so I’m going to blog about it some more 🙂

Today I’ll show you how to generate unique IDs using Boost::UUID header only library. There really isn’t much to be said about it other than it’s simple, consists of only few hearer files, and allows you to generate, hash, and serialize unique IDs.

Refer to the documentation for a complete overview. Here’s a sample program that generates and prints a unique ID:

#include <iostream>
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/uuid_io.hpp>
#include <boost/uuid/uuid_generators.hpp>

using namespace std;
using namespace boost::uuids;

int main(int argc, char** argv)
{
	auto gen = random_generator();
	uuid id = gen();

	cout << id << endl;

	return 1;
}

#include <iostream>

#include <boost/uuid/uuid.hpp>

#include <boost/uuid/uuid_io.hpp>

#include <boost/uuid/uuid_generators.hpp>

using namespace std;

using namespace boost::uuids;

int main(int argc, char** argv)

{

auto gen = random_generator();

uuid id = gen();

cout << id << endl;

return 1;

}

5023e2d9-8008-4142-b1c7-87cda7cb36b3
Program output.

RPC with Protocol Buffers

Posted on March 13, 2019March 28, 2021 by Martin

Just when I thought I covered the topic of RPC in C++ with Thrift, I came across a new and very promising RPC framework: Google’s gRPC. The more I read about it the more I like it: just like Thrift it supports many programing languages and internally uses Protocol Buffers to encode messages. So I decided to give it a try and see how easily I could create a simple client and serve programs…

Just like with Thrift, the first step is to define your RPC service. Here’s my definition of a simple service with one RPC method which accepts an input parameter and provides a return value:

syntax = "proto3";

message Input
{
	string input_msg = 1;
}

message Output
{
	string output_msg = 1;
}

service gRPCService
{
	rpc hello(Input) returns (Output) {}
}

syntax = "proto3";

message Input

{

string input_msg = 1;

}

message Output

{

string output_msg = 1;

}

service gRPCService

{

rpc hello(Input) returns (Output) {}

}

I saved it as grpc_service.proto and now was the time to generate the client and server stubs. Unlike Thrift, this is a two step process: 1) you must generate the protocol buffer files, and 2) generate the gRPC client/server stubs. Both actions are done with protoc compiler and a custom plugin that comes with gRPC:

protoc -I . –cpp_out=. grpc_service.proto
protoc -I . –grpc_out=. –plugin=protoc-gen-grpc=/usr/local/bin/grpc_cpp_plugin grpc_service.proto
gRPC code generation steps.

This will compile our service definition file and produce 4 output files: .h and .cc for protocol buffer messages, and .h and .cc for gRPC client/server stubs.

Alright, that was easy. Let’s see what it takes to implement a gRPC client program… after 15 minutes reading through the documentation I came up with the following:

#include <iostream>
#include <grpc/grpc.h>
#include <grpc++/channel.h>
#include <grpc++/client_context.h>
#include <grpc++/create_channel.h>
#include <grpc++/security/credentials.h>
#include "grpc_service.grpc.pb.h"

using namespace std;
using namespace grpc;

class gRPCClient
{
public:
	gRPCClient(shared_ptr<Channel> channel)
	: stub(gRPCService::NewStub(channel)) {}

	string hello(const std::string& msg)
	{
		Input input;
		Output output;
		ClientContext context;

		input.set_input_msg(msg);

		Status status = stub->hello(&context, input, &output);
		if (status.ok())
		{
			return output.output_msg();
		}
		else
		{
			cout << status.error_code() << ": " << status.error_message() << endl;
			return "RPC failed";
		}
	}

private:
	unique_ptr<gRPCService::Stub> stub;
};

int main(int argc, char** argv)
{
	gRPCClient client(grpc::CreateChannel("localhost:50051", grpc::InsecureChannelCredentials()));
	string reply = client.hello("gRPC client says hi!");
	cout << "Client received: " << reply << endl;

	return 1;
}

#include <iostream>

#include <grpc/grpc.h>

#include <grpc++/channel.h>

#include <grpc++/client_context.h>

#include <grpc++/create_channel.h>

#include <grpc++/security/credentials.h>

#include "grpc_service.grpc.pb.h"

using namespace std;

using namespace grpc;

class gRPCClient

{

public:

gRPCClient(shared_ptr<Channel> channel)

: stub(gRPCService::NewStub(channel)) {}

string hello(const std::string& msg)

{

Input input;

Output output;

ClientContext context;

input.set_input_msg(msg);

Status status = stub->hello(&context, input, &output);

if (status.ok())

{

return output.output_msg();

}

else

{

cout << status.error_code() << ": " << status.error_message() << endl;

return "RPC failed";

}

private:

unique_ptr<gRPCService::Stub> stub;

};

int main(int argc, char** argv)

{

gRPCClient client(grpc::CreateChannel("localhost:50051", grpc::InsecureChannelCredentials()));

string reply = client.hello("gRPC client says hi!");

cout << "Client received: " << reply << endl;

return 1;

}

Little more code needed to construct and invoke the RPC method as compared to Thrift, but not too bad. The extra lines are mostly around creating the protocol buffer message objects and setting their properties.

The corresponding gRPC server code is pretty much just as easy to implement as with Thrift. Here’s the simple server that prints out the message it receives and sends back a reply:

#include <iostream>
#include <grpc/grpc.h>
#include <grpc++/server.h>
#include <grpc++/server_builder.h>
#include <grpc++/server_context.h>
#include <grpc++/security/server_credentials.h>
#include "grpc_service.grpc.pb.h"

using namespace std;
using namespace grpc;

class gRPCServiceImpl final : public gRPCService::Service
{
public:
	Status hello(ServerContext* context, const Input* in, Output* out) override
	{
		cout << in->input_msg() << endl;
		out->set_output_msg("gRPC server says hi!");
		return Status::OK;
	}
};

int main(int argc, char** argv)
{
	gRPCServiceImpl service;
	ServerBuilder builder;
	builder.AddListeningPort("0.0.0.0:50051", grpc::InsecureServerCredentials());
	builder.RegisterService(&service);
	unique_ptr<Server> server(builder.BuildAndStart());
	cout << "Server starting..." << endl;
	server->Wait();

	return 1;
}

#include <iostream>

#include <grpc/grpc.h>

#include <grpc++/server.h>

#include <grpc++/server_builder.h>

#include <grpc++/server_context.h>

#include <grpc++/security/server_credentials.h>

#include "grpc_service.grpc.pb.h"

using namespace std;

using namespace grpc;

class gRPCServiceImpl final : public gRPCService::Service

{

public:

Status hello(ServerContext* context, const Input* in, Output* out) override

{

cout << in->input_msg() << endl;

out->set_output_msg("gRPC server says hi!");

return Status::OK;

}

};

int main(int argc, char** argv)

{

gRPCServiceImpl service;

ServerBuilder builder;

builder.AddListeningPort("0.0.0.0:50051", grpc::InsecureServerCredentials());

builder.RegisterService(&service);

unique_ptr<Server> server(builder.BuildAndStart());

cout << "Server starting..." << endl;

server->Wait();

return 1;

}

So there you have it folks! It is very easy to get started with this framework.

As far as which one you should prefer (Thrift vs gRPC) I can’t honestly say. I don’t know enough about them to claim one is better than the other. What I can say is that Thrift is a more mature framework that has been around for longer. You decide 🙂

P.S. On my Mac I was able to use Homebrew package manager to install the required headers/libraries/executables for gRPC. My Ubuntu 18.04 Linux has libgrpc available in its online repository. I also verified that Microsoft’s vcpkg has ports available for Windows.

Serialize data to XML

Posted on March 10, 2019March 28, 2021 by Martin

Protocol Buffers are not always an option and you just have to serialize your data to XML. Then what? Luckily there is a serialization library available from Boost and it makes that pretty easy for us. You don’t even have to modify your existing data structures to write them out as XML: the library is non-invasive.

Let’s say I want to serialize a list of people to a file, and read it back later. My data structures would be defined like this:

struct person
{
	string name;
	int dob;
	string email;
};

using people = vector<person>;

struct person

{

string name;

int dob;

string email;

};

using people = vector<person>;

And once a vector of person’s is serialized I want it to look something like this:

<people class_id="0" tracking_level="0" version="1">
	<count>2</count>
	<item_version>1</item_version>
	<item class_id="1" tracking_level="0" version="1">
		<person.name>Martin Vorbrodt</person.name>
		<person.dob>19800830</person.dob>
		<person.email>martin@vorbrodt.blog</person.email>
	</item>
	<item>
		<person.name>Dorota Vorbrodt</person.name>
		<person.dob>19810127</person.dob>
		<person.email>dorota@vorbrodt.blog</person.email>
	</item>
</people>

<item_version>1</item_version>

<person.name>Martin Vorbrodt</person.name>

<person.dob>19800830</person.dob>

<person.email>martin@vorbrodt.blog</person.email>

</item>

<item>

<person.name>Dorota Vorbrodt</person.name>

<person.dob>19810127</person.dob>

<person.email>dorota@vorbrodt.blog</person.email>

</item>

</people>

Easy! First you must define a generic serialize function for your data structure, then you instantiate an XML output archive with an ofstream object and pass it the data. Reading is done by instantiating an XML input archive with an ifstream object and loading the data into a variable. Like this:

#include <iostream>
#include <fstream>
#include <string>
#include <vector>
#include <boost/archive/xml_oarchive.hpp>
#include <boost/archive/xml_iarchive.hpp>
#include <boost/serialization/vector.hpp>
using namespace std;
using namespace boost::archive;
using namespace boost::serialization;

struct person
{
	string name;
	int dob;
	string email;
};

using people = vector<person>;

BOOST_CLASS_VERSION(person, 1);
BOOST_CLASS_VERSION(people, 1);

template<typename Archive>
void serialize(Archive& ar, person& person, const unsigned int version)
{
	ar & BOOST_SERIALIZATION_NVP(person.name);
	ar & BOOST_SERIALIZATION_NVP(person.dob);
	ar & BOOST_SERIALIZATION_NVP(person.email);
}

int main(int argc, char** argv)
{
	person me{"Martin Vorbrodt", 19800830, "martin@vorbrodt.blog"};
	person her{"Dorota Vorbrodt", 19810127, "dorota@vorbrodt.blog"};
	people us{me, her};
	{
		ofstream ofs("data.xml");
		xml_oarchive oa(ofs, boost::archive::no_header);
		oa << make_nvp("people", us);
	}

	people us_too{};
	{
		ifstream ifs("data.xml");
		xml_iarchive ia(ifs, boost::archive::no_header);
		ia >> make_nvp("people", us_too);
	}

	for(auto& p : us_too)
	{
		cout << "Name  : " << p.name << endl;
		cout << "DOB   : " << p.dob << endl;
		cout << "EMail : " << p.email << endl << endl;
	}

	return 1;
}

#include <iostream>

#include <fstream>

#include <string>

#include <vector>

#include <boost/archive/xml_oarchive.hpp>

#include <boost/archive/xml_iarchive.hpp>

#include <boost/serialization/vector.hpp>

using namespace std;

using namespace boost::archive;

using namespace boost::serialization;

struct person

{

string name;

int dob;

string email;

};

using people = vector<person>;

BOOST_CLASS_VERSION(person, 1);

BOOST_CLASS_VERSION(people, 1);

template<typename Archive>

void serialize(Archive& ar, person& person, const unsigned int version)

{

ar & BOOST_SERIALIZATION_NVP(person.name);

ar & BOOST_SERIALIZATION_NVP(person.dob);

ar & BOOST_SERIALIZATION_NVP(person.email);

}

int main(int argc, char** argv)

{

person me{"Martin Vorbrodt", 19800830, "[email protected]"};

person her{"Dorota Vorbrodt", 19810127, "[email protected]"};

people us{me, her};

{

ofstream ofs("data.xml");

xml_oarchive oa(ofs, boost::archive::no_header);

oa << make_nvp("people", us);

}

people us_too{};

{

ifstream ifs("data.xml");

xml_iarchive ia(ifs, boost::archive::no_header);

ia >> make_nvp("people", us_too);

}

for(auto& p : us_too)

{

cout << "Name : " << p.name << endl;

cout << "DOB : " << p.dob << endl;

cout << "EMail : " << p.email << endl << endl;

}

return 1;

}

Name : Martin Vorbrodt
DOB : 19800830
EMail : [email protected]

Name : Dorota Vorbrodt
DOB : 19810127
EMail : [email protected]
Program output.

The library has built in support for STL containers. It can also write the data in many output formats, not just XML. Luckily you only have to define one serialization function per data type and it will work with all input and output archives. Heck, you could even define a serialization function for your protocol buffers data types 😉

Thrift: or how to RPC

Posted on March 10, 2019March 28, 2021 by Martin

Just like my previous Protocol Buffers post, this one is also meant as a brief introduction that will point you in the right direction rather than an exhaustive tutorial. Here we go…

Again we are in search of a portable library, this time not for serialization, but for a portable RPC mechanism. On Windows we have the WCF, but what if we want support for many platforms and programming languages? All that is answered by Thrift (also see here). Initially developed at Facebook it is now free and open source project.

Let’s start by creating a simple thrift file that defines a “service”, or a RPC server, with functions and parameters:

namespace cpp Service

service Service
{
	void ping(),
	void hello(1:string msg),
	oneway void async_call()
}

namespace cpp Service

service Service

{

void ping(),

void hello(1:string msg),

oneway void async_call()

}

Here in a file service.thrift we have defined a RPC server called Service with three functions (one asynchronous) and a string parameter msg. Next we need to compile it. Just like protocol buffers, thrift is a code generator. It will produce everything needed to instantiate both the server and the client:

thrift –gen cpp -out . service.thrift
Thrift basic usage.

The above command will produce several header and source files for us. Now we are ready to implement our C++ client that will connect to the RPC server and issue remote procedure calls. The code is straight forward and easy to read and understand:

#include <iostream>
#include <thrift/transport/TSocket.h>
#include <thrift/protocol/TBinaryProtocol.h>
#include <thrift/transport/TTransportUtils.h>
#include "Service.h"

using namespace std;
using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;
using namespace Service;

int main(int argc, char** argv)
{
	auto socket = make_shared<TSocket>("localhost", 9090);
	auto transport = make_shared<TBufferedTransport>(socket);
	auto protocol = make_shared<TBinaryProtocol>(transport);

	ServiceClient client(protocol);

	try
	{
		transport->open();

		client.ping();
		client.hello("Martin says hi!");
		client.async_call();

		transport->close();
	}
	catch(TException& tx)
	{
		cout << "ERROR: " << tx.what() << endl;
	}

	return 1;
}

#include <iostream>

#include <thrift/transport/TSocket.h>

#include <thrift/protocol/TBinaryProtocol.h>

#include <thrift/transport/TTransportUtils.h>

#include "Service.h"

using namespace std;

using namespace apache::thrift;

using namespace apache::thrift::protocol;

using namespace apache::thrift::transport;

using namespace Service;

int main(int argc, char** argv)

{

auto socket = make_shared<TSocket>("localhost", 9090);

auto transport = make_shared<TBufferedTransport>(socket);

auto protocol = make_shared<TBinaryProtocol>(transport);

ServiceClient client(protocol);

try

{

transport->open();

client.ping();

client.hello("Martin says hi!");

client.async_call();

transport->close();

}

catch(TException& tx)

{

cout << "ERROR: " << tx.what() << endl;

}

return 1;

}

The server code is slightly more complicated, but not by much 🙂 In this post I’m using the most basic functions of thrift for illustration purposes. But know that it is quite capable of handling huge workloads and many connections. Here’s the corresponding server code:

#include <iostream>
#include <thrift/transport/TServerSocket.h>
#include <thrift/protocol/TBinaryProtocol.h>
#include <thrift/transport/TTransportUtils.h>
#include <thrift/server/TSimpleServer.h>
#include "Service.h"

using namespace std;
using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;
using namespace apache::thrift::server;
using namespace Service;

class ServiceHandler : public ServiceIf
{
public:
	ServiceHandler() = default;

	void ping() override { cout << "ping()" << endl; }
	void hello(const string& msg) override { cout << msg << endl; }
	void async_call() override { cout << "async_call()" << endl; }
};

int main(int argc, char** argv)
{
	TSimpleServer server(
		std::make_shared<ServiceProcessor>(std::make_shared<ServiceHandler>()),
		std::make_shared<TServerSocket>(9090),
		std::make_shared<TBufferedTransportFactory>(),
		std::make_shared<TBinaryProtocolFactory>());

	cout << "Starting the server..." << endl;

	server.serve();

	return 1;
}

#include <iostream>

#include <thrift/transport/TServerSocket.h>

#include <thrift/protocol/TBinaryProtocol.h>

#include <thrift/transport/TTransportUtils.h>

#include <thrift/server/TSimpleServer.h>

#include "Service.h"

using namespace std;

using namespace apache::thrift;

using namespace apache::thrift::protocol;

using namespace apache::thrift::transport;

using namespace apache::thrift::server;

using namespace Service;

class ServiceHandler : public ServiceIf

{

public:

ServiceHandler() = default;

void ping() override { cout << "ping()" << endl; }

void hello(const string& msg) override { cout << msg << endl; }

void async_call() override { cout << "async_call()" << endl; }

};

int main(int argc, char** argv)

{

TSimpleServer server(

std::make_shared<ServiceProcessor>(std::make_shared<ServiceHandler>()),

std::make_shared<TServerSocket>(9090),

std::make_shared<TBufferedTransportFactory>(),

std::make_shared<TBinaryProtocolFactory>());

cout << "Starting the server..." << endl;

server.serve();

return 1;

}

The extra code here is the ServiceHandler class which will do the actual RPC work. Let’s put it all together now. After I start the server and execute the client program on my machine, I get the following output:

Starting the server…
ping()
Martin says hi!
async_call()
Thrift RPC server output.

It works! I hope you enjoyed this little introduction to thrift. Now go read all about it!

P.S. As always, complete source and build files available at my GitHub.

Protocol Buffers: or how to serialize data

Posted on March 10, 2019April 2, 2021 by Martin

This post is meant as a brief introduction that will point you in the right direction rather than an exhaustive tutorial. Here we go…

Have you ever had to write code that serialized structured data into an efficient binary format, to be later saved to disk or sent over the network? Do you remember how difficult and time consuming it was? Haven’t you wished there was a standard C++ library to do it instead of reinventing the wheel? Well, today is your lucky day 🙂

Say hello to Google’s Protocol Buffers! It is a highly portable (and free!) library that allows you to define near arbitrary (numerical, strings, structures, lists, vectors, maps, you name it) data formats and easily serialize and deserialize them into a platform portable binary format. Why not use XML you may ask? How about 3 to 10 times smaller output, and 20 to 100 times faster serialization and deserialization just to name a few 🙂

Let’s start by defining a protocol buffer file that defines a “Person” data structure:

syntax = "proto2";
  
package data;

message Person
{
  required string name = 1;
  required int32 dob = 2;
  optional string email = 3;
}

syntax = "proto2";

package data;

message Person

{

required string name = 1;

required int32 dob = 2;

optional string email = 3;

}

Save that to protobuf.proto file and let’s compile it using the protoc code generator. Yes, Protocol Buffers is a code generator; it takes as input a .proto file and spits out C++ classes (it can also produce code in C#, Java, JS, ObjC, PHP, Python, and Ruby).

protoc -I=. –cpp_out=. protobuf.proto
protoc basic usage.

The above command will produce 2 files:

#include <iostream>
#include <string>
#include "protobuf.pb.h"
using namespace std;

int main(int argc, char** argv)
{
        GOOGLE_PROTOBUF_VERIFY_VERSION;

        data::Person p;

        p.set_name("Martin Vorbrodt");
        p.set_dob(19800830);
        p.set_email("martin@vorbrodt.blog");

        string binary;
        p.SerializeToString(&binary);

        data::Person p2;
        p2.ParseFromString(binary);

        cout << "Name  = " << p2.name() << endl;
        cout << "DOB   = " << p2.dob() << endl;
        cout << "EMail = " << p2.email() << endl;
        
        google::protobuf::ShutdownProtobufLibrary();
        return 1;
}

#include <iostream>

#include <string>

#include "protobuf.pb.h"

using namespace std;

int main(int argc, char** argv)

{

GOOGLE_PROTOBUF_VERIFY_VERSION;

data::Person p;

p.set_name("Martin Vorbrodt");

p.set_dob(19800830);

p.set_email("[email protected]");

string binary;

p.SerializeToString(&binary);

data::Person p2;

p2.ParseFromString(binary);

cout << "Name = " << p2.name() << endl;

cout << "DOB = " << p2.dob() << endl;

cout << "EMail = " << p2.email() << endl;

google::protobuf::ShutdownProtobufLibrary();

return 1;

}

This code is pretty self explanatory. We create, set and serialize one data structure of type data::Person, then deserialize it into another. The output is what we would expect:

Name = Martin Vorbrodt
DOB = 19800830
EMail = [email protected]
Program output.

That’s enough of an introduction. I hope you will read up on Protocol Buffers and realize the enormous potential of this library. Happy coding!

P.S. As always, complete source and build files available at my GitHub.

Well, that was no fun :(

Posted on March 9, 2019March 14, 2019 by Martin

No, not blogging, that’s still fun 🙂 moving my website to bluehost over the last 24 hours! But it’s finally done and the Vorbrodt’s C++ Blog is smooth sailing once again… on a brand spanking new domain! But have no fear, the old one will kindly redirect.

So what have I learned through this exercise? For starters website migrations, no matter how trivial (it’s only a basic blog with 2 pages and 46 posts after-all), just don’t go smooth. 95% of it will work, but then the annoying 5% that didn’t migrate will eat up 23 1/2 hours of your life trying to chase it down and patch it up! I can’t imagine moving a huge corporate or banking website… but I guess that’s what staging environments are for.

So my blog uses a custom theme, has few WordPress and Jetpack widgets, and few custom modifications to the php files by yours truly. Non of that migrated! See, I did an export followed by an import operation, and that only moved the data (posts, pages, comments). So I had to hunt down and reinstall theme and plugins on the new site; then I re-implemented the few php changes to make the theme to my liking again.

But the most irritating part of the export / import process was the fact that the post excerpts vanished into thin air! So when I fired up the site with the new theme all I cloud see were post titles… no excerpts on the front page! Luckily the post content was preserved perfectly. I then looked for and tried four or five different excerpts plugins to no avail, until I found the one I needed: Excerpt Editor. Nothing about it was automatic, but at least it let me rebuild, one by one, my post excerpts. Ufff. That took a while.

Once I got the page up and running the way it was before I immediately purchased a Jetpack Personal plan for $39/year which offers automatic site backups and restorations. It backs up the database (pages, posts, comments) as well as themes, plugins, uploads, etc. Hopefully I’ll never have to use it, but you know what they say… it’s better to have it and not need it, then need it and not have it 🙂
Oh, and the site hosting is $10/month, another $30/year for domain name and privacy protection on it. Pretty low price to pay for a total peace of mind!

Finally a word about my website hosting setup up to yesterday: yes it was done from home and a constant source of headaches! For starters I’m on a residential 1Gbit line with 40Mbit upload. Great for streaming 4K TV but not so hot for uploading website content (plus the upload latency was bad). Then there’s the whole dynamic IP address thing… so I had to create a sub domain with my google domain, enable DDNS service on it through my OpenWrt router, and pray it doesn’t change too often. Of course I couldn’t run the web server directly on the router, so I pushed it behind the firewall onto my Qnap NAS. Getting WordPress to run on this thing smoothly was an issue and required plenty of workarounds (like a custom cron job to pull wp-cron.php file every minute from the web server or else the internal WordPress tasks would get delayed and bad things would happen). Just a mess overall. Oh and don’t get me started on using LetsEncrypt certificates for https access. I love that they provide them for free, but for 90 days at a time! Really?! And then there was the weekend I spent figuring out how to serve the intermediate certificate along with my domain certificate from Qnap’s custom build Apache server… so I could get an A rating from SSL Labs 🙂

Anyways, too much venting and not enough C++ in this post so I’ll stop now!

P.S. If you got this far, please re-subscribe to email notifications if you’ve done so in the past. Those sadly didn’t survive the export / import process 🙁

Parallel STL

Posted on March 7, 2019March 18, 2021 by Martin

C++17 standard introduced execution policies to the standard algorithms; those allow for parallel and SIMD optimizations. I wanted to see how much faster the Parallel STL can be on my quad core system but non of my compilers currently support it. Luckily Intel has implemented it and made it available to the world 🙂

On a side note, in this post’s example I will be using several frameworks: TBB needed to compile the Parallel STL. And Catch2 to create the test benchmark. All are freely available on GitHub. BTW, thanks to Benjamin from Thoughts on Coding for pointing me toward the Catch2 library. It’s great for creating unit tests and benchmarks.

Let’s benchmark the following operations using STL and PSTL: generating random numbers, sorting the generated random numbers, finally verifying if they’re sorted. The performance increase on my quad core 2012 MacBook Pro with i7 2.3GHz is about 5x! Nice!

benchmark name iters elapsed ns average
———————————————————–
STL 1 10623612832 10.6236 s
PSTL 1 1967239761 1.96724 s
Program output.

#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>
#include <vector>
#include <random>
#include <algorithm>
#include <pstl/execution>
#include <pstl/algorithm>
using namespace std;
using namespace pstl;

const unsigned long long COUNT = 100'000'000;

TEST_CASE("STL vs PSTL", "[benchmark]")
{
	auto seed = random_device{}();

	vector<int> data(COUNT);

	BENCHMARK("STL")
	{
		generate(data.begin(), data.end(), mt19937{seed});
		sort(data.begin(), data.end());
		is_sorted(data.begin(), data.end());
	}

	BENCHMARK("PSTL")
	{
		generate(pstl::execution::par_unseq, data.begin(), data.end(), mt19937{seed});
		sort(pstl::execution::par_unseq, data.begin(), data.end());
		is_sorted(pstl::execution::par_unseq, data.begin(), data.end());
	}
}

#define CATCH_CONFIG_MAIN

#include <catch2/catch.hpp>

#include <vector>

#include <random>

#include <algorithm>

#include <pstl/execution>

#include <pstl/algorithm>

using namespace std;

using namespace pstl;

const unsigned long long COUNT = 100'000'000;

TEST_CASE("STL vs PSTL", "[benchmark]")

{

auto seed = random_device{}();

vector<int> data(COUNT);

BENCHMARK("STL")

{

generate(data.begin(), data.end(), mt19937{seed});

sort(data.begin(), data.end());

is_sorted(data.begin(), data.end());

}

BENCHMARK("PSTL")

{

generate(pstl::execution::par_unseq, data.begin(), data.end(), mt19937{seed});

sort(pstl::execution::par_unseq, data.begin(), data.end());

is_sorted(pstl::execution::par_unseq, data.begin(), data.end());

}

Advanced thread pool

Posted on February 27, 2019March 14, 2019 by Martin

Below is my implementation of the thread pool described in this talk and a benchmark comparing it against my simple thread pool implementation. The advanced pool is 15x faster at scheduling and dispatching short random length work items on my 2018 MacBook Pro with i5 CPU and 4 logical cores. It uses a queue per worker thread and a work stealing dispatcher. It tries to enqueue the work items onto a queue that is not currently locked by a dispatch thread. It also tries to steal work from other unblocked queues. As always the complete implementation is available at GitHub.

Benchmark program:

#include <iostream>
#include <chrono>
#include <cstdlib>
#include "pool.h"
using namespace std;
using namespace chrono;

const unsigned int COUNT = 10'000'000;
const unsigned int REPS = 10;

int main()
{
	srand(0);
	auto start = high_resolution_clock::now();
	{
		simple_thread_pool tp;
		for(int i = 0; i < COUNT; ++i)
			tp.enqueue_work([i]() {
				int x;
				int reps = REPS + (REPS * (rand() % 5));
				for(int n = 0; n < reps; ++n)
					x = i + rand();
			});
	}
	auto end = high_resolution_clock::now();
	auto duration = duration_cast<milliseconds>(end - start);
	cout << "simple_thread_pool duration = " << duration.count() / 1000.f << " s" << endl;

	srand(0);
	start = high_resolution_clock::now();
	{
		thread_pool tp;
		for(int i = 0; i < COUNT; ++i)
			tp.enqueue_work([i]() {
				int x;
				int reps = REPS + (REPS * (rand() % 5));
				for(int n = 0; n < reps; ++n)
					x = i + rand();
			});
	}
	end = high_resolution_clock::now();
	duration = duration_cast<milliseconds>(end - start);
	cout << "thread_pool duration = " << duration.count() / 1000.f << " s" << endl;
}

#include <iostream>

#include <chrono>

#include <cstdlib>

#include "pool.h"

using namespace std;

using namespace chrono;

const unsigned int COUNT = 10'000'000;

const unsigned int REPS = 10;

int main()

{

srand(0);

auto start = high_resolution_clock::now();

{

simple_thread_pool tp;

for(int i = 0; i < COUNT; ++i)

tp.enqueue_work([i]() {

int x;

int reps = REPS + (REPS * (rand() % 5));

for(int n = 0; n < reps; ++n)

x = i + rand();

});

}

auto end = high_resolution_clock::now();

auto duration = duration_cast<milliseconds>(end - start);

cout << "simple_thread_pool duration = " << duration.count() / 1000.f << " s" << endl;

srand(0);

start = high_resolution_clock::now();

{

thread_pool tp;

for(int i = 0; i < COUNT; ++i)

tp.enqueue_work([i]() {

int x;

int reps = REPS + (REPS * (rand() % 5));

for(int n = 0; n < reps; ++n)

x = i + rand();

});

}

end = high_resolution_clock::now();

duration = duration_cast<milliseconds>(end - start);

cout << "thread_pool duration = " << duration.count() / 1000.f << " s" << endl;

}

* Apple CLANG -Ofast -march=native -std=c++17 -lc++

simple_thread_pool duration = 30.337 s
thread_pool duration = 1.625 s

* LLVM -Ofast -march=native -std=c++17 -lc++

simple_thread_pool duration = 25.785 s
thread_pool duration = 1.615 s

* G++ -Ofast -march=native -std=c++17 -lstdc++ *

simple_thread_pool duration = 26.28 s
thread_pool duration = 1.614 s
Program output.

thread_pool class:

class thread_pool
{
public:
	thread_pool(unsigned int threads = std::thread::hardware_concurrency())
	: m_queues(threads), m_count(threads)
	{
		assert(threads != 0);
		auto worker = [&](unsigned int i)
		{
			while(true)
			{
				Proc f;
				for(unsigned int n = 0; n < m_count; ++n)
					if(m_queues[(i + n) % m_count].try_pop(f)) break;
				if(!f && !m_queues[i].pop(f)) break;
				f();
			}
		};
		for(unsigned int i = 0; i < threads; ++i)
			m_threads.emplace_back(worker, i);
	}

	~thread_pool() noexcept
	{
		for(auto& queue : m_queues)
			queue.done();
		for(auto& thread : m_threads)
			thread.join();
	}

	template<typename F, typename... Args>
	void enqueue_work(F&& f, Args&&... args)
	{
		auto work = [f,args...]() { f(args...); };
		unsigned int i = m_index++;
		for(unsigned int n = 0; n < m_count * K; ++n)
			if(m_queues[(i + n) % m_count].try_push(work)) return;
		m_queues[i % m_count].push(work);
	}

	template<typename F, typename... Args>
	auto enqueue_task(F&& f, Args&&... args) -> std::future<typename std::result_of<F(Args...)>::type>
	{
		using return_type = typename std::result_of<F(Args...)>::type;
		auto task = std::make_shared<std::packaged_task<return_type()>>(std::bind(std::forward<F>(f), std::forward<Args>(args)...));
		std::future<return_type> res = task->get_future();

		auto work = [task](){ (*task)(); };
		unsigned int i = m_index++;
		for(unsigned int n = 0; n < m_count * K; ++n)
			if(m_queues[(i + n) % m_count].try_push(work)) return res;
		m_queues[i % m_count].push(work);

		return res;
	}

private:
	using Proc = std::function<void(void)>;
	using Queues = std::vector<simple_blocking_queue<Proc>>;
	Queues m_queues;

	using Threads = std::vector<std::thread>;
	Threads m_threads;

	const unsigned int m_count;
	std::atomic_uint m_index = 0;

	inline static const unsigned int K = 3;
};

class thread_pool

{

public:

thread_pool(unsigned int threads = std::thread::hardware_concurrency())

: m_queues(threads), m_count(threads)

{

assert(threads != 0);

auto worker = [&](unsigned int i)

{

while(true)

{

Proc f;

for(unsigned int n = 0; n < m_count; ++n)

if(m_queues[(i + n) % m_count].try_pop(f)) break;

if(!f && !m_queues[i].pop(f)) break;

f();

}

};

for(unsigned int i = 0; i < threads; ++i)

m_threads.emplace_back(worker, i);

}

~thread_pool() noexcept

{

for(auto& queue : m_queues)

queue.done();

for(auto& thread : m_threads)

thread.join();

}

template<typename F, typename... Args>

void enqueue_work(F&& f, Args&&... args)

{

auto work = [f,args...]() { f(args...); };

unsigned int i = m_index++;

for(unsigned int n = 0; n < m_count * K; ++n)

if(m_queues[(i + n) % m_count].try_push(work)) return;

m_queues[i % m_count].push(work);

}

template<typename F, typename... Args>

auto enqueue_task(F&& f, Args&&... args) -> std::future<typename std::result_of<F(Args...)>::type>

{

using return_type = typename std::result_of<F(Args...)>::type;

auto task = std::make_shared<std::packaged_task<return_type()>>(std::bind(std::forward<F>(f), std::forward<Args>(args)...));

std::future<return_type> res = task->get_future();

auto work = [task](){ (*task)(); };

unsigned int i = m_index++;

for(unsigned int n = 0; n < m_count * K; ++n)

if(m_queues[(i + n) % m_count].try_push(work)) return res;

m_queues[i % m_count].push(work);

return res;

}

private:

using Proc = std::function<void(void)>;

using Queues = std::vector<simple_blocking_queue<Proc>>;

Queues m_queues;

using Threads = std::vector<std::thread>;

Threads m_threads;

const unsigned int m_count;

std::atomic_uint m_index = 0;

inline static const unsigned int K = 3;

};

Better Code: Concurrency

Posted on February 26, 2019March 14, 2019 by Martin

by Sean Parent

Better timer class

Posted on February 25, 2019March 28, 2021 by Martin

It bothered me that my previous simple timer implementation fired off a new thread for each timeout and interval. I knew things could be done better, but didn’t yet know how. Well this morning inspiration came and I implemented new and shiny timer class. The interface is simple: you create a timer with 1 parameter, its “tick”. The tick determines how frequently the internal thread wakes up and looks for work. Work can be a repeating interval event, or a one time timeout event. Each time you register an interval or a timeout you get back a pointer to an event object. Using this event object you can cancel the interval or the timeout, if it hasn’t fired already. The internal thread lives for as long as the timer object does. It also self-corrects any time drift caused by the firing of events and execution delay. Complete implementation can be found at GitHub.

Here’s how you use it:

#include <iostream>
#include <sstream>
#include "timer.h"
using namespace std;
using namespace chrono;

int main()
{
	auto start = high_resolution_clock::now();
	auto duration = [start]() {
		auto now = high_resolution_clock::now();
		auto msecs = duration_cast<milliseconds>(now - start).count();
		stringstream ss;
		ss << msecs / 1000.0;
		cout << "elapsed " << ss.str() << "s\t: ";
	};

	cout << "start" << endl;
	timer t(1ms);
	auto e1 = t.set_timeout(3s, [&]() { duration(); cout << "timeout 3s" << endl; });
	auto e2 = t.set_interval(1s, [&]() { duration(); cout << "interval 1s" << endl; });
	auto e3 = t.set_timeout(4s, [&]() { duration(); cout << "timeout 4s" << endl; });
	auto e4 = t.set_interval(2s, [&]() { duration(); cout << "interval 2s" << endl; });
	auto e5 = t.set_timeout(5s, [&]() { duration(); cout << "timeout that never happens" << endl; });
	e5->signal(); // cancel this timeout
	this_thread::sleep_for(5s);
	e4->signal(); // cancel this interval
	cout << "cancel interval 2" << endl;
	this_thread::sleep_for(5s);
	cout << "end" << endl;
}

#include <iostream>

#include <sstream>

#include "timer.h"

using namespace std;

using namespace chrono;

int main()

{

auto start = high_resolution_clock::now();

auto duration = [start]() {

auto now = high_resolution_clock::now();

auto msecs = duration_cast<milliseconds>(now - start).count();

stringstream ss;

ss << msecs / 1000.0;

cout << "elapsed " << ss.str() << "s\t: ";

};

cout << "start" << endl;

timer t(1ms);

auto e1 = t.set_timeout(3s, [&]() { duration(); cout << "timeout 3s" << endl; });

auto e2 = t.set_interval(1s, [&]() { duration(); cout << "interval 1s" << endl; });

auto e3 = t.set_timeout(4s, [&]() { duration(); cout << "timeout 4s" << endl; });

auto e4 = t.set_interval(2s, [&]() { duration(); cout << "interval 2s" << endl; });

auto e5 = t.set_timeout(5s, [&]() { duration(); cout << "timeout that never happens" << endl; });

e5->signal(); // cancel this timeout

this_thread::sleep_for(5s);

e4->signal(); // cancel this interval

cout << "cancel interval 2" << endl;

this_thread::sleep_for(5s);

cout << "end" << endl;

}

start
elapsed 1s : interval 1s
elapsed 2s : interval 1s
elapsed 2s : interval 2s
elapsed 3s : timeout 3s
elapsed 3s : interval 1s
elapsed 4s : interval 1s
elapsed 4s : timeout 4s
elapsed 4s : interval 2s
elapsed 5s : interval 1s
cancel interval 2
elapsed 6s : interval 1s
elapsed 7s : interval 1s
elapsed 8s : interval 1s
elapsed 9s : interval 1s
elapsed 10s : interval 1s
end
Program output.

The timer class:

#pragma once

#include <thread>
#include <chrono>
#include <memory>
#include <functional>
#include <set>
#include <iterator>
#include <cassert>
#include "event.h"

class timer
{
public:
	template<typename T>
	timer(T&& tick)
	: m_tick(std::chrono::duration_cast<std::chrono::nanoseconds>(tick)), m_thread([this]()
	{
		assert(m_tick.count() > 0);
		auto start = std::chrono::high_resolution_clock::now();
		std::chrono::nanoseconds drift{0};
		while(!m_event.wait_for(m_tick - drift))
		{
			++m_ticks;
			auto it = std::begin(m_events);
			auto end = std::end(m_events);
			while(it != end)
			{
				auto& event = *it;
				++event.elapsed;
				if(event.elapsed == event.ticks)
				{
					auto remove = event.proc();
					if(remove)
					{
						m_events.erase(it++);
						continue;
					}
					else
					{
						event.elapsed = 0;
					}
				}
				++it;
			}
			auto now = std::chrono::high_resolution_clock::now();
			auto realDuration = std::chrono::duration_cast<std::chrono::nanoseconds>(now - start);
			auto fakeDuration = std::chrono::duration_cast<std::chrono::nanoseconds>(m_tick * m_ticks);
			drift = realDuration - fakeDuration;
		}
	})
	{}

	~timer()
	{
		m_event.signal();
		m_thread.join();
	}

	template<typename T, typename F, typename... Args>
	auto set_timeout(T&& timeout, F f, Args&&... args)
	{
		assert(std::chrono::duration_cast<std::chrono::nanoseconds>(timeout).count() >= m_tick.count());
		auto event = std::make_shared<manual_event>();
		auto proc = [=]() {
			if(event->wait_for(std::chrono::seconds(0))) return true;
			f(args...);
			return true;
		};
		m_events.insert({ event_ctx::kNextSeqNum++, proc,
			static_cast<unsigned long long>(std::chrono::duration_cast<std::chrono::nanoseconds>(timeout).count() / m_tick.count()), 0, event });
		return event;
	}

	template<typename T, typename F, typename... Args>
	auto set_interval(T&& interval, F f, Args&&... args)
	{
		assert(std::chrono::duration_cast<std::chrono::nanoseconds>(interval).count() >= m_tick.count());
		auto event = std::make_shared<manual_event>();
		auto proc = [=]() {
			if(event->wait_for(std::chrono::seconds(0))) return true;
			f(args...);
			return false;
		};
		m_events.insert({ event_ctx::kNextSeqNum++, proc,
			static_cast<unsigned long long>(std::chrono::duration_cast<std::chrono::nanoseconds>(interval).count() / m_tick.count()), 0, event });
		return event;
	}

private:
	std::chrono::nanoseconds m_tick;
	unsigned long long m_ticks = 0;
	manual_event m_event;
	std::thread m_thread;

	struct event_ctx
	{
		bool operator < (const event_ctx& rhs) const { return seq_num < rhs.seq_num; }
		static inline unsigned long long kNextSeqNum = 0;
		unsigned long long seq_num;
		std::function<bool(void)> proc;
		unsigned long long ticks;
		mutable unsigned long long elapsed;
		std::shared_ptr<manual_event> event;
	};

	using set = std::set<event_ctx>;
	set m_events;
};

100

101

102

103

104

105

106

107

108

109

#pragma once

#include <thread>

#include <chrono>

#include <memory>

#include <functional>

#include <set>

#include <iterator>

#include <cassert>

#include "event.h"

class timer

{

public:

template<typename T>

timer(T&& tick)

: m_tick(std::chrono::duration_cast<std::chrono::nanoseconds>(tick)), m_thread([this]()

{

assert(m_tick.count() > 0);

auto start = std::chrono::high_resolution_clock::now();

std::chrono::nanoseconds drift{0};

while(!m_event.wait_for(m_tick - drift))

{

++m_ticks;

auto it = std::begin(m_events);

auto end = std::end(m_events);

while(it != end)

{

auto& event = *it;

++event.elapsed;

if(event.elapsed == event.ticks)

{

auto remove = event.proc();

if(remove)

{

m_events.erase(it++);

continue;

}

else

{

event.elapsed = 0;

}

++it;

}

auto now = std::chrono::high_resolution_clock::now();

auto realDuration = std::chrono::duration_cast<std::chrono::nanoseconds>(now - start);

auto fakeDuration = std::chrono::duration_cast<std::chrono::nanoseconds>(m_tick * m_ticks);

drift = realDuration - fakeDuration;

}

})

{}

~timer()

{

m_event.signal();

m_thread.join();

}

template<typename T, typename F, typename... Args>

auto set_timeout(T&& timeout, F f, Args&&... args)

{

assert(std::chrono::duration_cast<std::chrono::nanoseconds>(timeout).count() >= m_tick.count());

auto event = std::make_shared<manual_event>();

auto proc = [=]() {

if(event->wait_for(std::chrono::seconds(0))) return true;

f(args...);

return true;

};

m_events.insert({ event_ctx::kNextSeqNum++, proc,

static_cast<unsigned long long>(std::chrono::duration_cast<std::chrono::nanoseconds>(timeout).count() / m_tick.count()), 0, event });

return event;

}

template<typename T, typename F, typename... Args>

auto set_interval(T&& interval, F f, Args&&... args)

{

assert(std::chrono::duration_cast<std::chrono::nanoseconds>(interval).count() >= m_tick.count());

auto event = std::make_shared<manual_event>();

auto proc = [=]() {

if(event->wait_for(std::chrono::seconds(0))) return true;

f(args...);

return false;

};

m_events.insert({ event_ctx::kNextSeqNum++, proc,

static_cast<unsigned long long>(std::chrono::duration_cast<std::chrono::nanoseconds>(interval).count() / m_tick.count()), 0, event });

return event;

}

private:

std::chrono::nanoseconds m_tick;

unsigned long long m_ticks = 0;

manual_event m_event;

std::thread m_thread;

struct event_ctx

{

bool operator < (const event_ctx& rhs) const { return seq_num < rhs.seq_num; }

static inline unsigned long long kNextSeqNum = 0;

unsigned long long seq_num;

std::function<bool(void)> proc;

unsigned long long ticks;

mutable unsigned long long elapsed;

std::shared_ptr<manual_event> event;

};

using set = std::set<event_ctx>;

set m_events;

};

Random number generator

Posted on February 24, 2019March 14, 2019 by Martin

Examples based on this talk.

Below is the old and rusty way of generating random numbers. Don’t do it!

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main()
{
	srand((unsigned int)time(NULL));
	for(int n = 0; n < 10; ++n)
		printf("%d ", rand() % 1000);
	printf("\n");
}

#include <stdio.h>

#include <stdlib.h>

#include <time.h>

int main()

{

srand((unsigned int)time(NULL));

for(int n = 0; n < 10; ++n)

printf("%d ", rand() % 1000);

printf("\n");

}

Below is the new and shiny way of generating random numbers. Do that instead! Comments inline and a benchmark of each random number generator included. Program output first:

random_device min = 0, max = 4294967295
mt19937 min = 0, max = 4294967295
mt19937_64 min = 0, max = 18446744073709551615
10 -1 6 10 5 -4 -3 2 6 -3
8.73366 3.81724 2.11837 4.14365 9.58442
vector of ints: 0 1 2 3 4 5 6 7 8 9
shuffled to : 3 1 6 7 9 4 8 5 0 2
generating 100000000 random numbers…
random_device duration in ms = 142080
mt19937 duration in ms = 553.894
uniform_int_distribution duration in ms = 2719.63
uniform_real_distribution duration in ms = 1070.29
Program output.

Complete listing:

#include <iostream>
#include <random>
#include <algorithm>
#include <vector>
#include <chrono>
using namespace std;
using namespace chrono;

const int COUNT = 100'000'000;

int main()
{
	/* random_device MAY BE CRYPTOGRAPHICALLY STRONG */
	random_device rd;
	/* SEED mt19937 and mt19937_64 ENGINE WITH STRONG RANDOM NUMBER */
	mt19937 mt(rd());
	mt19937_64 mt64(rd());
	/* CREATE RANDOM DISTRIBUTION OBJECTS FOR INTS AND DOUBLES */
	uniform_int_distribution int_dist(-10, 10);
	uniform_real_distribution real_dist(1.0, 10.0);

	/* PRINT MIN AND MAX OF EACH RANDOM ENGINE */
	cout << "random_device min = " << rd.min() << ", max = " << rd.max() << endl;
	cout << "mt19937 min = " << mt.min() << ", max = " << mt.max() << endl;
	cout << "mt19937_64 min = " << mt64.min() << ", max = " << mt64.max() << endl;

	/* GENERATE 10 INTEGERS IN RANGE -10 TO 10 USING mt19937 GENERATOR */
	for(int n = 0; n < 10; ++n)
		cout << int_dist(mt) << " ";
	cout << endl;

	/* GENERATE 5 DOUBLES IN RANGE 1.0 TO 10.0 USING mt19937 GENERATOR */
	for(int n = 0; n < 5; ++n)
		cout << real_dist(mt) << " ";
	cout << endl;

	/* GENERATE A VECTOR OF CONSECUTIVE INTEGERS */
	vector<int> v;
	for(int n = 0; n < 10; ++n)
		v.push_back(n);

	/* PRINT IT */
	cout << "vector of ints: ";
	for(auto it : v) cout << it << " ";
	cout << endl;

	/* RANDOMLY SHUFFLE THE VECTOR OF INTEGERS USING mt19937 GENERATOR */
	shuffle(begin(v), end(v), mt);

	/* PRINT IT */
	cout << "shuffled to   : ";
	for(auto it : v) cout << it << " ";
	cout << endl;
	
	/* ********************* */
	/* BENCHMARK STARTS HERE */
	/* ********************* */
	cout << "generating " << COUNT << " random numbers..." << endl;

	/* TEST PERFORMANCE OF random_device */
	auto start = high_resolution_clock::now();
	int result{0};
	for(int i = 0; i < COUNT; ++i)
		result += rd();
	auto end = high_resolution_clock::now();
	auto duration = duration_cast<microseconds>(end - start).count() / 1000.0f;
	cout << "random_device duration in ms = " << duration << endl;

	/* TEST PERFORMANCE OF mt19937 */
	start = high_resolution_clock::now();
	result = 0;
	for(int i = 0; i < COUNT; ++i)
		result += mt();
	end = high_resolution_clock::now();
	duration = duration_cast<microseconds>(end - start).count() / 1000.0f;
	cout << "mt19937 duration in ms = " << duration << endl;

	/* TEST PERFORMANCE OF uniform_int_distribution */
	start = high_resolution_clock::now();
	result = 0;
	for(int i = 0; i < COUNT; ++i)
		result += int_dist(mt);
	end = high_resolution_clock::now();
	duration = duration_cast<microseconds>(end - start).count() / 1000.0f;
	cout << "uniform_int_distribution duration in ms = " << duration << endl;

	/* TEST PERFORMANCE OF uniform_real_distribution */
	start = high_resolution_clock::now();
	result = 0;
	for(int i = 0; i < COUNT; ++i)
		result += real_dist(mt);
	end = high_resolution_clock::now();
	duration = duration_cast<microseconds>(end - start).count() / 1000.0f;
	cout << "uniform_real_distribution duration in ms = " << duration << endl;
}

#include <iostream>

#include <random>

#include <algorithm>

#include <vector>

#include <chrono>

using namespace std;

using namespace chrono;

const int COUNT = 100'000'000;

int main()

{

/* random_device MAY BE CRYPTOGRAPHICALLY STRONG */

random_device rd;

/* SEED mt19937 and mt19937_64 ENGINE WITH STRONG RANDOM NUMBER */

mt19937 mt(rd());

mt19937_64 mt64(rd());

/* CREATE RANDOM DISTRIBUTION OBJECTS FOR INTS AND DOUBLES */

uniform_int_distribution int_dist(-10, 10);

uniform_real_distribution real_dist(1.0, 10.0);

/* PRINT MIN AND MAX OF EACH RANDOM ENGINE */

cout << "random_device min = " << rd.min() << ", max = " << rd.max() << endl;

cout << "mt19937 min = " << mt.min() << ", max = " << mt.max() << endl;

cout << "mt19937_64 min = " << mt64.min() << ", max = " << mt64.max() << endl;

/* GENERATE 10 INTEGERS IN RANGE -10 TO 10 USING mt19937 GENERATOR */

for(int n = 0; n < 10; ++n)

cout << int_dist(mt) << " ";

cout << endl;

/* GENERATE 5 DOUBLES IN RANGE 1.0 TO 10.0 USING mt19937 GENERATOR */

for(int n = 0; n < 5; ++n)

cout << real_dist(mt) << " ";

cout << endl;

/* GENERATE A VECTOR OF CONSECUTIVE INTEGERS */

vector<int> v;

for(int n = 0; n < 10; ++n)

v.push_back(n);

/* PRINT IT */

cout << "vector of ints: ";

for(auto it : v) cout << it << " ";

cout << endl;

/* RANDOMLY SHUFFLE THE VECTOR OF INTEGERS USING mt19937 GENERATOR */

shuffle(begin(v), end(v), mt);

/* PRINT IT */

cout << "shuffled to : ";

for(auto it : v) cout << it << " ";

cout << endl;

/* ********************* */

/* BENCHMARK STARTS HERE */

/* ********************* */

cout << "generating " << COUNT << " random numbers..." << endl;

/* TEST PERFORMANCE OF random_device */

auto start = high_resolution_clock::now();

int result{0};

for(int i = 0; i < COUNT; ++i)

result += rd();

auto end = high_resolution_clock::now();

auto duration = duration_cast<microseconds>(end - start).count() / 1000.0f;

cout << "random_device duration in ms = " << duration << endl;

/* TEST PERFORMANCE OF mt19937 */

start = high_resolution_clock::now();

result = 0;

for(int i = 0; i < COUNT; ++i)

result += mt();

end = high_resolution_clock::now();

duration = duration_cast<microseconds>(end - start).count() / 1000.0f;

cout << "mt19937 duration in ms = " << duration << endl;

/* TEST PERFORMANCE OF uniform_int_distribution */

start = high_resolution_clock::now();

result = 0;

for(int i = 0; i < COUNT; ++i)

result += int_dist(mt);

end = high_resolution_clock::now();

duration = duration_cast<microseconds>(end - start).count() / 1000.0f;

cout << "uniform_int_distribution duration in ms = " << duration << endl;

/* TEST PERFORMANCE OF uniform_real_distribution */

start = high_resolution_clock::now();

result = 0;

for(int i = 0; i < COUNT; ++i)

result += real_dist(mt);

end = high_resolution_clock::now();

duration = duration_cast<microseconds>(end - start).count() / 1000.0f;

cout << "uniform_real_distribution duration in ms = " << duration << endl;

}

Time to generate 100,000,000 random numbers on 2012 MacBook Pro i7 2.3GHz. On a logarithmic scale, where RD = std::random_device, MT = std::mt19937, UD = std::uniform_int_distribution, and UD-R = std::uniform_real_distribution.
Graph created using gnuplot.

rand() Considered Harmful

Posted on February 24, 2019March 14, 2019 by Martin

Hilarious and informative talk on C and C++ random number generation:

Practical Modern C++

Author: Martin

Parsing command line options

ANSI escape codes

Two-factor authentication

Plugins: loading code at runtime

Micro-benchmarks

SQL database access

Measuring CPU time

Printing stack traces

Generating Unique IDs

RPC with Protocol Buffers

Serialize data to XML

Thrift: or how to RPC

Protocol Buffers: or how to serialize data

Well, that was no fun :(

Parallel STL

Advanced thread pool

Better Code: Concurrency

Better timer class

Random number generator

rand() Considered Harmful