UPDATE: Better serializer
In my very first YouTube Channel video I discuss in detail the design of a serialization framework I came up with in response to a question I was recently asked. In this post I want to give a brief overview of how it works and how to get started using it.
The primary goal was to create a serialization framework that is uncoupled from the types it operates on, meaning no class nor struct needs to inherit any sort of
|
1 2 3 4 5 6 7 |
add_pack_transform<std::uint16_t>( [](std::uint16_t v, buffer_output_t& it) { pack_value(it, htons(v)); }); add_unpack_transform<std::uint16_t>( [](buffer_input_t& it, std::uint16_t* v) { *v = ntohs(unpack_value<std::uint16_t>(it)); }); |
By default every type is packed by doing bitwise copy from its memory location into a
|
1 2 3 |
auto buf_1 = pack(std::uint16_t(0x1234)); auto tup_1 = unpack<std::uint16_t>(buf_1); |
The code above packs a single
It is possible to perform type conversions inside the pack and unpack transforms:
|
1 2 3 4 5 |
add_pack_transform<std::size_t>([](std::size_t v, buffer_output_t& it) { pack_value(it, std::uint16_t(v)); }); add_unpack_transform<std::size_t>([](buffer_input_t& it, std::size_t* v) { *v = unpack_value<std::uint16_t>(it); }); |
Above code illustrates it: every
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
add_pack_transform<const char*>( [](const char* v, buffer_output_t& it) { auto len = std::strlen(v); pack_type(it, len); pack_bytes(it, v, len); }); add_pack_transform<std::string>( [](const std::string& v, buffer_output_t& it) { pack_type(it, v.length()); pack_bytes(it, v.data(), v.length()); }); add_unpack_transform<std::string>( [](buffer_input_t& it, std::string* v) { auto len = unpack_type<std::string::size_type>(it); new (v) std::string(len, std::string::value_type()); unpack_bytes(it, v->data(), len); }); |
Here I define how to serialize
Having defined transforms for simple types we can now add more complex types, let’s say
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
using strings_t = std::vector<std::string>; add_pack_transform<strings_t>( [](const strings_t& vs, buffer_output_t& it) { pack_type(it, vs.size()); std::for_each(std::begin(vs), std::end(vs), [&](const std::string& s) { pack_type(it, s); }); }); add_unpack_transform<strings_t>( [](buffer_input_t& it, strings_t* vs) { auto size = unpack_type<strings_t::size_type>(it); new (vs) strings_t(); vs->reserve(size); std::generate_n(std::back_inserter(*vs), size, [&]() { return unpack_type<std::string>(it); }); }); |
The packing code is trivial: pack the size of the vector followed by packing each entry one at a time. Notice here I am calling
One thing I have not yet mentioned is an optimization I introduced into the code: before types are packed the exact amount of memory is reserved inside the resulting
|
1 2 |
static auto k_default_pack_size_proc = pack_size_proc_t( [](const void*) { return sizeof(T); }); |
There is a void pointer parameter but in this case it is ignored. However it is required to compute sizes of more complex types like strings; that is because in order to store the pack size procs inside an internal data structure each one needs to be wrapped inside the default lambda. This lambda gets the memory location of each type as
|
1 2 3 4 5 6 7 |
add_pack_size_proc<const char*>( [](const char* v) { return pack_size(std::strlen(v)) + std::strlen(v); }); add_pack_size_proc<std::string>( [](const std::string& v) { return pack_size(v.length()) + v.length(); }); |
Here I tell the framework how to compute the number of bytes needed to serialize strings.
|
1 2 3 4 5 6 7 |
add_pack_size_proc<strings_t>( [](const strings_t& v) { return pack_size(v.size()) + std::accumulate(std::begin(v), std::end(v), 0, [](auto sum, const std::string& v) { return sum + pack_size(v); }); }); |
I initially tagged each type with a unique value so that I could verify it during unpacking to catch errors like packing an int but trying to unpack a string. This type tagging became more complicated than I first anticipated so I got rid of it, though I may revisit it and add an additional packing and unpacking functions that perform this verification optionally.
As always, the code is available on my GitHub page.
The framework is a single header file: lsf.hpp. Example program: lsf.cpp.
2 Replies to “Light-weight serialization framework”