First Commit

This commit is contained in:
Hamza committed 2026-02-26 13:00:32 +01:00
commit 8a6c7106e3
1312 files changed
+425396

No files matched your search

+37
View File
@@ -0,0 +1,37 @@
#-----------------------------------------------------------------------------
#
# CMake Config
#
# protozero documentation
#
#-----------------------------------------------------------------------------
message(STATUS "Configuring documentation")
message(STATUS "Looking for doxygen")
find_package(Doxygen)
if(DOXYGEN_FOUND AND DOXYGEN_DOT_FOUND)
message(STATUS "Looking for doxygen - found")
configure_file(Doxyfile.in ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile @ONLY)
file(GLOB HEADER_FILES "${CMAKE_SOURCE_DIR}/include/protozero/*.hpp")
add_custom_command(OUTPUT html/index.html
COMMAND ${DOXYGEN_EXECUTABLE}
ARGS ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile
DEPENDS Doxyfile.in advanced.md cheatsheet.md tutorial.md
${HEADER_FILES}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
COMMENT "Generating API documentation with Doxygen" VERBATIM)
add_custom_target(doc ALL
DEPENDS html/index.html)
else()
message(STATUS "Looking for doxygen - not found")
message(STATUS " Disabled making of documentation.")
endif()
#-----------------------------------------------------------------------------
message(STATUS "Configuring documentation - done")
#-----------------------------------------------------------------------------
File diff suppressed because it is too large. Load diff
+326
View File
@@ -0,0 +1,326 @@
# Protozero Advanced Topics
This documentation contains some mixed advanced topics for Protozero users.
Read the [tutorial](tutorial.md) first if you are new to Protozero.
## Limitations of Protozero
* A protobuf message has to fit into memory completely, otherwise it can not
be parsed with this library. There is no streaming support.
* The length of a string, bytes, or submessage can't be more than 2^31-1.
* There is no specific support for maps but they can be used as described in
the "Backwards compatibility" section of
https://developers.google.com/protocol-buffers/docs/proto3#maps.
## Checking the Protozero version number
If `protozero/version.hpp` is included, the following macros are set:
| Macro | Example | Description |
| -------------------------- | ------- | ---------------------------------------------- |
| `PROTOZERO_VERSION_MAJOR` | 1 | Major version number |
| `PROTOZERO_VERSION_MINOR` | 3 | Minor version number |
| `PROTOZERO_VERSION_PATCH` | 2 | Patch number |
| `PROTOZERO_VERSION_CODE` | 10302 | Version (major * 10,000 + minor * 100 + patch) |
| `PROTOZERO_VERSION_STRING` | "1.3.2" | Version string |
## Changing Protozero behaviour with macros
The behaviour of Protozero can be changed by defining the following macros.
They have to be set before including any of the Protozero headers.
### `PROTOZERO_STRICT_API`
If this is set, you will get some extra warnings or errors during compilation
if you are using an old (deprecated) interface to Protozero. Enable this if
you want to make sure your code will work with future versions of Protozero.
### `PROTOZERO_USE_VIEW`
Protozero uses the class `protozero::data_view` as the return type of the
`pbf_reader::get_view()` method and a few other functions take a
`protozero::data_view` as parameter.
If `PROTOZERO_USE_VIEW` is unset, `protozero::data_view` is Protozero's own
implementation of a *string view* class.
Set this macro if you want to use a different implementation such as the C++17
`std::string_view` class. In this case `protozero::data_view` will simply be
an alias to the class you specify.
#define PROTOZERO_USE_VIEW std::string_view
## Repeated fields in messages
The Google Protobuf spec documents that a non-repeated field can actually
appear several times in a message and the implementation is required to return
the value of the last version of that field in this case. `pbf_reader.hpp` does
not enforce this. If this feature is needed in your case, you have to do this
yourself.
The [spec also
says](https://developers.google.com/protocol-buffers/docs/encoding#packed)
that you must be able to read a packed repeated field where a not-packed
repeated field is expected and vice versa. Also there can be several (packed or
not-packed) repeated fields with the same tag and their contents must be
concatenated. It is your responsibility to do this, Protozero doesn't do that
for you.
### Using `tag_and_type()`
The `tag_and_type()` free function and the method of the same name on the
`pbf_reader` and `pbf_message` classes can be used to access both packed and
unpacked repeated fields. (It can also be used to check that you have the
right type of encoding for other fields.)
Here is the outline:
```cpp
enum class ExampleMsg : protozero::pbf_tag_type {
repeated_uint32_x = 1
};
std::string data = ...
pbf_message<ExampleMsg> message{data};
while (message.next()) {
switch (message.tag_and_type()) {
case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::length_delimited): {
auto xit = message.get_packed_uint32();
... // handle the repeated field when it is packed
}
break;
case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::varint): {
auto x = message.get_uint32();
... // handle the repeated field when it is not packed
}
break;
default:
message.skip();
}
}
```
All this works on `pbf_reader` in the same way as with `pbf_message` with the
usual difference that `pbf_reader` takes a numeric field tag and `pbf_message`
an enum field.
If you only want to check for one specific tag and type you can use the
two-argument version of `pbf_reader::next()`. In this case `17` is the field
tag we are looking for:
```cpp
std::string data = ...
pbf_reader message{data};
while (message.next(17, pbf_wire_type::varint)) {
auto foo = message.get_int32();
...
}
```
See the test under `test/t/tag_and_type/` for a complete example.
## Reserving memory when writing messages
If you know beforehand how large a message will become or can take an educated
guess, you can call the usual `std::string::reserve()` on the underlying string
before you give it to an `pbf_writer` or `pbf_builder` object.
Or you can (at any time) call `reserve()` on the `pbf_writer` or `pbf_builder`.
This will reserve the given amount of bytes *in addition to whatever is already
in that message*. (Note that this behaviour is different then what `reserve()`
does on `std::string` or `std::vector`.)
In the general case it is not easy to figure out how much memory you will need
because of the varint packing of integers. But sometimes you can make at least
a rough estimate. Still, you should probably only use this facility if you have
benchmarks proving that it actually makes your program faster.
## Using the low-level varint and zigzag encoding and decoding functions
Protozero gives you access to the low-level functions for encoding and
decoding varint and zigzag integer encodings, because these functions can
sometimes be useful outside the Protocol Buffer context.
### Using low-level functions
To use the low-level functions, add this include to your C++ program:
```cpp
#include <protozero/varint.hpp>
```
### Functions
The following functions are then available:
```cpp
decode_varint()
write_varint()
encode_zigzag32()
encode_zigzag64()
decode_zigzag32()
decode_zigzag64()
```
See the reference documentation created by `make doc` for details.
## Vectored input for length-delimited fields
Length-delimited fields (like string fields, byte fields and messages) are
usually set by calling `add_string()`, `add_message()`, etc. These functions
have several forms, but they basically all take a *tag*, a *size*, and a
*pointer to the data*. They write the length of the data into the message
and then copy the data over.
Sometimes you have the data not in one place, but spread over several
buffers. In this case you have to consolidate those buffers first, which needs
an extra copy. Say you have two very long strings that should be concatenated
into a message:
```cpp
std::string a{"very long string..."};
std::string b{"another very long string..."};
std::string data;
protozero::pbf_writer writer{data};
a.append(b); // expensive extra copy
writer.add_string(1, a);
```
To avoid this, the function `add_bytes_vectored()` can be used which allows
vectored (or scatter/gather) input like this:
```cpp
std::string a{"very long string..."};
std::string b{"another very long string..."};
std::string data;
protozero::pbf_writer writer{data};
writer.add_bytes_vectored(1, a, b);
```
`add_bytes_vectored()` will add up the sizes of all its arguments and copy over
all the data only once.
The function takes any number of arguments. The arguments must be of a type
supporting the `data()` and `size()` methods like `protozero::data_view()`,
`std::string` or the C++17 `std::string_view`.
Note that there is only one version of the function which can be used for any
length-delimited field including strings, bytes, messages and repeated packed
fields.
The function is also available in the `pbf_builder` class.
## Internal handling of varints
When varints are decoded they are always decoded as 64bit unsigned integers and
after that casted to the type you are requesting (using `static_cast`). This
means that if the protocol buffer message was created with a different integer
type than what you are reading it with, you might get wrong results without any
warning or error. This is the same behaviour as the Google Protocol Buffers
library has.
In normal use, this should never matter, because presumably you are using the
same types to write that data as you are using to read it later. It can happen
if the data is corrupted intentionally or unintentionally in some way. But
this can't be used to feed you any data that it wasn't possible to feed you
without this behaviour, so it doesn't open up any potential problems. You
always have to check anyway that the integers are in the range you expected
them to be in if the expected range is different than the range of the integer
type. This is especially true for enums which protozero will return as
`int32_t`.
## How many items are there in a repeated packed field?
Sometimes it is useful to know how many values there are in a repeated packed
field. For instance when you want to reserve space in a `std::vector`.
```cpp
protozero::pbf_reader message{...};
message.next(...);
const auto range = message.get_packed_sint32();
std::vector<int> myvalues;
myvalues.reserve(range.size());
for (auto value : range) {
myvalues.push_back(value);
}
```
It depends on the type of range how expensive the `size()` call is. For ranges
derived from packed repeated fixed sized values the effort will be constant,
for ranges derived from packed repeated varints, the effort will be linear, but
still considerably cheaper than decoding the varints. You have to benchmark
your use case to see whether the `reserve()` (or whatever you are using the
`size()` for) is worth it.
## Using a different buffer class than std::string
Normally you are using the `pbf_writer` or `pbf_builder` classes which use a
`std::string` that you supply as their buffer for building the actual protocol
buffers message into. But you can use a different buffer implementation
instead. This might be useful if you want to use a fixed-size buffer for
instance.
The `pbf_writer` and `pbf_builder` classes are actually only aliases for the
`basic_pbf_writer` and `basic_pbf_builder` template classes:
```cpp
using pbf_writer = basic_pbf_writer<std::string>;
template <typename T>
using pbf_builder = basic_pbf_builder<std::string, T>;
```
If you want to use a different buffer type, use the `basic_*` form of the
class and use the buffer class as template parameter. When instantiating the
`basic_pbf_writer` or `basic_pbf_builder`, the only parameter to the
constructor must always be a reference to an object of the buffer class.
```cpp
some_buffer_class buffer;
basic_pbf_writer<some_buffer_class> writer{buffer};
```
For this to work you must supply template specializations for some static
functions in the `protozero::buffer_customization` struct, see
`buffer_tmpl.hpp` for details.
Protozero already supports two buffer types:
* `std::string` (to use include `protozero/buffer_string.hpp`)
* `std::vector<char>` (to use include `protozero/buffer_vector.hpp`)
There is a class `protozero::fixed_size_buffer_adaptor` you can use as adaptor
for any fixed-sized buffer you might have. Include `protozero/buffer_fixed.hpp`
to use it:
```cpp
#include <protozero/buffer_fixed.hpp>
your_buffer_class some_buffer;
protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer.data(), some_buffer.size()};
basic_pbf_writer<protozero::fixed_size_buffer_adaptor> writer{buffer_adaptor};
```
The buffer adaptor can be initialized with any container if it supports the
`data()` and `size()` member functions:
```cpp
protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer};
```
+67
View File
@@ -0,0 +1,67 @@
# Protozero Cheat Sheet
See also this
[handy table](https://developers.google.com/protocol-buffers/docs/proto#scalar)
from the Google Protocol Buffers documentation.
## Scalar types
| PBF Type | Underlying Storage | C++ Type | Getter | Notes |
| -------- | ------------------ | ------------- | ---------------- | ----- |
| int32 | varint | `int32_t` | `get_int32()` | |
| sint32 | varint (zigzag) | `int32_t` | `get_sint32()` | |
| uint32 | varint | `uint32_t` | `get_uint32()` | |
| int64 | varint | `int64_t` | `get_int64()` | |
| sint64 | varint (zigzag) | `int64_t` | `get_sint64()` | |
| uint64 | varint | `uint64_t` | `get_uint64()` | |
| bool | varint | `bool` | `get_bool()` | |
| enum | varint | `int32_t` | `get_enum()` | |
| fixed32 | 32bit fixed | `uint32_t` | `get_fixed32()` | |
| sfixed32 | 32bit fixed | `int32_t` | `get_sfixed32()` | |
| fixed64 | 64bit fixed | `uint64_t` | `get_fixed64()` | |
| sfixed64 | 64bit fixed | `int64_t` | `get_sfixed64()` | |
| float | 32bit fixed | `float` | `get_float()` | |
| double | 64bit fixed | `double` | `get_double()` | |
| string | length-delimited | `data_view` | `get_view()` | (1) |
| string | length-delimited | pair | `get_data()` | (2) |
| string | length-delimited | `std::string` | `get_string()` | |
| bytes | length-delimited | `data_view` | `get_view()` | (1) |
| bytes | length-delimited | pair | `get_data()` | (2) |
| bytes | length-delimited | `std::string` | `get_bytes()` | |
| message | length-delimited | `data_view` | `get_view()` | (1) |
| message | length-delimited | pair | `get_data()` | (2) |
| message | length-delimited | `pbf_reader` | `get_message()` | |
### Notes:
* (1) preferred form, returns `protozero::data_view` which is convertible to
`std::string` if needed.
* (2) deprecated form, returns `std::pair<const char*, pbf_length_type>`,
use `get_view()` instead. This form is only available if
`PROTOZERO_STRICT_API` is not defined.
* The setter function of `pbf_writer` is always `add_` + the PBF type. Several
overloads are available.
## Packed repeated fields
| PBF Type | Getter |
| -------- | ----------------------- |
| int32 | `get_packed_int32()` |
| sint32 | `get_packed_sint32()` |
| uint32 | `get_packed_uint32()` |
| int64 | `get_packed_int64()` |
| sint64 | `get_packed_sint64()` |
| uint64 | `get_packed_uint64()` |
| bool | `get_packed_bool()` |
| enum | `get_packed_enum()` |
| fixed32 | `get_packed_fixed32()` |
| sfixed32 | `get_packed_sfixed32()` |
| fixed64 | `get_packed_fixed64()` |
| sfixed64 | `get_packed_sfixed64()` |
| float | `get_packed_float()` |
| double | `get_packed_double()` |
Packed repeated fields for string, bytes, and message types are not possible.
+626
View File
@@ -0,0 +1,626 @@
# Protozero Tutorial
## Getting to know Protocol Buffers
Protozero is a very low level library. You really have to know some of the
insides of Protocol Buffers to work with it!
So before reading any further in this document, read the following from the
Protocol Buffer documentation:
* [Developer Guide - Overview](https://developers.google.com/protocol-buffers/docs/overview)
* [Language Guide](https://developers.google.com/protocol-buffers/docs/proto)
* [Encoding](https://developers.google.com/protocol-buffers/docs/encoding)
Make sure you understand the basic types of values supported by Protocol
Buffers. Refer to this
[handy table](https://developers.google.com/protocol-buffers/docs/proto#scalar)
and [the cheat sheet](cheatsheet.md) if you are getting lost.
## Prerequisites
You need a C++11-capable compiler for Protozero to work. Copy the files in the
`include/protozero` directory somewhere where your build system can find them.
Keep the `protozero` directory and include the files in the form
```cpp
#include <protozero/FILENAME.hpp>
```
## Parsing protobuf-encoded messages
### Using `pbf_reader`
To use the `pbf_reader` class, add this include to your C++ program:
```cpp
#include <protozero/pbf_reader.hpp>
```
The `pbf_reader` class contains asserts that will detect some programming
errors. We encourage you to compile with asserts enabled in your debug builds.
### An introductory example
Lets say you have a protocol description in a `.proto` file like this:
```cpp
message Example1 {
required uint32 x = 1;
optional string s = 2;
repeated fixed64 r = 17;
}
```
To read messages created according to that description, you will have code that
looks somewhat like this:
```cpp
#include <protozero/pbf_reader.hpp>
// get data from somewhere into the input string
std::string input = get_input_data();
// initialize pbf message with this data
protozero::pbf_reader message{input};
// iterate over fields in the message
while (message.next()) {
// switch depending on the field tag (the field name is not available)
switch (message.tag()) {
case 1:
// get data for tag 1 (in this case an uint32)
auto x = message.get_uint32();
break;
case 2:
// get data for tag 2 (in this case a string)
std::string s = message.get_string();
break;
case 17:
// ignore data for tag 17
message.skip();
break;
default:
// ignore data for unknown tags to allow for future extensions
message.skip();
}
}
```
You always have to call `next()` and then either one of the accessor functions
(like `get_uint32()` or `get_string()`) to get the field value or `skip()` to
ignore this field. Then call `next()` again, and so forth. Never call `next()`
twice in a row or any if the accessor or skip functions twice in a row.
Because the `pbf_reader` class doesn't know the `.proto` file it doesn't know
which field names or tags there are and it doesn't known the types of the
fields. You have to make sure to call the right `get_...()` function for each
tag. Some `assert()s` are done to check you are calling the right functions,
but not all errors can be detected.
Note that it doesn't matter whether a field is defined as `required`,
`optional`, or `repeated`. You always have to be prepared to get zero, one, or
more instances of a field and you always have to be prepared to get other
fields, too, unless you want your program to break if somebody adds a new
field.
### If you only need a single field
If, out of a protocol buffer message, you only need the value of a single
field, you can use the version of the `next()` function with a parameter:
```cpp
// same .proto file and initialization as above
// get all fields with tag 17, skip all others
while (message.next(17)) {
auto r = message.get_fixed64();
std::cout << r << "\n";
}
```
### Handling scalar fields
As you saw in the example, handling scalar field types is reasonably easy. You
just check the `.proto` file for the type of a field and call the corresponding
function called `get_` + _field type_.
For `string` and `bytes` types the internal handling is exactly the same, but
both `get_string()` and `get_bytes()` are provided to make the code
self-documenting. Both theses calls allocate and return a `std::string` which
can add some overhead. You can call the `get_view()` function instead which
returns a `data_view` containing a pointer into the data (access with `data()`)
and the length of the data (access with `size()`).
### Handling repeated packed fields
Fields that are marked as `[packed=true]` in the `.proto` file are handled
somewhat differently. `get_packed_...()` functions returning an iterator range
are used to access the data.
So, for example, if you have a protocol description in a `.proto` file like
this:
```cpp
message Example2 {
repeated sint32 i = 1 [packed=true];
}
```
You can get to the data like this:
```cpp
protozero::pbf_reader message{input.data(), input.size()};
// set current field
message.next(1);
// get an iterator range
auto pi = message.get_packed_sint32();
// iterate to get to all values
for (auto it = pi.begin(); it != pi.end(); ++it) {
std::cout << *it << '\n';
}
```
Or, with a range-based for-loop:
```cpp
for (auto value : pi) {
std::cout << v << '\n';
}
```
So you are getting a pair of normal forward iterators wrapped in an iterator
range object. The iterators can be used with any STL algorithms etc.
Note that the previous only applies to repeated **packed** fields, normal
repeated fields are handled in the usual way for scalar fields.
### Handling embedded messages
Protocol Buffers can embed any message inside another message. To access an
embedded message use the `get_message()` function. So for this description:
```cpp
message Point {
required double x = 1;
required double y = 2;
}
message Example3 {
repeated Point point = 10;
}
```
you can parse with this code:
```cpp
protozero::pbf_reader message{input};
while (message.next(10)) {
protozero::pbf_reader point = message.get_message();
double x, y;
while (point.next()) {
switch (point.tag()) {
case 1:
x = point.get_double();
break;
case 2:
y = point.get_double();
break;
default:
point.skip();
}
}
std::cout << "x=" << x << " y=" << y << "\n";
}
```
### Handling enums
Enums are stored as varints and they can't be differentiated from them. Use
the `get_enum()` function to get the value of the enum, you have to translate
this into the symbolic name yourself. See the `enum` test case for an example.
### Asserts and exceptions in the Protozero library
Protozero uses `assert()` liberally to help you find bugs in your own code when
compiled in debug mode (ie with `NDEBUG` not set). If such an assert "fires",
this is a very strong indication that there is a bug in your code somewhere.
(Protozero will disable those asserts and "convert" them into exception in its
own test code. This is done to make sure the asserts actually work as intended.
Your test code will not need this!)
Exceptions, on the other hand, are thrown by Protozero if some kind of data
corruption was detected while it is trying to parse the data. This could also
be an indicator for a bug in the user code, but because it can happen if the
data was (intentionally or not intentionally) been messed with, it is reported
to the user code using exceptions.
Most of the functions on the writer side can throw a `std::bad_alloc`
exception if there is no space to grow a buffer. Other than that no exceptions
can occur on the writer side.
All exceptions thrown by the reader side derive from `protozero::exception`.
Note that all exceptions can also happen if you are expecting a data field of
a certain type in your code but the field actually has a different type. In
that case the `pbf_reader` class might interpret the bytes in the buffer in
the wrong way and anything can happen.
#### `end_of_buffer_exception`
This will be thrown whenever any of the functions "runs out of input data".
It means you either have an incomplete message in your input or some other
data corruption has taken place.
#### `unknown_pbf_wire_type_exception`
This will be thrown if an unsupported wire type is encountered. Either your
input data is corrupted or it was written with an unsupported version of a
Protocol Buffers implementation.
#### `varint_too_long_exception`
This exception indicates an illegal encoding of a varint. It means your input
data is corrupted in some way.
#### `invalid_tag_exception`
This exception is thrown when a tag has an invalid value. Tags must be
unsigned integers between 1 and 2^29-1. Tags between 19000 and 19999 are not
allowed. See
https://developers.google.com/protocol-buffers/docs/proto#assigning-tags
#### `invalid_length_exception`
This exception is thrown when a length field of a packed repeated field is
invalid. For fixed size types the length must be a multiple of the size of
the type.
### The `pbf_reader` class
The `pbf_reader` class behaves like a value type. Objects are reasonably small
(two pointers and two `uint32_t`, so 24 bytes on a 64bit system) and they can
be copied and moved around trivially.
`pbf_reader` objects can be constructed from a `std::string` or a `const char*`
and a length field (either supplied as separate arguments or as a `std::pair`).
In all cases objects of the `pbf_reader` class store a pointer into the input
data that was given to the constructor. You have to make sure this pointer
stays valid for the duration of the objects lifetime.
## Parsing protobuf-encoded messages using `pbf_message`
One problem in the code above are the "magic numbers" used as tags for the
different fields that you got from the `.proto` file. Instead of spreading
these magic numbers around your code you can define them once in an `enum
class` and then use the `pbf_message` template class instead of the
`pbf_reader` class.
Here is the first example again, this time using this new technique. So you
have the following in a `.proto` file:
```cpp
message Example1 {
required uint32 x = 1;
optional string s = 2;
repeated fixed64 r = 17;
}
```
Add the following declaration in one of your header files:
```cpp
enum class Example1 : protozero::pbf_tag_type {
required_uint32_x = 1,
optional_string_s = 2,
repeated_fixed64_r = 17
};
```
The message name becomes the name of the `enum class` which is always built
on top of the `protozero::pbf_tag_type` type. Each field in the message
becomes one value of the enum. In this case the name is created from the
type (including the modifiers like `required` or `optional`) and the name of
the field. You can use any name you want, but this convention makes it easier
later, to get everything right.
To read messages created according to that description, you will have code that
looks somewhat like this, this time using `pbf_message` instead of
`pbf_reader`:
```cpp
#include <protozero/pbf_message.hpp>
// get data from somewhere into the input string
std::string input = get_input_data();
// initialize pbf message with this data
protozero::pbf_message<Example1> message{input};
// iterate over fields in the message
while (message.next()) {
// switch depending on the field tag (the field name is not available)
switch (message.tag()) {
case Example1::required_uint32_x:
auto x = message.get_uint32();
break;
case Example1::optional_string_s:
std::string s = message.get_string();
break;
case Example1::repeated_fixed64_r:
message.skip();
break;
default:
// ignore data for unknown tags to allow for future extensions
message.skip();
}
}
```
Note the correspondance between the enum value (for instance
`required_uint32_x`) and the name of the getter function (for instance
`get_uint32()`). This makes it easier to get the correct types. Also the
naming makes it easier to keep different message types apart if you have
multiple (or embedded) messages.
See the `test/t/complex` test case for a complete example using this interface.
Using `pbf_message` in favour of `pbf_reader` is recommended for all code.
Note that `pbf_message` derives from `pbf_reader`, so you can always fall
back to the more generic interface if necessary.
One problem you might run into is the following: The enum class lists all
possible values you know about and you'll have lots of `switch` statements
checking those values. Some compilers will know that your `switch` covers
all possible cases and warn you if you have a `default` case that looks
unneccessary to the compiler. But you still want that `default` case to allow
for future extension of those messages (and maybe also to detect corrupted
data). You can switch of this warning with `-Wno-covered-switch-default`).
## Writing protobuf-encoded messages
### Using `pbf_writer`
To use the `pbf_writer` class, add this include to your C++ program:
```cpp
#include <protozero/pbf_writer.hpp>
```
The `pbf_writer` class contains asserts that will detect some programming
errors. We encourage you to compile with asserts enabled in your debug builds.
### An introductory example
Lets say you have a protocol description in a `.proto` file like this:
```cpp
message Example {
required uint32 x = 1;
optional string s = 2;
repeated fixed64 r = 17;
}
```
To write messages created according to that description, you will have code
that looks somewhat like this:
```cpp
#include <protozero/pbf_writer.hpp>
std::string data;
protozero::pbf_writer pbf_example{data};
pbf_example.add_uint32(1, 27); // uint32_t x
pbf_example.add_fixed64(17, 1); // fixed64 r
pbf_example.add_fixed64(17, 2);
pbf_example.add_fixed64(17, 3);
pbf_example.add_string(2, "foobar"); // string s
```
First you need a string which will be used as buffer to assemble the
protobuf-formatted message. The `pbf_writer` object contains a reference to
this string buffer and through it you add data to that buffer piece by piece.
The buffer doesn't have to be empty, the `pbf_writer` will simply append its
data to whatever is there already.
### Handling scalar fields
As you could see in the introductory example handling any kind of scalar field
is easy. The type of field doesn't matter and it doesn't matter whether it is
optional, required or repeated. You always call one of the `add_TYPE()` method
on the pbf writer object.
The first parameter of these methods is always the *tag* of the field (the
field number) from the `.proto` file. The second parameter is the value you
want to set. For the `bytes` and `string` types several versions of the add
method are available taking a `const std::string&` or a `const char*` and a
length.
For `enum` types you have to use the numeric value as the symbolic names from
the `.proto` file are not available.
### Handling repeated packed fields
Repeated packed fields can easily be set from a pair of iterators:
```cpp
std::string data;
protozero::pbf_writer pw{data};
std::vector<int> v = { 1, 4, 9, 16, 25, 36 };
pw.add_packed_int32(1, std::begin(v), std::end(v));
```
If you don't have an iterator you can use the alternative form:
```cpp
std::string data;
protozero::pbf_writer pw{data};
{
protozero::packed_field_int32 field{pw, 1};
field.add_element(1);
field.add_element(10);
field.add_element(100);
}
```
Of course you can add as many elements as you want. If you add no elements
at all, this code will still work, Protozero detects this special case and
pretends you never even initialized this field.
The nested scope is important in this case, because the destructor of the
`field` object will make sure the length stored inside the field is set to
the right value. You must close that scope before adding other fields to the
`pw` pbf writer.
If you know how many elements you will add to the field and your field contains
fixed length elements, you can tell Protozero and it can optimize this case:
```cpp
std::string data;
protozero::pbf_writer pw{data};
{
protozero::packed_field_fixed32 field{pw, 1, 2}; // exactly two elements
field.add_element(42);
field.add_element(13);
}
```
In this case you have to supply exactly as many elements as you promised,
otherwise you will get a broken protobuf message.
This works for `packed_field_fixed32`, `packed_field_sfixed32`,
`packed_field_fixed64`, `packed_field_sfixed64`, `packed_field_float`, and
`packed_field_double`.
You can abandon writing of the packed field if this becomes necessary by
calling `rollback()`:
```cpp
std::string data;
protozero::pbf_writer pw{data};
{
protozero::packed_field_int32 field{pw, 1};
field.add_element(42);
// some error occurs, you don't want to have this field at all
field.rollback();
}
```
The result is the same as if the lines inside the nested brackets had never
been called. Do not try to call `add_element()` after a rollback.
### Handling sub-messages
Nested sub-messages can be handled by first creating the submessage and then
adding to the parent message:
```cpp
std::string buffer_sub;
protozero::pbf_writer pbf_sub{buffer_sub};
// add fields to sub-message
pbf_sub.add_...(...);
// ...
// sub-message is finished here
std::string buffer_parent;
protozero::pbf_writer pbf_parent{buffer_parent};
pbf_parent.add_message(1, buffer_sub);
```
This is easy to do but it has the drawback of needing a separate `std::string`
buffer. If this concerns you (and why would you use protozero and not the
Google protobuf library if it doesn't?) there is another way:
```cpp
std::string data;
protozero::pbf_writer pbf_parent{data};
// optionally add fields to parent here
pbf_parent.add_...(...);
// open a new scope
{
// create new pbf_writer with parent and the tag (field number)
// as parameters
protozero::pbf_writer pbf_sub{pbf_parent, 1};
// add fields to sub here...
pbf_sub.add_...(...);
} // closing the scope will close the sub-message
// optionally add more fields to parent here
pbf_parent.add_...(...);
```
This can be nested arbitrarily deep.
Internally the sub-message writer re-uses the buffer from the parent. It
reserves enough space in the buffer to later write the length of the submessage
into it. It then adds the contents of the submessage to the buffer. When the
`pbf_sub` writer is destructed the length of the submessage is calculated and
written in the reserved space. If less space was needed for the length field
than was available, the rest of the buffer is moved over a few bytes.
You can abandon writing of submessage if this becomes necessary by
calling `rollback()`:
```cpp
std::string data;
protozero::pbf_writer pbf_parent{data};
// open a new scope
{
// create new pbf_writer with parent and the tag (field number)
// as parameters
protozero::pbf_writer pbf_sub{pbf_parent, 1};
// add fields to sub here...
pbf_sub.add_...(...);
// some problem occurs and you want to abandon the submessage:
pbf_sub.rollback();
}
// optionally add more fields to parent here
pbf_parent.add_...(...);
```
The result is the same as if the lines inside the nested brackets had never
been called. Do not try to call any of the `add_*` functions on the submessage
after a rollback.
## Writing protobuf-encoded messages using `pbf_builder`
Just like the `pbf_message` template class wraps the `pbf_reader` class, there
is a `pbf_builder` template class wrapping the `pbf_writer` class. It is
instantiated using the same `enum class` described above and used exactly
like the `pbf_writer` class but using the values of the enum instead of bare
integers.
See the `test/t/complex` test case for a complete example using this interface.