First Commit
This commit is contained in:
commit
8a6c7106e3
1312 files changed
+425396
No files matched your search
+37
@@ -0,0 +1,37 @@
|
||||
#-----------------------------------------------------------------------------
|
||||
#
|
||||
# CMake Config
|
||||
#
|
||||
# protozero documentation
|
||||
#
|
||||
#-----------------------------------------------------------------------------
|
||||
|
||||
message(STATUS "Configuring documentation")
|
||||
|
||||
message(STATUS "Looking for doxygen")
|
||||
find_package(Doxygen)
|
||||
|
||||
if(DOXYGEN_FOUND AND DOXYGEN_DOT_FOUND)
|
||||
message(STATUS "Looking for doxygen - found")
|
||||
configure_file(Doxyfile.in ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile @ONLY)
|
||||
|
||||
file(GLOB HEADER_FILES "${CMAKE_SOURCE_DIR}/include/protozero/*.hpp")
|
||||
add_custom_command(OUTPUT html/index.html
|
||||
COMMAND ${DOXYGEN_EXECUTABLE}
|
||||
ARGS ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile
|
||||
DEPENDS Doxyfile.in advanced.md cheatsheet.md tutorial.md
|
||||
${HEADER_FILES}
|
||||
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
|
||||
COMMENT "Generating API documentation with Doxygen" VERBATIM)
|
||||
add_custom_target(doc ALL
|
||||
DEPENDS html/index.html)
|
||||
else()
|
||||
message(STATUS "Looking for doxygen - not found")
|
||||
message(STATUS " Disabled making of documentation.")
|
||||
endif()
|
||||
|
||||
#-----------------------------------------------------------------------------
|
||||
message(STATUS "Configuring documentation - done")
|
||||
|
||||
|
||||
#-----------------------------------------------------------------------------
|
||||
+2684
File diff suppressed because it is too large.
Load diff
+326
@@ -0,0 +1,326 @@
|
||||
|
||||
# Protozero Advanced Topics
|
||||
|
||||
This documentation contains some mixed advanced topics for Protozero users.
|
||||
Read the [tutorial](tutorial.md) first if you are new to Protozero.
|
||||
|
||||
|
||||
## Limitations of Protozero
|
||||
|
||||
* A protobuf message has to fit into memory completely, otherwise it can not
|
||||
be parsed with this library. There is no streaming support.
|
||||
* The length of a string, bytes, or submessage can't be more than 2^31-1.
|
||||
* There is no specific support for maps but they can be used as described in
|
||||
the "Backwards compatibility" section of
|
||||
https://developers.google.com/protocol-buffers/docs/proto3#maps.
|
||||
|
||||
|
||||
## Checking the Protozero version number
|
||||
|
||||
If `protozero/version.hpp` is included, the following macros are set:
|
||||
|
||||
| Macro | Example | Description |
|
||||
| -------------------------- | ------- | ---------------------------------------------- |
|
||||
| `PROTOZERO_VERSION_MAJOR` | 1 | Major version number |
|
||||
| `PROTOZERO_VERSION_MINOR` | 3 | Minor version number |
|
||||
| `PROTOZERO_VERSION_PATCH` | 2 | Patch number |
|
||||
| `PROTOZERO_VERSION_CODE` | 10302 | Version (major * 10,000 + minor * 100 + patch) |
|
||||
| `PROTOZERO_VERSION_STRING` | "1.3.2" | Version string |
|
||||
|
||||
|
||||
## Changing Protozero behaviour with macros
|
||||
|
||||
The behaviour of Protozero can be changed by defining the following macros.
|
||||
They have to be set before including any of the Protozero headers.
|
||||
|
||||
### `PROTOZERO_STRICT_API`
|
||||
|
||||
If this is set, you will get some extra warnings or errors during compilation
|
||||
if you are using an old (deprecated) interface to Protozero. Enable this if
|
||||
you want to make sure your code will work with future versions of Protozero.
|
||||
|
||||
### `PROTOZERO_USE_VIEW`
|
||||
|
||||
Protozero uses the class `protozero::data_view` as the return type of the
|
||||
`pbf_reader::get_view()` method and a few other functions take a
|
||||
`protozero::data_view` as parameter.
|
||||
|
||||
If `PROTOZERO_USE_VIEW` is unset, `protozero::data_view` is Protozero's own
|
||||
implementation of a *string view* class.
|
||||
|
||||
Set this macro if you want to use a different implementation such as the C++17
|
||||
`std::string_view` class. In this case `protozero::data_view` will simply be
|
||||
an alias to the class you specify.
|
||||
|
||||
#define PROTOZERO_USE_VIEW std::string_view
|
||||
|
||||
|
||||
## Repeated fields in messages
|
||||
|
||||
The Google Protobuf spec documents that a non-repeated field can actually
|
||||
appear several times in a message and the implementation is required to return
|
||||
the value of the last version of that field in this case. `pbf_reader.hpp` does
|
||||
not enforce this. If this feature is needed in your case, you have to do this
|
||||
yourself.
|
||||
|
||||
The [spec also
|
||||
says](https://developers.google.com/protocol-buffers/docs/encoding#packed)
|
||||
that you must be able to read a packed repeated field where a not-packed
|
||||
repeated field is expected and vice versa. Also there can be several (packed or
|
||||
not-packed) repeated fields with the same tag and their contents must be
|
||||
concatenated. It is your responsibility to do this, Protozero doesn't do that
|
||||
for you.
|
||||
|
||||
### Using `tag_and_type()`
|
||||
|
||||
The `tag_and_type()` free function and the method of the same name on the
|
||||
`pbf_reader` and `pbf_message` classes can be used to access both packed and
|
||||
unpacked repeated fields. (It can also be used to check that you have the
|
||||
right type of encoding for other fields.)
|
||||
|
||||
Here is the outline:
|
||||
|
||||
```cpp
|
||||
enum class ExampleMsg : protozero::pbf_tag_type {
|
||||
repeated_uint32_x = 1
|
||||
};
|
||||
|
||||
std::string data = ...
|
||||
pbf_message<ExampleMsg> message{data};
|
||||
while (message.next()) {
|
||||
switch (message.tag_and_type()) {
|
||||
case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::length_delimited): {
|
||||
auto xit = message.get_packed_uint32();
|
||||
... // handle the repeated field when it is packed
|
||||
}
|
||||
break;
|
||||
case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::varint): {
|
||||
auto x = message.get_uint32();
|
||||
... // handle the repeated field when it is not packed
|
||||
}
|
||||
break;
|
||||
default:
|
||||
message.skip();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All this works on `pbf_reader` in the same way as with `pbf_message` with the
|
||||
usual difference that `pbf_reader` takes a numeric field tag and `pbf_message`
|
||||
an enum field.
|
||||
|
||||
If you only want to check for one specific tag and type you can use the
|
||||
two-argument version of `pbf_reader::next()`. In this case `17` is the field
|
||||
tag we are looking for:
|
||||
|
||||
```cpp
|
||||
std::string data = ...
|
||||
pbf_reader message{data};
|
||||
while (message.next(17, pbf_wire_type::varint)) {
|
||||
auto foo = message.get_int32();
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
See the test under `test/t/tag_and_type/` for a complete example.
|
||||
|
||||
|
||||
## Reserving memory when writing messages
|
||||
|
||||
If you know beforehand how large a message will become or can take an educated
|
||||
guess, you can call the usual `std::string::reserve()` on the underlying string
|
||||
before you give it to an `pbf_writer` or `pbf_builder` object.
|
||||
|
||||
Or you can (at any time) call `reserve()` on the `pbf_writer` or `pbf_builder`.
|
||||
This will reserve the given amount of bytes *in addition to whatever is already
|
||||
in that message*. (Note that this behaviour is different then what `reserve()`
|
||||
does on `std::string` or `std::vector`.)
|
||||
|
||||
In the general case it is not easy to figure out how much memory you will need
|
||||
because of the varint packing of integers. But sometimes you can make at least
|
||||
a rough estimate. Still, you should probably only use this facility if you have
|
||||
benchmarks proving that it actually makes your program faster.
|
||||
|
||||
|
||||
## Using the low-level varint and zigzag encoding and decoding functions
|
||||
|
||||
Protozero gives you access to the low-level functions for encoding and
|
||||
decoding varint and zigzag integer encodings, because these functions can
|
||||
sometimes be useful outside the Protocol Buffer context.
|
||||
|
||||
### Using low-level functions
|
||||
|
||||
To use the low-level functions, add this include to your C++ program:
|
||||
|
||||
```cpp
|
||||
#include <protozero/varint.hpp>
|
||||
```
|
||||
|
||||
### Functions
|
||||
|
||||
The following functions are then available:
|
||||
|
||||
```cpp
|
||||
decode_varint()
|
||||
write_varint()
|
||||
encode_zigzag32()
|
||||
encode_zigzag64()
|
||||
decode_zigzag32()
|
||||
decode_zigzag64()
|
||||
```
|
||||
|
||||
See the reference documentation created by `make doc` for details.
|
||||
|
||||
|
||||
## Vectored input for length-delimited fields
|
||||
|
||||
Length-delimited fields (like string fields, byte fields and messages) are
|
||||
usually set by calling `add_string()`, `add_message()`, etc. These functions
|
||||
have several forms, but they basically all take a *tag*, a *size*, and a
|
||||
*pointer to the data*. They write the length of the data into the message
|
||||
and then copy the data over.
|
||||
|
||||
Sometimes you have the data not in one place, but spread over several
|
||||
buffers. In this case you have to consolidate those buffers first, which needs
|
||||
an extra copy. Say you have two very long strings that should be concatenated
|
||||
into a message:
|
||||
|
||||
```cpp
|
||||
std::string a{"very long string..."};
|
||||
std::string b{"another very long string..."};
|
||||
|
||||
std::string data;
|
||||
protozero::pbf_writer writer{data};
|
||||
|
||||
a.append(b); // expensive extra copy
|
||||
|
||||
writer.add_string(1, a);
|
||||
```
|
||||
|
||||
To avoid this, the function `add_bytes_vectored()` can be used which allows
|
||||
vectored (or scatter/gather) input like this:
|
||||
|
||||
```cpp
|
||||
std::string a{"very long string..."};
|
||||
std::string b{"another very long string..."};
|
||||
|
||||
std::string data;
|
||||
protozero::pbf_writer writer{data};
|
||||
|
||||
writer.add_bytes_vectored(1, a, b);
|
||||
```
|
||||
|
||||
`add_bytes_vectored()` will add up the sizes of all its arguments and copy over
|
||||
all the data only once.
|
||||
|
||||
The function takes any number of arguments. The arguments must be of a type
|
||||
supporting the `data()` and `size()` methods like `protozero::data_view()`,
|
||||
`std::string` or the C++17 `std::string_view`.
|
||||
|
||||
Note that there is only one version of the function which can be used for any
|
||||
length-delimited field including strings, bytes, messages and repeated packed
|
||||
fields.
|
||||
|
||||
The function is also available in the `pbf_builder` class.
|
||||
|
||||
|
||||
## Internal handling of varints
|
||||
|
||||
When varints are decoded they are always decoded as 64bit unsigned integers and
|
||||
after that casted to the type you are requesting (using `static_cast`). This
|
||||
means that if the protocol buffer message was created with a different integer
|
||||
type than what you are reading it with, you might get wrong results without any
|
||||
warning or error. This is the same behaviour as the Google Protocol Buffers
|
||||
library has.
|
||||
|
||||
In normal use, this should never matter, because presumably you are using the
|
||||
same types to write that data as you are using to read it later. It can happen
|
||||
if the data is corrupted intentionally or unintentionally in some way. But
|
||||
this can't be used to feed you any data that it wasn't possible to feed you
|
||||
without this behaviour, so it doesn't open up any potential problems. You
|
||||
always have to check anyway that the integers are in the range you expected
|
||||
them to be in if the expected range is different than the range of the integer
|
||||
type. This is especially true for enums which protozero will return as
|
||||
`int32_t`.
|
||||
|
||||
|
||||
## How many items are there in a repeated packed field?
|
||||
|
||||
Sometimes it is useful to know how many values there are in a repeated packed
|
||||
field. For instance when you want to reserve space in a `std::vector`.
|
||||
|
||||
```cpp
|
||||
protozero::pbf_reader message{...};
|
||||
message.next(...);
|
||||
const auto range = message.get_packed_sint32();
|
||||
|
||||
std::vector<int> myvalues;
|
||||
myvalues.reserve(range.size());
|
||||
|
||||
for (auto value : range) {
|
||||
myvalues.push_back(value);
|
||||
}
|
||||
```
|
||||
|
||||
It depends on the type of range how expensive the `size()` call is. For ranges
|
||||
derived from packed repeated fixed sized values the effort will be constant,
|
||||
for ranges derived from packed repeated varints, the effort will be linear, but
|
||||
still considerably cheaper than decoding the varints. You have to benchmark
|
||||
your use case to see whether the `reserve()` (or whatever you are using the
|
||||
`size()` for) is worth it.
|
||||
|
||||
|
||||
## Using a different buffer class than std::string
|
||||
|
||||
Normally you are using the `pbf_writer` or `pbf_builder` classes which use a
|
||||
`std::string` that you supply as their buffer for building the actual protocol
|
||||
buffers message into. But you can use a different buffer implementation
|
||||
instead. This might be useful if you want to use a fixed-size buffer for
|
||||
instance.
|
||||
|
||||
The `pbf_writer` and `pbf_builder` classes are actually only aliases for the
|
||||
`basic_pbf_writer` and `basic_pbf_builder` template classes:
|
||||
|
||||
```cpp
|
||||
using pbf_writer = basic_pbf_writer<std::string>;
|
||||
|
||||
template <typename T>
|
||||
using pbf_builder = basic_pbf_builder<std::string, T>;
|
||||
```
|
||||
|
||||
If you want to use a different buffer type, use the `basic_*` form of the
|
||||
class and use the buffer class as template parameter. When instantiating the
|
||||
`basic_pbf_writer` or `basic_pbf_builder`, the only parameter to the
|
||||
constructor must always be a reference to an object of the buffer class.
|
||||
|
||||
```cpp
|
||||
some_buffer_class buffer;
|
||||
basic_pbf_writer<some_buffer_class> writer{buffer};
|
||||
```
|
||||
|
||||
For this to work you must supply template specializations for some static
|
||||
functions in the `protozero::buffer_customization` struct, see
|
||||
`buffer_tmpl.hpp` for details.
|
||||
|
||||
Protozero already supports two buffer types:
|
||||
* `std::string` (to use include `protozero/buffer_string.hpp`)
|
||||
* `std::vector<char>` (to use include `protozero/buffer_vector.hpp`)
|
||||
|
||||
There is a class `protozero::fixed_size_buffer_adaptor` you can use as adaptor
|
||||
for any fixed-sized buffer you might have. Include `protozero/buffer_fixed.hpp`
|
||||
to use it:
|
||||
|
||||
```cpp
|
||||
#include <protozero/buffer_fixed.hpp>
|
||||
|
||||
your_buffer_class some_buffer;
|
||||
protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer.data(), some_buffer.size()};
|
||||
basic_pbf_writer<protozero::fixed_size_buffer_adaptor> writer{buffer_adaptor};
|
||||
```
|
||||
|
||||
The buffer adaptor can be initialized with any container if it supports the
|
||||
`data()` and `size()` member functions:
|
||||
|
||||
```cpp
|
||||
protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer};
|
||||
```
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
|
||||
# Protozero Cheat Sheet
|
||||
|
||||
See also this
|
||||
[handy table](https://developers.google.com/protocol-buffers/docs/proto#scalar)
|
||||
from the Google Protocol Buffers documentation.
|
||||
|
||||
## Scalar types
|
||||
|
||||
| PBF Type | Underlying Storage | C++ Type | Getter | Notes |
|
||||
| -------- | ------------------ | ------------- | ---------------- | ----- |
|
||||
| int32 | varint | `int32_t` | `get_int32()` | |
|
||||
| sint32 | varint (zigzag) | `int32_t` | `get_sint32()` | |
|
||||
| uint32 | varint | `uint32_t` | `get_uint32()` | |
|
||||
| int64 | varint | `int64_t` | `get_int64()` | |
|
||||
| sint64 | varint (zigzag) | `int64_t` | `get_sint64()` | |
|
||||
| uint64 | varint | `uint64_t` | `get_uint64()` | |
|
||||
| bool | varint | `bool` | `get_bool()` | |
|
||||
| enum | varint | `int32_t` | `get_enum()` | |
|
||||
| fixed32 | 32bit fixed | `uint32_t` | `get_fixed32()` | |
|
||||
| sfixed32 | 32bit fixed | `int32_t` | `get_sfixed32()` | |
|
||||
| fixed64 | 64bit fixed | `uint64_t` | `get_fixed64()` | |
|
||||
| sfixed64 | 64bit fixed | `int64_t` | `get_sfixed64()` | |
|
||||
| float | 32bit fixed | `float` | `get_float()` | |
|
||||
| double | 64bit fixed | `double` | `get_double()` | |
|
||||
| string | length-delimited | `data_view` | `get_view()` | (1) |
|
||||
| string | length-delimited | pair | `get_data()` | (2) |
|
||||
| string | length-delimited | `std::string` | `get_string()` | |
|
||||
| bytes | length-delimited | `data_view` | `get_view()` | (1) |
|
||||
| bytes | length-delimited | pair | `get_data()` | (2) |
|
||||
| bytes | length-delimited | `std::string` | `get_bytes()` | |
|
||||
| message | length-delimited | `data_view` | `get_view()` | (1) |
|
||||
| message | length-delimited | pair | `get_data()` | (2) |
|
||||
| message | length-delimited | `pbf_reader` | `get_message()` | |
|
||||
|
||||
### Notes:
|
||||
|
||||
* (1) preferred form, returns `protozero::data_view` which is convertible to
|
||||
`std::string` if needed.
|
||||
* (2) deprecated form, returns `std::pair<const char*, pbf_length_type>`,
|
||||
use `get_view()` instead. This form is only available if
|
||||
`PROTOZERO_STRICT_API` is not defined.
|
||||
* The setter function of `pbf_writer` is always `add_` + the PBF type. Several
|
||||
overloads are available.
|
||||
|
||||
|
||||
## Packed repeated fields
|
||||
|
||||
| PBF Type | Getter |
|
||||
| -------- | ----------------------- |
|
||||
| int32 | `get_packed_int32()` |
|
||||
| sint32 | `get_packed_sint32()` |
|
||||
| uint32 | `get_packed_uint32()` |
|
||||
| int64 | `get_packed_int64()` |
|
||||
| sint64 | `get_packed_sint64()` |
|
||||
| uint64 | `get_packed_uint64()` |
|
||||
| bool | `get_packed_bool()` |
|
||||
| enum | `get_packed_enum()` |
|
||||
| fixed32 | `get_packed_fixed32()` |
|
||||
| sfixed32 | `get_packed_sfixed32()` |
|
||||
| fixed64 | `get_packed_fixed64()` |
|
||||
| sfixed64 | `get_packed_sfixed64()` |
|
||||
| float | `get_packed_float()` |
|
||||
| double | `get_packed_double()` |
|
||||
|
||||
Packed repeated fields for string, bytes, and message types are not possible.
|
||||
|
||||
+626
@@ -0,0 +1,626 @@
|
||||
|
||||
# Protozero Tutorial
|
||||
|
||||
## Getting to know Protocol Buffers
|
||||
|
||||
Protozero is a very low level library. You really have to know some of the
|
||||
insides of Protocol Buffers to work with it!
|
||||
|
||||
So before reading any further in this document, read the following from the
|
||||
Protocol Buffer documentation:
|
||||
|
||||
* [Developer Guide - Overview](https://developers.google.com/protocol-buffers/docs/overview)
|
||||
* [Language Guide](https://developers.google.com/protocol-buffers/docs/proto)
|
||||
* [Encoding](https://developers.google.com/protocol-buffers/docs/encoding)
|
||||
|
||||
Make sure you understand the basic types of values supported by Protocol
|
||||
Buffers. Refer to this
|
||||
[handy table](https://developers.google.com/protocol-buffers/docs/proto#scalar)
|
||||
and [the cheat sheet](cheatsheet.md) if you are getting lost.
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You need a C++11-capable compiler for Protozero to work. Copy the files in the
|
||||
`include/protozero` directory somewhere where your build system can find them.
|
||||
Keep the `protozero` directory and include the files in the form
|
||||
|
||||
```cpp
|
||||
#include <protozero/FILENAME.hpp>
|
||||
```
|
||||
|
||||
|
||||
## Parsing protobuf-encoded messages
|
||||
|
||||
### Using `pbf_reader`
|
||||
|
||||
To use the `pbf_reader` class, add this include to your C++ program:
|
||||
|
||||
```cpp
|
||||
#include <protozero/pbf_reader.hpp>
|
||||
```
|
||||
|
||||
The `pbf_reader` class contains asserts that will detect some programming
|
||||
errors. We encourage you to compile with asserts enabled in your debug builds.
|
||||
|
||||
|
||||
### An introductory example
|
||||
|
||||
Lets say you have a protocol description in a `.proto` file like this:
|
||||
|
||||
```cpp
|
||||
message Example1 {
|
||||
required uint32 x = 1;
|
||||
optional string s = 2;
|
||||
repeated fixed64 r = 17;
|
||||
}
|
||||
```
|
||||
|
||||
To read messages created according to that description, you will have code that
|
||||
looks somewhat like this:
|
||||
|
||||
```cpp
|
||||
#include <protozero/pbf_reader.hpp>
|
||||
|
||||
// get data from somewhere into the input string
|
||||
std::string input = get_input_data();
|
||||
|
||||
// initialize pbf message with this data
|
||||
protozero::pbf_reader message{input};
|
||||
|
||||
// iterate over fields in the message
|
||||
while (message.next()) {
|
||||
|
||||
// switch depending on the field tag (the field name is not available)
|
||||
switch (message.tag()) {
|
||||
case 1:
|
||||
// get data for tag 1 (in this case an uint32)
|
||||
auto x = message.get_uint32();
|
||||
break;
|
||||
case 2:
|
||||
// get data for tag 2 (in this case a string)
|
||||
std::string s = message.get_string();
|
||||
break;
|
||||
case 17:
|
||||
// ignore data for tag 17
|
||||
message.skip();
|
||||
break;
|
||||
default:
|
||||
// ignore data for unknown tags to allow for future extensions
|
||||
message.skip();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
You always have to call `next()` and then either one of the accessor functions
|
||||
(like `get_uint32()` or `get_string()`) to get the field value or `skip()` to
|
||||
ignore this field. Then call `next()` again, and so forth. Never call `next()`
|
||||
twice in a row or any if the accessor or skip functions twice in a row.
|
||||
|
||||
Because the `pbf_reader` class doesn't know the `.proto` file it doesn't know
|
||||
which field names or tags there are and it doesn't known the types of the
|
||||
fields. You have to make sure to call the right `get_...()` function for each
|
||||
tag. Some `assert()s` are done to check you are calling the right functions,
|
||||
but not all errors can be detected.
|
||||
|
||||
Note that it doesn't matter whether a field is defined as `required`,
|
||||
`optional`, or `repeated`. You always have to be prepared to get zero, one, or
|
||||
more instances of a field and you always have to be prepared to get other
|
||||
fields, too, unless you want your program to break if somebody adds a new
|
||||
field.
|
||||
|
||||
|
||||
### If you only need a single field
|
||||
|
||||
If, out of a protocol buffer message, you only need the value of a single
|
||||
field, you can use the version of the `next()` function with a parameter:
|
||||
|
||||
```cpp
|
||||
// same .proto file and initialization as above
|
||||
|
||||
// get all fields with tag 17, skip all others
|
||||
while (message.next(17)) {
|
||||
auto r = message.get_fixed64();
|
||||
std::cout << r << "\n";
|
||||
}
|
||||
```
|
||||
|
||||
### Handling scalar fields
|
||||
|
||||
As you saw in the example, handling scalar field types is reasonably easy. You
|
||||
just check the `.proto` file for the type of a field and call the corresponding
|
||||
function called `get_` + _field type_.
|
||||
|
||||
For `string` and `bytes` types the internal handling is exactly the same, but
|
||||
both `get_string()` and `get_bytes()` are provided to make the code
|
||||
self-documenting. Both theses calls allocate and return a `std::string` which
|
||||
can add some overhead. You can call the `get_view()` function instead which
|
||||
returns a `data_view` containing a pointer into the data (access with `data()`)
|
||||
and the length of the data (access with `size()`).
|
||||
|
||||
|
||||
### Handling repeated packed fields
|
||||
|
||||
Fields that are marked as `[packed=true]` in the `.proto` file are handled
|
||||
somewhat differently. `get_packed_...()` functions returning an iterator range
|
||||
are used to access the data.
|
||||
|
||||
So, for example, if you have a protocol description in a `.proto` file like
|
||||
this:
|
||||
|
||||
```cpp
|
||||
message Example2 {
|
||||
repeated sint32 i = 1 [packed=true];
|
||||
}
|
||||
```
|
||||
|
||||
You can get to the data like this:
|
||||
|
||||
```cpp
|
||||
protozero::pbf_reader message{input.data(), input.size()};
|
||||
|
||||
// set current field
|
||||
message.next(1);
|
||||
|
||||
// get an iterator range
|
||||
auto pi = message.get_packed_sint32();
|
||||
|
||||
// iterate to get to all values
|
||||
for (auto it = pi.begin(); it != pi.end(); ++it) {
|
||||
std::cout << *it << '\n';
|
||||
}
|
||||
```
|
||||
|
||||
Or, with a range-based for-loop:
|
||||
|
||||
```cpp
|
||||
for (auto value : pi) {
|
||||
std::cout << v << '\n';
|
||||
}
|
||||
```
|
||||
|
||||
So you are getting a pair of normal forward iterators wrapped in an iterator
|
||||
range object. The iterators can be used with any STL algorithms etc.
|
||||
|
||||
Note that the previous only applies to repeated **packed** fields, normal
|
||||
repeated fields are handled in the usual way for scalar fields.
|
||||
|
||||
|
||||
### Handling embedded messages
|
||||
|
||||
Protocol Buffers can embed any message inside another message. To access an
|
||||
embedded message use the `get_message()` function. So for this description:
|
||||
|
||||
```cpp
|
||||
message Point {
|
||||
required double x = 1;
|
||||
required double y = 2;
|
||||
}
|
||||
|
||||
message Example3 {
|
||||
repeated Point point = 10;
|
||||
}
|
||||
```
|
||||
|
||||
you can parse with this code:
|
||||
|
||||
```cpp
|
||||
protozero::pbf_reader message{input};
|
||||
|
||||
while (message.next(10)) {
|
||||
protozero::pbf_reader point = message.get_message();
|
||||
double x, y;
|
||||
while (point.next()) {
|
||||
switch (point.tag()) {
|
||||
case 1:
|
||||
x = point.get_double();
|
||||
break;
|
||||
case 2:
|
||||
y = point.get_double();
|
||||
break;
|
||||
default:
|
||||
point.skip();
|
||||
}
|
||||
}
|
||||
std::cout << "x=" << x << " y=" << y << "\n";
|
||||
}
|
||||
```
|
||||
|
||||
### Handling enums
|
||||
|
||||
Enums are stored as varints and they can't be differentiated from them. Use
|
||||
the `get_enum()` function to get the value of the enum, you have to translate
|
||||
this into the symbolic name yourself. See the `enum` test case for an example.
|
||||
|
||||
|
||||
### Asserts and exceptions in the Protozero library
|
||||
|
||||
Protozero uses `assert()` liberally to help you find bugs in your own code when
|
||||
compiled in debug mode (ie with `NDEBUG` not set). If such an assert "fires",
|
||||
this is a very strong indication that there is a bug in your code somewhere.
|
||||
|
||||
(Protozero will disable those asserts and "convert" them into exception in its
|
||||
own test code. This is done to make sure the asserts actually work as intended.
|
||||
Your test code will not need this!)
|
||||
|
||||
Exceptions, on the other hand, are thrown by Protozero if some kind of data
|
||||
corruption was detected while it is trying to parse the data. This could also
|
||||
be an indicator for a bug in the user code, but because it can happen if the
|
||||
data was (intentionally or not intentionally) been messed with, it is reported
|
||||
to the user code using exceptions.
|
||||
|
||||
Most of the functions on the writer side can throw a `std::bad_alloc`
|
||||
exception if there is no space to grow a buffer. Other than that no exceptions
|
||||
can occur on the writer side.
|
||||
|
||||
All exceptions thrown by the reader side derive from `protozero::exception`.
|
||||
|
||||
Note that all exceptions can also happen if you are expecting a data field of
|
||||
a certain type in your code but the field actually has a different type. In
|
||||
that case the `pbf_reader` class might interpret the bytes in the buffer in
|
||||
the wrong way and anything can happen.
|
||||
|
||||
#### `end_of_buffer_exception`
|
||||
|
||||
This will be thrown whenever any of the functions "runs out of input data".
|
||||
It means you either have an incomplete message in your input or some other
|
||||
data corruption has taken place.
|
||||
|
||||
#### `unknown_pbf_wire_type_exception`
|
||||
|
||||
This will be thrown if an unsupported wire type is encountered. Either your
|
||||
input data is corrupted or it was written with an unsupported version of a
|
||||
Protocol Buffers implementation.
|
||||
|
||||
#### `varint_too_long_exception`
|
||||
|
||||
This exception indicates an illegal encoding of a varint. It means your input
|
||||
data is corrupted in some way.
|
||||
|
||||
#### `invalid_tag_exception`
|
||||
|
||||
This exception is thrown when a tag has an invalid value. Tags must be
|
||||
unsigned integers between 1 and 2^29-1. Tags between 19000 and 19999 are not
|
||||
allowed. See
|
||||
https://developers.google.com/protocol-buffers/docs/proto#assigning-tags
|
||||
|
||||
#### `invalid_length_exception`
|
||||
|
||||
This exception is thrown when a length field of a packed repeated field is
|
||||
invalid. For fixed size types the length must be a multiple of the size of
|
||||
the type.
|
||||
|
||||
### The `pbf_reader` class
|
||||
|
||||
The `pbf_reader` class behaves like a value type. Objects are reasonably small
|
||||
(two pointers and two `uint32_t`, so 24 bytes on a 64bit system) and they can
|
||||
be copied and moved around trivially.
|
||||
|
||||
`pbf_reader` objects can be constructed from a `std::string` or a `const char*`
|
||||
and a length field (either supplied as separate arguments or as a `std::pair`).
|
||||
In all cases objects of the `pbf_reader` class store a pointer into the input
|
||||
data that was given to the constructor. You have to make sure this pointer
|
||||
stays valid for the duration of the objects lifetime.
|
||||
|
||||
## Parsing protobuf-encoded messages using `pbf_message`
|
||||
|
||||
One problem in the code above are the "magic numbers" used as tags for the
|
||||
different fields that you got from the `.proto` file. Instead of spreading
|
||||
these magic numbers around your code you can define them once in an `enum
|
||||
class` and then use the `pbf_message` template class instead of the
|
||||
`pbf_reader` class.
|
||||
|
||||
Here is the first example again, this time using this new technique. So you
|
||||
have the following in a `.proto` file:
|
||||
|
||||
```cpp
|
||||
message Example1 {
|
||||
required uint32 x = 1;
|
||||
optional string s = 2;
|
||||
repeated fixed64 r = 17;
|
||||
}
|
||||
```
|
||||
|
||||
Add the following declaration in one of your header files:
|
||||
|
||||
```cpp
|
||||
enum class Example1 : protozero::pbf_tag_type {
|
||||
required_uint32_x = 1,
|
||||
optional_string_s = 2,
|
||||
repeated_fixed64_r = 17
|
||||
};
|
||||
```
|
||||
|
||||
The message name becomes the name of the `enum class` which is always built
|
||||
on top of the `protozero::pbf_tag_type` type. Each field in the message
|
||||
becomes one value of the enum. In this case the name is created from the
|
||||
type (including the modifiers like `required` or `optional`) and the name of
|
||||
the field. You can use any name you want, but this convention makes it easier
|
||||
later, to get everything right.
|
||||
|
||||
To read messages created according to that description, you will have code that
|
||||
looks somewhat like this, this time using `pbf_message` instead of
|
||||
`pbf_reader`:
|
||||
|
||||
```cpp
|
||||
#include <protozero/pbf_message.hpp>
|
||||
|
||||
// get data from somewhere into the input string
|
||||
std::string input = get_input_data();
|
||||
|
||||
// initialize pbf message with this data
|
||||
protozero::pbf_message<Example1> message{input};
|
||||
|
||||
// iterate over fields in the message
|
||||
while (message.next()) {
|
||||
|
||||
// switch depending on the field tag (the field name is not available)
|
||||
switch (message.tag()) {
|
||||
case Example1::required_uint32_x:
|
||||
auto x = message.get_uint32();
|
||||
break;
|
||||
case Example1::optional_string_s:
|
||||
std::string s = message.get_string();
|
||||
break;
|
||||
case Example1::repeated_fixed64_r:
|
||||
message.skip();
|
||||
break;
|
||||
default:
|
||||
// ignore data for unknown tags to allow for future extensions
|
||||
message.skip();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note the correspondance between the enum value (for instance
|
||||
`required_uint32_x`) and the name of the getter function (for instance
|
||||
`get_uint32()`). This makes it easier to get the correct types. Also the
|
||||
naming makes it easier to keep different message types apart if you have
|
||||
multiple (or embedded) messages.
|
||||
|
||||
See the `test/t/complex` test case for a complete example using this interface.
|
||||
|
||||
Using `pbf_message` in favour of `pbf_reader` is recommended for all code.
|
||||
Note that `pbf_message` derives from `pbf_reader`, so you can always fall
|
||||
back to the more generic interface if necessary.
|
||||
|
||||
One problem you might run into is the following: The enum class lists all
|
||||
possible values you know about and you'll have lots of `switch` statements
|
||||
checking those values. Some compilers will know that your `switch` covers
|
||||
all possible cases and warn you if you have a `default` case that looks
|
||||
unneccessary to the compiler. But you still want that `default` case to allow
|
||||
for future extension of those messages (and maybe also to detect corrupted
|
||||
data). You can switch of this warning with `-Wno-covered-switch-default`).
|
||||
|
||||
|
||||
## Writing protobuf-encoded messages
|
||||
|
||||
### Using `pbf_writer`
|
||||
|
||||
To use the `pbf_writer` class, add this include to your C++ program:
|
||||
|
||||
```cpp
|
||||
#include <protozero/pbf_writer.hpp>
|
||||
```
|
||||
|
||||
The `pbf_writer` class contains asserts that will detect some programming
|
||||
errors. We encourage you to compile with asserts enabled in your debug builds.
|
||||
|
||||
|
||||
### An introductory example
|
||||
|
||||
Lets say you have a protocol description in a `.proto` file like this:
|
||||
|
||||
```cpp
|
||||
message Example {
|
||||
required uint32 x = 1;
|
||||
optional string s = 2;
|
||||
repeated fixed64 r = 17;
|
||||
}
|
||||
```
|
||||
|
||||
To write messages created according to that description, you will have code
|
||||
that looks somewhat like this:
|
||||
|
||||
```cpp
|
||||
#include <protozero/pbf_writer.hpp>
|
||||
|
||||
std::string data;
|
||||
protozero::pbf_writer pbf_example{data};
|
||||
|
||||
pbf_example.add_uint32(1, 27); // uint32_t x
|
||||
pbf_example.add_fixed64(17, 1); // fixed64 r
|
||||
pbf_example.add_fixed64(17, 2);
|
||||
pbf_example.add_fixed64(17, 3);
|
||||
pbf_example.add_string(2, "foobar"); // string s
|
||||
```
|
||||
|
||||
First you need a string which will be used as buffer to assemble the
|
||||
protobuf-formatted message. The `pbf_writer` object contains a reference to
|
||||
this string buffer and through it you add data to that buffer piece by piece.
|
||||
The buffer doesn't have to be empty, the `pbf_writer` will simply append its
|
||||
data to whatever is there already.
|
||||
|
||||
|
||||
### Handling scalar fields
|
||||
|
||||
As you could see in the introductory example handling any kind of scalar field
|
||||
is easy. The type of field doesn't matter and it doesn't matter whether it is
|
||||
optional, required or repeated. You always call one of the `add_TYPE()` method
|
||||
on the pbf writer object.
|
||||
|
||||
The first parameter of these methods is always the *tag* of the field (the
|
||||
field number) from the `.proto` file. The second parameter is the value you
|
||||
want to set. For the `bytes` and `string` types several versions of the add
|
||||
method are available taking a `const std::string&` or a `const char*` and a
|
||||
length.
|
||||
|
||||
For `enum` types you have to use the numeric value as the symbolic names from
|
||||
the `.proto` file are not available.
|
||||
|
||||
|
||||
### Handling repeated packed fields
|
||||
|
||||
Repeated packed fields can easily be set from a pair of iterators:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pw{data};
|
||||
|
||||
std::vector<int> v = { 1, 4, 9, 16, 25, 36 };
|
||||
pw.add_packed_int32(1, std::begin(v), std::end(v));
|
||||
```
|
||||
|
||||
If you don't have an iterator you can use the alternative form:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pw{data};
|
||||
{
|
||||
protozero::packed_field_int32 field{pw, 1};
|
||||
field.add_element(1);
|
||||
field.add_element(10);
|
||||
field.add_element(100);
|
||||
}
|
||||
```
|
||||
|
||||
Of course you can add as many elements as you want. If you add no elements
|
||||
at all, this code will still work, Protozero detects this special case and
|
||||
pretends you never even initialized this field.
|
||||
|
||||
The nested scope is important in this case, because the destructor of the
|
||||
`field` object will make sure the length stored inside the field is set to
|
||||
the right value. You must close that scope before adding other fields to the
|
||||
`pw` pbf writer.
|
||||
|
||||
If you know how many elements you will add to the field and your field contains
|
||||
fixed length elements, you can tell Protozero and it can optimize this case:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pw{data};
|
||||
{
|
||||
protozero::packed_field_fixed32 field{pw, 1, 2}; // exactly two elements
|
||||
field.add_element(42);
|
||||
field.add_element(13);
|
||||
}
|
||||
```
|
||||
|
||||
In this case you have to supply exactly as many elements as you promised,
|
||||
otherwise you will get a broken protobuf message.
|
||||
|
||||
This works for `packed_field_fixed32`, `packed_field_sfixed32`,
|
||||
`packed_field_fixed64`, `packed_field_sfixed64`, `packed_field_float`, and
|
||||
`packed_field_double`.
|
||||
|
||||
You can abandon writing of the packed field if this becomes necessary by
|
||||
calling `rollback()`:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pw{data};
|
||||
{
|
||||
protozero::packed_field_int32 field{pw, 1};
|
||||
field.add_element(42);
|
||||
// some error occurs, you don't want to have this field at all
|
||||
field.rollback();
|
||||
}
|
||||
```
|
||||
|
||||
The result is the same as if the lines inside the nested brackets had never
|
||||
been called. Do not try to call `add_element()` after a rollback.
|
||||
|
||||
|
||||
### Handling sub-messages
|
||||
|
||||
Nested sub-messages can be handled by first creating the submessage and then
|
||||
adding to the parent message:
|
||||
|
||||
```cpp
|
||||
std::string buffer_sub;
|
||||
protozero::pbf_writer pbf_sub{buffer_sub};
|
||||
|
||||
// add fields to sub-message
|
||||
pbf_sub.add_...(...);
|
||||
// ...
|
||||
|
||||
// sub-message is finished here
|
||||
|
||||
std::string buffer_parent;
|
||||
protozero::pbf_writer pbf_parent{buffer_parent};
|
||||
pbf_parent.add_message(1, buffer_sub);
|
||||
```
|
||||
|
||||
This is easy to do but it has the drawback of needing a separate `std::string`
|
||||
buffer. If this concerns you (and why would you use protozero and not the
|
||||
Google protobuf library if it doesn't?) there is another way:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pbf_parent{data};
|
||||
|
||||
// optionally add fields to parent here
|
||||
pbf_parent.add_...(...);
|
||||
|
||||
// open a new scope
|
||||
{
|
||||
// create new pbf_writer with parent and the tag (field number)
|
||||
// as parameters
|
||||
protozero::pbf_writer pbf_sub{pbf_parent, 1};
|
||||
|
||||
// add fields to sub here...
|
||||
pbf_sub.add_...(...);
|
||||
|
||||
} // closing the scope will close the sub-message
|
||||
|
||||
// optionally add more fields to parent here
|
||||
pbf_parent.add_...(...);
|
||||
```
|
||||
|
||||
This can be nested arbitrarily deep.
|
||||
|
||||
Internally the sub-message writer re-uses the buffer from the parent. It
|
||||
reserves enough space in the buffer to later write the length of the submessage
|
||||
into it. It then adds the contents of the submessage to the buffer. When the
|
||||
`pbf_sub` writer is destructed the length of the submessage is calculated and
|
||||
written in the reserved space. If less space was needed for the length field
|
||||
than was available, the rest of the buffer is moved over a few bytes.
|
||||
|
||||
You can abandon writing of submessage if this becomes necessary by
|
||||
calling `rollback()`:
|
||||
|
||||
```cpp
|
||||
std::string data;
|
||||
protozero::pbf_writer pbf_parent{data};
|
||||
|
||||
// open a new scope
|
||||
{
|
||||
// create new pbf_writer with parent and the tag (field number)
|
||||
// as parameters
|
||||
protozero::pbf_writer pbf_sub{pbf_parent, 1};
|
||||
|
||||
// add fields to sub here...
|
||||
pbf_sub.add_...(...);
|
||||
|
||||
// some problem occurs and you want to abandon the submessage:
|
||||
pbf_sub.rollback();
|
||||
}
|
||||
|
||||
// optionally add more fields to parent here
|
||||
pbf_parent.add_...(...);
|
||||
```
|
||||
|
||||
The result is the same as if the lines inside the nested brackets had never
|
||||
been called. Do not try to call any of the `add_*` functions on the submessage
|
||||
after a rollback.
|
||||
|
||||
## Writing protobuf-encoded messages using `pbf_builder`
|
||||
|
||||
Just like the `pbf_message` template class wraps the `pbf_reader` class, there
|
||||
is a `pbf_builder` template class wrapping the `pbf_writer` class. It is
|
||||
instantiated using the same `enum class` described above and used exactly
|
||||
like the `pbf_writer` class but using the values of the enum instead of bare
|
||||
integers.
|
||||
|
||||
See the `test/t/complex` test case for a complete example using this interface.
|
||||
|
||||
Reference in new issue
Block a user