序列化库 FlatBuffers

简介

https://github.com/google/flatbuffers
https://google.github.io/flatbuffers/index.html#flatbuffers_overview

FlatBuffers是Google发布的一个高效的跨平台的序列化库,支持多种编程语言。

Google最初是将其用于游戏开发和其他对性能要求比较高的场合。

FlatBuffers与普通序列化方案的区别:

一般序列化通常采用以下步骤:将数据结构序列化为JSON、XML、二进制或其他格式的数据流,然后接收方或使用方将数据流再解析为数据结构。

以C++为例,FlatBuffers的步骤是:

  1. 编写Schema文件(schema file),即使用接口定义语言(IDL)定义数据结构
  2. 使用flatc编译器编译Schema文件,生成C++文件
  3. 将生成的C++文件添加到项目中一起编译
  4. 使用生成的C++文件提供的接口序列化/反序列化数据

C++例子

编写Schema文件

文件monster.fbs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// Example IDL file for our monster's schema.

namespace MyGame.Sample;

enum Color:byte { Red = 0, Green, Blue = 2 }

union Equipment { Weapon } // Optionally add more tables.

struct Vec3 {
x:float;
y:float;
z:float;
}

table Monster {
pos:Vec3; // Struct.
mana:short = 150;
hp:short = 100;
name:string;
friendly:bool = false (deprecated);
inventory:[ubyte]; // Vector of scalars.
color:Color = Blue; // Enum.
weapons:[Weapon]; // Vector of tables.
equipped:Equipment; // Union.
path:[Vec3]; // Vector of structs.
}

table Weapon {
name:string;
damage:short;
}

root_type Monster;

编写Schema文件的教程
https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html

编译Schema文件

1
2
cd flatbuffers/samples
./../flatc --cpp monster.fbs

使用

序列化,写FlatBuffers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#include "monster_generated.h" // This was generated by `flatc`.
using namespace MyGame::Sample; // Specified in the schema.

// Create a `FlatBufferBuilder`, which will be used to create our
// monsters' FlatBuffers.
flatbuffers::FlatBufferBuilder builder(1024);

auto weapon_one_name = builder.CreateString("Sword");
short weapon_one_damage = 3;
auto weapon_two_name = builder.CreateString("Axe");
short weapon_two_damage = 5;

// Use the `CreateWeapon` shortcut to create Weapons with all the fields set.
auto sword = CreateWeapon(builder, weapon_one_name, weapon_one_damage);
auto axe = CreateWeapon(builder, weapon_two_name, weapon_two_damage);

// Serialize a name for our monster, called "Orc".
auto name = builder.CreateString("Orc");

// Create a `vector` representing the inventory of the Orc. Each number
// could correspond to an item that can be claimed after he is slain.
unsigned char treasure[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
auto inventory = builder.CreateVector(treasure, 10);

// Place the weapons into a `std::vector`, then convert that into a FlatBuffer `vector`.
std::vector<flatbuffers::Offset<Weapon>> weapons_vector;
weapons_vector.push_back(sword);
weapons_vector.push_back(axe);
auto weapons = builder.CreateVector(weapons_vector);

Vec3 points[] = { Vec3(1.0f, 2.0f, 3.0f), Vec3(4.0f, 5.0f, 6.0f) };
auto path = builder.CreateVectorOfStructs(points, 2);

// Create the position struct
auto position = Vec3(1.0f, 2.0f, 3.0f);

// Set his hit points to 300 and his mana to 150.
int hp = 300;
int mana = 150;

// Finally, create the monster using the `CreateMonster` helper function
// to set all fields.
auto orc = CreateMonster(builder, &position, mana, hp, name, inventory,
Color_Red, weapons, Equipment_Weapon, axe.Union(),
path);

// You can use this code instead of `CreateMonster()`, to create our orc
// manually.
MonsterBuilder monster_builder(builder);
monster_builder.add_pos(&position);
monster_builder.add_hp(hp);
monster_builder.add_name(name);
monster_builder.add_inventory(inventory);
monster_builder.add_color(Color_Red);
monster_builder.add_weapons(weapons);
monster_builder.add_equipped_type(Equipment_Weapon);
monster_builder.add_equipped(axe.Union());
auto orc = monster_builder.Finish();

monster_builder.add_equipped_type(Equipment_Weapon); // Union type
monster_builder.add_equipped(axe); // Union data

// Call `Finish()` to instruct the builder that this monster is complete.
// Note: Regardless of how you created the `orc`, you still need to call
// `Finish()` on the `FlatBufferBuilder`.
builder.Finish(orc); // You could also call `FinishMonsterBuffer(builder,
// orc);`.

// This must be called after `Finish()`.
uint8_t *buf = builder.GetBufferPointer();
int size = builder.GetSize(); // Returns the size of the buffer that
// `GetBufferPointer()` points to.

反序列化,读FlatBuffers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include "monster_generated.h" // This was generated by `flatc`.
using namespace MyGame::Sample; // Specified in the schema.

uint8_t *buffer_pointer = /* the data you just read */;

// Get a pointer to the root object inside the buffer.
auto monster = GetMonster(buffer_pointer);

// `monster` is of type `Monster *`.
// Note: root object pointers are NOT the same as `buffer_pointer`.
// `GetMonster` is a convenience function that calls `GetRoot<Monster>`,
// the latter is also available for non-root types.

auto hp = monster->hp();
auto mana = monster->mana();
auto name = monster->name()->c_str();

auto pos = monster->pos();
auto x = pos->x();
auto y = pos->y();
auto z = pos->z();

auto inv = monster->inventory(); // A pointer to a `flatbuffers::Vector<>`.
auto inv_len = inv->size();
auto third_item = inv->Get(2);

auto weapons = monster->weapons(); // A pointer to a `flatbuffers::Vector<>`.
auto weapon_len = weapons->size();
auto second_weapon_name = weapons->Get(1)->name()->str();
auto second_weapon_damage = weapons->Get(1)->damage();

auto union_type = monster.equipped_type();
if (union_type == Equipment_Weapon) {
auto weapon = static_cast<const Weapon*>(monster->equipped()); // Requires `static_cast`
// to type `const Weapon*`.
auto weapon_name = weapon->name()->str(); // "Axe"
auto weapon_damage = weapon->damage(); // 5
}

扩展阅读

https://zhuanlan.zhihu.com/p/77644854
http://www.voidcn.com/article/p-xtbacuzn-bqm.html
https://blog.csdn.net/qq_35559358/article/details/79443327