Ask HN: What occurred to flatbuffers? Are they getting used?

If your main concern is “faster than JSON” then you’re better off using Protocol Buffers simply because they’re way more popular and better supported. FlatBuffers are cool because they let you decode on demand. Say you have an array of 10,000 complex objects. With JSON or Protocol Buffers you’re going to need to decode and load into memory all 10,000 before you’re able to access the one you want. But with FlatBuffers you can decode item X without touching 99% of the rest of the data. Quicker and much more memory efficient.

But it’s not simple to implement. You have to write a schema then turn that schema into source files in your target language. There’s an impressive array of target languages but it’s a custom executable and that adds complexity to any build. Then the generated API is difficult to use (in JS at least) because of course an array isn’t a JavaScript array, it’s an object with decoder helpers.

It’s also quite easy to trip yourself up in terms of performance by decoding the same data over and over again rather than re-using the first decode like you would with JSON or PB. So you have to think about which decoded items to store in memory, where, for how long, etc… I kind of think of it as the data equivalent of a programming language with manual memory management. Definitely has a place. But the majority of projects are going to be fine with automatic memory management.

Edit: I was able to find these at https://github.com/hnakamur/protobuf-deb/blob/master/docs/pe… however these numbers do not appear conclusive. Protobuf decode throughput for many schemas examined is way slower than JSON, however protobufs will most likely even be a bit smaller. One must evaluate decode throughput for a similar paperwork serialized each methods slightly than simply a desk.

Worth noting that all these things are true for protobuf as well.

(I work for Google but don’t speak for it.)

Is the capn’n proto use case similar to something like ZeroMQ or NNG? I’m still not fully sure.

Games, data visualization, … numerically heavy applications mainly.

On a side-note; JSON has been somewhat of a curse. The developer ergonomics of it are so good, that web devs completely disregard how they should layout their data. You know, sending a table as a bunch of nested arrays, that sort of thing. Yuck.

In web apps, data is essentially unusable until it has been unmarshalled. Fine for small things, horrible for data-heavy apps, which really so many apps are now.

Sometimes I wonder if it will change. I’m optimistic that the popularity of mem-efficient formats like this will establish a new base paradigm of data transfer, and be adopted broadly on the web.

grpc and Thrift are mostly backend service interconnects in lieu of RESTful.

Capnproto is also awesome.

If yes, is it a less hare-brained idea than using the ctypes Python module to mmap a file as a C struct? That’s what I’m currently doing to get 10x speedup relative to SQLite for an application bottlenecked on disk bandwidth, but it’s unergonomic to say the least.

Flatbuffers look like a way to get the same performance with better ergonomics, but maybe there’s a catch. (E.g. I thought the same thing about Apache Arrow before, but then I realized it’s basically read-only. I don’t expect to need to resize my tables often, but I do need to be able to twiddle individual values inside the file.)

Tell me more! Is your data larger than memory? You need persistence?

You might take a look at Aerospike, even on a single node if you need low latency persistence.

2. No, writers can directly embed structs and primitive data types into binary buffers at runtime through an API generated from an IDL file. Readers use direct memory access to pull values out of the buffers. If you set it up right, this can result in a massive perf boost by eliminating the encoding and decoding steps.

3. Facebook uses them in their mobile app. Another commenter mentioned use of the in the Arrow format. The flatbuffers website isn’t the best, but clearly documents the flatbuffers IDL.

They aren’t hard to use, but they aren’t the easiest thing either.

Once you get the gist of the API through the tutorial, the other important topics that come up immediately are version control; git repo design; headers and framing.

In production, they’ve been bulletproof. As long as you account for the compile time issues (schema versioning, repo, headers and framing, etc.).

I did this one for reading json on the fast path, the sending system laid out the arrays in a periodic pattern in memory that enabled parseless retrieval of individual values.

https://github.com/simdjson/simdjson

Also I assume you’d have to have some sort of binary portion bundled with it to hold the field offsets, no?

Also browser has JSON parsing built in. Less dependencies. Easier tooling overall.

In my experience people overuse protobuf. But I also worked at Google, where it’s the hammer in constant search of any nail it can find.

At the very least, endpoints should provide the option to provide a JSON form through content representation negotiation.

We moved to flatbuffers and back to JSON because in the end of the day, for our data, data compression with JSON+gzip was similarly-sized than the original one (which had some other fields that we were not using) and 10-20 times faster to decode.

That said, the use case for flatbuffers and capnproto isn’t really about data size, it’s about avoiding unnecessary copies in the processing pipeline. “Zero copy” really does pay dividends where performance is a concern if you write your code the right way.

Most people working on typical “web stack” type applications won’t hit these concerns. But there are classes of applications where what flatbuffers (and other zerocopy payload formats) offer is important.

The difference in computation time between operating on something sitting in L1 cache vs not-in-cache is orders of magnitude. And memory bandwidth is a bottleneck in some applications and on some machines (particularly embedded.)