Why I quit using Flexible Arrays (C++)

At ex-job my experience with Flexible Arrays (FAs further) have been positive. I initially got there thanks to institute, and then started working fulltime, and part of my work was to develop a server and client for delivering updates, and afterwards for other activity. Until recently I didn’t even know FAs are invalid as far as the standard concerned.

Also, client have been written in completely different language than server, so my experience with FAs was limited to the server.

Deserialization was as simple as casting a buffer-pointer to a struct-pointer, e.g.:

struct Foo {
uint8_t someField;
uint8_t data[]; // my FA ♥
}__attribute__((packed));
// [snip]
Foo* foo = (Foo*) received_buf;

Serialization could have been harder, but barely I noticed it — I’ve been in furious-learning mode, everything was hard and new, and I was afraid of missing a deadline. So sometimes I just couldn’t see if something can be improved, othertimes I simply couldn’t afford time for exploration, so had to do best I can and move on.

Now I’m implementing a pidgin plugin for IMPP (trillian) protocol, and I dare you, FAs stands for Fucking Annoying! Too bad I understood it very late, and now I have to rewrite lots of code, but oh, well. The more you know…

First of all, let me explain what made the problem so dramatic as opposed to my previous experience. Protocol of my ex-job software was pretty simple: some struct with fields, and FA in the end. The struct Foo you see above is pretty much it. IMPP on the other hand consist of TLVs — depending on different situations both header, and the body, and the units of data can be of at least 2 different sizes. For example a TLV unit — the structure which carries the data — depending on certain bit of its type can either have the next field (being the size of the data) as uint16_t or uint32_t.

This means that “simplicity of deserialization” is no more. Instead of casting whole thing I had to cast a small part, decrease size of data left (alternatively shift a pointer), check certain field, then check if I have enough data to proceed, otherwise return a specific error, but if everything okay, make again some calculation with sizes left, repeat whole thing again until next field, and again and again. Does it sound confusing? I hope it does, because in actual code it is! Too many places to make a mistake.

I’m sure it’s possible to automate somehow, but to figure it out I’d have to write down all use-cases. And to avoid building up unnecessary abstractions the best way would be to write half the code, then check the boilerplate. Which is a problem — I once caught myself staring for more than an hour in the code trying to understand everything I need just to move to the next level. Needless to say I spent many hours debugging various parts of such code that upon running under libasan been throwing buffer overflow. Last time it recidivated, I gave up, and decided to go look what API serialization libs are using out there; ATM I’m in process of hooking up Cereal.


  1. -st problem is a simple struct initialization. How would you initialize our hero Foo from above? Well, you can do Foo foo = {1, {2,3,4,5}}, fine. Now consider the following problem: your array is actually some other particular struct, so you want to initialize this struct, and then cast it to the array. Something like Foo foo = {1, *(uint8_t(*)[5])myOtherStruct}. Now, I have explored every possible way, I have even tried rarely known beast “pointer to array” which you can see in the prev. sentence. It never works, array always decays to a pointer. Your only bet is taking pointers, then using memcpy/std::copy. And unless you’re a robot, adding another pointer arithmetic in the code that already crowded by it for aforementioned reasons doesn’t make it look any better.

Now that I think of it though, I could probably make a constructor accepting a pointer and a size. It’d have to blindly write beyond this->data pointer though, which I’m pretty sure is undefined behavior; but then again, the concept of FAs doesn’t even exist in standard! I dunno if it’s a good idea.
I think issue per se arises from FAs not being valid in standard, so interaction with the rest of C++ haven’t been explored by whoever invented the concept.

  1. -nd problem kind of grows from the first. When you have lots of structs with FAs in the end, you’d have to slowly initialize them from the bottom, one-by-one, and then copy, initialize, copy, rinse and repeat. It is a lot of boilerplate code, and a lot of unnecessary copying. And remember: it is real lot of arithmetic, because you can’t just take sizeof(mystruct) and be done with, instead you have to check certain fields to make sure whether particular field have one size or another.

  2. -rd problem: you can’t have a proper default constructor for a struct with FA. Well, technically you can, but it will obviously initialize the FA to not have any data. There is a subtle consequence of this problem — you can’t use Cereal serialization library for such a struct, because the lib doesn’t know how much size needs to be preallocated.


As an alternative I gonna use a pointer. It might not follow the logic of protocol as close because the place where it would point upon deserialization will probably be in a completely different place than the rest of the struct, but at least I can avoid a bunch of horrible hacks and a maintainance nightmare if this plugin gets finally released. For the same reason I’m going to stick to Cereal — it unfortunately doesn’t allow zero-copy deserialization, but, well… consider me an asshole.

Advertisements

opcontrol: command not found

Wanted to share how to use oprofile nowadays. Every tutorial on the Internet I found uses opcontrol, but, well…

$ sudo opcontrol --reset
sudo: opcontrol: command not found

Searching for package with the filename tells that opcontrol exists neither in Archlinux nor in Ubuntu. Perhaps the command just was renamed? Let’s search for an option of opcontrol in manuals:

$ man -k start-daemon
start-daemon: nothing appropriate.

I don’t know what happened, but the way it works now is through operf. E.g. sudo operf mycommand — or rather, since we don’t want to run the profiled app with elevated rights, just run the app, and connect operf to its pid, like sudo operf --pid=8888.

For some reason operf stay running even when the app exited, just press ^C to stop operf, it would create a profiling data, then you can use opreport as usual, like in tutorials.

Missing icons of kmix and klipper in bar of i3

It’s a quick note for anyone coming from search. If you’re not using a full-featured DE, rather something as simple as i3, or Awesome, or XMonad, or Fluxbox, or anything else not including bells’n’whistles; and you’ve missing or wrong icon of klipper/kmix in bar(and, probably, some other Qt apps), then you’ve to set XDG_CURRENT_DESKTOP="xfce" for those apps.

E.g. I’m using i3, and have the variable set to “kde” (for dolphin, and likes), so I’m just autostarting klipper and kmix like XDG_CURRENT_DESKTOP="xfce" kmix.

The reasons is probably that those DEs provides some API which applets expect to use. Using "xfce" variable in the case of absent DE is fine though.

Archlinux: Compose key

It was hard to find, so I want to share how I got Compose key configured’n’working on Archlinux. Note: if you’re reading it in the future, I hope some of the problems (e.g. with ibus) are outdated. Leave a comment if that’s true.

At the moment of writing this, the Internet has pretty scarce info about text input internals for GNU/Linux. Most of the info I got by arguing with ibus maintainer here, here, and here. What you may want to know with relation to Compose key, is that the input (including XCompose) is done with XKB protocol. It sounds pretty X11, but is used on Wayland either.

An important note: the Compose Input Methods in either ibus or fcitx has lame imitations, they doesn’t even read ~/.XCompose. That’s not what you want, the real XCompose have to be configured with XKB, and is not an IM.

To make XCompose working with ibus you’d have to set globally (in the /etc/profile) GTK_IM_MODULE and QT_IM_MODULE variables to ibus, then “XMODIFIERS=@im=ibus” (this one may be not mandatory, Idk), and either execute setxkbmap -option compose:rwin (substitute rwin with the favorite key if needed), or set it up in the dungeon of XKB configuration (I couldn’t find quickly for where and how, and configured via X-files, see below, but it won’t work obviously with Wayland).

The ibus way worked with gnome-on-wayland as well as on X11, but you didn’t really want ibus as it’s pretty raw. To me the most annoyng are non-working LED on layout switch, and missing support for multicharacters, i.e. you can’t configure getting the on pressing Compose+t. CJK people doesn’t like ibus either. Fortunately, even though ibus maintainer doesn’t seem to be interested in fixing anything personally, at least they maintains it, so you can write a patch, and hope the PR to be merged.


I’m using instead the fcitx implementation of Compose (and I didn’t test it with Wayland), but it is tricky, so follow the hands!

  1. Install fcitx.
  2. Add into the /etc/profile (or wherever to be exported globally, overwrite if you had the variables set to anything else):
    export XMODIFIERS=@im=none
    export GTK_IM_MODULE="fcitx"
    export QT_IM_MODULE="fcitx"
    
  3. Important: make sure that fcitx daemon is not running! Idk why, but for it starts, everything breaks. The daemon shalt not pass! From what I understand, Qt and GTK, and may be EFL, are having some internally supplied fcitx modules (about GTK it’s for sure) that probably get called. Still, why fcitx daemon kills the fun, is a puzzle.

  4. Configure the Compose key. For starters, just to test if it works, you could run setxkbmap -option compose:rwin (to set the right win-logo key as Compose), then press some combinations. Ideally you’d have to have the ~/.XCompose file, and to press its combinations, to make sure it’s being read. If everything is fine, you could configure it via X11 files (and may be XKB ones, but Idk how). Here the X11 example I’m using:

    $ cat /etc/X11/xorg.conf.d/00-keyboard.conf
    # Read and parsed by systemd-localed. It's probably wise not to edit this file
    # manually too freely.
    Section "InputClass"
            Identifier "system-keyboard"
            MatchIsKeyboard "on"
            Option "XkbLayout" "us,ru"
            Option "XkbModel" "pc105"
            Option "XkbOptions"    "grp:rctrl_rshift_toggle,grp_led:scroll,lv3:ralt_switch,compose:rwin"
    EndSection
    

Now, if you look at 00-keyboard.conf example thinking of ibus, make sure to remove the language switch option, so that it wouldn’t conflict with ibus config.

Bonus:

You may want a ready to go XCompose file (the dotxcompose in the repository) as a ~/.XCompose. It’s made to be backward-compatible with system XCompose file, so you may want to overwrite some combinations, in particular these to press a ^ just single time.

Pass +RTS options to GHC’n’GHCi

For working with projects, I’m using apps from Stack. Besides, I usually also have GHCi running for quick checks. And for I didn’t really want duplicating shared libraries and file cache in RAM (i.e. the ones the system GHC is using, plus the ones that Stack does), I’m trying to stick to Stack for everything.

So to force Stack to pass RTS options to ghc and ghci you’d have to use --ghc-options and --ghci-options respectively.

I.e. I saved aliases to my ~/.zshrc (if you’re using bash, it’d be a ~/.bashrc).

alias ghci512="stack ghci --ghci-options '+RTS -M512m -RTS'"
alias ghci="stack ghci --ghci-options ''"
alias ghc="stack ghc --ghc-options ''"

Aliases won’t always work (like if you’re using a script not knowing about them), nevertheless are useful most of the time.

UPD: with aliases were problems that I fed up with, so I made up scripts instead:

$ cat /usr/bin/ghciMath512 
#!/bin/sh
stack ghci --ghci-options "+RTS -M512m -RTS -ghci-script /home/constantine/.ghciMath `echo $@`"
$ cat /usr/bin/ghci       
#!/bin/sh
stack ghci --ghci-options "`echo $@`"

Note the echo — it’s needed to collapse multiple arguments into a single with space-separated words.

Compile Yi with Pango

By default Yi doesn’t compile Pango frontend, only vty (and “batch”, whatever it is). To compile Pango with stack (btw, use stack, as it would remove some problems; also stack install rebuilds faster, then cabal install does), execute in the yi directory:

$ stack init
$ stack install --flag yi:pango

Then, to run the editor with Pango frontend:

$ stack exec yi -- -f pango

P.S.: There’s annoying thing, that in Vim’s normal mode the cursor — the Pango frontend problem — is not a big black box. It’s the problem, specific to GTK2-3, it just didn’t have an API to make such a cursor. Technically, there’s a workaround to enable “insert” mode, i.e. like if one would press “Insert” key. The first time when I met the editor, I tried to fix this bug as part of a student paper, but didn’t manage to go so far as to finally get it fixed. I did much tho, and wrote on IRC about it, but peoples there wasn’t interested. Granted, I ought to at least report a bug. Well, let’s see what can I do today.