Federico's Blog

GNOME themes, an incomplete status report, and how you can help

2021-09-01T13:57:47-05:00

"Themes in GNOME" is a complicated topic in technical and social Err codemadness.org 70 i 8 terms. Technically there are a lot of incomplete moving parts; Err codemadness.org 70 i 9 socially there is a lot of missing documentation to be written, a lot Err codemadness.org 70 i 10 of miscommunication and mismatched expectations.

Err codemadness.org 70 i 11

The following is a brief and incomplete, but hopefully encouraging, Err codemadness.org 70 i 12 summary of the status of themes in GNOME. I want to give you an Err codemadness.org 70 i 13 overall picture of the status of things, and more importantly, an idea Err codemadness.org 70 i 14 of how you can help. This is not a problem that can be solved by a Err codemadness.org 70 i 15 small team of platform developers.

Err codemadness.org 70 i 16

I wish to thank Alexander Mikhaylenko for providing most of the Err codemadness.org 70 i 17 knowledge in this post.

Err codemadness.org 70 i 18

Frame of reference

Err codemadness.org 70 i 19

First, I urge you to read Cassidy James Blaede's comprehensive "The Err codemadness.org 70 i 20 Need for a FreeDesktop Dark Style Err codemadness.org 70 i 21 Preference". Err codemadness.org 70 i 22 That gives an excellent, well-researched introduction to the "dark Err codemadness.org 70 i 23 style" problem, the status quo on other platforms, and exploratory Err codemadness.org 70 i 24 plans for GNOME and Elementary from 2019.

Err codemadness.org 70 i 25

Go ahead, read it. It's very good.

Err codemadness.org 70 i 26

There is also a GUADEC talk about Cassidy's Err codemadness.org 70 i 27 research Err codemadness.org 70 i 28 if you prefer to watch a video.

Err codemadness.org 70 i 29

Two key take-aways from this: First, about this being a Err codemadness.org 70 i 30 preference, not a system-enforced setting:

Err codemadness.org 70 i 31

Err codemadness.org 70 i 32
I’m explicitly using the language “Dark Style Preference” for a Err codemadness.org 70 i 33 reason! As you’ll read further on, it’s important that this is Err codemadness.org 70 i 34 treated as a user “preference,” not an explicit “mode” or Err codemadness.org 70 i 35 strictly-enforced “setting.” It’s also not a “theme” in the sense Err codemadness.org 70 i 36 that it just swaps out some assets, but is a way for the OS to Err codemadness.org 70 i 37 support a user expressing a preference, and apps to respond to that Err codemadness.org 70 i 38 preference.
Err codemadness.org 70 i 39

Err codemadness.org 70 i 40

Second, about the accessibility implications:

Err codemadness.org 70 i 41

Err codemadness.org 70 i 42
Clearly there’s an accessibility and usability angle here. And as Err codemadness.org 70 i 43 with other accessibility efforts, it’s important to not relegate a Err codemadness.org 70 i 44 dark style preference to a buried “Universal Access” or Err codemadness.org 70 i 45 “Accessibility” feature, as that makes it less discoverable, less Err codemadness.org 70 i 46 tested, and less likely to be used by folks who could greatly Err codemadness.org 70 i 47 benefit, but don’t consider themselves “disabled.”
Err codemadness.org 70 i 48

Err codemadness.org 70 i 49

Libadwaita and the rest of the ecosystem

Err codemadness.org 70 i 50

Read the libadwaita Err codemadness.org 70 i 51 roadmap; Err codemadness.org 70 i 52 it is very short, but links to very interesting issues on gitlab.

Err codemadness.org 70 i 53

For example, this merge request is for an API to query the dark style Err codemadness.org 70 i 54 and high-contrast Err codemadness.org 70 i 55 preferences. Err codemadness.org 70 i 56 It has links to pending work in other parts of the platform: libhandy, Err codemadness.org 70 i 57 gsettings schemas, portals so that containerized applications can Err codemadness.org 70 i 58 query those preferences.

Err codemadness.org 70 i 59

As far as I understand it, applications that just use GTK3 or libhandy Err codemadness.org 70 i 60 can opt in to supporting the dark style preference — it is opt-in Err codemadness.org 70 i 61 because doing that unconditionally in GTK/libhandy right now would Err codemadness.org 70 i 62 break existing applications.. If your app uses libadwaita, it is Err codemadness.org 70 i 63 assumed that you have opted into supporting that preference, since Err codemadness.org 70 i 64 libadwaita's widgets already make that assumption, and it is not Err codemadness.org 70 i 65 API-stable yet — so it can make that assumption from the beginning.

Err codemadness.org 70 i 66

There is discussion of the accessibility implications in the design Err codemadness.org 70 i 67 mockups.

Err codemadness.org 70 i 68

CSS parity across implementations

Err codemadness.org 70 i 69

In GNOME we have three implementations of CSS:

Err codemadness.org 70 i 70

Err codemadness.org 70 i 72
librsvg uses servo's engine for CSS selector matching, and micro-parsers for CSS values based on servo's cssparser.
Err codemadness.org 70 i 73
Err codemadness.org 70 i 75
GTK has its own CSS parser and processor.
Err codemadness.org 70 i 76
Err codemadness.org 70 i 78
Gnome-shell uses an embedded version of libcroco for parsing, but it Err codemadness.org 70 i 79 does most of the selector matching and cascading with gnome-shell's Err codemadness.org 70 i 80 own Shell Toolkit code.
Err codemadness.org 70 i 81

Err codemadness.org 70 i 83

None of those implementations supports @media queries nor custom Err codemadness.org 70 i 84 properties with var(). That is, unlike in the web platform, GNOME Err codemadness.org 70 i 85 applications cannot have this in their CSS:

Err codemadness.org 70 i 86

@media (prefers-color-scheme: dark) {	Err	codemadness.org	70
i    87   /* styles for dark style */	Err	codemadness.org	70
i    88 }	Err	codemadness.org	70
i    89 	Err	codemadness.org	70
i    90 @media (prefers-color-scheme: light) {	Err	codemadness.org	70
i    91   /* styles for light style */	Err	codemadness.org	70
i    92 }	Err	codemadness.org	70
i    93

Err codemadness.org 70 i 94 Err codemadness.org 70 i 95

Or even declaring colors in a civilized fashion:

Err codemadness.org 70 i 96

:root {	Err	codemadness.org	70
i    97   --main-bg-color: pink;	Err	codemadness.org	70
i    98 }	Err	codemadness.org	70
i    99 	Err	codemadness.org	70
i   100 some_widget {	Err	codemadness.org	70
i   101   background-color: var(--main-bg-color);	Err	codemadness.org	70
i   102 }	Err	codemadness.org	70
i   103

Err codemadness.org 70 i 104 Err codemadness.org 70 i 105

Or combining the two:

Err codemadness.org 70 i 106

@media (prefers-color-scheme: dark) {	Err	codemadness.org	70
i   107   :root {	Err	codemadness.org	70
i   108     --main-bg-color: /* some nice dark background color */;	Err	codemadness.org	70
i   109     --main-fg-color: /* a contrasty light foreground */;	Err	codemadness.org	70
i   110   }	Err	codemadness.org	70
i   111 }	Err	codemadness.org	70
i   112 	Err	codemadness.org	70
i   113 @media (prefers-color-scheme: light) {	Err	codemadness.org	70
i   114   :root {	Err	codemadness.org	70
i   115     --main-bg-color: /* some nice light background color */;	Err	codemadness.org	70
i   116     --main-fg-color: /* a contrasty dark foreground */;	Err	codemadness.org	70
i   117   }	Err	codemadness.org	70
i   118 }	Err	codemadness.org	70
i   119 	Err	codemadness.org	70
i   120 some_widget {	Err	codemadness.org	70
i   121   background-color: var(--main-bg-color);	Err	codemadness.org	70
i   122 }	Err	codemadness.org	70
i   123

Err codemadness.org 70 i 124 Err codemadness.org 70 i 125

Boom. I think this would remove some workarounds we have right now:

Err codemadness.org 70 i 126

Err codemadness.org 70 i 128
Just like GTK, libadwaita generates four variants of the system's Err codemadness.org 70 i 129 stylesheet using scss (regular, dark, high-contrast, Err codemadness.org 70 i 130 high-contrast-dark). This would be obviated with @media Err codemadness.org 70 i 131 queries Err codemadness.org 70 i 132 for prefers-color-scheme, prefers-contrast, inverted-colors as Err codemadness.org 70 i 133 in the web platform.
Err codemadness.org 70 i 134
Err codemadness.org 70 i 136
GTK has a custom @define-color keyword, but neither gnome-shell Err codemadness.org 70 i 137 nor librsvg support that. This would be obviated with CSS custom Err codemadness.org 70 i 138 properties - Err codemadness.org 70 i 139 the var() mechanism. (I don't know if some "environmental" stuff Err codemadness.org 70 i 140 would be better done as Err codemadness.org 70 i 141 env(), Err codemadness.org 70 i 142 but none of the three implementations support that, either.)
Err codemadness.org 70 i 143

Err codemadness.org 70 i 145

Accent colors

Err codemadness.org 70 i 146

They are currently implemented with GTK's @define-color, which is Err codemadness.org 70 i 147 not ideal if the colors have to trickle down from GTK to SVG icons, Err codemadness.org 70 i 148 since librsvg doesn't do @define-color - it would rather have Err codemadness.org 70 i 149 var() Err codemadness.org 70 i 150 instead.

Err codemadness.org 70 i 151

Of course, gnome-shell's libcroco doesn't do @define-color either.

Err codemadness.org 70 i 152

Look for @accent_color, @accent_bg_color, @warning_color, etc. in the default Err codemadness.org 70 i 153 stylesheet, Err codemadness.org 70 i 154 or better yet, write documentation!

Err codemadness.org 70 i 155

The default style:

Err codemadness.org 70 i 156

Err codemadness.org 70 i 157

Accent color set to orange (e.g. tweak it in GTK's CSS inspector):

Err codemadness.org 70 i 158

Err codemadness.org 70 i 159

/* Standalone, e.g. the "Page 1" label */	Err	codemadness.org	70
i   160 @define-color accent_color @orange_5;	Err	codemadness.org	70
i   161 	Err	codemadness.org	70
i   162 /* background+text pair */	Err	codemadness.org	70
i   163 @define-color accent_bg_color @orange_4;	Err	codemadness.org	70
i   164 @define-color accent_fg_color white;	Err	codemadness.org	70
i   165

Err codemadness.org 70 i 166 Err codemadness.org 70 i 167

Custom widgets

Err codemadness.org 70 i 168

Again, your app's custom stylesheet for its custom widgets can use the Err codemadness.org 70 i 169 colors defined through @define-color from the system's stylesheet.

Err codemadness.org 70 i 170

Recoloring styles

Err codemadness.org 70 i 171

You will be able to do this after it gets merged into the main branch, Err codemadness.org 70 i 172 e.g. recolor everything to sepia:

Err codemadness.org 70 i 173

Err codemadness.org 70 i 174

@define-color headerbar_bg_color #eedcbf;	Err	codemadness.org	70
i   175 @define-color headerbar_fg_color #483a22;	Err	codemadness.org	70
i   176 	Err	codemadness.org	70
i   177 @define-color bg_color #f9f3e9;	Err	codemadness.org	70
i   178 @define-color fg_color #483a22;	Err	codemadness.org	70
i   179 	Err	codemadness.org	70
i   180 @define-color dark_fill_color shade(#f9f3e9, .95);	Err	codemadness.org	70
i   181 	Err	codemadness.org	70
i   182 @define-color accent_bg_color @orange_4;	Err	codemadness.org	70
i   183 @define-color accent_color @orange_5;	Err	codemadness.org	70
i   184

Err codemadness.org 70 i 185 Err codemadness.org 70 i 186

Of course shade() is not web-platform CSS, either. We could keep Err codemadness.org 70 i 187 it, or redo it by implementing calc() Err codemadness.org 70 i 188 function for Err codemadness.org 70 i 189 color values.

Err codemadness.org 70 i 190

Recoloring icons

Err codemadness.org 70 i 191

Currently GTK takes some defined colors and creates a chunk of CSS to Err codemadness.org 70 i 192 inject into SVG for Err codemadness.org 70 i 193 icons. Err codemadness.org 70 i 194 This has some Err codemadness.org 70 i 195 problems.

Err codemadness.org 70 i 196

There is also some discussion about standardizing recolorable Err codemadness.org 70 i 197 icons across Err codemadness.org 70 i 198 desktop environments.

Err codemadness.org 70 i 199

How you can help

Err codemadness.org 70 i 200

Implement support for @media Err codemadness.org 70 i 201 queries Err codemadness.org 70 i 202 in our three CSS implementations (librsvg, gnome-shell, GTK). Decide Err codemadness.org 70 i 203 how CSS media features like prefers-color-scheme, Err codemadness.org 70 i 204 prefers-contrast, inverted-colors should interact with the GNOME's Err codemadness.org 70 i 205 themes and accessibility, and decide if we should use them for Err codemadness.org 70 i 206 familiarity with the web platform, or if we need media features with Err codemadness.org 70 i 207 different names.

Err codemadness.org 70 i 208

Implement support for CSS custom properties - Err codemadness.org 70 i 209 var() Err codemadness.org 70 i 210 in our three CSS implementations. Decide if we should replace the Err codemadness.org 70 i 211 current @define-color with that (note that @define-color is only Err codemadness.org 70 i 212 in GTK, but not in librsvg or gnome-shell).

Err codemadness.org 70 i 213

See the libadwaita Err codemadness.org 70 i 214 roadmap Err codemadness.org 70 i 215 and help out!

Err codemadness.org 70 i 216

Port applications to use the proposed APIs for querying the dark style Err codemadness.org 70 i 217 preference. There are a bunch of hacky ways of doing it right now; Err codemadness.org 70 i 218 they need to be migrated to the new system.

Err codemadness.org 70 i 219

Personally I would love help with finishing to port gnome-shell's Err codemadness.org 70 i 220 styles to Err codemadness.org 70 i 221 Rust - Err codemadness.org 70 i 222 this is part of unifying librsvg's and gnome-shell's CSS machinery.

Bzip2's experimental repository is changing maintainership

2021-06-03T19:21:04-05:00

Bzip2's stable repository is maintained at Sourceware by Err codemadness.org 70 i 228 Mark Wielaard. In 2019 I started maintaining an experimental Err codemadness.org 70 i 229 repository in GitLab, with the intention of updating Err codemadness.org 70 i 230 the build system and starting a Rust port of bzip2. Unfortunately I Err codemadness.org 70 i 231 have left this project slip by.

Err codemadness.org 70 i 232

The new maintainer of the experimental repository for Err codemadness.org 70 i 233 Bzip2 is Micah Snyder. Thanks, Micah, for picking it up!

Librsvg, Rust, and non-mainstream architectures

2021-02-24T19:14:55-06:00

Almost five years ago librsvg introduced Rust into its source Err codemadness.org 70 i 238 code. Around the same time, Linux distributions Err codemadness.org 70 i 239 started shipping the first versions of Firefox that also required Err codemadness.org 70 i 240 Rust. I unashamedly wanted to ride that wave: distros would have to Err codemadness.org 70 i 241 integrate a new language in their build infrastructure, or they would Err codemadness.org 70 i 242 be left without Firefox. I was hoping that having a working Rust Err codemadness.org 70 i 243 toolchain would make it easier for the rustified librsvg to get into Err codemadness.org 70 i 244 distros.

Err codemadness.org 70 i 245

Two years after that, someone from Debian complained Err codemadness.org 70 i 246 that this made it hard or impossible to build librsvg (and all the Err codemadness.org 70 i 247 software that depends on it, which is A Lot) on all the architectures Err codemadness.org 70 i 248 that Debian builds on — specifically, on things like HP PA-RISC or Err codemadness.org 70 i 249 Alpha, which even Debian marks as "discontinued" now.

Err codemadness.org 70 i 250

Recently there was a similar kerfuffle, this time from someone from Err codemadness.org 70 i 251 Gentoo, specifically about how Python's cryptography Err codemadness.org 70 i 252 package now requires Rust. So, it doesn't build for platforms that Err codemadness.org 70 i 253 Rust/LLVM don't support, like hppa, alpha, and Itanium. It also Err codemadness.org 70 i 254 doesn't build for platforms for which there are no Rust packages from Err codemadness.org 70 i 255 Gentoo yet (mips, s390x, riscv among them).

Err codemadness.org 70 i 256

Memories of discontinued architectures

Err codemadness.org 70 i 257

Let me reminisce about a couple of discontinued architectures. If I'm Err codemadness.org 70 i 258 reading Wikipedia correctly, the DEC Alpha ceased to be Err codemadness.org 70 i 259 developed in 2001, and HP, who purchased Compaq, who purchased DEC, Err codemadness.org 70 i 260 stopped selling Alpha systems in 2007. Notably, Compaq phased out the Err codemadness.org 70 i 261 Alpha in favor of the Itanium, which stopped being developed in 2017.

Err codemadness.org 70 i 262

I used an Alpha machine in 1997-1998, back at the University. Err codemadness.org 70 i 263 Miguel kindly let me program and learn from him at the Institute Err codemadness.org 70 i 264 where he worked, and the computer lab there got an Alpha box to let Err codemadness.org 70 i 265 the scientists run mathematical models on a machine with really fast Err codemadness.org 70 i 266 floating-point. This was a time when people actually regularly ssh'ed Err codemadness.org 70 i 267 into machines to run X11 applications remotely — in their case, I Err codemadness.org 70 i 268 think it was Matlab and Mathematica. Good times.

Err codemadness.org 70 i 269

The Alpha had fast floating point, much faster than Intel x86 CPUs, Err codemadness.org 70 i 270 and I was delighted to do graphics work on it. That was the first Err codemadness.org 70 i 271 64-bit machine I used, and it let me learn how to fix code that only Err codemadness.org 70 i 272 assumed 32 bits. It had a really picky floating-point unit. Whereas Err codemadness.org 70 i 273 x86 would happily throw you a NaN if you used uninitialized memory as Err codemadness.org 70 i 274 floats, the Alpha would properly fault and crash the program. I fixed Err codemadness.org 70 i 275 so many bugs thanks to that!

Err codemadness.org 70 i 276

I also have fond memories of the 32-bit SPARC Err codemadness.org 70 i 277 boxes at the University and their flat-screen fixed-frequency CRT Err codemadness.org 70 i 278 displays, but you know, I haven't seen one of those machines since Err codemadness.org 70 i 279 1998. Because I was doing graphics work, I used the single SPARC Err codemadness.org 70 i 280 machine in the computer lab at the Institute that had 24-bit graphics, Err codemadness.org 70 i 281 with a humongous 21" CRT display. PCs at the time still had 8-bit video Err codemadness.org 70 i 282 cards and shitty little monitors.

Err codemadness.org 70 i 283

At about the same time that the Institute got its Alpha, it also got Err codemadness.org 70 i 284 one of the first 64-bit UltraSPARCs from Sun — a very expensive Err codemadness.org 70 i 285 machine definitely not targeted to hobbyists. I think it had two CPUs! Err codemadness.org 70 i 286 Multicore did not exist!

Err codemadness.org 70 i 287

I think I saw a single Itanium machine in my life, probably around Err codemadness.org 70 i 288 2002-2005. The Ximian/Novell office in Mexico City got one, for QA Err codemadness.org 70 i 289 purposes — an incredibly loud and unstable machine. I don't think we Err codemadness.org 70 i 290 ever did any actual development on that box; it was a "can you Err codemadness.org 70 i 291 reproduce this bug there" kind of thing. I think Ximian/Novell had a Err codemadness.org 70 i 292 contract with HP to test the distro there, I don't remember.

Err codemadness.org 70 i 293

Unsupported architectures at the LLVM level

Err codemadness.org 70 i 294

Platforms like the Alpha and Itanium that Rust/LLVM don't support — Err codemadness.org 70 i 295 those platforms are dead in the water. The compiler cannot target Err codemadness.org 70 i 296 them, as Rust generates machine code via LLVM, and LLVM doesn't Err codemadness.org 70 i 297 support them.

Err codemadness.org 70 i 298

I don't know why distributions maintained by volunteers give Err codemadness.org 70 i 299 themselves the responsibility to keep their software running on Err codemadness.org 70 i 300 platforms that have not been manufactured for years, and that were Err codemadness.org 70 i 301 never even hobbyist machines.

Err codemadness.org 70 i 302

I read the other day, and now I regret not keeping the link, something Err codemadness.org 70 i 303 like this: don't assume that your hobby computing entitles you to free Err codemadness.org 70 i 304 labor on the part of compiler writers, software maintainers, and Err codemadness.org 70 i 305 distro volunteers. (If someone helps me find the source, I'll happily Err codemadness.org 70 i 306 link to it and quote it properly.)

Err codemadness.org 70 i 307

Non-tier-1 platforms and "$distro does not build Rust there yet"

Err codemadness.org 70 i 308

I think people are discovering these once again:

Err codemadness.org 70 i 309

Err codemadness.org 70 i 311
Writing and supporting a compiler for a certain architecture takes Real Work.
Err codemadness.org 70 i 312
Err codemadness.org 70 i 314
Supporting a distro for a certain architecture takes Real Work.
Err codemadness.org 70 i 315
Err codemadness.org 70 i 317
Fixing software to work on a certain architecture takes Real Work.
Err codemadness.org 70 i 318

Err codemadness.org 70 i 320

Rust divides its support for different platforms into tiers, going Err codemadness.org 70 i 321 from tier 1, the most supported, to tier 3, the least supported. Or, Err codemadness.org 70 i 322 I should say, taken care of, which is a combination of people who Err codemadness.org 70 i 323 actually have the hardware in question, and whether the general CI and Err codemadness.org 70 i 324 build tooling is prepared to deal with them as effectively as it does Err codemadness.org 70 i 325 for tier 1 platforms.

Err codemadness.org 70 i 326

In other words: there are more people capable of paying attention to, and Err codemadness.org 70 i 327 testing things on, x86_64 PCs than there are for Err codemadness.org 70 i 328 sparc-unknown-linux-gnu.

Err codemadness.org 70 i 329

Some anecdotes from Suse

Err codemadness.org 70 i 330

At Suse we actually support IBM's s390x big iron; those mainframes run Err codemadness.org 70 i 331 Suse Linux Enterprise Server. You have to pay a lot of money to get a Err codemadness.org 70 i 332 machine like that and support for it. It's a room-sized beast that Err codemadness.org 70 i 333 requires professional babysitting.

Err codemadness.org 70 i 334

When librsvg and Firefox started getting rustified, there was of Err codemadness.org 70 i 335 course concern about getting Rust to work properly on the s390x. Err codemadness.org 70 i 336 I worked sporadically with the people who made the distro work there, Err codemadness.org 70 i 337 and who had to deal with building Rust and Firefox on it (librsvg Err codemadness.org 70 i 338 was a non-issue after getting Rust and Firefox to work).

Err codemadness.org 70 i 339

I think all the LLVM work for the s390x was done at IBM. There were Err codemadness.org 70 i 340 probably a couple of miscompilations that affected Firefox; they got fixed.

Err codemadness.org 70 i 341

One would expect bugs in software for IBM mainframes to be fixed by Err codemadness.org 70 i 342 IBM or its contractors, not by volunteers maintaining a distro in Err codemadness.org 70 i 343 their spare time.

Err codemadness.org 70 i 344

Giving computing time on mainframes to volunteers in distros could seem Err codemadness.org 70 i 345 like a good samaritan move, or a trap to extract free labor from Err codemadness.org 70 i 346 unsuspecting people.

Err codemadness.org 70 i 347

Endianness bugs

Err codemadness.org 70 i 348

Firefox's problems on the s390x were more around big-endian bugs than Err codemadness.org 70 i 349 anything. You see, all the common architectures these days (x86_64 Err codemadness.org 70 i 350 and arm64) are little-endian. However, s390x is big-endian, Err codemadness.org 70 i 351 which means that all multi-byte numbers in memory are stored backwards Err codemadness.org 70 i 352 from what most software expects.

Err codemadness.org 70 i 353

It is not a problem to write software that assumes little-endian or Err codemadness.org 70 i 354 big-endian all the time, but it takes a little care to write software Err codemadness.org 70 i 355 that works on either.

Err codemadness.org 70 i 356

Most of the software that volunteers and paid people write assumes Err codemadness.org 70 i 357 little-endian CPUs, because that is likely what they are targeting. Err codemadness.org 70 i 358 It is a pain in the ass to encounter code that works incorrectly on Err codemadness.org 70 i 359 big-endian — a pain because knowing where to look for evidence of Err codemadness.org 70 i 360 bugs is tricky, and fixing existing code to work with either Err codemadness.org 70 i 361 endianness can be either very simple, or a major adventure in Err codemadness.org 70 i 362 refactoring and testing.

Err codemadness.org 70 i 363

Two cases in point:

Err codemadness.org 70 i 364

Firefox. When Suse started dealing with Rust and Firefox in the Err codemadness.org 70 i 365 s390x, there were endianness bugs in the graphics code in Firefox that Err codemadness.org 70 i 366 deals with pixel formats. Whether pixels get stored in memory as Err codemadness.org 70 i 367 ARGB/ABGR/RGBA/etc. is a platform-specific thing, and is generally a Err codemadness.org 70 i 368 combination of the graphics hardware for that platform, plus the Err codemadness.org 70 i 369 actual CPU architecture. At that time, it looked like the C++ code in Err codemadness.org 70 i 370 Firefox that deals with pixels had been rewritten/refactored, and had Err codemadness.org 70 i 371 lost big-endian support along the way. I don't know the current Err codemadness.org 70 i 372 status (not a single big-endian CPU in my vincinity), but I haven't Err codemadness.org 70 i 373 seen related bugs come in the Suse bug tracker? Maybe it's fixed now?

Err codemadness.org 70 i 374

Librsvg had two root causes of bugs for Err codemadness.org 70 i 375 big-endian. One was in the old code for SVG Err codemadness.org 70 i 376 filter effects that was written in C; it never supported big-endian. The Err codemadness.org 70 i 377 initial port to Rust inherited the same bug (think of a line-by-line Err codemadness.org 70 i 378 port, althought it wasn't exactly like that), but it got fixed when my Err codemadness.org 70 i 379 Summer of Code intern Ivan Molodetskikh refactored the code to have a Err codemadness.org 70 i 380 Pixel abstraction that works for little-endian and big-endian, and Err codemadness.org 70 i 381 wraps Cairo's funky requirements.

Err codemadness.org 70 i 382

The other endian-related bug in librsvg was when computing Err codemadness.org 70 i 383 masks. Again, a little refactoring with that Pixel Err codemadness.org 70 i 384 abstraction fixed it.

Err codemadness.org 70 i 385

I knew that the original C code for SVG filter effects didn't work on Err codemadness.org 70 i 386 big-endian. But even back then, at Suse we never got Err codemadness.org 70 i 387 reports of it producing incorrect results on the s390x... maybe people don't use Err codemadness.org 70 i 388 their mainframes to run rsvg-convert? I was hoping that the port to Err codemadness.org 70 i 389 Rust of that code would automatically fix that bug, and it kind of Err codemadness.org 70 i 390 happened that way through Ivan's careful work.

Err codemadness.org 70 i 391

And the code for masks? There were two bugs reported with that same Err codemadness.org 70 i 392 root cause: one from Err codemadness.org 70 i 393 Debian as a Err codemadness.org 70 i 394 failure in librsvg's test suite (yay, it caught that bug!), and one Err codemadness.org 70 i 395 from someone running an Apple PowerBook Err codemadness.org 70 i 396 G4 with a MATE Err codemadness.org 70 i 397 desktop and seeing incorrectly-rendered SVG icons.

Err codemadness.org 70 i 398

And you know what? I am delighted to see people trying to keep Err codemadness.org 70 i 399 those lovely machines alive. A laptop that doesn't get warm enough to Err codemadness.org 70 i 400 burn your thighs, what a concept. A perfectly serviceable 32-bit Err codemadness.org 70 i 401 laptop with a maximum of about 1 GB of RAM and a 40 GB hard drive (it Err codemadness.org 70 i 402 didn't have HDMI!)... But you know, it's the same kind of delight I Err codemadness.org 70 i 403 feel when people talk about doing film photography on a Err codemadness.org 70 i 404 Rollei 35. A lot of Err codemadness.org 70 i 405 nostalgia for hardware of days past, and a lot of mixed feelings about Err codemadness.org 70 i 406 not throwing out working things and creating more trash.

Err codemadness.org 70 i 407

As a graphics programmer I feel the responsibility to write code that Err codemadness.org 70 i 408 works on little-endian and big-endian, but you know, it's not exactly Err codemadness.org 70 i 409 an everyday concern anymore. The last big-endian machine I used on an Err codemadness.org 70 i 410 everyday basis was the SPARCs in the university, more than 20 years Err codemadness.org 70 i 411 ago.

Err codemadness.org 70 i 412

Who gets paid to fix this?

Err codemadness.org 70 i 413

That's the question. Suse got paid to support Firefox on the s390x; I Err codemadness.org 70 i 414 suppose IBM has an interest in fixing LLVM there; both actually have Err codemadness.org 70 i 415 people and hardware and money to that effect.

Err codemadness.org 70 i 416

Within Suse, I am by default responsible for keeping librsvg Err codemadness.org 70 i 417 working for the s390x as well — it gets built as part of the distro, Err codemadness.org 70 i 418 after all. I have never gotten an endianness bug report from the Suse Err codemadness.org 70 i 419 side of things.

Err codemadness.org 70 i 420

Which leads me to suspect that, probably similar to Debian and Gentoo, Err codemadness.org 70 i 421 we build a lot of software because it's in the build chain, but we Err codemadness.org 70 i 422 don't run it to its fullest extent. Do people run GNOME desktops on Err codemadness.org 70 i 423 s390x virtual machines? Maybe? Did they not notice endianness bugs Err codemadness.org 70 i 424 because they were not in the code path that most GNOME icons Err codemadness.org 70 i 425 actually use? Who knows!

Err codemadness.org 70 i 426

I'm thankful to Simon from the Debian bug for pointing out Err codemadness.org 70 i 427 the failure in librsvg's test case for masks, and to Mingcong for Err codemadness.org 70 i 428 actually showing a screenshot of a MATE desktop running on a PPC Err codemadness.org 70 i 429 PowerBook. Those were useful things for them to do.

Err codemadness.org 70 i 430

Also — they were kind about it. It was a pleasure to interact with them.

Do not use librsvg 2.40.x

2020-11-27T10:28:46-06:00

Please do not use librsvg 2.40.x; it cannot render recent Adwaita icon themes correctly.

Err codemadness.org 70 i 435

The librsvg 2.40.x series is the last "C only" version of the library; Err codemadness.org 70 i 436 it was deprecated in 2017.

Err codemadness.org 70 i 437

During the port to Rust, I rewrote the path parser to be Err codemadness.org 70 i 438 spec-compliant, and fixed a few cases that the C version did not Err codemadness.org 70 i 439 handle. One of this cases is for compact Arc data.

Err codemadness.org 70 i 440

The SVG path grammar allows Err codemadness.org 70 i 441 one to remove whitespace between numbers if the next number starts Err codemadness.org 70 i 442 with a sign. For example, 23-45 gets parsed as two numbers 23 Err codemadness.org 70 i 443 -45.

Err codemadness.org 70 i 444

In addition, the arguments of the Arc commands have two flags in the Err codemadness.org 70 i 445 middle of a bunch of numbers. The flags can be 0 or 1, and there Err codemadness.org 70 i 446 may be no whitespace between the flags and the next number. For Err codemadness.org 70 i 447 example, A1.98 1.98 0 0015 13.96 gets parsed as A1.98 1.98 0 0 0 15 Err codemadness.org 70 i 448 13.96 — note the two 0 0 flags before the 15.

Err codemadness.org 70 i 449

Librsvg 2.40.x does not parse this correctly. Err codemadness.org 70 i 450 Adwaita-icon-theme-3.36, and possibly earlier versions, uses minimized Err codemadness.org 70 i 451 SVG files with compressed whitespace, and will not render correctly Err codemadness.org 70 i 452 with the C-only version of librsvg.

Err codemadness.org 70 i 453

This is help-contents-symbolic.svg rendered with librsvg 2.40.21:

Err codemadness.org 70 i 454

Err codemadness.org 70 i 455

And this is help-contents-symbolic.svg rendered with librsvg 2.50.2:

Err codemadness.org 70 i 456

Err codemadness.org 70 i 457

This is not the only icon with compact Arc commands; there are many Err codemadness.org 70 i 458 others that will also be mis-rendered in 2.40.x.

Err codemadness.org 70 i 459

I don't know when Adwaita started using SVGs with compressed Err codemadness.org 70 i 460 whitespace; probably it didn't when librsvg 2.40.x was the latest Err codemadness.org 70 i 461 version, or everyone would have noticed mis-rendered icons.

Err codemadness.org 70 i 462

Background: Someone recently filed a Err codemadness.org 70 i 463 bug about Err codemadness.org 70 i 464 memory unsafety in librsvg 2.40.x's path parser, which mysteriously Err codemadness.org 70 i 465 enough only manifests itself in big-endian platforms. I wouldn't be Err codemadness.org 70 i 466 surprised if this had latent bugs on little-endian as well.

Err codemadness.org 70 i 467

Please use at least librsvg 2.48.x; any earlier versions are not Err codemadness.org 70 i 468 supported. Generally I keep an eye on the last two stable release Err codemadness.org 70 i 469 sets (2.48.x and 2.50.x as of this writing), but only commit fixes to Err codemadness.org 70 i 470 the latest stable series (2.50.x currently).

Librsvg's test suite is now in Rust

2020-10-26T10:38:12-06:00

Some important changes are afoot in librsvg.

Err codemadness.org 70 i 477

Changes to continuous integration

Err codemadness.org 70 i 478

Some days ago, Dunja Lalic rewrote the continuous integration Err codemadness.org 70 i 479 scripts to be much faster. A complete pipeline used to take Err codemadness.org 70 i 480 about 90 minutes to run, now it takes about 15 minutes on average.

Err codemadness.org 70 i 481

Err codemadness.org 70 i 482

The description of changes is interesting. The idea is to make tests fail as fast as possible, close to the beginning of the pipeline. To speed up the whole pipeline, Dunja did the following:

Err codemadness.org 70 i 483

Err codemadness.org 70 i 485
Move the cargo check stage to the beginning. This test means, Err codemadness.org 70 i 486 "does this even have a chance of compiling?".
Err codemadness.org 70 i 487
Err codemadness.org 70 i 489
Have the code style and formatting tests, cargo clippy and cargo Err codemadness.org 70 i 490 fmt, run in parallel with the unit tests. These lints can fail, Err codemadness.org 70 i 491 but they are easy to fix after one is finished modifying the main Err codemadness.org 70 i 492 code.
Err codemadness.org 70 i 493
Err codemadness.org 70 i 495
Run the unit tests and the smoke tests in debug mode, so they compile Err codemadness.org 70 i 496 quickly.
Err codemadness.org 70 i 497
Err codemadness.org 70 i 499
Run the complete integration test suite in release mode. This takes Err codemadness.org 70 i 500 longer to compile, but there are some slow tests that benefit a lot Err codemadness.org 70 i 501 from faster execution.
Err codemadness.org 70 i 502
Err codemadness.org 70 i 504
Move the release tests until the end, and only run them once a week Err codemadness.org 70 i 505 — or whenever, by hand. These take a good amount of time to run, Err codemadness.org 70 i 506 because they do a full make distcheck and autotools is slow. Even Err codemadness.org 70 i 507 then, now these tests take 30-40 minutes, instead of the 90 from Err codemadness.org 70 i 508 before.
Err codemadness.org 70 i 509
Err codemadness.org 70 i 511
Between each stage of the pipeline, don't cache what doesn't help Err codemadness.org 70 i 512 reduce compilation time. It seems that keeping around a big cache, Err codemadness.org 70 i 513 with the whole build target, between each pipeline stage can be Err codemadness.org 70 i 514 worse than not having one at all.
Err codemadness.org 70 i 515

Err codemadness.org 70 i 517

Err codemadness.org 70 i 518

Test suite in Rust

Err codemadness.org 70 i 519

Beteen Sven Neumann, Dunja Lalic, and myself we have finally ported Err codemadness.org 70 i 520 the test suite to Rust: all of librsvg's tests are Err codemadness.org 70 i 521 now in Rust, except for the C API tests. We had to do a few things:

Err codemadness.org 70 i 522

Err codemadness.org 70 i 524
Review the old tests and remove some obsolete ones.
Err codemadness.org 70 i 525
Err codemadness.org 70 i 527
Port each of the test modules to Rust. They are small, but each one Err codemadness.org 70 i 528 has special little things — test for crashes in the XML loading Err codemadness.org 70 i 529 code, test for crashes during rendering, test the library's security Err codemadness.org 70 i 530 limits.
Err codemadness.org 70 i 531
Err codemadness.org 70 i 533
Fix the small tests that come as part of the documentation.
Err codemadness.org 70 i 534
Err codemadness.org 70 i 536
Untangle the reference tests and port them to Rust.
Err codemadness.org 70 i 537
Err codemadness.org 70 i 539
Move little chunks of code around so the unit tests and integration Err codemadness.org 70 i 540 tests can share utilities to compare images, compute file paths for Err codemadness.org 70 i 541 test fixtures, etc.
Err codemadness.org 70 i 542

Err codemadness.org 70 i 544

The most complicated thing to port was the reference tests. These are Err codemadness.org 70 i 545 the most important ones; each test loads an SVG document, renders it, Err codemadness.org 70 i 546 and compares the result to a reference PNG image. There are some Err codemadness.org 70 i 547 complications in the tests; they have to create a special Err codemadness.org 70 i 548 configuration for Fontconfig and Pango, so as to have reproducible Err codemadness.org 70 i 549 font rendering. The pango-rs bindings do not cover this part of Err codemadness.org 70 i 550 Pango, so we had to do some things by hand.

Err codemadness.org 70 i 551

Anyway, the tests are now in Rust. One nice thing is that Err codemadness.org 70 i 552 now the tests run automatically in parallel, across all CPU cores, so Err codemadness.org 70 i 553 we save on total testing time.

Err codemadness.org 70 i 554

What's next: cargo-c and publish to crates.io

Err codemadness.org 70 i 555

We want to be able to publish librsvg in crates.io as a Err codemadness.org 70 i 556 normal crate; this implies being able to compile, test, and publish Err codemadness.org 70 i 557 entirely from Cargo. The compilation and testing part is done.

Err codemadness.org 70 i 558

Now, we have to reorganize the code so it can be published to Err codemadness.org 70 i 559 crates.io. Librsvg comes in three parts, rsvg_internals with the Err codemadness.org 70 i 560 implementation of the library, librsvg with the traditional C API, Err codemadness.org 70 i 561 and librsvg_crate with the Rust API. However, to publish the Rust Err codemadness.org 70 i 562 API to crates.io, it would be more convenient to have a single crate Err codemadness.org 70 i 563 instead of one with the internals and one with the API.

Err codemadness.org 70 i 564

The next step is thus to reorganize the code:

Err codemadness.org 70 i 565

Err codemadness.org 70 i 567
Make it possible to implement the C API as a compile-time option on Err codemadness.org 70 i 568 top of the normal Rust code. We want to use cargo-c to Err codemadness.org 70 i 569 compile the traditional shared library librsvg.so, instead of Err codemadness.org 70 i 570 depending on C tools for compiling and linking.
Err codemadness.org 70 i 571
Err codemadness.org 70 i 573
Combine rsvg_internals and librsvg_crate in a single crate, to Err codemadness.org 70 i 574 publish them together. Crates.io has a 10 MB limit per crate; now Err codemadness.org 70 i 575 that the test suite lives in a separate tests crate, this Err codemadness.org 70 i 576 shouldn't be a problem.
Err codemadness.org 70 i 577
Err codemadness.org 70 i 579
I would like to polish the public error types before Err codemadness.org 70 i 580 publishing the Rust API; right now they expose some implementation Err codemadness.org 70 i 581 details that are of no interest to callers of the library.
Err codemadness.org 70 i 582

Err codemadness.org 70 i 584

What remains to be ported to Rust?

Err codemadness.org 70 i 585

Only two things, which amount to less than 900 lines of C code:

Err codemadness.org 70 i 586

Err codemadness.org 70 i 588
rsvg-convert - the command-line program that everyone uses to Err codemadness.org 70 i 589 convert SVG to PNG and other formats. Fortunately, Sven Neumann Err codemadness.org 70 i 590 wrote some fantastic tests for rsvg-convert, Err codemadness.org 70 i 591 as it is like an API that we need to keep stable: if we change the Err codemadness.org 70 i 592 command-line options or the program's behavior, we would break Err codemadness.org 70 i 593 everyone's scripts.
Err codemadness.org 70 i 594
Err codemadness.org 70 i 596
The gdk-pixbuf module for loading SVG. Alberto Ruiz has started Err codemadness.org 70 i 597 porting it to Rust. The generic part of this code Err codemadness.org 70 i 598 could later serve to wrap other Rust image codecs and plug them to Err codemadness.org 70 i 599 gdk-pixbuf.
Err codemadness.org 70 i 600

Librsvg is accepting interns for Outreachy's December 2020 round

2020-10-05T09:52:27-05:00

There are two projects in librsvg available for Outreachy applicants Err codemadness.org 70 i 608 in the December 2020 / March 2021 round:

Err codemadness.org 70 i 609

Err codemadness.org 70 i 611
Revamp the text engine - Do you know about international text Err codemadness.org 70 i 612 layout? Can you read a right-to-left language, or do you write in a Err codemadness.org 70 i 613 language that requires complex shaping? Would you like to implement Err codemadness.org 70 i 614 the SVG 2 text specification in a pleasant Rust code Err codemadness.org 70 i 615 base? This project requires someone who can write Rust Err codemadness.org 70 i 616 comfortably; it will require reading and refactoring some Err codemadness.org 70 i 617 existing code. You don't need to be an expert in exotic lifetimes Err codemadness.org 70 i 618 and trait bounds and such; the code doesn't use them.
Err codemadness.org 70 i 619
Err codemadness.org 70 i 621
Implement SVG2/CSS3 features - Are you excited by all the SVG2 Err codemadness.org 70 i 622 features in Inkscape, and would like to add support for them Err codemadness.org 70 i 623 in librsvg? Would you like to do small changes to many parts of the Err codemadness.org 70 i 624 code to implement small features, one at a time? Do you like Err codemadness.org 70 i 625 test-driven development? This project requires someone who can Err codemadness.org 70 i 626 write Rust code at a medium level; you'll learn a lot by Err codemadness.org 70 i 627 cutting&pasting from existing code and refactoring things to Err codemadness.org 70 i 628 implement SVG2 features.
Err codemadness.org 70 i 629

Err codemadness.org 70 i 631

Important: Outreachy's December 2020 / March 2021 round is available only Err codemadness.org 70 i 632 for students in the Southern hemisphere. People in the Err codemadness.org 70 i 633 Northern hemisphere can wait until the 2021 mid-year round.

Err codemadness.org 70 i 634

You can see GNOME's projects in Outreachy for this round. Err codemadness.org 70 i 635 The deadline for initial contributions and project applications is Err codemadness.org 70 i 636 October 31, 2020 at 16:00 UTC.

"Rust does not have a stable ABI"

2020-08-12T22:01:44-05:00

I've seen GNOME people (often, people who have been working for a long Err codemadness.org 70 i 643 time on C libraries) express concerns along the following lines:

Err codemadness.org 70 i 644

Compiled Rust code doesn't have a stable ABI (application binary interface).
So, we can't have shared libraries in the traditional fashion of Err codemadness.org 70 i 647 Linux distributions.
Also Rust bundles its entire standard library with every binary it compiles, which makes Rust-built libraries huge.

Err codemadness.org 70 i 650

These are extremely valid concerns to be addressed by people like Err codemadness.org 70 i 651 myself who propose that chunks of infrastructural libraries Err codemadness.org 70 i 652 should be done in Rust.

Err codemadness.org 70 i 653

So, let's begin.

Err codemadness.org 70 i 654

The first part of this article is a super-quick introduction to shared Err codemadness.org 70 i 655 libraries and how Linux distributions use them. If you already know Err codemadness.org 70 i 656 those things, feel free to skip to the "Rust does not have a stable Err codemadness.org 70 i 657 ABI" section.

Err codemadness.org 70 i 658

How do distributions use shared libraries?

Err codemadness.org 70 i 659

If several programs run at the same time and use the same shared library Err codemadness.org 70 i 660 (say, libgtk-3.so), the operating system can load a single copy of Err codemadness.org 70 i 661 the library in memory and share the read-only parts of the code/data Err codemadness.org 70 i 662 through the magic of virtual memory.

Err codemadness.org 70 i 663

In theory, if a library gets a bugfix but does not change its Err codemadness.org 70 i 664 interface, one can just recompile the library, stick the new .so in Err codemadness.org 70 i 665 /usr/lib or whatever, and be done with it. Programs that depend on Err codemadness.org 70 i 666 the library do not need to be recompiled.

Err codemadness.org 70 i 667

If libraries limit their public interface to a plain C ABI Err codemadness.org 70 i 668 (application binary interface), they are relatively easy to consume Err codemadness.org 70 i 669 from other programming languages. Those languages don't have to deal Err codemadness.org 70 i 670 with name mangling of C++ symbols, exception handlers, constructors, Err codemadness.org 70 i 671 and all that complexity. Pretty much every language has some form of Err codemadness.org 70 i 672 C FFI (foreign function interface), which roughly means "call C Err codemadness.org 70 i 673 functions without too much trouble".

Err codemadness.org 70 i 674

For the purposes of a library, what's an Err codemadness.org 70 i 675 ABI? Err codemadness.org 70 i 676 Wikipedia says, "An ABI defines how data structures or computational Err codemadness.org 70 i 677 routines are accessed in machine code [...] A common aspect of an Err codemadness.org 70 i 678 ABI is the calling convention", which means that to call a function in Err codemadness.org 70 i 679 machine code you need to frob the call and stack pointers, pass some Err codemadness.org 70 i 680 function arguments in registers or push some others to the stack, etc. Err codemadness.org 70 i 681 Really low-level stuff. Each machine architecture or operating system Err codemadness.org 70 i 682 usually defines a C standard ABI.

Err codemadness.org 70 i 683

For libraries, we commonly understand an ABI to mean the machine-code Err codemadness.org 70 i 684 implications of their programming interface. Which functions are Err codemadness.org 70 i 685 available as public symbols in the .so file? To which numeric Err codemadness.org 70 i 686 values do C enum values correspond, so that they can be passed to Err codemadness.org 70 i 687 those functions? What is the exact order and type of arguments that Err codemadness.org 70 i 688 the functions take? What are the struct sizes, and the order and Err codemadness.org 70 i 689 types and padding of the fields that those functions take? Does one Err codemadness.org 70 i 690 pass arguments in CPU registers or on the stack? Does the caller or Err codemadness.org 70 i 691 the callee clean up the stack after a function call?

Err codemadness.org 70 i 692

Bug fixes and security fixes

Err codemadness.org 70 i 693

Linux distributions generally try really hard to have a single Err codemadness.org 70 i 694 version of each shared library installed in the system: a single Err codemadness.org 70 i 695 libjpeg.so, a single libpng.so, a single libc.so, etc.

Err codemadness.org 70 i 696

This is helpful when there needs to be an update to fix a bug, Err codemadness.org 70 i 697 security-related or not: users can just download the updated package Err codemadness.org 70 i 698 for the library, which when installed will just stick in a new .so Err codemadness.org 70 i 699 in the right place, and the calling software won't need to be updated.

Err codemadness.org 70 i 700

This is possible only if the bug really only changes the internal Err codemadness.org 70 i 701 code without changing behavior or interface. If a bug fix requires Err codemadness.org 70 i 702 part of the public API or ABI to change, then you are screwed; all Err codemadness.org 70 i 703 calling software needs to be recompiled. "Irresponsible" library Err codemadness.org 70 i 704 authors either learn really fast when distros complain loudly about Err codemadness.org 70 i 705 this sort of change, or they don't learn and get forever marked by Err codemadness.org 70 i 706 distros as "that irresponsible library" which always requires special Err codemadness.org 70 i 707 handling in order not to break other software.

Err codemadness.org 70 i 708

Sidenote: sometimes it's more complicated. Poppler (the PDF Err codemadness.org 70 i 709 rendering library) ships at least two stable APIs, one Glib-based in Err codemadness.org 70 i 710 C, and one Qt-based in C++. However, some software like texlive uses Err codemadness.org 70 i 711 Poppler's internals library directly, which of course does not have a Err codemadness.org 70 i 712 stable API, and thus texlive breaks frequently as Poppler evolves. Err codemadness.org 70 i 713 Someone should extend the public, stable API so that texlive doesn't Err codemadness.org 70 i 714 have to use the library's internals!

Err codemadness.org 70 i 715

Bundled libraries

Err codemadness.org 70 i 716

Sometimes it is not irresponsible authors of libraries, but rather Err codemadness.org 70 i 717 that people who use the libraries find out that over time the behavior Err codemadness.org 70 i 718 of the library changes subtly, maybe without breaking the API or ABI, Err codemadness.org 70 i 719 and they are better off bundling a specific version of the library Err codemadness.org 70 i 720 with their software. That version is what they test their software Err codemadness.org 70 i 721 against, and they try to learn its quirks.

Err codemadness.org 70 i 722

Distros inevitably complain about this, and either patch the calling Err codemadness.org 70 i 723 software by hand to force it to use the system's shared library, or Err codemadness.org 70 i 724 succeed in getting patches accepted by the software so that they have Err codemadness.org 70 i 725 a --use-system-libjpeg option or similar.

Err codemadness.org 70 i 726

This doesn't work very well if the bundled version of the library has Err codemadness.org 70 i 727 extra patches that are not in a distro's usual patches. Or Err codemadness.org 70 i 728 vice-versa; it may actually work better to use the distro's version of Err codemadness.org 70 i 729 the library, if it has extra fixes that the bundled library doesn't. Err codemadness.org 70 i 730 Who knows! It's a case-by-case situation.

Err codemadness.org 70 i 731

Rust does not have a stable ABI

Err codemadness.org 70 i 732

By default indeed it doesn't, because the compiler team wants to have Err codemadness.org 70 i 733 the freedom to change the data layout and Rust-to-Rust calling Err codemadness.org 70 i 734 conventions, often for performance reasons, at any time. For example, Err codemadness.org 70 i 735 it is not guaranteed that struct fields will be laid out in memory in Err codemadness.org 70 i 736 the same order as they are written in the code:

Err codemadness.org 70 i 737

struct Foo {	Err	codemadness.org	70
i   738     bar: bool,	Err	codemadness.org	70
i   739     baz: f64,	Err	codemadness.org	70
i   740     beep: bool,	Err	codemadness.org	70
i   741     qux: i32,	Err	codemadness.org	70
i   742 }	Err	codemadness.org	70
i   743

Err codemadness.org 70 i 744 Err codemadness.org 70 i 745

The compiler is free to rearrange the struct fields in memory as it Err codemadness.org 70 i 746 sees fit. Maybe it decides to put the two bool fields next to each Err codemadness.org 70 i 747 other to save on inter-field padding due to alignment requirements; Err codemadness.org 70 i 748 maybe it does static analysis or profile-guided optimizations and Err codemadness.org 70 i 749 picks an optmal ordering.

Err codemadness.org 70 i 750

But we can override this! Let's look at data layout first, and then Err codemadness.org 70 i 751 calling conventions.

Err codemadness.org 70 i 752

Data layout for C versus Rust

Err codemadness.org 70 i 753

The following is the same struct as above, but with an extra #[repr(C)] attribute:

Err codemadness.org 70 i 754

#[repr(C)]	Err	codemadness.org	70
i   755 struct Foo {	Err	codemadness.org	70
i   756     bar: bool,	Err	codemadness.org	70
i   757     baz: f64,	Err	codemadness.org	70
i   758     beep: bool,	Err	codemadness.org	70
i   759     qux: i32,	Err	codemadness.org	70
i   760 }	Err	codemadness.org	70
i   761

Err codemadness.org 70 i 762 Err codemadness.org 70 i 763

With that attribute, the struct will be laid out just as this C struct:

Err codemadness.org 70 i 764

#include <stdbool.h>	Err	codemadness.org	70
i   765 #include <stdint.h>	Err	codemadness.org	70
i   766 	Err	codemadness.org	70
i   767 struct Foo {	Err	codemadness.org	70
i   768     bool bar;	Err	codemadness.org	70
i   769     double baz;	Err	codemadness.org	70
i   770     bool beep;	Err	codemadness.org	70
i   771     int32_t qux;	Err	codemadness.org	70
i   772 }	Err	codemadness.org	70
i   773

Err codemadness.org 70 i 774 Err codemadness.org 70 i 775

(Aside: it is unfortunate that gboolean is not bool, Err codemadness.org 70 i 776 but that's because gboolean predates C99, and clearly standards from Err codemadness.org 70 i 777 20 years ago are too new to use. (Aside aside: since I wrote that Err codemadness.org 70 i 778 other post, Rust's repr(C) for bool is actually defined as C99's bool; Err codemadness.org 70 i 779 it's no longer undefined.))

Err codemadness.org 70 i 780

Even Rust's data-carrying enums can be laid out in a manner friendly Err codemadness.org 70 i 781 to C and C++:

Err codemadness.org 70 i 782

#[repr(C, u8)]	Err	codemadness.org	70
i   783 enum MyEnum {	Err	codemadness.org	70
i   784     A(u32),	Err	codemadness.org	70
i   785     B(f32, bool),	Err	codemadness.org	70
i   786 }	Err	codemadness.org	70
i   787

Err codemadness.org 70 i 788 Err codemadness.org 70 i 789

This means, use C layout, and a u8 for the enum's discriminant. It Err codemadness.org 70 i 790 will be laid out like this:

Err codemadness.org 70 i 791

#include <stdbool.h>	Err	codemadness.org	70
i   792 #include <stdint.h>	Err	codemadness.org	70
i   793 	Err	codemadness.org	70
i   794 enum MyEnumTag {	Err	codemadness.org	70
i   795         A,	Err	codemadness.org	70
i   796         B	Err	codemadness.org	70
i   797 };	Err	codemadness.org	70
i   798 	Err	codemadness.org	70
i   799 typedef uint32_t MyEnumPayloadA;	Err	codemadness.org	70
i   800 	Err	codemadness.org	70
i   801 typedef struct {	Err	codemadness.org	70
i   802         float x;	Err	codemadness.org	70
i   803         bool y;	Err	codemadness.org	70
i   804 } MyEnumPayloadB;	Err	codemadness.org	70
i   805 	Err	codemadness.org	70
i   806 typedef union {	Err	codemadness.org	70
i   807         MyEnumPayloadA a;	Err	codemadness.org	70
i   808         MyEnumPayloadB b;	Err	codemadness.org	70
i   809 } MyEnumPayload;	Err	codemadness.org	70
i   810 	Err	codemadness.org	70
i   811 typedef struct {	Err	codemadness.org	70
i   812         uint8_t tag;	Err	codemadness.org	70
i   813         MyEnumPayload payload;	Err	codemadness.org	70
i   814 } MyEnum;	Err	codemadness.org	70
i   815

Err codemadness.org 70 i 816 Err codemadness.org 70 i 817

The gory details of data layout are in the Alternative Representations section of the Err codemadness.org 70 i 818 Rustonomicon and Err codemadness.org 70 i 819 the Unsafe Code Err codemadness.org 70 i 820 Guidelines.

Err codemadness.org 70 i 821

Calling conventions

Err codemadness.org 70 i 822

An ABI's calling conventions detail things like how to call functions Err codemadness.org 70 i 823 in machine code, and how to lay out function arguments in registers or Err codemadness.org 70 i 824 the stack. The wikipedia page on X86 calling Err codemadness.org 70 i 825 conventions Err codemadness.org 70 i 826 has a good cheat-sheet, useful when you are looking at assembly code Err codemadness.org 70 i 827 and registers in a low-level debugger.

Err codemadness.org 70 i 828

I've already written about how it is possible to write Rust code to Err codemadness.org 70 i 829 export functions callable from C; one uses the extern "C" in the Err codemadness.org 70 i 830 function definition and a #[no_mangle] attribute to keep the symbol Err codemadness.org 70 i 831 name pristine. This is how librsvg is able to have the following:

Err codemadness.org 70 i 832

#[no_mangle]	Err	codemadness.org	70
i   833 pub unsafe extern "C" fn rsvg_handle_new_from_file(	Err	codemadness.org	70
i   834     filename: *const libc::c_char,	Err	codemadness.org	70
i   835     error: *mut *mut glib_sys::GError,	Err	codemadness.org	70
i   836 ) -> *const RsvgHandle {	Err	codemadness.org	70
i   837     // ...	Err	codemadness.org	70
i   838 }	Err	codemadness.org	70
i   839

Err codemadness.org 70 i 840 Err codemadness.org 70 i 841

Which compiles to what a C compiler would produce for this:

Err codemadness.org 70 i 842

RsvgHandle *rsvg_handle_new_from_file (const gchar *filename, GError **error);	Err	codemadness.org	70
i   843

Err codemadness.org 70 i 844 Err codemadness.org 70 i 845

(Aside: librsvg still uses an intermediate C library full of Err codemadness.org 70 i 846 stubs that just Err codemadness.org 70 i 847 call the Rust-exported functions, but there is now tooling to produce a .so Err codemadness.org 70 i 848 directly from Err codemadness.org 70 i 849 Rust which I Err codemadness.org 70 i 850 just haven't had time to investigate. Help is appreciated!)

Err codemadness.org 70 i 851

Summary of ABI so far

Err codemadness.org 70 i 852

It is one's decision to export a stable C ABI from a Rust library. Err codemadness.org 70 i 853 There is some awkwardness in how types are laid out in C, because the Err codemadness.org 70 i 854 Rust type system is richer, but things can be made to work well with a Err codemadness.org 70 i 855 little thought. Certainly no more thought than the burden of Err codemadness.org 70 i 856 designing and maintaining a stable API/ABI in plain C.

Err codemadness.org 70 i 857

I'll fold the second concern into here — "we can't have shared Err codemadness.org 70 i 858 libraries in traditional distro fashion". Yes, we can, API/ABI-wise, Err codemadness.org 70 i 859 but read on.

Err codemadness.org 70 i 860

Rust bundles its entire standard library with Rust-built .so's

Err codemadness.org 70 i 861

I.e. it statically links all the Rust dependencies. This produces a Err codemadness.org 70 i 862 large .so:

Err codemadness.org 70 i 863

librsvg-2.so (version 2.40.21, C only) - 1408840 bytes
librsvg-2.so (version 2.49.3, Rust only) - 9899120 bytes

Err codemadness.org 70 i 867

Holy crap! What's all that?

Err codemadness.org 70 i 868

(And I'm cheating: this is both with link-time optimization turned on, Err codemadness.org 70 i 869 and by running strip(1) on the .so. If you just autogen.sh && make Err codemadness.org 70 i 870 it will be bigger.)

Err codemadness.org 70 i 871

This has Rust's standard library statically linked (or at least the Err codemadness.org 70 i 872 bits of that librsvg actually uses), plus all the Rust dependencies Err codemadness.org 70 i 873 (cssparser, selectors, nalgebra, glib-rs, cairo-rs, locale_config, Err codemadness.org 70 i 874 rayon, xml5ever, and an assload of crates). I could explain why each Err codemadness.org 70 i 875 one is needed:

Err codemadness.org 70 i 876

cssparser - librsvg needs to parse CSS.
selectors - librsvg needs to run the CSS selector matching Err codemadness.org 70 i 879 algorithm.
nalgebra - the code for SVG filter effects uses vectors and Err codemadness.org 70 i 881 matrices.
glib-rs, cairo-rs - draw to Cairo and export GObject types.
locale_config - so that localized SVG files can work.
rayon - so filters can use all your CPU cores instead of processing Err codemadness.org 70 i 885 one pixel at a time.
Etcetera. SVG is big and requires a lot of helper code!

Err codemadness.org 70 i 888

Is this a problem?

Err codemadness.org 70 i 889

Or more exactly, why does this happen, and why do people perceive it Err codemadness.org 70 i 890 as a problem?

Err codemadness.org 70 i 891

Stable APIs/ABIs and distros

Err codemadness.org 70 i 892

Many Linux distributions have worked really hard to ensure that Err codemadness.org 70 i 893 there is a single copy of "system libraries" in an installation. Err codemadness.org 70 i 894 There is Just One Copy of /usr/lib/libc.so, /usr/lib/libjpeg.so, Err codemadness.org 70 i 895 etc., and packages are compiled with special options to tell them to Err codemadness.org 70 i 896 really use the sytem libraries instead of their bundled versions, or Err codemadness.org 70 i 897 patched to do so if they don't provide build-time options for that.

Err codemadness.org 70 i 898

In a way, this works well for distros:

Err codemadness.org 70 i 899

Err codemadness.org 70 i 901
A bug in a library can be fixed in a single place, and all Err codemadness.org 70 i 902 applications that use it get the fix automatically.
Err codemadness.org 70 i 903
Err codemadness.org 70 i 905
A security bug can be patched in a single place, and in theory Err codemadness.org 70 i 906 applications don't need to be audited further.
Err codemadness.org 70 i 907

Err codemadness.org 70 i 909

If you maintain a library that is shipped in Linux distros, and you Err codemadness.org 70 i 910 break the ABI, you'll get complaints from distros very quickly.

Err codemadness.org 70 i 911

This is good because it creates responsible maintainers for libraries Err codemadness.org 70 i 912 that can be depended on. It's how Inkscape/GIMP can have a stable Err codemadness.org 70 i 913 toolkit to be written in.

Err codemadness.org 70 i 914

This is bad because it encourages stagnation in the long term. It's Err codemadness.org 70 i 915 how we get a horrible, unsafe, error-prone API in libjpeg that can Err codemadness.org 70 i 916 never ever be improved because it would requires changes in tons of Err codemadness.org 70 i 917 software; it's why gboolean is still a 32-bit int after Err codemadness.org 70 i 918 twenty-something years, even though everything else close to C has Err codemadness.org 70 i 919 decided that booleans are 1 byte. It's how Inkscape/GIMP take many Err codemadness.org 70 i 920 years to move from GTK2 to GTK3 (okay, that's lack of paid developers Err codemadness.org 70 i 921 to do the grunt work, but it is enabled by having forever-stable APIs).

Err codemadness.org 70 i 922

However, a long-term stable API/ABI has a lot of value. It is why Err codemadness.org 70 i 923 the Windows API is the crown jewels; it is why people can rely on glib Err codemadness.org 70 i 924 and glibc to not break their code for many years and take them for granted.

Err codemadness.org 70 i 925

But we only have a single stable ABI anyway

Err codemadness.org 70 i 926

And that is the C ABI. Even C++ libraries have trouble with this, and Err codemadness.org 70 i 927 people sometimes write the internals of a library in C++ for Err codemadness.org 70 i 928 convenience, but export a stable C API/ABI from it.

Err codemadness.org 70 i 929

High level languages like Python have real trouble calling C++ code Err codemadness.org 70 i 930 precisely because of ABI issues.

Err codemadness.org 70 i 931

Actually, in GNOME we have gone further than that

Err codemadness.org 70 i 932

In GNOME we have constructed a sweet little universe where GObject Err codemadness.org 70 i 933 Introspection is Err codemadness.org 70 i 934 basically a C ABI with a ton of machine-generated annotations to make Err codemadness.org 70 i 935 it friendly to language bindings.

Err codemadness.org 70 i 936

Still, we rely on a C ABI underneath. See this exploratory twitter Err codemadness.org 70 i 937 thread on advancing the C ABI from Rust for Err codemadness.org 70 i 938 lots of food for thought.

Err codemadness.org 70 i 939

Single copies of libraries with a C ABI

Err codemadness.org 70 i 940

Okay, let's go back to this. What price do we pay for single copies Err codemadness.org 70 i 941 of libraries that, by necessity, must export a C ABI?

Err codemadness.org 70 i 942

Err codemadness.org 70 i 944
Code that can be conveniently called from C, maybe from C++, and Err codemadness.org 70 i 945 moderately to very inconvently from ANYTHING ELSE. With most new Err codemadness.org 70 i 946 application code being written definitely not in C, maybe we should Err codemadness.org 70 i 947 reconsider our priorities here.
Err codemadness.org 70 i 948
Err codemadness.org 70 i 950
No language facilities like generics or field visibility, which are Err codemadness.org 70 i 951 not even "modern language" features. Even C++ templates get Err codemadness.org 70 i 952 compiled and statically linked into the calling code, because Err codemadness.org 70 i 953 there's no way to pass information like the size of T in Err codemadness.org 70 i 954 Array<T> across a C ABI. You wanted to make some struct fields Err codemadness.org 70 i 955 public and some private? You are out of luck.
Err codemadness.org 70 i 956
Err codemadness.org 70 i 958
No knowledge of data ownership except by careful reading of the C Err codemadness.org 70 i 959 function's documentation. Does the function free its arguments? Err codemadness.org 70 i 960 How - with free() or g_free() or my_thing_free()? Or does the Err codemadness.org 70 i 961 caller just lend it a reference? Can the data be copied bit-by-bit Err codemadness.org 70 i 962 or must a special function be called to make a copy? Err codemadness.org 70 i 963 GObject-Introspection carries this information in its annotations, Err codemadness.org 70 i 964 while the C ABI has no idea and just ships raw pointers around.
Err codemadness.org 70 i 965

Err codemadness.org 70 i 967

More food for thought note: this twitter Err codemadness.org 70 i 968 thread says Err codemadness.org 70 i 969 this about the C++ ABI: "Also, the ABI matters for whether the actual Err codemadness.org 70 i 970 level of practicality of complying with LGPL matches the level of Err codemadness.org 70 i 971 practicality intended years ago when some project picked LGPL as its Err codemadness.org 70 i 972 license. Of course, the standard does not talk about LGPL, either. Err codemadness.org 70 i 973 LGPL has rather different implications for Rust and Go than it does Err codemadness.org 70 i 974 for C and Java. It was obviously written with C in mind."

Err codemadness.org 70 i 975

Monomorphization and template bloat

Err codemadness.org 70 i 976

While C++ had the problem of "lots of template code in header files", Err codemadness.org 70 i 977 Rust has the problem that monomorphization of generics creates a lot Err codemadness.org 70 i 978 of compiled Err codemadness.org 70 i 979 code. Err codemadness.org 70 i 980 There are tricks to avoid this and they are all the decision of the Err codemadness.org 70 i 981 library/crate author. Both share the root cause that templated or Err codemadness.org 70 i 982 generic code must be recompiled for every specific use, and thus Err codemadness.org 70 i 983 cannot live in a shared library.

Err codemadness.org 70 i 984

Also, see this wonderful article on how different languages implement Err codemadness.org 70 i 985 generics, Err codemadness.org 70 i 986 and think that a plain C ABI means we have NOTHING of the sort.

Err codemadness.org 70 i 987

Also, see How Swift Achieved Dynamic Linking Where Rust Err codemadness.org 70 i 988 Couldn't for more food for Err codemadness.org 70 i 989 thought. This is extremely roughly equivalent to GObject's boxed Err codemadness.org 70 i 990 types; callers keep values on the heap but know the type layout via Err codemadness.org 70 i 991 annotation magic, while the library's actual implementation Err codemadness.org 70 i 992 is free to have the values on the stack or wherever for its own use.

Err codemadness.org 70 i 993

Should all libraries export APIs with generics and exotic types?

Err codemadness.org 70 i 994

No!

Err codemadness.org 70 i 995

You probably want something like a low-level array of values, Err codemadness.org 70 i 996 Vec<T>, to be inlined everywhere and with code that knows the Err codemadness.org 70 i 997 type of the vector's elements. Element accesses can be inlined to a Err codemadness.org 70 i 998 single machine instruction in the best case.

Err codemadness.org 70 i 999

But not everything requires this absolute raw performance with Err codemadness.org 70 i 1000 everything inlined everywhere. It is fine to pass references or Err codemadness.org 70 i 1001 pointers to things and do dynamic dispatch from a vtable if you are Err codemadness.org 70 i 1002 not in a super-tight loop, as we love to do in the GObject world.

Err codemadness.org 70 i 1003

Library sizes

Err codemadness.org 70 i 1004

I don't have a good answer to librsvg's compiled size. If gnome-shell Err codemadness.org 70 i 1005 merges my branch to rustify the CSS code, it will also grow its binary Err codemadness.org 70 i 1006 size by quite a bit.

Err codemadness.org 70 i 1007

It is my intention to have a Rust crate that both librsvg and Err codemadness.org 70 i 1008 gnome-shell share for their CSS styling needs, but right now I have no Err codemadness.org 70 i 1009 idea if this would be a shared library or just a normal Rust crate. Err codemadness.org 70 i 1010 Maybe it's possible to have a very general CSS library, and the Err codemadness.org 70 i 1011 application registers which properties it can parse and how? Is it Err codemadness.org 70 i 1012 possible to do this as a shared library without essentially Err codemadness.org 70 i 1013 reinventing libcroco? I don't know yet. We'll see.

Err codemadness.org 70 i 1014

A metaphor which I haven't fully explored

Err codemadness.org 70 i 1015

If every application or end-user package is kind of like a living Err codemadness.org 70 i 1016 organism, with its own cycles and behaviors and organs (dependent Err codemadness.org 70 i 1017 libraries) that make it possible...

Err codemadness.org 70 i 1018

Why do distros expect all the living organisms on your machine to Err codemadness.org 70 i 1019 share The World's Single Lungs Service, and The World's Single Stomach Err codemadness.org 70 i 1020 Service, and The World's Single Liver Service?

Err codemadness.org 70 i 1021

You know, instead of letting every organism have its own slightly Err codemadness.org 70 i 1022 different version of those organs, customized for it? We humans know Err codemadness.org 70 i 1023 how to do vaccination campaigns and everything; maybe we need better Err codemadness.org 70 i 1024 tools to apply bug fixes where they are needed?

Err codemadness.org 70 i 1025

I know this metaphor is extremely imperfect and not how things work in Err codemadness.org 70 i 1026 software, but it makes me wonder.

Looking for candidates for the 2020 GNOME Foundation elections

2020-05-26T17:05:31-05:00

I forgot to write this a few days ago; I hope it is not too late.

Err codemadness.org 70 i 1031

The GNOME Foundation's elections for the Board are coming Err codemadness.org 70 i 1032 up, and we are looking for candidates. Of the 7 directors, we are Err codemadness.org 70 i 1033 replacing 4, and the 3 remaining positions remain for another year. Err codemadness.org 70 i 1034 You could be one of those four.

Err codemadness.org 70 i 1035

I would like it very much if there were candidates and directors that Err codemadness.org 70 i 1036 fall outside the box of "white male programmer"; it is unfortunate Err codemadness.org 70 i 1037 that for the current Board we ended up with all dudes. GNOME has a Err codemadness.org 70 i 1038 Code of Conduct to make it a good place to be.

Err codemadness.org 70 i 1039

Allan Day wrote a review of the Board's activies for the last Err codemadness.org 70 i 1040 year. We are moving from a model where the Board does a Err codemadness.org 70 i 1041 little bit of everything, to one with a more strategic role — now that Err codemadness.org 70 i 1042 the Foundation has full-time employees, they take care of most of the Err codemadness.org 70 i 1043 executive work.

Err codemadness.org 70 i 1044

The call-for-candidates is open until May 29, so hurry up!

Err codemadness.org 70 i 1045

GNOME Foundation Board of Directors: a Year in Review
Call for candidates and elections

Bringing my Emacs from the past

2020-04-28T18:03:22-05:00

I started using Emacs in 1995, and since then I have been carrying a .emacs Err codemadness.org 70 i 1052 that by now has a lot of accumulated crap. It is such an old configuration that Err codemadness.org 70 i 1053 it didn't even use the modern convention of ~/.emacs.d/init.el (and it looks Err codemadness.org 70 i 1054 like a newer Emacs version will allow .config/emacs as per the XDG Err codemadness.org 70 i 1055 standard... at last).

Err codemadness.org 70 i 1056

I have wanted to change my Emacs configuration for some time, and give it all Err codemadness.org 70 i 1057 the pretty and modern toys.

Err codemadness.org 70 i 1058

The things that matter the most to me:

Err codemadness.org 70 i 1059

Not have a random dumpster in ~/.emacs if possible.
Pretty colors.
Magit.
Rust-mode or whatever the new thing is for rust-analyzer and the Language Server.

Err codemadness.org 70 i 1065

After looking at several examples of configurations that mention use-package Err codemadness.org 70 i 1066 as a unified way of loading packages and configuring them, I found this Err codemadness.org 70 i 1067 configuration which is extremely well Err codemadness.org 70 i 1068 documented. The author does literate programming with org-mode and elisp — Err codemadness.org 70 i 1069 something which I'm casually interested in, but not just now — but that way Err codemadness.org 70 i 1070 everything ends up very well explained and easy to read.

Err codemadness.org 70 i 1071

I extracted bits of that configuration and ended up with the following.

Err codemadness.org 70 i 1072

Everything in `~/.emacs/init.el` and with `use-package`

Err codemadness.org 70 i 1073

;; Initialize package system	Err	codemadness.org	70
i  1074 	Err	codemadness.org	70
i  1075 (require 'package)	Err	codemadness.org	70
i  1076 	Err	codemadness.org	70
i  1077 (setq package-archives	Err	codemadness.org	70
i  1078       '(("org"     .       "https://orgmode.org/elpa/")	Err	codemadness.org	70
i  1079         ("gnu"     .       "https://elpa.gnu.org/packages/")	Err	codemadness.org	70
i  1080         ("melpa"   .       "https://melpa.org/packages/")))	Err	codemadness.org	70
i  1081 	Err	codemadness.org	70
i  1082 (package-initialize)	Err	codemadness.org	70
i  1083 ;(package-refresh-contents)	Err	codemadness.org	70
i  1084 	Err	codemadness.org	70
i  1085 ;; Use-package for civilized configuration	Err	codemadness.org	70
i  1086 	Err	codemadness.org	70
i  1087 (unless (package-installed-p 'use-package)	Err	codemadness.org	70
i  1088   (package-install 'use-package))	Err	codemadness.org	70
i  1089 (require 'use-package)	Err	codemadness.org	70
i  1090 	Err	codemadness.org	70
i  1091 (setq use-package-always-ensure t)	Err	codemadness.org	70
i  1092

Err codemadness.org 70 i 1093 Err codemadness.org 70 i 1094

`~/.emacs.d/custom.el` for `M-x customize` stuff

Err codemadness.org 70 i 1095

;; Set customization data in a specific file, without littering	Err	codemadness.org	70
i  1096 ;; my init files.	Err	codemadness.org	70
i  1097 	Err	codemadness.org	70
i  1098 (setq custom-file "~/.emacs.d/custom.el")	Err	codemadness.org	70
i  1099 (load custom-file)	Err	codemadness.org	70
i  1100

Err codemadness.org 70 i 1101 Err codemadness.org 70 i 1102

Which-key to get hints when typing command prefixes

Err codemadness.org 70 i 1103

;; Make it easier to discover key shortcuts	Err	codemadness.org	70
i  1104 	Err	codemadness.org	70
i  1105 (use-package which-key	Err	codemadness.org	70
i  1106   :diminish	Err	codemadness.org	70
i  1107   :config	Err	codemadness.org	70
i  1108   (which-key-mode)	Err	codemadness.org	70
i  1109   (which-key-setup-side-window-bottom)	Err	codemadness.org	70
i  1110   (setq which-key-idle-delay 0.1))	Err	codemadness.org	70
i  1111

Err codemadness.org 70 i 1112 Err codemadness.org 70 i 1113

Don't pollute the modeline with common modes

Err codemadness.org 70 i 1114

;; Do not show some common modes in the modeline, to save space	Err	codemadness.org	70
i  1115 	Err	codemadness.org	70
i  1116 (use-package diminish	Err	codemadness.org	70
i  1117   :defer 5	Err	codemadness.org	70
i  1118   :config	Err	codemadness.org	70
i  1119   (diminish 'org-indent-mode))	Err	codemadness.org	70
i  1120

Err codemadness.org 70 i 1121 Err codemadness.org 70 i 1122

Magit to use git in a civilized fashion

Err codemadness.org 70 i 1123

;; Magit	Err	codemadness.org	70
i  1124 	Err	codemadness.org	70
i  1125 (use-package magit	Err	codemadness.org	70
i  1126   :config	Err	codemadness.org	70
i  1127   (global-set-key (kbd "C-x g") 'magit-status))	Err	codemadness.org	70
i  1128

Err codemadness.org 70 i 1129 Err codemadness.org 70 i 1130

Move between windows with Shift-arrows

Err codemadness.org 70 i 1131

;; Let me switch windows with shift-arrows instead of "C-x o" all the time	Err	codemadness.org	70
i  1132 (windmove-default-keybindings)	Err	codemadness.org	70
i  1133

Err codemadness.org 70 i 1134 Err codemadness.org 70 i 1135

Pretty colors

Err codemadness.org 70 i 1136

I was using solarized-dark but I like this one even better:

Err codemadness.org 70 i 1137

;; Pretty colors	Err	codemadness.org	70
i  1138 	Err	codemadness.org	70
i  1139 (use-package flatland-theme	Err	codemadness.org	70
i  1140   :config	Err	codemadness.org	70
i  1141   (custom-theme-set-faces 'flatland	Err	codemadness.org	70
i  1142    '(show-paren-match ((t (:background "dark gray" :foreground "black" :weight bold))))	Err	codemadness.org	70
i  1143    '(show-paren-mismatch ((t (:background "firebrick" :foreground "orange" :weight bold))))))	Err	codemadness.org	70
i  1144

Err codemadness.org 70 i 1145 Err codemadness.org 70 i 1146

Nyan cat instead of scrollbars

Err codemadness.org 70 i 1147

;; Nyan cat instead of scrollbar	Err	codemadness.org	70
i  1148 ;; scroll-bar-mode is turned off in custom.el	Err	codemadness.org	70
i  1149 	Err	codemadness.org	70
i  1150 (use-package nyan-mode	Err	codemadness.org	70
i  1151   :config	Err	codemadness.org	70
i  1152   (nyan-mode 1))	Err	codemadness.org	70
i  1153

Err codemadness.org 70 i 1154 Err codemadness.org 70 i 1155

Move buffers to adjacent windows

Err codemadness.org 70 i 1156

;; Move buffers between windows	Err	codemadness.org	70
i  1157 	Err	codemadness.org	70
i  1158 (use-package buffer-move	Err	codemadness.org	70
i  1159   :config	Err	codemadness.org	70
i  1160   (global-set-key (kbd "<C-S-up>")     'buf-move-up)	Err	codemadness.org	70
i  1161   (global-set-key (kbd "<C-S-down>")   'buf-move-down)	Err	codemadness.org	70
i  1162   (global-set-key (kbd "<C-S-left>")   'buf-move-left)	Err	codemadness.org	70
i  1163   (global-set-key (kbd "<C-S-right>")  'buf-move-right))	Err	codemadness.org	70
i  1164

Err codemadness.org 70 i 1165 Err codemadness.org 70 i 1166

Change buffer names for files with the same name

Err codemadness.org 70 i 1167

;; Note that ‘uniquify’ is builtin.	Err	codemadness.org	70
i  1168 (require 'uniquify)	Err	codemadness.org	70
i  1169 (setq uniquify-separator "/"               ;; The separator in buffer names.	Err	codemadness.org	70
i  1170       uniquify-buffer-name-style 'forward) ;; names/in/this/style	Err	codemadness.org	70
i  1171

Err codemadness.org 70 i 1172 Err codemadness.org 70 i 1173

Helm to auto-complete in grand style

Err codemadness.org 70 i 1174

(use-package helm	Err	codemadness.org	70
i  1175  :diminish	Err	codemadness.org	70
i  1176  :init (helm-mode t)	Err	codemadness.org	70
i  1177  :bind (("M-x"     . helm-M-x)	Err	codemadness.org	70
i  1178         ("C-x C-f" . helm-find-files)	Err	codemadness.org	70
i  1179         ("C-x b"   . helm-mini)     ;; See buffers & recent files; more useful.	Err	codemadness.org	70
i  1180         ("C-x r b" . helm-filtered-bookmarks)	Err	codemadness.org	70
i  1181         ("C-x C-r" . helm-recentf)  ;; Search for recently edited files	Err	codemadness.org	70
i  1182         ("C-c i"   . helm-imenu)	Err	codemadness.org	70
i  1183         ("C-h a"   . helm-apropos)	Err	codemadness.org	70
i  1184         ;; Look at what was cut recently & paste it in.	Err	codemadness.org	70
i  1185         ("M-y" . helm-show-kill-ring)	Err	codemadness.org	70
i  1186 	Err	codemadness.org	70
i  1187         :map helm-map	Err	codemadness.org	70
i  1188         ;; We can list ‘actions’ on the currently selected item by C-z.	Err	codemadness.org	70
i  1189         ("C-z" . helm-select-action)	Err	codemadness.org	70
i  1190         ;; Let's keep tab-completetion anyhow.	Err	codemadness.org	70
i  1191         ("TAB"   . helm-execute-persistent-action)	Err	codemadness.org	70
i  1192         ("<tab>" . helm-execute-persistent-action)))	Err	codemadness.org	70
i  1193

Err codemadness.org 70 i 1194 Err codemadness.org 70 i 1195

Ripgrep to search in grand style

Err codemadness.org 70 i 1196

;; Ripgrep	Err	codemadness.org	70
i  1197 	Err	codemadness.org	70
i  1198 (use-package rg	Err	codemadness.org	70
i  1199   :config	Err	codemadness.org	70
i  1200   (global-set-key (kbd "M-s g") 'rg)	Err	codemadness.org	70
i  1201   (global-set-key (kbd "M-s d") 'rg-dwim))	Err	codemadness.org	70
i  1202 	Err	codemadness.org	70
i  1203 (use-package helm-rg)	Err	codemadness.org	70
i  1204

Err codemadness.org 70 i 1205 Err codemadness.org 70 i 1206

Rust mode and Language Server

Err codemadness.org 70 i 1207

Now that RLS is in the process of being deprecated, it's getting substituted Err codemadness.org 70 i 1208 with rust-analyzer. Also, rust-mode goes away in favor of rustic.

Err codemadness.org 70 i 1209

;; Rustic, LSP	Err	codemadness.org	70
i  1210 	Err	codemadness.org	70
i  1211 (use-package flycheck)	Err	codemadness.org	70
i  1212 	Err	codemadness.org	70
i  1213 (use-package rustic)	Err	codemadness.org	70
i  1214 	Err	codemadness.org	70
i  1215 (use-package lsp-ui)	Err	codemadness.org	70
i  1216 	Err	codemadness.org	70
i  1217 (use-package helm-lsp	Err	codemadness.org	70
i  1218   :config	Err	codemadness.org	70
i  1219   (define-key lsp-mode-map [remap xref-find-apropos] #'helm-lsp-workspace-symbol))	Err	codemadness.org	70
i  1220

Err codemadness.org 70 i 1221 Err codemadness.org 70 i 1222

Performatively not get distracted

Err codemadness.org 70 i 1223

;;; Show a notification when compilation finishes	Err	codemadness.org	70
i  1224 	Err	codemadness.org	70
i  1225 (setq compilation-finish-functions	Err	codemadness.org	70
i  1226       (append compilation-finish-functions	Err	codemadness.org	70
i  1227           '(fmq-compilation-finish)))	Err	codemadness.org	70
i  1228 	Err	codemadness.org	70
i  1229 (defun fmq-compilation-finish (buffer status)	Err	codemadness.org	70
i  1230   (when (not (member mode-name '("Grep" "rg")))	Err	codemadness.org	70
i  1231     (call-process "notify-send" nil nil nil	Err	codemadness.org	70
i  1232           "-t" "0"	Err	codemadness.org	70
i  1233           "-i" "emacs"	Err	codemadness.org	70
i  1234           "Compilation finished in Emacs"	Err	codemadness.org	70
i  1235           status)))	Err	codemadness.org	70
i  1236

Err codemadness.org 70 i 1237 Err codemadness.org 70 i 1238

Stuff from custom.el

Err codemadness.org 70 i 1239

The interesting bits here are making LSP work; everything else is preferences.

Err codemadness.org 70 i 1240

(custom-set-variables	Err	codemadness.org	70
i  1241  ;; custom-set-variables was added by Custom.	Err	codemadness.org	70
i  1242  ;; If you edit it by hand, you could mess it up, so be careful.	Err	codemadness.org	70
i  1243  ;; Your init file should contain only one such instance.	Err	codemadness.org	70
i  1244  ;; If there is more than one, they won't work right.	Err	codemadness.org	70
i  1245  '(column-number-mode t)	Err	codemadness.org	70
i  1246  '(custom-safe-themes	Err	codemadness.org	70
i  1247    (quote	Err	codemadness.org	70
i  1248     ("2540689fd0bc5d74c4682764ff6c94057ba8061a98be5dd21116bf7bf301acfb" "bffa9739ce0752a37d9b1eee78fc00ba159748f50dc328af4be661484848e476" "0fffa9669425ff140ff2ae8568c7719705ef33b7a927a0ba7c5e2ffcfac09b75" "2809bcb77ad21312897b541134981282dc455ccd7c14d74cc333b6e549b824f3" default)))	Err	codemadness.org	70
i  1249  '(delete-selection-mode nil)	Err	codemadness.org	70
i  1250  '(lsp-rust-analyzer-display-chaining-hints t)	Err	codemadness.org	70
i  1251  '(lsp-rust-analyzer-display-parameter-hints nil)	Err	codemadness.org	70
i  1252  '(lsp-rust-analyzer-macro-expansion-method (quote rustic-analyzer-macro-expand))	Err	codemadness.org	70
i  1253  '(lsp-rust-analyzer-server-command (quote ("/home/federico/.cargo/bin/rust-analyzer")))	Err	codemadness.org	70
i  1254  '(lsp-rust-analyzer-server-display-inlay-hints nil)	Err	codemadness.org	70
i  1255  '(lsp-rust-full-docs t)	Err	codemadness.org	70
i  1256  '(lsp-rust-server (quote rust-analyzer))	Err	codemadness.org	70
i  1257  '(lsp-ui-doc-alignment (quote window))	Err	codemadness.org	70
i  1258  '(lsp-ui-doc-position (quote top))	Err	codemadness.org	70
i  1259  '(lsp-ui-sideline-enable nil)	Err	codemadness.org	70
i  1260  '(menu-bar-mode nil)	Err	codemadness.org	70
i  1261  '(package-selected-packages	Err	codemadness.org	70
i  1262    (quote	Err	codemadness.org	70
i  1263     (helm-lsp lsp-ui lsp-mode flycheck rustic rg helm-rg ripgrep helm-projectile helm buffer-move nyan-mode flatland-black-theme flatland-theme afternoon-theme spacemacs-theme solarized-theme magit diminish which-key use-package)))	Err	codemadness.org	70
i  1264  '(rustic-lsp-server (quote rust-analyzer))	Err	codemadness.org	70
i  1265  '(scroll-bar-mode nil)	Err	codemadness.org	70
i  1266  '(scroll-step 0)	Err	codemadness.org	70
i  1267  '(tool-bar-mode nil))	Err	codemadness.org	70
i  1268 (custom-set-faces	Err	codemadness.org	70
i  1269  ;; custom-set-faces was added by Custom.	Err	codemadness.org	70
i  1270  ;; If you edit it by hand, you could mess it up, so be careful.	Err	codemadness.org	70
i  1271  ;; Your init file should contain only one such instance.	Err	codemadness.org	70
i  1272  ;; If there is more than one, they won't work right.	Err	codemadness.org	70
i  1273  )	Err	codemadness.org	70
i  1274

Err codemadness.org 70 i 1275 Err codemadness.org 70 i 1276

Results

Err codemadness.org 70 i 1277

I am very happy with rustic / rust-analyzer and the Language Server. Having Err codemadness.org 70 i 1278 documentation on each thing when one moves the cursor around code is something Err codemadness.org 70 i 1279 that I never thought would work well in Emacs. I haven't decided if I love M-x Err codemadness.org 70 i 1280 lsp-rust-analyzer-inlay-hints-mode or if it drives me nuts; it shows you the Err codemadness.org 70 i 1281 names of function arguments and inferred types among the code. I suppose I'll Err codemadness.org 70 i 1282 turn it off and on as needed.

Err codemadness.org 70 i 1283

Some days ago, before using helm, I had projectile-mode to work with git Err codemadness.org 70 i 1284 checkouts and I was quite liking it. I haven't found how to configure Err codemadness.org 70 i 1285 helm-projectile to work; I'll have to keep experimenting.

Reducing memory consumption in librsvg, part 4: compact representation for Bézier paths

2020-03-26T18:58:36-06:00

Let's continue with the enormous SVG from the last time, a Err codemadness.org 70 i 1292 map extracted from OpenStreetMap.

Err codemadness.org 70 i 1293

According to Massif, peak memory consumption for that file occurs at Err codemadness.org 70 i 1294 the following point during the execution of rsvg-convert. I pasted Err codemadness.org 70 i 1295 only the part that refers to Bézier paths:

Err codemadness.org 70 i 1296

    --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1297       n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)	Err	codemadness.org	70
i  1298     --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1299 1    33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0	Err	codemadness.org	70
i  1300 2   ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:84)	Err	codemadness.org	70
i  1301     | ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:172)	Err	codemadness.org	70
i  1302     |   ->24.88% (352,523,448B) 0x4A2727E: allocate_in<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:98)	Err	codemadness.org	70
i  1303     |     ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (raw_vec.rs:167)	Err	codemadness.org	70
i  1304     |       ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (vec.rs:358)	Err	codemadness.org	70
i  1305     |         ->24.88% (352,523,448B) 0x4A2727E: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter (vec.rs:1992)	Err	codemadness.org	70
i  1306     |           ->24.88% (352,523,448B) 0x49D212C: from_iter<rsvg_internals::path_builder::PathCommand,smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>> (vec.rs:1901)	Err	codemadness.org	70
i  1307     |             ->24.88% (352,523,448B) 0x49D212C: collect<smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>,alloc::vec::Vec<rsvg_internals::path_builder::PathCommand>> (iterator.rs:1493)	Err	codemadness.org	70
i  1308     |               ->24.88% (352,523,448B) 0x49D212C: into_vec<[rsvg_internals::path_builder::PathCommand; 32]> (lib.rs:893)	Err	codemadness.org	70
i  1309     |                 ->24.88% (352,523,448B) 0x49D212C: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)	Err	codemadness.org	70
i  1310 3   |                   ->24.88% (352,523,016B) 0x4A0394C: into_path (path_builder.rs:320)	Err	codemadness.org	70
i  1311     |	Err	codemadness.org	70
i  1312 4   ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:128)	Err	codemadness.org	70
i  1313     | ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:187)	Err	codemadness.org	70
i  1314     |   ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:633)	Err	codemadness.org	70
i  1315     |     ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand> (vec.rs:623)	Err	codemadness.org	70
i  1316     |       ->03.60% (50,990,328B) 0x4A242F0: alloc::vec::Vec<T>::into_boxed_slice (vec.rs:679)	Err	codemadness.org	70
i  1317     |         ->03.60% (50,990,328B) 0x49D2136: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)	Err	codemadness.org	70
i  1318 5   |           ->03.60% (50,990,328B) 0x4A0394C: into_path (path_builder.rs:320)	Err	codemadness.org	70
i  1319

Err codemadness.org 70 i 1320 Err codemadness.org 70 i 1321

Line 1 has the totals, and we see that at that point the program uses Err codemadness.org 70 i 1322 1,329,943,212 bytes on the heap.

Err codemadness.org 70 i 1323

Lines 3 and 5 give us a hint that into_path is being called; this is Err codemadness.org 70 i 1324 the function that converts a temporary/mutable PathBuilder into a Err codemadness.org 70 i 1325 permanent/immutable Path.

Err codemadness.org 70 i 1326

Lines 2 and 4 indicate that the arrays of PathCommand, which are Err codemadness.org 70 i 1327 inside those immutable Paths, use 24.88% + 3.60% = 28.48% of the Err codemadness.org 70 i 1328 program's memory; between both they use Err codemadness.org 70 i 1329 352,523,448 + 50,990,328 = 403,513,776 bytes.

Err codemadness.org 70 i 1330

That is about 400 MB of PathCommand. Let's see what's going on.

Err codemadness.org 70 i 1331

What is in a PathCommand?

Err codemadness.org 70 i 1332

A Path is a list of commands similar to PostScript, which get used Err codemadness.org 70 i 1333 in SVG to draw Bézier paths. It is a flat array of PathCommand:

Err codemadness.org 70 i 1334

pub struct Path {	Err	codemadness.org	70
i  1335     path_commands: Box<[PathCommand]>,	Err	codemadness.org	70
i  1336 }	Err	codemadness.org	70
i  1337 	Err	codemadness.org	70
i  1338 pub enum PathCommand {	Err	codemadness.org	70
i  1339     MoveTo(f64, f64),	Err	codemadness.org	70
i  1340     LineTo(f64, f64),	Err	codemadness.org	70
i  1341     CurveTo(CubicBezierCurve),	Err	codemadness.org	70
i  1342     Arc(EllipticalArc),	Err	codemadness.org	70
i  1343     ClosePath,	Err	codemadness.org	70
i  1344 }	Err	codemadness.org	70
i  1345

Err codemadness.org 70 i 1346 Err codemadness.org 70 i 1347

Let's see the variants of PathCommand:

Err codemadness.org 70 i 1348

MoveTo: 2 double-precision floating-point numbers.
LineTo: same.
CurveTo: 6 double-precision floating-point numbers.
EllipticalArc: 7 double-precision floating-point numbers, plus 2 Err codemadness.org 70 i 1353 flags (see below).
ClosePath: no extra data.

Err codemadness.org 70 i 1356

These variants vary a lot in terms of size, and each element of the Err codemadness.org 70 i 1357 Path.path_commands array occupies the maximum of their sizes Err codemadness.org 70 i 1358 (i.e. sizeof::<EllipticalArc>).

Err codemadness.org 70 i 1359

A more compact representation

Err codemadness.org 70 i 1360

Ideally, each command in the array would only occupy as much space as Err codemadness.org 70 i 1361 it needs.

Err codemadness.org 70 i 1362

We can represent a Path in a different way, as two separate arrays:

Err codemadness.org 70 i 1363

A very compact array of commands without coordinates.
An array with coordinates only.

Err codemadness.org 70 i 1367

That is, the following:

Err codemadness.org 70 i 1368

pub struct Path {	Err	codemadness.org	70
i  1369     commands: Box<[PackedCommand]>,	Err	codemadness.org	70
i  1370     coords: Box<[f64]>,	Err	codemadness.org	70
i  1371 }	Err	codemadness.org	70
i  1372

Err codemadness.org 70 i 1373 Err codemadness.org 70 i 1374

The coords array is obvious; it is just a flat array with all the Err codemadness.org 70 i 1375 coordinates in the Path in the order in which they appear.

Err codemadness.org 70 i 1376

And the commands array?

Err codemadness.org 70 i 1377

PackedCommand

Err codemadness.org 70 i 1378

We saw above that the biggest variant in PathCommand is Err codemadness.org 70 i 1379 Arc(EllipticalArc). Let's look inside it:

Err codemadness.org 70 i 1380

pub struct EllipticalArc {	Err	codemadness.org	70
i  1381     pub r: (f64, f64),	Err	codemadness.org	70
i  1382     pub x_axis_rotation: f64,	Err	codemadness.org	70
i  1383     pub large_arc: LargeArc,	Err	codemadness.org	70
i  1384     pub sweep: Sweep,	Err	codemadness.org	70
i  1385     pub from: (f64, f64),	Err	codemadness.org	70
i  1386     pub to: (f64, f64),	Err	codemadness.org	70
i  1387 }	Err	codemadness.org	70
i  1388

Err codemadness.org 70 i 1389 Err codemadness.org 70 i 1390

There are 7 f64 floating-point numbers there. The other two fields, Err codemadness.org 70 i 1391 large_arc and sweep, are effectively booleans (they are just enums Err codemadness.org 70 i 1392 with two variants, with pretty names instead of just true and Err codemadness.org 70 i 1393 false).

Err codemadness.org 70 i 1394

Thus, we have 7 doubles and two flags. Between the two flags there Err codemadness.org 70 i 1395 are 4 possibilities.

Err codemadness.org 70 i 1396

Since no other PathCommand variant has flags, we can have the Err codemadness.org 70 i 1397 following enum, which fits in a single byte:

Err codemadness.org 70 i 1398

#[repr(u8)]	Err	codemadness.org	70
i  1399 enum PackedCommand {	Err	codemadness.org	70
i  1400     MoveTo,	Err	codemadness.org	70
i  1401     LineTo,	Err	codemadness.org	70
i  1402     CurveTo,	Err	codemadness.org	70
i  1403     ArcSmallNegative,	Err	codemadness.org	70
i  1404     ArcSmallPositive,	Err	codemadness.org	70
i  1405     ArcLargeNegative,	Err	codemadness.org	70
i  1406     ArcLargePositive,	Err	codemadness.org	70
i  1407     ClosePath,	Err	codemadness.org	70
i  1408 }	Err	codemadness.org	70
i  1409

Err codemadness.org 70 i 1410 Err codemadness.org 70 i 1411

That is, simple values for MoveTo/etc. and four special values for Err codemadness.org 70 i 1412 the different types of Arc.

Err codemadness.org 70 i 1413

Packing a PathCommand into a PackedCommand

Err codemadness.org 70 i 1414

In order to pack the array of PathCommand, we must first know how Err codemadness.org 70 i 1415 many coordinates each of its variants will produce:

Err codemadness.org 70 i 1416

impl PathCommand {	Err	codemadness.org	70
i  1417     fn num_coordinates(&self) -> usize {	Err	codemadness.org	70
i  1418         match *self {	Err	codemadness.org	70
i  1419             PathCommand::MoveTo(..) => 2,	Err	codemadness.org	70
i  1420             PathCommand::LineTo(..) => 2,	Err	codemadness.org	70
i  1421             PathCommand::CurveTo(_) => 6,	Err	codemadness.org	70
i  1422             PathCommand::Arc(_) => 7,	Err	codemadness.org	70
i  1423             PathCommand::ClosePath => 0,	Err	codemadness.org	70
i  1424         }	Err	codemadness.org	70
i  1425     }	Err	codemadness.org	70
i  1426 }	Err	codemadness.org	70
i  1427

Err codemadness.org 70 i 1428 Err codemadness.org 70 i 1429

Then, we need to convert each PathCommand into a PackedCommand and Err codemadness.org 70 i 1430 write its coordinates into an array:

Err codemadness.org 70 i 1431

impl PathCommand {	Err	codemadness.org	70
i  1432     fn to_packed(&self, coords: &mut [f64]) -> PackedCommand {	Err	codemadness.org	70
i  1433         match *self {	Err	codemadness.org	70
i  1434             PathCommand::MoveTo(x, y) => {	Err	codemadness.org	70
i  1435                 coords[0] = x;	Err	codemadness.org	70
i  1436                 coords[1] = y;	Err	codemadness.org	70
i  1437                 PackedCommand::MoveTo	Err	codemadness.org	70
i  1438             }	Err	codemadness.org	70
i  1439 	Err	codemadness.org	70
i  1440             // etc. for the other simple commands	Err	codemadness.org	70
i  1441 	Err	codemadness.org	70
i  1442             PathCommand::Arc(ref a) => a.to_packed_and_coords(coords),	Err	codemadness.org	70
i  1443         }	Err	codemadness.org	70
i  1444     }	Err	codemadness.org	70
i  1445 }	Err	codemadness.org	70
i  1446

Err codemadness.org 70 i 1447 Err codemadness.org 70 i 1448

Let's look at that to_packed_and_coords more closely:

Err codemadness.org 70 i 1449

impl EllipticalArc {	Err	codemadness.org	70
i  1450     fn to_packed_and_coords(&self, coords: &mut [f64]) -> PackedCommand {	Err	codemadness.org	70
i  1451         coords[0] = self.r.0;	Err	codemadness.org	70
i  1452         coords[1] = self.r.1;	Err	codemadness.org	70
i  1453         coords[2] = self.x_axis_rotation;	Err	codemadness.org	70
i  1454         coords[3] = self.from.0;	Err	codemadness.org	70
i  1455         coords[4] = self.from.1;	Err	codemadness.org	70
i  1456         coords[5] = self.to.0;	Err	codemadness.org	70
i  1457         coords[6] = self.to.1;	Err	codemadness.org	70
i  1458 	Err	codemadness.org	70
i  1459         match (self.large_arc, self.sweep) {	Err	codemadness.org	70
i  1460             (LargeArc(false), Sweep::Negative) => PackedCommand::ArcSmallNegative,	Err	codemadness.org	70
i  1461             (LargeArc(false), Sweep::Positive) => PackedCommand::ArcSmallPositive,	Err	codemadness.org	70
i  1462             (LargeArc(true), Sweep::Negative) => PackedCommand::ArcLargeNegative,	Err	codemadness.org	70
i  1463             (LargeArc(true), Sweep::Positive) => PackedCommand::ArcLargePositive,	Err	codemadness.org	70
i  1464         }	Err	codemadness.org	70
i  1465     }	Err	codemadness.org	70
i  1466 }	Err	codemadness.org	70
i  1467

Err codemadness.org 70 i 1468 Err codemadness.org 70 i 1469

Creating the compact Path

Err codemadness.org 70 i 1470

Let's look at PathBuilder::into_path line by line:

Err codemadness.org 70 i 1471

impl PathBuilder {	Err	codemadness.org	70
i  1472     pub fn into_path(self) -> Path {	Err	codemadness.org	70
i  1473         let num_commands = self.path_commands.len();	Err	codemadness.org	70
i  1474         let num_coords = self	Err	codemadness.org	70
i  1475             .path_commands	Err	codemadness.org	70
i  1476             .iter()	Err	codemadness.org	70
i  1477             .fold(0, |acc, cmd| acc + cmd.num_coordinates());	Err	codemadness.org	70
i  1478

Err codemadness.org 70 i 1479 Err codemadness.org 70 i 1480

First we compute the total number of coordinates using fold; we ask Err codemadness.org 70 i 1481 each command cmd its num_coordinates() and add it into the acc Err codemadness.org 70 i 1482 accumulator.

Err codemadness.org 70 i 1483

Now we know how much memory to allocate:

Err codemadness.org 70 i 1484

        let mut packed_commands = Vec::with_capacity(num_commands);	Err	codemadness.org	70
i  1485         let mut coords = vec![0.0; num_coords];	Err	codemadness.org	70
i  1486

Err codemadness.org 70 i 1487 Err codemadness.org 70 i 1488

We use Vec::with_capacity to allocate exactly as much memory as we will Err codemadness.org 70 i 1489 need for the packed_commands; adding elements will not need a Err codemadness.org 70 i 1490 realloc(), since we already know how many elements we will have.

Err codemadness.org 70 i 1491

We use the vec! macro to create an array of 0.0 repeated Err codemadness.org 70 i 1492 num_coords times; that macro uses with_capacity internally. That is the Err codemadness.org 70 i 1493 array we will use to store the coordinates for all the commands.

Err codemadness.org 70 i 1494

        let mut coords_slice = coords.as_mut_slice();	Err	codemadness.org	70
i  1495

Err codemadness.org 70 i 1496 Err codemadness.org 70 i 1497

We get a mutable slice out of the whole array of coordinates.

Err codemadness.org 70 i 1498

        for c in self.path_commands {	Err	codemadness.org	70
i  1499             let n = c.num_coordinates();	Err	codemadness.org	70
i  1500             packed_commands.push(c.to_packed(coords_slice.get_mut(0..n).unwrap()));	Err	codemadness.org	70
i  1501             coords_slice = &mut coords_slice[n..];	Err	codemadness.org	70
i  1502         }	Err	codemadness.org	70
i  1503

Err codemadness.org 70 i 1504 Err codemadness.org 70 i 1505

For each command, we see how many coordinates it will generate and we Err codemadness.org 70 i 1506 put that number in n. We get a mutable sub-slice from Err codemadness.org 70 i 1507 coords_slice with only that number of elements, and pass it to Err codemadness.org 70 i 1508 to_packed for each command.

Err codemadness.org 70 i 1509

At the end of each iteration we move the mutable slice to where the Err codemadness.org 70 i 1510 next command's coordinates will go.

Err codemadness.org 70 i 1511

        Path {	Err	codemadness.org	70
i  1512             commands: packed_commands.into_boxed_slice(),	Err	codemadness.org	70
i  1513             coords: coords.into_boxed_slice(),	Err	codemadness.org	70
i  1514         }	Err	codemadness.org	70
i  1515     }	Err	codemadness.org	70
i  1516

Err codemadness.org 70 i 1517 Err codemadness.org 70 i 1518

At the end, we create the final and immutable Path by converting Err codemadness.org 70 i 1519 each array into_boxed_slice like the last time. That way each of Err codemadness.org 70 i 1520 the two arrays, the one with PackedCommands and the one with Err codemadness.org 70 i 1521 coordinates, occupy the minimum space they need.

Err codemadness.org 70 i 1522

An iterator for Path

Err codemadness.org 70 i 1523

This is all very well, but we also want it to be easy to iterate on Err codemadness.org 70 i 1524 that compact representation; the PathCommand enums from the Err codemadness.org 70 i 1525 beginning are very convenient to use and that's what the rest of the Err codemadness.org 70 i 1526 code already uses. Let's make an iterator that unpacks what is inside Err codemadness.org 70 i 1527 a Path and produces a PathCommand for each element.

Err codemadness.org 70 i 1528

pub struct PathIter<'a> {	Err	codemadness.org	70
i  1529     commands: slice::Iter<'a, PackedCommand>,	Err	codemadness.org	70
i  1530     coords: &'a [f64],	Err	codemadness.org	70
i  1531 }	Err	codemadness.org	70
i  1532

Err codemadness.org 70 i 1533 Err codemadness.org 70 i 1534

We need an iterator over the array of PackedCommand so we can visit Err codemadness.org 70 i 1535 each command. However, to get elements of coords, I am going to Err codemadness.org 70 i 1536 use a slice of f64 instead of an iterator.

Err codemadness.org 70 i 1537

Let's look at the implementation of the iterator:

Err codemadness.org 70 i 1538

impl<'a> Iterator for PathIter<'a> {	Err	codemadness.org	70
i  1539     type Item = PathCommand;	Err	codemadness.org	70
i  1540 	Err	codemadness.org	70
i  1541     fn next(&mut self) -> Option<Self::Item> {	Err	codemadness.org	70
i  1542         if let Some(cmd) = self.commands.next() {	Err	codemadness.org	70
i  1543             let cmd = PathCommand::from_packed(cmd, self.coords);	Err	codemadness.org	70
i  1544             let num_coords = cmd.num_coordinates();	Err	codemadness.org	70
i  1545             self.coords = &self.coords[num_coords..];	Err	codemadness.org	70
i  1546             Some(cmd)	Err	codemadness.org	70
i  1547         } else {	Err	codemadness.org	70
i  1548             None	Err	codemadness.org	70
i  1549         }	Err	codemadness.org	70
i  1550     }	Err	codemadness.org	70
i  1551 }	Err	codemadness.org	70
i  1552

Err codemadness.org 70 i 1553 Err codemadness.org 70 i 1554

Since we want each iteration to produce a PathCommand, we declare it Err codemadness.org 70 i 1555 as having the associated type Item = PathCommand.

Err codemadness.org 70 i 1556

If the self.commands iterator has another element, it means there is Err codemadness.org 70 i 1557 another PackedCommand available.

Err codemadness.org 70 i 1558

We call PathCommand::from_packed with the self.coords slice to Err codemadness.org 70 i 1559 unpack a command and its coordinates. We see how many coordinates the Err codemadness.org 70 i 1560 command consumed and re-slice self.coords according to the number of Err codemadness.org 70 i 1561 commands, so that it now points to the coordinates for the next Err codemadness.org 70 i 1562 command.

Err codemadness.org 70 i 1563

We return Some(cmd) if there was an element, or None if the Err codemadness.org 70 i 1564 iterator is empty.

Err codemadness.org 70 i 1565

The implementation of from_packed is obvious and I'll just paste a Err codemadness.org 70 i 1566 bit from it:

Err codemadness.org 70 i 1567

impl PathCommand {	Err	codemadness.org	70
i  1568     fn from_packed(packed: &PackedCommand, coords: &[f64]) -> PathCommand {	Err	codemadness.org	70
i  1569         match *packed {	Err	codemadness.org	70
i  1570             PackedCommand::MoveTo => {	Err	codemadness.org	70
i  1571                 let x = coords[0];	Err	codemadness.org	70
i  1572                 let y = coords[1];	Err	codemadness.org	70
i  1573                 PathCommand::MoveTo(x, y)	Err	codemadness.org	70
i  1574             }	Err	codemadness.org	70
i  1575 	Err	codemadness.org	70
i  1576             // etc. for the other variants in PackedCommand	Err	codemadness.org	70
i  1577 	Err	codemadness.org	70
i  1578             PackedCommand::ArcSmallNegative => PathCommand::Arc(EllipticalArc::from_coords(	Err	codemadness.org	70
i  1579                 LargeArc(false),	Err	codemadness.org	70
i  1580                 Sweep::Negative,	Err	codemadness.org	70
i  1581                 coords,	Err	codemadness.org	70
i  1582             )),	Err	codemadness.org	70
i  1583 	Err	codemadness.org	70
i  1584             PackedCommand::ArcSmallPositive => // etc.	Err	codemadness.org	70
i  1585 	Err	codemadness.org	70
i  1586             PackedCommand::ArcLargeNegative => // etc.	Err	codemadness.org	70
i  1587 	Err	codemadness.org	70
i  1588             PackedCommand::ArcLargePositive => // etc.	Err	codemadness.org	70
i  1589         }	Err	codemadness.org	70
i  1590     }	Err	codemadness.org	70
i  1591 }	Err	codemadness.org	70
i  1592

Err codemadness.org 70 i 1593 Err codemadness.org 70 i 1594

Results

Err codemadness.org 70 i 1595

Before the changes (this is the same Massif heading as above):

Err codemadness.org 70 i 1596

--------------------------------------------------------------------------------	Err	codemadness.org	70
i  1597   n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)	Err	codemadness.org	70
i  1598 --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1599  33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0	Err	codemadness.org	70
i  1600                                        ^^^^^^^^^^^^^	Err	codemadness.org	70
i  1601                                            boo	Err	codemadness.org	70
i  1602

Err codemadness.org 70 i 1603 Err codemadness.org 70 i 1604

After:

Err codemadness.org 70 i 1605

--------------------------------------------------------------------------------	Err	codemadness.org	70
i  1606   n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)	Err	codemadness.org	70
i  1607 --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1608  28 26,611,886,993    1,093,747,888    1,023,147,907    70,599,981            0	Err	codemadness.org	70
i  1609                                        ^^^^^^^^^^^^^	Err	codemadness.org	70
i  1610                                           oh yeah	Err	codemadness.org	70
i  1611

Err codemadness.org 70 i 1612 Err codemadness.org 70 i 1613

We went from using 1,329,943,212 bytes down to 1,023,147,907 bytes, Err codemadness.org 70 i 1614 that is, we knocked it down by 300 MB.

Err codemadness.org 70 i 1615

However, that is for the whole program. Above we saw that Path data Err codemadness.org 70 i 1616 occupies 403,513,776 bytes; how about now?

Err codemadness.org 70 i 1617

->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:84)	Err	codemadness.org	70
i  1618 | ->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:172)	Err	codemadness.org	70
i  1619 |   ->07.45% (81,525,328B) 0x4A34C6F: allocate_in<f64,alloc::alloc::Global> (raw_vec.rs:98)	Err	codemadness.org	70
i  1620 |     ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (raw_vec.rs:167)	Err	codemadness.org	70
i  1621 |       ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (vec.rs:358)	Err	codemadness.org	70
i  1622 |         ->07.45% (81,525,328B) 0x4A34C6F: rsvg_internals::path_builder::PathBuilder::into_path (path_builder.rs:486)	Err	codemadness.org	70
i  1623

Err codemadness.org 70 i 1624 Err codemadness.org 70 i 1625

Perfect. We went from occupying 403,513,776 bytes to just Err codemadness.org 70 i 1626 81,525,328 bytes. Instead of Path data amounting to 28.48% of the Err codemadness.org 70 i 1627 heap, it is just 7.45%.

Err codemadness.org 70 i 1628

I think we can stop worrying about Path data for now. I like how Err codemadness.org 70 i 1629 this turned out without having to use unsafe.

Err codemadness.org 70 i 1630

References

Err codemadness.org 70 i 1631

Refactoring to use an iterator
Adding tests for Path/PathBuilder
Using a more compact representation for Path

Reducing memory consumption in librsvg, part 3: slack space in Bézier paths

2020-03-24T17:14:55-06:00

We got a bug with a gigantic SVG of a map extracted from Err codemadness.org 70 i 1643 OpenStreetMap, and it has about 600,000 elements. Most of them are Err codemadness.org 70 i 1644 <path>, that is, specifications for Bézier paths.

Err codemadness.org 70 i 1645

A <path> can look like this:

Err codemadness.org 70 i 1646

<path d="m 2239.05,1890.28 5.3,-1.81"/>	Err	codemadness.org	70
i  1647

Err codemadness.org 70 i 1648 Err codemadness.org 70 i 1649

The d attribute contains a list of commands to Err codemadness.org 70 i 1650 create a Bézier path, very similar to PostScript's operators. Librsvg Err codemadness.org 70 i 1651 has the following to represent those commands:

Err codemadness.org 70 i 1652

pub enum PathCommand {	Err	codemadness.org	70
i  1653     MoveTo(f64, f64),	Err	codemadness.org	70
i  1654     LineTo(f64, f64),	Err	codemadness.org	70
i  1655     CurveTo(CubicBezierCurve),	Err	codemadness.org	70
i  1656     Arc(EllipticalArc),	Err	codemadness.org	70
i  1657     ClosePath,	Err	codemadness.org	70
i  1658 }	Err	codemadness.org	70
i  1659

Err codemadness.org 70 i 1660 Err codemadness.org 70 i 1661

Those commands get stored in an array, a Vec inside a PathBuilder:

Err codemadness.org 70 i 1662

pub struct PathBuilder {	Err	codemadness.org	70
i  1663     path_commands: Vec<PathCommand>,	Err	codemadness.org	70
i  1664 }	Err	codemadness.org	70
i  1665

Err codemadness.org 70 i 1666 Err codemadness.org 70 i 1667

Librsvg translates each of the commands inside a <path d="..."/> Err codemadness.org 70 i 1668 into a PathCommand and pushes it into the Vec in the Err codemadness.org 70 i 1669 PathBuilder. When it is done parsing the attribute, the Err codemadness.org 70 i 1670 PathBuilder remains as the final version of the path.

Err codemadness.org 70 i 1671

To let a Vec grow efficiently as items are pushed into Err codemadness.org 70 i 1672 it, Rust makes the Vec grow by powers of 2. When we add an item, if Err codemadness.org 70 i 1673 the capacity of the Vec is full, its buffer gets realloc()ed to Err codemadness.org 70 i 1674 twice its capacity. That way there are only O(log₂n) calls to Err codemadness.org 70 i 1675 realloc(), where n is the total number of items in the array.

Err codemadness.org 70 i 1676

However, this means that once we are done adding items to the Vec, Err codemadness.org 70 i 1677 there may still be some free space in it: the capacity exceeds the Err codemadness.org 70 i 1678 length of the array. The invariant is that Err codemadness.org 70 i 1679 vec.capacity() >= vec.len().

Err codemadness.org 70 i 1680

First I wanted to shrink the PathBuilders so that they have no extra Err codemadness.org 70 i 1681 capacity in the end.

Err codemadness.org 70 i 1682

First step: convert to Box<[T]>

Err codemadness.org 70 i 1683

A "boxed slice" is a contiguous array in the heap, that cannot grow or Err codemadness.org 70 i 1684 shrink. That is, it has no extra capacity, only a length.

Err codemadness.org 70 i 1685

Vec has a method into_boxed_slice which does Err codemadness.org 70 i 1686 eactly that: it consumes the vector and converts it into a boxed Err codemadness.org 70 i 1687 slice without extra capacity. In its innards, it does a realloc() Err codemadness.org 70 i 1688 on the Vec's buffer to match its length.

Err codemadness.org 70 i 1689

Let's see the numbers that Massif reports:

Err codemadness.org 70 i 1690

--------------------------------------------------------------------------------	Err	codemadness.org	70
i  1691   n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)	Err	codemadness.org	70
i  1692 --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1693  23 22,751,613,855    1,560,916,408    1,493,746,540    67,169,868            0	Err	codemadness.org	70
i  1694                                        ^^^^^^^^^^^^^	Err	codemadness.org	70
i  1695                                            before	Err	codemadness.org	70
i  1696 	Err	codemadness.org	70
i  1697  30 22,796,106,012    1,553,581,072    1,329,943,324   223,637,748            0	Err	codemadness.org	70
i  1698                                        ^^^^^^^^^^^^^	Err	codemadness.org	70
i  1699                                            after	Err	codemadness.org	70
i  1700

Err codemadness.org 70 i 1701 Err codemadness.org 70 i 1702

That is, we went from using 1,493,746,540 bytes on the heap to using Err codemadness.org 70 i 1703 1,329,943,324 bytes. Simply removing extra capacity from the path Err codemadness.org 70 i 1704 commands saves about 159 MB for this particular file.

Err codemadness.org 70 i 1705

Second step: make the allocator do less work

Err codemadness.org 70 i 1706

However, the extra-heap column in that table has a number I don't Err codemadness.org 70 i 1707 like: there are 223,637,748 bytes in malloc() metadata and unused Err codemadness.org 70 i 1708 space in the heap.

Err codemadness.org 70 i 1709

I suppose that so many calls to realloc() make the heap a bit Err codemadness.org 70 i 1710 fragmented.

Err codemadness.org 70 i 1711

It would be good to be able to read most of the <path d="..."/> to Err codemadness.org 70 i 1712 temporary buffers that don't need so many calls to realloc(), and Err codemadness.org 70 i 1713 that in the end get copied to exact-sized buffers, without extra Err codemadness.org 70 i 1714 capacity.

Err codemadness.org 70 i 1715

We can do just that with the smallvec crate. A SmallVec has the Err codemadness.org 70 i 1716 same API as Vec, but it can store small arrays directly in the Err codemadness.org 70 i 1717 stack, without an extra heap allocation. Once the capacity is full, Err codemadness.org 70 i 1718 the stack buffer "spills" into a heap buffer automatically.

Err codemadness.org 70 i 1719

Most of the d attributes in the huge file in the bug have Err codemadness.org 70 i 1720 fewer than 32 commands. That is, if we use the following:

Err codemadness.org 70 i 1721

pub struct PathBuilder {	Err	codemadness.org	70
i  1722     path_commands: SmallVec<[PathCommand; 32]>,	Err	codemadness.org	70
i  1723 }	Err	codemadness.org	70
i  1724

Err codemadness.org 70 i 1725 Err codemadness.org 70 i 1726

We are saying that there can be up to 32 items in the SmallVec Err codemadness.org 70 i 1727 without causing a heap allocation; once that is exceeded, it will work Err codemadness.org 70 i 1728 like a normal Vec.

Err codemadness.org 70 i 1729

At the end we still do into_boxed_slice to turn it into an Err codemadness.org 70 i 1730 independent heap allocation with an exact size.

Err codemadness.org 70 i 1731

This reduces the extra-heap quite a bit:

Err codemadness.org 70 i 1732

--------------------------------------------------------------------------------	Err	codemadness.org	70
i  1733   n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)	Err	codemadness.org	70
i  1734 --------------------------------------------------------------------------------	Err	codemadness.org	70
i  1735  33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0	Err	codemadness.org	70
i  1736                                                         ^^^^^^^^^^	Err	codemadness.org	70
i  1737

Err codemadness.org 70 i 1738 Err codemadness.org 70 i 1739

Also, the total bytes shrink from 1,553,581,072 to Err codemadness.org 70 i 1740 1,416,831,176 — we have a smaller heap because there is not so much Err codemadness.org 70 i 1741 work for the allocator, and there are a lot fewer temporary blocks Err codemadness.org 70 i 1742 when parsing the d attributes.

Err codemadness.org 70 i 1743

Making the code prettier

Err codemadness.org 70 i 1744

I put in the following:

Err codemadness.org 70 i 1745

/// This one is mutable	Err	codemadness.org	70
i  1746 pub struct PathBuilder {	Err	codemadness.org	70
i  1747     path_commands: SmallVec<[PathCommand; 32]>,	Err	codemadness.org	70
i  1748 }	Err	codemadness.org	70
i  1749 	Err	codemadness.org	70
i  1750 /// This one is immutable	Err	codemadness.org	70
i  1751 pub struct Path {	Err	codemadness.org	70
i  1752     path_commands: Box<[PathCommand]>,	Err	codemadness.org	70
i  1753 }	Err	codemadness.org	70
i  1754 	Err	codemadness.org	70
i  1755 impl PathBuilder {	Err	codemadness.org	70
i  1756     /// Consumes the PathBuilder and converts it into an immutable Path	Err	codemadness.org	70
i  1757     pub fn into_path(self) -> Path {	Err	codemadness.org	70
i  1758         Path {	Err	codemadness.org	70
i  1759             path_commands: self.path_commands.into_boxed_slice(),	Err	codemadness.org	70
i  1760         }	Err	codemadness.org	70
i  1761     }	Err	codemadness.org	70
i  1762 }	Err	codemadness.org	70
i  1763

Err codemadness.org 70 i 1764 Err codemadness.org 70 i 1765

With that, PathBuilder is just a temporary struct that turns into an Err codemadness.org 70 i 1766 immutable Path once we are done feeding it. Path contains a boxed Err codemadness.org 70 i 1767 slice of the exact size, without any extra capacity.

Err codemadness.org 70 i 1768

Next steps

Err codemadness.org 70 i 1769

All the coordinates in librsvg are stored as f64, double-precision Err codemadness.org 70 i 1770 floating point numbers. The SVG/CSS spec says that single-precision Err codemadness.org 70 i 1771 floats are enough, and that 64-bit floats should be used only for Err codemadness.org 70 i 1772 geometric transformations.

Err codemadness.org 70 i 1773

I'm a bit scared to make that change; I'll have to look closely at the Err codemadness.org 70 i 1774 results of the test suite to see if rendered files change very much. Err codemadness.org 70 i 1775 I suppose even big maps require only as much precision as f32 — Err codemadness.org 70 i 1776 after all, that is what OpenStreetMap uses.

Err codemadness.org 70 i 1777

References

Err codemadness.org 70 i 1778

Convert the Vec into a Err codemadness.org 70 i 1780 Box<[T]>
Convert to SmallVec

Reducing memory consumption in librsvg, part 2: SpecifiedValues

2020-03-20T14:29:16-06:00

To continue with last time's topic, let's see how to make Err codemadness.org 70 i 1787 librsvg's DOM nodes smaller in memory. Since that time, there have Err codemadness.org 70 i 1788 been some changes to the code; that is why in this post some of the Err codemadness.org 70 i 1789 type names are different from last time's.

Err codemadness.org 70 i 1790

Every SVG element is represented with this struct:

Err codemadness.org 70 i 1791

pub struct Element {	Err	codemadness.org	70
i  1792     element_type: ElementType,	Err	codemadness.org	70
i  1793     element_name: QualName,	Err	codemadness.org	70
i  1794     id: Option<String>,	Err	codemadness.org	70
i  1795     class: Option<String>,	Err	codemadness.org	70
i  1796     specified_values: SpecifiedValues,	Err	codemadness.org	70
i  1797     important_styles: HashSet<QualName>,	Err	codemadness.org	70
i  1798     result: ElementResult,	Err	codemadness.org	70
i  1799     transform: Transform,	Err	codemadness.org	70
i  1800     values: ComputedValues,	Err	codemadness.org	70
i  1801     cond: bool,	Err	codemadness.org	70
i  1802     style_attr: String,	Err	codemadness.org	70
i  1803     element_impl: Box<dyn ElementTrait>,	Err	codemadness.org	70
i  1804 }	Err	codemadness.org	70
i  1805

Err codemadness.org 70 i 1806 Err codemadness.org 70 i 1807

The two biggest fields are the ones with types SpecifiedValues and Err codemadness.org 70 i 1808 ComputedValues. These are the sizes of the whole Element struct Err codemadness.org 70 i 1809 and those two types:

Err codemadness.org 70 i 1810

sizeof Element: 1808	Err	codemadness.org	70
i  1811 sizeof SpecifiedValues: 824	Err	codemadness.org	70
i  1812 sizeof ComputedValues: 704	Err	codemadness.org	70
i  1813

Err codemadness.org 70 i 1814 Err codemadness.org 70 i 1815

In this post, we'll reduce the size of SpecifiedValues.

Err codemadness.org 70 i 1816

What is SpecifiedValues?

Err codemadness.org 70 i 1817

If we have an element like this:

Err codemadness.org 70 i 1818

<circle cx="10" cy="10" r="10" stroke-width="4" stroke="blue"/>	Err	codemadness.org	70
i  1819

Err codemadness.org 70 i 1820 Err codemadness.org 70 i 1821

The values of the style properties stroke-width and stroke get Err codemadness.org 70 i 1822 stored in a SpecifiedValues struct. This struct has a bunch of Err codemadness.org 70 i 1823 fields, one for each possible style property:

Err codemadness.org 70 i 1824

pub struct SpecifiedValues {	Err	codemadness.org	70
i  1825     baseline_shift:              SpecifiedValue<BaselineShift>,	Err	codemadness.org	70
i  1826     clip_path:                   SpecifiedValue<ClipPath>,	Err	codemadness.org	70
i  1827     clip_rule:                   SpecifiedValue<ClipRule>,	Err	codemadness.org	70
i  1828     /// ...	Err	codemadness.org	70
i  1829     stroke:                      SpecifiedValue<Stroke>,	Err	codemadness.org	70
i  1830     stroke_width:                SpecifiedValue<StrokeWidth>,	Err	codemadness.org	70
i  1831     /// ...	Err	codemadness.org	70
i  1832 }	Err	codemadness.org	70
i  1833

Err codemadness.org 70 i 1834 Err codemadness.org 70 i 1835

Each field is a SpecifiedValue<T> for the following reason. In Err codemadness.org 70 i 1836 CSS/SVG, a style property can be unspecified, or it can have an Err codemadness.org 70 i 1837 inherit value to force the property to be copied from the element's Err codemadness.org 70 i 1838 parent, or it can actually have a specified value. Librsvg represents Err codemadness.org 70 i 1839 these as follows:

Err codemadness.org 70 i 1840

pub enum SpecifiedValue<T>	Err	codemadness.org	70
i  1841 where	Err	codemadness.org	70
i  1842     T: // some trait bounds here	Err	codemadness.org	70
i  1843 {	Err	codemadness.org	70
i  1844     Unspecified,	Err	codemadness.org	70
i  1845     Inherit,	Err	codemadness.org	70
i  1846     Specified(T),	Err	codemadness.org	70
i  1847 }	Err	codemadness.org	70
i  1848

Err codemadness.org 70 i 1849 Err codemadness.org 70 i 1850

Now, SpecifiedValues has a bunch of fields, 47 of them to be exact — Err codemadness.org 70 i 1851 one for each of the style properties that librsvg supports. That is Err codemadness.org 70 i 1852 why SpecifiedValues has a size of 824 bytes; it is the largest Err codemadness.org 70 i 1853 sub-structure within Element, and it would be good to reduce its Err codemadness.org 70 i 1854 size.

Err codemadness.org 70 i 1855

Not all properties are specified

Err codemadness.org 70 i 1856

Let's go back to the chunk of SVG from above:

Err codemadness.org 70 i 1857

<circle cx="10" cy="10" r="10" stroke-width="4" stroke="blue"/>	Err	codemadness.org	70
i  1858

Err codemadness.org 70 i 1859 Err codemadness.org 70 i 1860

Here we only have two specified properties, so the stroke_width and Err codemadness.org 70 i 1861 stroke fields of SpecifiedValues will be set as Err codemadness.org 70 i 1862 SpecifiedValue::Specified(something) and all the other fields will Err codemadness.org 70 i 1863 be left as SpecifiedValue::Unspecified.

Err codemadness.org 70 i 1864

It would be good to store only complete values for the properties that Err codemadness.org 70 i 1865 are specified, and just a small flag for unset properties.

Err codemadness.org 70 i 1866

Another way to represent the set of properties

Err codemadness.org 70 i 1867

Since there is a maximum of 47 properties per element (or more if Err codemadness.org 70 i 1868 librsvg adds support for extra ones), we can have a small array of Err codemadness.org 70 i 1869 47 bytes. Each byte contains the index within another array that Err codemadness.org 70 i 1870 contains only the values of specified properties, or a sentinel value Err codemadness.org 70 i 1871 for properties that are unset.

Err codemadness.org 70 i 1872

First, I made an enum that fits in a u8 for all the properties, plus Err codemadness.org 70 i 1873 the sentinel value, which also gives us the total number of Err codemadness.org 70 i 1874 properties. The #[repr(u8)] guarantees that this enum fits in a Err codemadness.org 70 i 1875 byte.

Err codemadness.org 70 i 1876

#[repr(u8)]	Err	codemadness.org	70
i  1877 enum PropertyId {	Err	codemadness.org	70
i  1878     BaselineShift,	Err	codemadness.org	70
i  1879     ClipPath,	Err	codemadness.org	70
i  1880     ClipRule,	Err	codemadness.org	70
i  1881     Color,	Err	codemadness.org	70
i  1882     // ...	Err	codemadness.org	70
i  1883     WritingMode,	Err	codemadness.org	70
i  1884     XmlLang,	Err	codemadness.org	70
i  1885     XmlSpace,	Err	codemadness.org	70
i  1886     UnsetProperty, // the number of properties and also the sentinel value	Err	codemadness.org	70
i  1887 }	Err	codemadness.org	70
i  1888

Err codemadness.org 70 i 1889 Err codemadness.org 70 i 1890

Also, since before these changes there was the following monster to Err codemadness.org 70 i 1891 represent "which property is this" plus the property's value:

Err codemadness.org 70 i 1892

pub enum ParsedProperty {	Err	codemadness.org	70
i  1893     BaselineShift(SpecifiedValue<BaselineShift>),	Err	codemadness.org	70
i  1894     ClipPath(SpecifiedValue<ClipPath>),	Err	codemadness.org	70
i  1895     ClipRule(SpecifiedValue<ClipRule>),	Err	codemadness.org	70
i  1896     Color(SpecifiedValue<Color>),	Err	codemadness.org	70
i  1897     // ...	Err	codemadness.org	70
i  1898 }	Err	codemadness.org	70
i  1899

Err codemadness.org 70 i 1900 Err codemadness.org 70 i 1901

I changed the definition of SpecifiedValues to have two arrays, one Err codemadness.org 70 i 1902 to store which properties are specified, and another only with the Err codemadness.org 70 i 1903 values for the properties that are actually specified:

Err codemadness.org 70 i 1904

pub struct SpecifiedValues {	Err	codemadness.org	70
i  1905     indices: [u8; PropertyId::UnsetProperty as usize],	Err	codemadness.org	70
i  1906     props: Vec<ParsedProperty>,	Err	codemadness.org	70
i  1907 }	Err	codemadness.org	70
i  1908

Err codemadness.org 70 i 1909 Err codemadness.org 70 i 1910

There is a thing that is awkward in Rust, or which I haven't found how Err codemadness.org 70 i 1911 to solve in a nicer way: given a ParsedProperty, find the Err codemadness.org 70 i 1912 corresponding PropertyId for its discriminant. I did the obvious Err codemadness.org 70 i 1913 thing:

Err codemadness.org 70 i 1914

impl ParsedProperty {	Err	codemadness.org	70
i  1915     fn get_property_id(&self) -> PropertyId {	Err	codemadness.org	70
i  1916         use ParsedProperty::*;	Err	codemadness.org	70
i  1917 	Err	codemadness.org	70
i  1918         match *self {	Err	codemadness.org	70
i  1919             BaselineShift(_) => PropertyId::BaselineShift,	Err	codemadness.org	70
i  1920             ClipPath(_)      => PropertyId::ClipPath,	Err	codemadness.org	70
i  1921             ClipRule(_)      => PropertyId::ClipRule,	Err	codemadness.org	70
i  1922             Color(_)         => PropertyId::Color,	Err	codemadness.org	70
i  1923             // ...	Err	codemadness.org	70
i  1924         }	Err	codemadness.org	70
i  1925     }	Err	codemadness.org	70
i  1926 }	Err	codemadness.org	70
i  1927

Err codemadness.org 70 i 1928 Err codemadness.org 70 i 1929

Initialization

Err codemadness.org 70 i 1930

First, we want to initialize an empty SpecifiedValues, where every Err codemadness.org 70 i 1931 element of the the indices array is set to the sentinel value that Err codemadness.org 70 i 1932 means that the corresponding property is not set:

Err codemadness.org 70 i 1933

impl Default for SpecifiedValues {	Err	codemadness.org	70
i  1934     fn default() -> Self {	Err	codemadness.org	70
i  1935         SpecifiedValues {	Err	codemadness.org	70
i  1936             indices: [PropertyId::UnsetProperty.as_u8(); PropertyId::UnsetProperty as usize],	Err	codemadness.org	70
i  1937             props: Vec::new(),	Err	codemadness.org	70
i  1938         }	Err	codemadness.org	70
i  1939     }	Err	codemadness.org	70
i  1940 }	Err	codemadness.org	70
i  1941

Err codemadness.org 70 i 1942 Err codemadness.org 70 i 1943

That sets the indices field to an array full of the same Err codemadness.org 70 i 1944 PropertyId::UnsetProperty sentinel value. Also, the props array Err codemadness.org 70 i 1945 is empty; it hasn't even had a block of memory allocated for it yet. Err codemadness.org 70 i 1946 That way, SVG elements without style properties don't use any extra Err codemadness.org 70 i 1947 memory.

Err codemadness.org 70 i 1948

Which properties are specified and what are their indices?

Err codemadness.org 70 i 1949

Second, we want a function that will give us the index in props for Err codemadness.org 70 i 1950 some property, or that will tell us if the property has not been set Err codemadness.org 70 i 1951 yet:

Err codemadness.org 70 i 1952

impl SpecifiedValues {	Err	codemadness.org	70
i  1953     fn property_index(&self, id: PropertyId) -> Option<usize> {	Err	codemadness.org	70
i  1954         let v = self.indices[id.as_usize()];	Err	codemadness.org	70
i  1955 	Err	codemadness.org	70
i  1956         if v == PropertyId::UnsetProperty.as_u8() {	Err	codemadness.org	70
i  1957             None	Err	codemadness.org	70
i  1958         } else {	Err	codemadness.org	70
i  1959             Some(v as usize)	Err	codemadness.org	70
i  1960         }	Err	codemadness.org	70
i  1961     }	Err	codemadness.org	70
i  1962 }	Err	codemadness.org	70
i  1963

Err codemadness.org 70 i 1964 Err codemadness.org 70 i 1965

(If someone passes id = PropertyId::UnsetProperty, the array access Err codemadness.org 70 i 1966 to indices will panic, which is what we want, since that is not a Err codemadness.org 70 i 1967 valid property id.)

Err codemadness.org 70 i 1968

Change a property's value

Err codemadness.org 70 i 1969

Third, we want to set the value of a property that has not been set, Err codemadness.org 70 i 1970 or change the value of one that was already specified:

Err codemadness.org 70 i 1971

impl SpecifiedValues {	Err	codemadness.org	70
i  1972     fn replace_property(&mut self, prop: &ParsedProperty) {	Err	codemadness.org	70
i  1973         let id = prop.get_property_id();	Err	codemadness.org	70
i  1974 	Err	codemadness.org	70
i  1975         if let Some(index) = self.property_index(id) {	Err	codemadness.org	70
i  1976             self.props[index] = prop.clone();	Err	codemadness.org	70
i  1977         } else {	Err	codemadness.org	70
i  1978             self.props.push(prop.clone());	Err	codemadness.org	70
i  1979             let pos = self.props.len() - 1;	Err	codemadness.org	70
i  1980             self.indices[id.as_usize()] = pos as u8;	Err	codemadness.org	70
i  1981         }	Err	codemadness.org	70
i  1982     }	Err	codemadness.org	70
i  1983 }	Err	codemadness.org	70
i  1984

Err codemadness.org 70 i 1985 Err codemadness.org 70 i 1986

In the first case in the if, the property was already set and we Err codemadness.org 70 i 1987 just replace its value. In the second case, the property was not set; Err codemadness.org 70 i 1988 we add it to the props array and store its resulting index in Err codemadness.org 70 i 1989 indices.

Err codemadness.org 70 i 1990

Results

Err codemadness.org 70 i 1991

Before:

Err codemadness.org 70 i 1992

sizeof Element: 1808	Err	codemadness.org	70
i  1993 sizeof SpecifiedValues: 824	Err	codemadness.org	70
i  1994

Err codemadness.org 70 i 1995 Err codemadness.org 70 i 1996

After:

Err codemadness.org 70 i 1997

sizeof Element: 1056	Err	codemadness.org	70
i  1998 sizeof SpecifiedValues: 72	Err	codemadness.org	70
i  1999

Err codemadness.org 70 i 2000 Err codemadness.org 70 i 2001

The pathological file from the last time used Err codemadness.org 70 i 2002 463,412,720 bytes in memory before these changes. After the changes, Err codemadness.org 70 i 2003 it uses 314,526,136 bytes.

Err codemadness.org 70 i 2004

I also measured memory consumption for a normal file, in this case Err codemadness.org 70 i 2005 one with a bunch of GNOME's symbolic icons. The old version Err codemadness.org 70 i 2006 uses 17 MB; the new version only 13 MB.

Err codemadness.org 70 i 2007

How to keep fine-tuning this

Err codemadness.org 70 i 2008

For now, I am satisfied with SpecifiedValues, although it could Err codemadness.org 70 i 2009 still be made smaller:

Err codemadness.org 70 i 2010

Err codemadness.org 70 i 2012
The crate tagged-box converts an enum like ParsedProperty into Err codemadness.org 70 i 2013 an enum-of-boxes, and codifies the enum's discriminant into the Err codemadness.org 70 i 2014 box's pointer. This way each variant occupies the minimum possible Err codemadness.org 70 i 2015 memory, although in a separately-allocated block, and the container Err codemadness.org 70 i 2016 itself uses only a pointer. I am not sure if this is worth it; each Err codemadness.org 70 i 2017 ParsedProperty is 64 bytes, but the flat array props: Err codemadness.org 70 i 2018 Vec<ParsedProperty> is very appealing in a single block of memory. Err codemadness.org 70 i 2019 I have not checked the sizes of each individual property to see if Err codemadness.org 70 i 2020 they vary a lot among them.
Err codemadness.org 70 i 2021
Err codemadness.org 70 i 2023
Look for a crate that lets us have the properties in a single memory Err codemadness.org 70 i 2024 block, a kind of arena with variable types. This can be implemented Err codemadness.org 70 i 2025 with a bit of unsafe, but one has to be careful with the alignment Err codemadness.org 70 i 2026 of different types.
Err codemadness.org 70 i 2027
Err codemadness.org 70 i 2029
The crate enum_set2 represents an array of field-less enums as a Err codemadness.org 70 i 2030 compact bit array. If we changed the representation of Err codemadness.org 70 i 2031 SpecifiedValue, this would reduce the indices array to a Err codemadness.org 70 i 2032 minimum.
Err codemadness.org 70 i 2033

Err codemadness.org 70 i 2035

If someone wants to dedicate some time to implement and measure this, Err codemadness.org 70 i 2036 I would be very grateful.

Err codemadness.org 70 i 2037

Next steps

Err codemadness.org 70 i 2038

According to Massif, the next thing is to keep making Element Err codemadness.org 70 i 2039 smaller. The next thing to shrink is ComputedValues. The obvious Err codemadness.org 70 i 2040 route is to do exactly the same as I did for SpecifiedValues. I am Err codemadness.org 70 i 2041 not sure if it would be better to try to share the style Err codemadness.org 70 i 2042 structs between elements.

Librsvg accepting interns for Summer of Code 2020

2020-03-16T17:53:17-06:00

Are you a student qualified to run for Summer of Code 2020? I'm Err codemadness.org 70 i 2051 willing to mentor the following project for librsvg.

Err codemadness.org 70 i 2052

Project: Revamp the text engine in librsvg

Err codemadness.org 70 i 2053

Librsvg supports only a few features of the SVG Text Err codemadness.org 70 i 2054 specification. It requires Err codemadness.org 70 i 2055 extra features to be really useful:

Err codemadness.org 70 i 2056

Err codemadness.org 70 i 2058
Proper bidirectional support. Librsvg supports the direction and Err codemadness.org 70 i 2059 unicode-bidi properties for text elements, among others, but in a Err codemadness.org 70 i 2060 very rudimentary fashion. It just translates those properties to Err codemadness.org 70 i 2061 Pango terminology and asks PangoLayout to lay out the text. SVG Err codemadness.org 70 i 2062 really wants finer control of that, for which...
Err codemadness.org 70 i 2063
Err codemadness.org 70 i 2065
... ideally you would make librsvg use Harfbuzz directly, or a Err codemadness.org 70 i 2066 wrapper that is close to its level of operation. Pango is a bit too high Err codemadness.org 70 i 2067 level for the needs of SVG.
Err codemadness.org 70 i 2068
Err codemadness.org 70 i 2070
Manual layout of text glyphs. After a text engine like Harfbuzz Err codemadness.org 70 i 2071 does the shaping, librsvg would need to lay out the produced glyphs in Err codemadness.org 70 i 2072 the way of the SVG attributes dx, dy, x, y, etc. The SVG Text Err codemadness.org 70 i 2073 specification has the algorithms for this.
Err codemadness.org 70 i 2074
Err codemadness.org 70 i 2076
The cherry on top: text-on-a-path. Again, the spec has the details. Err codemadness.org 70 i 2077 You would make Wikimedia content creators very happy with this!
Err codemadness.org 70 i 2078

Err codemadness.org 70 i 2080

Requirements: Rust for programming language; some familiarity with Err codemadness.org 70 i 2081 Unicode concepts and text layout. Familiarity with Cairo and Harfbuzz Err codemadness.org 70 i 2082 would help a lot. Preference will be given to people who can write a Err codemadness.org 70 i 2083 right-to-left human language, or a language that requires complex Err codemadness.org 70 i 2084 shaping.

Err codemadness.org 70 i 2085

Details for students

Reducing memory consumption in librsvg, part 1: text nodes

2020-03-12T18:57:35-06:00

Librsvg's memory consumption has not been a problem so far for GNOME's Err codemadness.org 70 i 2091 use cases, which is basically rendering icons. But for SVG files with Err codemadness.org 70 i 2092 thousands of elements, it could do a lot better.

Err codemadness.org 70 i 2093

Memory consumption in the DOM

Err codemadness.org 70 i 2094

Librsvg shares some common problems with web browsers: it must Err codemadness.org 70 i 2095 construct a DOM tree in memory with SVG elements, and keep a bunch of Err codemadness.org 70 i 2096 information for each of the tree's nodes. For example, each SVG Err codemadness.org 70 i 2097 element may have an id attribute, or a class; each one has a Err codemadness.org 70 i 2098 transformation matrix; etc.

Err codemadness.org 70 i 2099

Apart from the tree node metadata (pointers to sibling and parent Err codemadness.org 70 i 2100 nodes), each node has this:

Err codemadness.org 70 i 2101

/// Contents of a tree node	Err	codemadness.org	70
i  2102 pub struct NodeData {	Err	codemadness.org	70
i  2103     node_type: NodeType,	Err	codemadness.org	70
i  2104     element_name: QualName,	Err	codemadness.org	70
i  2105     id: Option<String>,    // id attribute from XML element	Err	codemadness.org	70
i  2106     class: Option<String>, // class attribute from XML element	Err	codemadness.org	70
i  2107     specified_values: SpecifiedValues,	Err	codemadness.org	70
i  2108     important_styles: HashSet<QualName>,	Err	codemadness.org	70
i  2109     result: NodeResult,	Err	codemadness.org	70
i  2110     transform: Transform,	Err	codemadness.org	70
i  2111     values: ComputedValues,	Err	codemadness.org	70
i  2112     cond: bool,	Err	codemadness.org	70
i  2113     style_attr: String,	Err	codemadness.org	70
i  2114 	Err	codemadness.org	70
i  2115     node_impl: Box<dyn NodeTrait>, // concrete struct for node types	Err	codemadness.org	70
i  2116 }	Err	codemadness.org	70
i  2117

Err codemadness.org 70 i 2118 Err codemadness.org 70 i 2119

On a 64-bit box, that NodeData struct is 1808 bytes. And the biggest fields Err codemadness.org 70 i 2120 are the SpecifiedValues (824 bytes) and ComputedValues (704 bytes).

Err codemadness.org 70 i 2121

Librsvg represents all tree nodes with that struct. Consider an SVG Err codemadness.org 70 i 2122 like this:

Err codemadness.org 70 i 2123

<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100">	Err	codemadness.org	70
i  2124   <rect x="10" y="20"/>	Err	codemadness.org	70
i  2125   <path d="..."/>	Err	codemadness.org	70
i  2126   <text x="10" y="20">Hello</text>	Err	codemadness.org	70
i  2127   <!-- etc -->	Err	codemadness.org	70
i  2128 </svg>	Err	codemadness.org	70
i  2129

Err codemadness.org 70 i 2130 Err codemadness.org 70 i 2131

There are 4 elements in that file. However, there are also tree nodes Err codemadness.org 70 i 2132 for the XML text nodes, that is, the whitespace between tags and the Err codemadness.org 70 i 2133 "Hello" inside the <text> element.

Err codemadness.org 70 i 2134

The contents of each of those text nodes is tiny (a newline and maybe Err codemadness.org 70 i 2135 a couple of spaces), but each node still takes up at least 1808 bytes Err codemadness.org 70 i 2136 from the NodeData struct, plus the size of the text string.

Err codemadness.org 70 i 2137

Let's refactor this to make it easier to remove that overhead.

Err codemadness.org 70 i 2138

First step: separate text nodes from element nodes

Err codemadness.org 70 i 2139

Internally, librsvg represents XML text nodes with a NodeChars struct Err codemadness.org 70 i 2140 which is basically a string with some extra stuff. All the concrete Err codemadness.org 70 i 2141 structs for tree node types must implement a trait called NodeTrait, Err codemadness.org 70 i 2142 and NodeChars is no exception:

Err codemadness.org 70 i 2143

pub struct NodeChars {	Err	codemadness.org	70
i  2144    // a string with the text node's contents	Err	codemadness.org	70
i  2145 }	Err	codemadness.org	70
i  2146 	Err	codemadness.org	70
i  2147 impl NodeTrait for NodeChars {	Err	codemadness.org	70
i  2148    // a mostly empty impl with methods that do nothing	Err	codemadness.org	70
i  2149 }	Err	codemadness.org	70
i  2150

Err codemadness.org 70 i 2151 Err codemadness.org 70 i 2152

You don't see it in the definition of NodeData in the previous Err codemadness.org 70 i 2153 section, but for a text node, the NodeData.node_impl field would Err codemadness.org 70 i 2154 point to a heap-allocated NodeChars (it can do that, since Err codemadness.org 70 i 2155 NodeChars implements NodeTrait, so it can go into node_impl: Err codemadness.org 70 i 2156 Box<dyn NodeTrait>).

Err codemadness.org 70 i 2157

First, I turned the NodeData struct into an enum with two variants, Err codemadness.org 70 i 2158 and moved all of its previous fields to an Element struct:

Err codemadness.org 70 i 2159

// This one is new	Err	codemadness.org	70
i  2160 pub enum NodeData {	Err	codemadness.org	70
i  2161     Element(Element),	Err	codemadness.org	70
i  2162     Text(NodeChars),	Err	codemadness.org	70
i  2163 }	Err	codemadness.org	70
i  2164 	Err	codemadness.org	70
i  2165 // This is the old struct with a different name	Err	codemadness.org	70
i  2166 pub enum Element {	Err	codemadness.org	70
i  2167     node_type: NodeType,	Err	codemadness.org	70
i  2168     element_name: QualName,	Err	codemadness.org	70
i  2169     id: Option<String>,	Err	codemadness.org	70
i  2170     class: Option<String>,	Err	codemadness.org	70
i  2171     specified_values: SpecifiedValues,	Err	codemadness.org	70
i  2172     important_styles: HashSet<QualName>,	Err	codemadness.org	70
i  2173     result: NodeResult,	Err	codemadness.org	70
i  2174     transform: Transform,	Err	codemadness.org	70
i  2175     values: ComputedValues,	Err	codemadness.org	70
i  2176     cond: bool,	Err	codemadness.org	70
i  2177     style_attr: String,	Err	codemadness.org	70
i  2178     node_impl: Box<dyn NodeTrait>,	Err	codemadness.org	70
i  2179 }	Err	codemadness.org	70
i  2180

Err codemadness.org 70 i 2181 Err codemadness.org 70 i 2182

The size of a Rust enum is the maximum of the sizes of its variants, Err codemadness.org 70 i 2183 plus a little extra for the discriminant (you can think of a C struct Err codemadness.org 70 i 2184 with an int for the discriminant, and a union of variants).

Err codemadness.org 70 i 2185

The code needed a few Err codemadness.org 70 i 2186 changes Err codemadness.org 70 i 2187 to split NodeData in this way, by adding accessor Err codemadness.org 70 i 2188 functions to each of the Element or Text cases conveniently. This Err codemadness.org 70 i 2189 is one of those refactors where you can just change the declaration, Err codemadness.org 70 i 2190 and walk down the compiler's errors to make each case use the accesors Err codemadness.org 70 i 2191 instead of whatever was done before.

Err codemadness.org 70 i 2192

Second step: move the Element variant to a separate allocation

Err codemadness.org 70 i 2193

Now, we turn NodeData into Err codemadness.org 70 i 2194 this:

Err codemadness.org 70 i 2195

pub enum NodeData {	Err	codemadness.org	70
i  2196     Element(Box<Element>), // This goes inside a Box	Err	codemadness.org	70
i  2197     Text(NodeChars),	Err	codemadness.org	70
i  2198 }	Err	codemadness.org	70
i  2199

Err codemadness.org 70 i 2200 Err codemadness.org 70 i 2201

That way, the Element variant is the size of a pointer (i.e. a Err codemadness.org 70 i 2202 pointer to the heap-allocated Box), and the Text variant is as big Err codemadness.org 70 i 2203 as NodeChars as usual.

Err codemadness.org 70 i 2204

This means that Element nodes are just as big as before, plus an Err codemadness.org 70 i 2205 extra pointer, plus an extra heap allocation.

Err codemadness.org 70 i 2206

However, the Text nodes get a lot smaller!

Err codemadness.org 70 i 2207

Before: sizeof::<NodeData>() = 1808
After: sizeof::<NodeData>() = 72

Err codemadness.org 70 i 2211

By making the Element variant a lot smaller (the size of a Box, Err codemadness.org 70 i 2212 which is just a pointer), it has no extra overhead on the Text Err codemadness.org 70 i 2213 variant.

Err codemadness.org 70 i 2214

This means that in the SVG file, all the whitespace between XML Err codemadness.org 70 i 2215 elements now takes a lot less memory.

Err codemadness.org 70 i 2216

Some numbers from a pathological file

Err codemadness.org 70 i 2217

Issue 42 is about Err codemadness.org 70 i 2218 an SVG file that is just a <use> element repeated many times, once Err codemadness.org 70 i 2219 per line:

Err codemadness.org 70 i 2220

<svg xmlns="http://www.w3.org/2000/svg">	Err	codemadness.org	70
i  2221   <defs>	Err	codemadness.org	70
i  2222     <symbol id="glyph0-0">	Err	codemadness.org	70
i  2223       <!-- a few elements here -->	Err	codemadness.org	70
i  2224     </symbol>	Err	codemadness.org	70
i  2225   </defs>	Err	codemadness.org	70
i  2226 	Err	codemadness.org	70
i  2227   <use xlink:href="#glyph0-0" x="1" y="10"/>	Err	codemadness.org	70
i  2228   <use xlink:href="#glyph0-0" x="1" y="10"/>	Err	codemadness.org	70
i  2229   <use xlink:href="#glyph0-0" x="1" y="10"/>	Err	codemadness.org	70
i  2230   <!-- about 196,000 similar lines -->	Err	codemadness.org	70
i  2231 </svg>	Err	codemadness.org	70
i  2232

Err codemadness.org 70 i 2233 Err codemadness.org 70 i 2234

So we have around 196,000 elements. According to Valgrind's Massif Err codemadness.org 70 i 2235 tool, this makes rsvg-convert allocate 800,501,568 bytes in the Err codemadness.org 70 i 2236 old version, versus 463,412,720 bytes in the new version, or about 60% Err codemadness.org 70 i 2237 of the space.

Err codemadness.org 70 i 2238

Next steps

Err codemadness.org 70 i 2239

There is a lot of repetition in the text nodes of a typical SVG file. Err codemadness.org 70 i 2240 For example, in that pathological file above, most of the whitespace is Err codemadness.org 70 i 2241 identical: between each element there is a newline and two spaces. Err codemadness.org 70 i 2242 Instead of having thousands of little allocations, all with the same Err codemadness.org 70 i 2243 string, there could be a pool of shared strings. Files with "real" Err codemadness.org 70 i 2244 indentation could get benefits from sharing the whitespace-only text Err codemadness.org 70 i 2245 nodes.

Err codemadness.org 70 i 2246

Real browser engines are very careful to share the style structs Err codemadness.org 70 i 2247 across elements if possible. Look for "style struct sharing" in Err codemadness.org 70 i 2248 "Inside a super fast CSS engine: Quantum CSS". This is Err codemadness.org 70 i 2249 going to take some good work in librsvg, but we can get there Err codemadness.org 70 i 2250 gradually.

Err codemadness.org 70 i 2251

References

Err codemadness.org 70 i 2252

Commits for the whole Err codemadness.org 70 i 2254 refactoring
Turn the NodeData struct into an enum with variants for Element and Err codemadness.org 70 i 2256 Text
Box the Element variant to make Text nodes Err codemadness.org 70 i 2258 smaller. Err codemadness.org 70 i 2259 The commit message has parts of the massif log with all the Err codemadness.org 70 i 2260 interesting numbers.
Tracker bug for memory Err codemadness.org 70 i 2262 consumption

Exposing C and Rust APIs: some thoughts from librsvg

2020-01-15T11:15:06-06:00

Librsvg exports two public APIs: the C API that is in turn available Err codemadness.org 70 i 2268 to other languages through GObject Introspection, and the Rust API.

Err codemadness.org 70 i 2269

You could call this a use of the facade pattern on top of the Err codemadness.org 70 i 2270 rsvg_internals crate. That crate is the actual Err codemadness.org 70 i 2271 implementation of librsvg, and exports an interface with many knobs Err codemadness.org 70 i 2272 that are not exposed from the public APIs. The knobs are to allow for Err codemadness.org 70 i 2273 the variations in each of those APIs.

Err codemadness.org 70 i 2274

This post is about some interesting things that have come up during Err codemadness.org 70 i 2275 the creation/separation of those public APIs, and the implications of Err codemadness.org 70 i 2276 having an internals library that implements both.

Err codemadness.org 70 i 2277

Initial code organization

Err codemadness.org 70 i 2278

When librsvg was being ported to Rust, it just had an rsvg_internals Err codemadness.org 70 i 2279 crate that compiled as a staticlib to a .a library, which was Err codemadness.org 70 i 2280 later linked into the final librsvg.so.

Err codemadness.org 70 i 2281

Eventually the code got to the point where it was feasible to port the Err codemadness.org 70 i 2282 toplevel C API to Rust. This was relatively easy to do, since Err codemadness.org 70 i 2283 everything else underneath was already in Rust. At that point I Err codemadness.org 70 i 2284 became interested in also having a Rust API for librsvg — Err codemadness.org 70 i 2285 first to port the test suite to Rust and be able to run tests in Err codemadness.org 70 i 2286 parallel, and then to actually have a public API in Rust with more Err codemadness.org 70 i 2287 modern idioms than the historical, GObject-based API in C.

Err codemadness.org 70 i 2288

Version 2.45.5, from February 2019, is the last release that only had Err codemadness.org 70 i 2289 a C API.

Err codemadness.org 70 i 2290

Most of the C API of librsvg is in the RsvgHandle class. An Err codemadness.org 70 i 2291 RsvgHandle gets loaded with SVG data from a file or a stream, and Err codemadness.org 70 i 2292 then gets rendered to a Cairo context. The naming of Rust source Err codemadness.org 70 i 2293 files more or less matched the C source files, so where there was Err codemadness.org 70 i 2294 rsvg-handle.c initially, later we had handle.rs with the Rustified Err codemadness.org 70 i 2295 part of that code.

Err codemadness.org 70 i 2296

So, handle.rs had the Rust internals of the RsvgHandle class, and Err codemadness.org 70 i 2297 a bunch of extern "C" functions callable from C. For example, for Err codemadness.org 70 i 2298 this function in the public C API:

Err codemadness.org 70 i 2299

void rsvg_handle_set_base_gfile (RsvgHandle *handle,	Err	codemadness.org	70
i  2300                                  GFile      *base_file);	Err	codemadness.org	70
i  2301

Err codemadness.org 70 i 2302 Err codemadness.org 70 i 2303

The corresponding Rust implementation was this:

Err codemadness.org 70 i 2304

#[no_mangle]	Err	codemadness.org	70
i  2305 pub unsafe extern "C" fn rsvg_handle_rust_set_base_gfile(	Err	codemadness.org	70
i  2306     raw_handle: *mut RsvgHandle,	Err	codemadness.org	70
i  2307     raw_gfile: *mut gio_sys::GFile,	Err	codemadness.org	70
i  2308 ) {	Err	codemadness.org	70
i  2309     let rhandle = get_rust_handle(raw_handle);        // 1	Err	codemadness.org	70
i  2310 	Err	codemadness.org	70
i  2311     assert!(!raw_gfile.is_null());                    // 2	Err	codemadness.org	70
i  2312     let file: gio::File = from_glib_none(raw_gfile);  // 3	Err	codemadness.org	70
i  2313 	Err	codemadness.org	70
i  2314     rhandle.set_base_gfile(&file);                    // 4	Err	codemadness.org	70
i  2315 }	Err	codemadness.org	70
i  2316

Err codemadness.org 70 i 2317 Err codemadness.org 70 i 2318

Get the Rust struct corresponding to the C GObject.
Check the arguments.
Convert from C GObject reference to Rust reference.
Call the actual implementation of set_base_gfile in the Rust Err codemadness.org 70 i 2323 struct.

Err codemadness.org 70 i 2325

You can see that this function takes in arguments with C types, and Err codemadness.org 70 i 2326 converts them to Rust types. It's basically just glue between the C Err codemadness.org 70 i 2327 code and the actual implementation.

Err codemadness.org 70 i 2328

Then, the actual implementation of set_base_gfile looked like Err codemadness.org 70 i 2329 this:

Err codemadness.org 70 i 2330

impl Handle {	Err	codemadness.org	70
i  2331     fn set_base_gfile(&self, file: &gio::File) {	Err	codemadness.org	70
i  2332         if let Some(uri) = file.get_uri() {	Err	codemadness.org	70
i  2333             self.set_base_url(&uri);	Err	codemadness.org	70
i  2334         } else {	Err	codemadness.org	70
i  2335             rsvg_g_warning("file has no URI; will not set the base URI");	Err	codemadness.org	70
i  2336         }	Err	codemadness.org	70
i  2337     }	Err	codemadness.org	70
i  2338 }	Err	codemadness.org	70
i  2339

Err codemadness.org 70 i 2340 Err codemadness.org 70 i 2341

This is an actual method for a Rust Handle struct, and takes Rust Err codemadness.org 70 i 2342 types as arguments — no conversions are necessary here. However, Err codemadness.org 70 i 2343 there is a pesky call to rsvg_g_warning, about which I'll talk later.

Err codemadness.org 70 i 2344

I found it cleanest, although not the shortest code, to structure Err codemadness.org 70 i 2345 things like this:

Err codemadness.org 70 i 2346

Err codemadness.org 70 i 2348
C code: bunch of stub functions where rsvg_blah just calls a Err codemadness.org 70 i 2349 corresponding rsvg_rust_blah.
Err codemadness.org 70 i 2350
Err codemadness.org 70 i 2352
Toplevel Rust code: bunch of #[no_mangle] unsafe extern "C" fn rust_blah() that Err codemadness.org 70 i 2353 convert from C argument types to Rust types, and call safe Rust Err codemadness.org 70 i 2354 functions — for librsvg, these happened to be methods for a struct. Err codemadness.org 70 i 2355 Before returning, the toplevel functions convert Rust return values Err codemadness.org 70 i 2356 to C return values, and do things like converting the Err(E) of a Err codemadness.org 70 i 2357 Result<> into a GError or a boolean or whatever the traditional Err codemadness.org 70 i 2358 C API required.
Err codemadness.org 70 i 2359

Err codemadness.org 70 i 2361

In the very first versions of the code where the public API was Err codemadness.org 70 i 2362 implemented in Rust, the extern "C" functions actually contained Err codemadness.org 70 i 2363 their implementation. However, after some refactoring, it turned out Err codemadness.org 70 i 2364 to be cleaner to leave those functions just with the task of Err codemadness.org 70 i 2365 converting C to Rust types and vice-versa, and put the actual Err codemadness.org 70 i 2366 implementation in very Rust-y code. This made it easier to keep the Err codemadness.org 70 i 2367 unsafe conversion code (unsafe because it deals with raw pointers Err codemadness.org 70 i 2368 coming from C) only in the toplevel functions.

Err codemadness.org 70 i 2369

Growing out a Rust API

Err codemadness.org 70 i 2370

This commit is where the new, public Rust API Err codemadness.org 70 i 2371 started. That commit just created a Cargo workspace Err codemadness.org 70 i 2372 with two crates; the rsvg_internals crate that we already had, and a Err codemadness.org 70 i 2373 librsvg_crate with the public Rust API.

Err codemadness.org 70 i 2374

The commits over the subsequent couple of months are of intense Err codemadness.org 70 i 2375 refactoring:

Err codemadness.org 70 i 2376

Err codemadness.org 70 i 2378
This commit moves the unsafe extern "C" Err codemadness.org 70 i 2379 functions to a separate c_api.rs source file. This leaves Err codemadness.org 70 i 2380 handle.rs with only the safe Rust implementation of the toplevel Err codemadness.org 70 i 2381 API, and c_api.rs with the unsafe entry points that mostly just Err codemadness.org 70 i 2382 convert argument types, return values, and errors.
Err codemadness.org 70 i 2383
Err codemadness.org 70 i 2385
The API primitives get expanded to allow for a public Rust API that Err codemadness.org 70 i 2386 is "hard to misuse" unlike the C API, which needs to be called Err codemadness.org 70 i 2387 in a certain order.
Err codemadness.org 70 i 2388

Err codemadness.org 70 i 2390

Needing to call a C macro

Err codemadness.org 70 i 2391

However, there was a little problem. The Rust code cannot call Err codemadness.org 70 i 2392 g_warning, a C macro in glib that prints a message to Err codemadness.org 70 i 2393 stderr or uses structured logging. Librsvg used that to signal Err codemadness.org 70 i 2394 conditions where something went (recoverably) wrong, but there was no Err codemadness.org 70 i 2395 way to return a proper error code to the caller — it's mainly used as Err codemadness.org 70 i 2396 a debugging aid.

Err codemadness.org 70 i 2397

This is what the rsvg_internals used to be able to call that C macro:

Err codemadness.org 70 i 2398

First, the C code exports a function that just calls the macro:

Err codemadness.org 70 i 2399

/* This function exists just so that we can effectively call g_warning() from Rust,	Err	codemadness.org	70
i  2400  * since glib-rs doesn't bind the g_log functions yet.	Err	codemadness.org	70
i  2401  */	Err	codemadness.org	70
i  2402 void	Err	codemadness.org	70
i  2403 rsvg_g_warning_from_c(const char *msg)	Err	codemadness.org	70
i  2404 {	Err	codemadness.org	70
i  2405     g_warning ("%s", msg);	Err	codemadness.org	70
i  2406 }	Err	codemadness.org	70
i  2407

Err codemadness.org 70 i 2408 Err codemadness.org 70 i 2409

Second, the Rust code binds that function to be callable from Rust:

Err codemadness.org 70 i 2410

pub fn rsvg_g_warning(msg: &str) {	Err	codemadness.org	70
i  2411     extern "C" {	Err	codemadness.org	70
i  2412         fn rsvg_g_warning_from_c(msg: *const libc::c_char);	Err	codemadness.org	70
i  2413     }	Err	codemadness.org	70
i  2414 	Err	codemadness.org	70
i  2415     unsafe {	Err	codemadness.org	70
i  2416         rsvg_g_warning_from_c(msg.to_glib_none().0);	Err	codemadness.org	70
i  2417     }	Err	codemadness.org	70
i  2418 }	Err	codemadness.org	70
i  2419

Err codemadness.org 70 i 2420 Err codemadness.org 70 i 2421

However! Since the standalone librsvg_crate does not link to the C Err codemadness.org 70 i 2422 code from the public librsvg.so, the helper rsvg_g_warning_from_c Err codemadness.org 70 i 2423 is not available!

Err codemadness.org 70 i 2424

A configuration feature for the internals library

Err codemadness.org 70 i 2425

And yet! Those warnings are only meaningful for the C API, which is Err codemadness.org 70 i 2426 not able to return error codes from all situations. However, the Rust Err codemadness.org 70 i 2427 API is able to do that, and so doesn't need the warnings printed to Err codemadness.org 70 i 2428 stderr. My first solution was to add a build-time option for whether Err codemadness.org 70 i 2429 the rsvg_internals library is being build for the C library, or for Err codemadness.org 70 i 2430 the Rust one.

Err codemadness.org 70 i 2431

In case we are building for the C library, the code calls Err codemadness.org 70 i 2432 rsvg_g_warning_from_c as usual.

Err codemadness.org 70 i 2433

But in case we are building for the Rust library, that code is a Err codemadness.org 70 i 2434 no-op.

Err codemadness.org 70 i 2435

This is the bit in rsvg_internals/Cargo.toml to declare the feature:

Err codemadness.org 70 i 2436

[features]	Err	codemadness.org	70
i  2437 # Enables calling g_warning() when built as part of librsvg.so	Err	codemadness.org	70
i  2438 c-library = []	Err	codemadness.org	70
i  2439

Err codemadness.org 70 i 2440 Err codemadness.org 70 i 2441

And this is the corresponding code:

Err codemadness.org 70 i 2442

#[cfg(feature = "c-library")]	Err	codemadness.org	70
i  2443 pub fn rsvg_g_warning(msg: &str) {	Err	codemadness.org	70
i  2444     unsafe {	Err	codemadness.org	70
i  2445         extern "C" {	Err	codemadness.org	70
i  2446             fn rsvg_g_warning_from_c(msg: *const libc::c_char);	Err	codemadness.org	70
i  2447         }	Err	codemadness.org	70
i  2448 	Err	codemadness.org	70
i  2449         rsvg_g_warning_from_c(msg.to_glib_none().0);	Err	codemadness.org	70
i  2450     }	Err	codemadness.org	70
i  2451 }	Err	codemadness.org	70
i  2452 	Err	codemadness.org	70
i  2453 #[cfg(not(feature = "c-library"))]	Err	codemadness.org	70
i  2454 pub fn rsvg_g_warning(_msg: &str) {	Err	codemadness.org	70
i  2455     // The only callers of this are in handle.rs. When those functions	Err	codemadness.org	70
i  2456     // are called from the Rust API, they are able to return a	Err	codemadness.org	70
i  2457     // meaningful error code, but the C API isn't - so they issues a	Err	codemadness.org	70
i  2458     // g_warning() instead.	Err	codemadness.org	70
i  2459 }	Err	codemadness.org	70
i  2460

Err codemadness.org 70 i 2461 Err codemadness.org 70 i 2462

The first function is the one that is compiled when the c-library Err codemadness.org 70 i 2463 feature is enabled; this happens when building rsvg_internals to Err codemadness.org 70 i 2464 link into librsvg.so.

Err codemadness.org 70 i 2465

The second function does nothing; it is what is compiled when Err codemadness.org 70 i 2466 rsvg_internals is being used just from the librsvg_crate crate Err codemadness.org 70 i 2467 with the Rust API.

Err codemadness.org 70 i 2468

While this worked well, it meant that the internals library was Err codemadness.org 70 i 2469 built twice on each compilation run of the whole librsvg module: Err codemadness.org 70 i 2470 once for librsvg.so, and once for librsvg_crate.

Err codemadness.org 70 i 2471

Making programming errors a `g_critical`

Err codemadness.org 70 i 2472

While g_warning() means "something went wrong, but the program will Err codemadness.org 70 i 2473 continue", g_critical() means "there is a programming error". For Err codemadness.org 70 i 2474 historical reasons Glib does not abort when g_critical() is called, Err codemadness.org 70 i 2475 except by setting G_DEBUG=fatal-criticals, or by Err codemadness.org 70 i 2476 running a development version of Glib.

Err codemadness.org 70 i 2477

This commit turned warnings into critical errors when the Err codemadness.org 70 i 2478 C API was called out of order, by using a similar Err codemadness.org 70 i 2479 rsvg_g_critical_from_c() wrapper for a C macro.

Err codemadness.org 70 i 2480

Separating the C-callable code into yet another crate

Err codemadness.org 70 i 2481

To recapitulate, at that point we had this:

Err codemadness.org 70 i 2482

librsvg/	Err	codemadness.org	70
i  2483 |  Cargo.toml - declares the Cargo workspace	Err	codemadness.org	70
i  2484 |	Err	codemadness.org	70
i  2485 +- rsvg_internals/	Err	codemadness.org	70
i  2486 |  |  Cargo.toml	Err	codemadness.org	70
i  2487 |  +- src/	Err	codemadness.org	70
i  2488 |       c_api.rs - convert types and return values, call into implementation	Err	codemadness.org	70
i  2489 |       handle.rs - actual implementation	Err	codemadness.org	70
i  2490 |       *.rs - all the other internals	Err	codemadness.org	70
i  2491 |	Err	codemadness.org	70
i  2492 +- librsvg/	Err	codemadness.org	70
i  2493 |    *.c - stub functions that call into Rust	Err	codemadness.org	70
i  2494 |    rsvg-base.c - contains rsvg_g_warning_from_c() among others	Err	codemadness.org	70
i  2495 |	Err	codemadness.org	70
i  2496 +- librsvg_crate/	Err	codemadness.org	70
i  2497    |  Cargo.toml	Err	codemadness.org	70
i  2498    +- src/	Err	codemadness.org	70
i  2499    |    lib.rs - public Rust API	Err	codemadness.org	70
i  2500    +- tests/ - tests for the public Rust API	Err	codemadness.org	70
i  2501         *.rs	Err	codemadness.org	70
i  2502

Err codemadness.org 70 i 2503 Err codemadness.org 70 i 2504

At this point c_api.rs with all the unsafe functions looked out of Err codemadness.org 70 i 2505 place. That code is only relevant to librsvg.so — the public C API Err codemadness.org 70 i 2506 —, not to the Rust API in librsvg_crate.

Err codemadness.org 70 i 2507

I started moving the C API glue to a separate librsvg_c_api crate that lives Err codemadness.org 70 i 2508 along with the C stubs:

Err codemadness.org 70 i 2509

+- librsvg/	Err	codemadness.org	70
i  2510 |    *.c - stub functions that call into Rust	Err	codemadness.org	70
i  2511 |    rsvg-base.c - contains rsvg_g_warning_from_c() among others	Err	codemadness.org	70
i  2512 |    Cargo.toml	Err	codemadness.org	70
i  2513 |    c_api.rs - what we had before	Err	codemadness.org	70
i  2514

Err codemadness.org 70 i 2515 Err codemadness.org 70 i 2516

This made the dependencies look like the following:

Err codemadness.org 70 i 2517

      rsvg_internals	Err	codemadness.org	70
i  2518        ^           ^	Err	codemadness.org	70
i  2519        |             \	Err	codemadness.org	70
i  2520        |               \	Err	codemadness.org	70
i  2521 librsvg_crate     librsvg_c_api	Err	codemadness.org	70
i  2522   (Rust API)             ^	Err	codemadness.org	70
i  2523                          |	Err	codemadness.org	70
i  2524                     librsvg.so	Err	codemadness.org	70
i  2525                       (C API)	Err	codemadness.org	70
i  2526

Err codemadness.org 70 i 2527 Err codemadness.org 70 i 2528

And also, this made it possible to remove the configuration feature Err codemadness.org 70 i 2529 for rsvg_internals, since the code that calls Err codemadness.org 70 i 2530 rsvg_g_warning_from_c now lives in librsvg_c_api.

Err codemadness.org 70 i 2531

With that, rsvg_internals is compiled only once, as it should be.

Err codemadness.org 70 i 2532

This also helped clean up some code in the internals library. Err codemadness.org 70 i 2533 Deprecated functions that render SVGs directly to GdkPixbuf are now Err codemadness.org 70 i 2534 in librsvg_c_api and don't clutter the rsvg_internals library. Err codemadness.org 70 i 2535 All the GObject boilerplate is there as well now; rsvg_internals is Err codemadness.org 70 i 2536 mostly safe code except for the glue to libxml2.

Err codemadness.org 70 i 2537

Summary

Err codemadness.org 70 i 2538

It was useful to move all the code that dealt with incoming C types, Err codemadness.org 70 i 2539 our outgoing C return values and errors, into the same place, and Err codemadness.org 70 i 2540 separate it from the "pure Rust" code.

Err codemadness.org 70 i 2541

This took gradual refactoring and was not done in a single step, but Err codemadness.org 70 i 2542 it left the resulting Rust code rather nice and clean.

Err codemadness.org 70 i 2543

When we added a new public Rust API, we had to shuffle some code Err codemadness.org 70 i 2544 around that could only be linked in the context of a C library.

Err codemadness.org 70 i 2545

Compile-time configuration features are useful (like #ifdef in the C Err codemadness.org 70 i 2546 world), but they do cause double compilation if you need a C-internals Err codemadness.org 70 i 2547 and a Rust-internals library from the same code.

Err codemadness.org 70 i 2548

Having proper error reporting throughout the Rust code is a lot of Err codemadness.org 70 i 2549 work, but pretty much invaluable. The glue code to C can then convert Err codemadness.org 70 i 2550 and expose those errors as needed.

Err codemadness.org 70 i 2551

If you need both C and Rust APIs into the same code base, you may end Err codemadness.org 70 i 2552 up naturally using a facade pattern for each. It helps to gradually Err codemadness.org 70 i 2553 refactor the internals to be as "pure idiomatic Rust" as possible, Err codemadness.org 70 i 2554 while letting API idiosyncrasies bubble up to each individual facade.

Moving gnome-shell's styles to Rust

2019-11-25T21:10:06-06:00

Gnome-shell uses CSS processing code that dates from Err codemadness.org 70 i 2564 HippoCanvas, Err codemadness.org 70 i 2565 a CSS-aware canvas from around 2006. It uses libcroco to parse CSS, Err codemadness.org 70 i 2566 and implements selector matching by hand in C.

Err codemadness.org 70 i 2567

This code is getting rather dated, and libcroco is unmaintained.

Err codemadness.org 70 i 2568

I've been reading the code for Err codemadness.org 70 i 2569 StTheme Err codemadness.org 70 i 2570 and Err codemadness.org 70 i 2571 StThemeNode, Err codemadness.org 70 i 2572 and it looks very feasible to port it gradually to Rust, by using the Err codemadness.org 70 i 2573 same crates that librsvg uses, and eventually removing libcroco Err codemadness.org 70 i 2574 altogether: gnome-shell is the last module that uses libcroco in Err codemadness.org 70 i 2575 distro packages.

Err codemadness.org 70 i 2576

Strategy

Err codemadness.org 70 i 2577

StTheme and StThemeNode use libcroco to load CSS stylesheets and Err codemadness.org 70 i 2578 keep them in memory. The values of individual properties are just Err codemadness.org 70 i 2579 tokenized and kept around as a linked list of CRTerm; this struct Err codemadness.org 70 i 2580 represents a single token.

Err codemadness.org 70 i 2581

Later, the drawing code uses functions like Err codemadness.org 70 i 2582 st_theme_node_lookup_color(node, "property_name") or Err codemadness.org 70 i 2583 st_theme_node_lookup_length() to query the various properties that Err codemadness.org 70 i 2584 it needs. It is then that the type of each property gets Err codemadness.org 70 i 2585 determined: prior to that step, property values are just tokenized, Err codemadness.org 70 i 2586 not parsed into usable values.

Err codemadness.org 70 i 2587

I am going to start by porting the individual parsers to Rust, similar Err codemadness.org 70 i 2588 to what Paolo and I did for librsvg. It turns out that there's some Err codemadness.org 70 i 2589 code we can share.

Err codemadness.org 70 i 2590

So far I have the parser for Err codemadness.org 70 i 2591 colors Err codemadness.org 70 i 2592 implemented in Rust. This removes a little bunch of Err codemadness.org 70 i 2593 code Err codemadness.org 70 i 2594 from the C parsers, and replaces it with a little Rust code, since the Err codemadness.org 70 i 2595 cssparser crate can already parse CSS colors with alpha with no extra Err codemadness.org 70 i 2596 work — libcroco didn't support alpha.

Err codemadness.org 70 i 2597

As a bonus, this supports hsl() colors in addition to rgb() ones Err codemadness.org 70 i 2598 out of the box!

Err codemadness.org 70 i 2599

After all the parsers are done, the next step would be to convert the Err codemadness.org 70 i 2600 representation of complete stylesheets into pure Rust code.

Err codemadness.org 70 i 2601

What can we expect?

Err codemadness.org 70 i 2602

A well-maintained CSS stack. Firefox and Servo both use the Err codemadness.org 70 i 2603 crates in question, so librsvg and gnome-shell should get maintenance Err codemadness.org 70 i 2604 of a robust CSS stack "for free", for the foreseeable future.

Err codemadness.org 70 i 2605

Speed. Caveat: I have no profile data for gnome-shell yet, so I don't Err codemadness.org 70 i 2606 know how much time it spends doing CSS parsing and cascading, but it Err codemadness.org 70 i 2607 looks like the Rust version has a good chance of being more efficient.

Err codemadness.org 70 i 2608

The selectors crate has some very Err codemadness.org 70 i 2609 interesting Err codemadness.org 70 i 2610 optimizations Err codemadness.org 70 i 2611 from Mozilla Servo, and it is also now used in Firefox. It supports Err codemadness.org 70 i 2612 doing selector matching using Bloom filters, and can also avoid Err codemadness.org 70 i 2613 re-cascading child nodes if a change to a parent would not cause its Err codemadness.org 70 i 2614 children to change.

Err codemadness.org 70 i 2615

All the parsing is done with zero-copy parsers thanks to Rust's string Err codemadness.org 70 i 2616 slices; without so many malloc() calls in the parsing code path, Err codemadness.org 70 i 2617 the parsing stage should really fly.

Err codemadness.org 70 i 2618

More CSS features. The selectors crate can do matching on Err codemadness.org 70 i 2619 basically all kinds of selectors as defined by recent CSS specs; one Err codemadness.org 70 i 2620 just has to provide the correct hooks into the calling code's Err codemadness.org 70 i 2621 representation of the DOM tree. The kind of matching that StTheme Err codemadness.org 70 i 2622 can do is somewhat limited; the rustification should make it match Err codemadness.org 70 i 2623 much more closely to what people expect from CSS engines in web Err codemadness.org 70 i 2624 browsers.

Err codemadness.org 70 i 2625

A well-defined model of property inheritance. StThemeNode's Err codemadness.org 70 i 2626 model for CSS property inheritance is a bit ad-hoc and inconsistent. Err codemadness.org 70 i 2627 I haven't quite tested it, but from looking at the code, it seems that Err codemadness.org 70 i 2628 not all properties get inherited in the same way. I hope to move it Err codemadness.org 70 i 2629 to something closer to what librsvg already does, which should make it Err codemadness.org 70 i 2630 match people's expectations from the web.

Err codemadness.org 70 i 2631

In the meantime

Err codemadness.org 70 i 2632

I have a merge request ready to simply move the libcroco source code Err codemadness.org 70 i 2633 directly inside gnome-shell's source tree. This should let distros Err codemadness.org 70 i 2634 remove their libcroco package as soon as possible. That MR does not Err codemadness.org 70 i 2635 require Rust yet.

Err codemadness.org 70 i 2636

My playground is here:

Err codemadness.org 70 i 2637

Gnome-shell branch to rustify the Err codemadness.org 70 i 2639 styles
Stylish, a Rust Err codemadness.org 70 i 2641 library that will implement gnome-shell's styling code.

Err codemadness.org 70 i 2643

This does not compile yet! I'll plug things together tomorrow.

Err codemadness.org 70 i 2644

(Oh, yes, the project to redo Firefox's CSS stack in Rust used to be Err codemadness.org 70 i 2645 called Stylo. I'm calling this Stylish, as in Styles for the Shell.)

Refactoring the Length type

2019-11-19T10:01:14-06:00

CSS length values have a number and a unit, e.g. 5cm Err codemadness.org 70 i 2651 or 6px. Sometimes the unit is a percentage, like 50%, and SVG Err codemadness.org 70 i 2652 says that lengths with percentage units should be resolved with Err codemadness.org 70 i 2653 respect to a certain rectangle. For example, consider this circle Err codemadness.org 70 i 2654 element:

Err codemadness.org 70 i 2655

<circle cx="50%" cy="75%" r="4px" fill="black"/>	Err	codemadness.org	70
i  2656

Err codemadness.org 70 i 2657 Err codemadness.org 70 i 2658

This means, draw a solid black circle whose center is at 50% of the Err codemadness.org 70 i 2659 width and 75% of the height of the current viewport. The circle Err codemadness.org 70 i 2660 should have a 4-pixel radius.

Err codemadness.org 70 i 2661

The process of converting that kind of units into absolute pixels for Err codemadness.org 70 i 2662 the final drawing is called normalization. In SVG, percentage Err codemadness.org 70 i 2663 units sometimes need to be normalized with respect to the current Err codemadness.org 70 i 2664 viewport (a local coordinate system), or with respect to the size of Err codemadness.org 70 i 2665 another object (e.g. when a clipping path is used to cut the current Err codemadness.org 70 i 2666 shape in half).

Err codemadness.org 70 i 2667

One detail about normalization is that it can be with respect to the Err codemadness.org 70 i 2668 horizontal dimension of the current viewport, the vertical dimension, Err codemadness.org 70 i 2669 or both. Keep this in mind: at normalization time, we need to be able Err codemadness.org 70 i 2670 to distinguish between those three modes.

Err codemadness.org 70 i 2671

The original C version

Err codemadness.org 70 i 2672

I have talked about the original C code for lengths before; the Err codemadness.org 70 i 2673 following is a small summary.

Err codemadness.org 70 i 2674

The original C code had this struct to represent lengths:

Err codemadness.org 70 i 2675

typedef struct {	Err	codemadness.org	70
i  2676     double length;	Err	codemadness.org	70
i  2677     char factor;	Err	codemadness.org	70
i  2678 } RsvgLength;	Err	codemadness.org	70
i  2679

Err codemadness.org 70 i 2680 Err codemadness.org 70 i 2681

The parsing code would set the factor field to a Err codemadness.org 70 i 2682 character depending on the length's unit: 'p' for percentages, 'i' Err codemadness.org 70 i 2683 for inches, etc., and '\0' for the default unit, which is pixels.

Err codemadness.org 70 i 2684

Along with that, the normalization code needed to know Err codemadness.org 70 i 2685 the direction (horizontal, vertical, both) to which the length in Err codemadness.org 70 i 2686 question refers. It did this by taking another character as an Err codemadness.org 70 i 2687 argument to the normalization function:

Err codemadness.org 70 i 2688

double	Err	codemadness.org	70
i  2689 _rsvg_css_normalize_length (const RsvgLength * in, RsvgDrawingCtx * ctx, char dir)	Err	codemadness.org	70
i  2690 {	Err	codemadness.org	70
i  2691     if (in->factor == '\0')            /* pixels, no need to normalize */	Err	codemadness.org	70
i  2692         return in->length;	Err	codemadness.org	70
i  2693     else if (in->factor == 'p') {      /* percentages; need to consider direction */	Err	codemadness.org	70
i  2694         if (dir == 'h')                                     /* horizontal */	Err	codemadness.org	70
i  2695             return in->length * ctx->vb.rect.width;	Err	codemadness.org	70
i  2696         if (dir == 'v')                                     /* vertical */	Err	codemadness.org	70
i  2697             return in->length * ctx->vb.rect.height;	Err	codemadness.org	70
i  2698         if (dir == 'o')                                     /* both */	Err	codemadness.org	70
i  2699             return in->length * rsvg_viewport_percentage (ctx->vb.rect.width,	Err	codemadness.org	70
i  2700                                                           ctx->vb.rect.height);	Err	codemadness.org	70
i  2701     } else { ... }	Err	codemadness.org	70
i  2702 }	Err	codemadness.org	70
i  2703

Err codemadness.org 70 i 2704 Err codemadness.org 70 i 2705

The original post talks about how I found a couple of bugs with Err codemadness.org 70 i 2706 how the directions are identified at normalization time. The function Err codemadness.org 70 i 2707 above expects one of 'h'/'v'/'o' for horizontal/vertical/both, and one or Err codemadness.org 70 i 2708 two places in the code passed the wrong character.

Err codemadness.org 70 i 2709

Making the C version cleaner

Err codemadness.org 70 i 2710

Before converting that code to Rust, I removed the pesky Err codemadness.org 70 i 2711 characters and made the code use proper enums to Err codemadness.org 70 i 2712 identify a length's units.

Err codemadness.org 70 i 2713

+typedef enum {	Err	codemadness.org	70
i  2714 +    LENGTH_UNIT_DEFAULT,	Err	codemadness.org	70
i  2715 +    LENGTH_UNIT_PERCENT,	Err	codemadness.org	70
i  2716 +    LENGTH_UNIT_FONT_EM,	Err	codemadness.org	70
i  2717 +    LENGTH_UNIT_FONT_EX,	Err	codemadness.org	70
i  2718 +    LENGTH_UNIT_INCH,	Err	codemadness.org	70
i  2719 +    LENGTH_UNIT_RELATIVE_LARGER,	Err	codemadness.org	70
i  2720 +    LENGTH_UNIT_RELATIVE_SMALLER	Err	codemadness.org	70
i  2721 +} LengthUnit;	Err	codemadness.org	70
i  2722 +	Err	codemadness.org	70
i  2723  typedef struct {	Err	codemadness.org	70
i  2724      double length;	Err	codemadness.org	70
i  2725 -    char factor;	Err	codemadness.org	70
i  2726 +    LengthUnit unit;	Err	codemadness.org	70
i  2727  } RsvgLength;	Err	codemadness.org	70
i  2728

Err codemadness.org 70 i 2729 Err codemadness.org 70 i 2730

Then, do the same for the normalization function, so it will get the Err codemadness.org 70 i 2731 direction in which to normalize as an enum instead of a char.

Err codemadness.org 70 i 2732

+typedef enum {	Err	codemadness.org	70
i  2733 +    LENGTH_DIR_HORIZONTAL,	Err	codemadness.org	70
i  2734 +    LENGTH_DIR_VERTICAL,	Err	codemadness.org	70
i  2735 +    LENGTH_DIR_BOTH	Err	codemadness.org	70
i  2736 +} LengthDir;	Err	codemadness.org	70
i  2737 	Err	codemadness.org	70
i  2738  double	Err	codemadness.org	70
i  2739 -_rsvg_css_normalize_length (const RsvgLength * in, RsvgDrawingCtx * ctx, char dir)	Err	codemadness.org	70
i  2740 +_rsvg_css_normalize_length (const RsvgLength * in, RsvgDrawingCtx * ctx, LengthDir dir)	Err	codemadness.org	70
i  2741

Err codemadness.org 70 i 2742 Err codemadness.org 70 i 2743

Making the C version easier to get right

Err codemadness.org 70 i 2744

While doing the last change above, I found a place in the code that Err codemadness.org 70 i 2745 used the wrong direction by mistake, probably due to a cut&paste Err codemadness.org 70 i 2746 error. Part of the problem here is that the code was specifying the Err codemadness.org 70 i 2747 direction at normalization time.

Err codemadness.org 70 i 2748

I decided to change it so that each direction value carried its own Err codemadness.org 70 i 2749 direction since initialization, so that subsequent Err codemadness.org 70 i 2750 code wouldn't have to worry about that. Hopefully, initializing a Err codemadness.org 70 i 2751 width field should make it obvious that it needed Err codemadness.org 70 i 2752 LENGTH_DIR_HORIZONTAL.

Err codemadness.org 70 i 2753

 typedef struct {	Err	codemadness.org	70
i  2754      double length;	Err	codemadness.org	70
i  2755      LengthUnit unit;	Err	codemadness.org	70
i  2756 +    LengthDir dir;	Err	codemadness.org	70
i  2757  } RsvgLength;	Err	codemadness.org	70
i  2758

Err codemadness.org 70 i 2759 Err codemadness.org 70 i 2760

That is, so that instead of

Err codemadness.org 70 i 2761

  /* at initialization time */	Err	codemadness.org	70
i  2762   foo.width = _rsvg_css_parse_length (str);	Err	codemadness.org	70
i  2763 	Err	codemadness.org	70
i  2764   ...	Err	codemadness.org	70
i  2765 	Err	codemadness.org	70
i  2766   /* at rendering time */	Err	codemadness.org	70
i  2767   double final_width = _rsvg_css_normalize_length (&foo.width, ctx, LENGTH_DIR_HORIZONTAL);	Err	codemadness.org	70
i  2768

Err codemadness.org 70 i 2769 Err codemadness.org 70 i 2770

we would instead do this:

Err codemadness.org 70 i 2771

  /* at initialization time */	Err	codemadness.org	70
i  2772   foo.width = _rsvg_css_parse_length (str, LENGTH_DIR_HORIZONTAL);	Err	codemadness.org	70
i  2773 	Err	codemadness.org	70
i  2774   ...	Err	codemadness.org	70
i  2775 	Err	codemadness.org	70
i  2776   /* at rendering time */	Err	codemadness.org	70
i  2777   double final_width = _rsvg_css_normalize_length (&foo.width, ctx);	Err	codemadness.org	70
i  2778

Err codemadness.org 70 i 2779 Err codemadness.org 70 i 2780

This made the drawing code, which deals with a lot of coordinates at Err codemadness.org 70 i 2781 the same time, a lot less noisy.

Err codemadness.org 70 i 2782

Initial port to Rust

Err codemadness.org 70 i 2783

To recap, this was the state of the structs after the initial Err codemadness.org 70 i 2784 refactoring in C:

Err codemadness.org 70 i 2785

typedef enum {	Err	codemadness.org	70
i  2786     LENGTH_UNIT_DEFAULT,	Err	codemadness.org	70
i  2787     LENGTH_UNIT_PERCENT,	Err	codemadness.org	70
i  2788     LENGTH_UNIT_FONT_EM,	Err	codemadness.org	70
i  2789     LENGTH_UNIT_FONT_EX,	Err	codemadness.org	70
i  2790     LENGTH_UNIT_INCH,	Err	codemadness.org	70
i  2791     LENGTH_UNIT_RELATIVE_LARGER,	Err	codemadness.org	70
i  2792     LENGTH_UNIT_RELATIVE_SMALLER	Err	codemadness.org	70
i  2793 } LengthUnit;	Err	codemadness.org	70
i  2794 	Err	codemadness.org	70
i  2795 typedef enum {	Err	codemadness.org	70
i  2796     LENGTH_DIR_HORIZONTAL,	Err	codemadness.org	70
i  2797     LENGTH_DIR_VERTICAL,	Err	codemadness.org	70
i  2798     LENGTH_DIR_BOTH	Err	codemadness.org	70
i  2799 } LengthDir;	Err	codemadness.org	70
i  2800 	Err	codemadness.org	70
i  2801 typedef struct {	Err	codemadness.org	70
i  2802     double length;	Err	codemadness.org	70
i  2803     LengthUnit unit;	Err	codemadness.org	70
i  2804     LengthDir dir;	Err	codemadness.org	70
i  2805 } RsvgLength;	Err	codemadness.org	70
i  2806

Err codemadness.org 70 i 2807 Err codemadness.org 70 i 2808

This ported to Rust in a straightforward fashion:

Err codemadness.org 70 i 2809

pub enum LengthUnit {	Err	codemadness.org	70
i  2810     Default,	Err	codemadness.org	70
i  2811     Percent,	Err	codemadness.org	70
i  2812     FontEm,	Err	codemadness.org	70
i  2813     FontEx,	Err	codemadness.org	70
i  2814     Inch,	Err	codemadness.org	70
i  2815     RelativeLarger,	Err	codemadness.org	70
i  2816     RelativeSmaller	Err	codemadness.org	70
i  2817 }	Err	codemadness.org	70
i  2818 	Err	codemadness.org	70
i  2819 pub enum LengthDir {	Err	codemadness.org	70
i  2820     Horizontal,	Err	codemadness.org	70
i  2821     Vertical,	Err	codemadness.org	70
i  2822     Both	Err	codemadness.org	70
i  2823 }	Err	codemadness.org	70
i  2824 	Err	codemadness.org	70
i  2825 pub struct RsvgLength {	Err	codemadness.org	70
i  2826     length: f64,	Err	codemadness.org	70
i  2827     unit: LengthUnit,	Err	codemadness.org	70
i  2828     dir: LengthDir	Err	codemadness.org	70
i  2829 }	Err	codemadness.org	70
i  2830

Err codemadness.org 70 i 2831 Err codemadness.org 70 i 2832

It got a similar constructor that took the Err codemadness.org 70 i 2833 direction and produced an RsvgLength:

Err codemadness.org 70 i 2834

impl RsvgLength {	Err	codemadness.org	70
i  2835     pub fn parse (string: &str, dir: LengthDir) -> RsvgLength { ... }	Err	codemadness.org	70
i  2836 }	Err	codemadness.org	70
i  2837

Err codemadness.org 70 i 2838 Err codemadness.org 70 i 2839

(This was before using Result; remember that the original Err codemadness.org 70 i 2840 C code did very little error checking!)

Err codemadness.org 70 i 2841

The initial Parse trait

Err codemadness.org 70 i 2842

It was at that point that it seemed convenient to introduce a Parse Err codemadness.org 70 i 2843 trait, which all CSS value types would implement to Err codemadness.org 70 i 2844 parse themselves from string.

Err codemadness.org 70 i 2845

However, parsing an RsvgLength also needed an extra piece of data, Err codemadness.org 70 i 2846 the LengthDir. My initial version of the Parse trait had an Err codemadness.org 70 i 2847 associated called Data, through which one could pass an extra piece Err codemadness.org 70 i 2848 of data during parsing/initialization:

Err codemadness.org 70 i 2849

pub trait Parse: Sized {	Err	codemadness.org	70
i  2850     type Data;	Err	codemadness.org	70
i  2851     type Err;	Err	codemadness.org	70
i  2852 	Err	codemadness.org	70
i  2853     fn parse (s: &str, data: Self::Data) -> Result<Self, Self::Err>;	Err	codemadness.org	70
i  2854 }	Err	codemadness.org	70
i  2855

Err codemadness.org 70 i 2856 Err codemadness.org 70 i 2857

This was explicitly to be able to pass a LengthDir to the parser for Err codemadness.org 70 i 2858 RsvgLength:

Err codemadness.org 70 i 2859

impl Parse for RsvgLength {	Err	codemadness.org	70
i  2860     type Data = LengthDir;	Err	codemadness.org	70
i  2861     type Err = AttributeError;	Err	codemadness.org	70
i  2862 	Err	codemadness.org	70
i  2863     fn parse (string: &str, dir: LengthDir) -> Result <RsvgLength, AttributeError> { ... }	Err	codemadness.org	70
i  2864 }	Err	codemadness.org	70
i  2865

Err codemadness.org 70 i 2866 Err codemadness.org 70 i 2867

This was okay for lengths, but very noisy for everything else that Err codemadness.org 70 i 2868 didn't require an extra bit of data. In the rest of the code, the Err codemadness.org 70 i 2869 helper type was Data = () and there was a pair of extra parentheses () Err codemadness.org 70 i 2870 in every place that parse() was called.

Err codemadness.org 70 i 2871

Removing the helper Data type

Err codemadness.org 70 i 2872

Introducing one type per direction

Err codemadness.org 70 i 2873

Over a year later, that () bit of data everywhere was driving me Err codemadness.org 70 i 2874 nuts. I started refactoring the Length module to remove it.

Err codemadness.org 70 i 2875

First, I introduced three newtypes to wrap Length, and indicate Err codemadness.org 70 i 2876 their direction at the same time:

Err codemadness.org 70 i 2877

pub struct LengthHorizontal(Length);	Err	codemadness.org	70
i  2878 pub struct LengthVertical(Length);	Err	codemadness.org	70
i  2879 pub struct LengthBoth(Length);	Err	codemadness.org	70
i  2880

Err codemadness.org 70 i 2881 Err codemadness.org 70 i 2882

This was done with a macro because now each wrapper type Err codemadness.org 70 i 2883 needed to know the relevant LengthDir.

Err codemadness.org 70 i 2884

Now, for example, the declaration for the <circle> element looked Err codemadness.org 70 i 2885 like this:

Err codemadness.org 70 i 2886

pub struct NodeCircle {	Err	codemadness.org	70
i  2887     cx: Cell<LengthHorizontal>,	Err	codemadness.org	70
i  2888     cy: Cell<LengthVertical>,	Err	codemadness.org	70
i  2889     r: Cell<LengthBoth>,	Err	codemadness.org	70
i  2890 }	Err	codemadness.org	70
i  2891

Err codemadness.org 70 i 2892 Err codemadness.org 70 i 2893

(Ignore the Cell everywhere; we got rid of that later.)

Err codemadness.org 70 i 2894

Removing the `dir` field

Err codemadness.org 70 i 2895

Since now the information about the length's direction is embodied in Err codemadness.org 70 i 2896 the LengthHorizontal/LengthVertical/LengthBoth types, this made it Err codemadness.org 70 i 2897 possible to remove the dir field from the inner Err codemadness.org 70 i 2898 Length struct.

Err codemadness.org 70 i 2899

 pub struct RsvgLength {	Err	codemadness.org	70
i  2900      length: f64,	Err	codemadness.org	70
i  2901      unit: LengthUnit,	Err	codemadness.org	70
i  2902 -    dir: LengthDir	Err	codemadness.org	70
i  2903  }	Err	codemadness.org	70
i  2904 	Err	codemadness.org	70
i  2905 +pub struct LengthHorizontal(Length);	Err	codemadness.org	70
i  2906 +pub struct LengthVertical(Length);	Err	codemadness.org	70
i  2907 +pub struct LengthBoth(Length);	Err	codemadness.org	70
i  2908 +	Err	codemadness.org	70
i  2909 +define_length_type!(LengthHorizontal, LengthDir::Horizontal);	Err	codemadness.org	70
i  2910 +define_length_type!(LengthVertical, LengthDir::Vertical);	Err	codemadness.org	70
i  2911 +define_length_type!(LengthBoth, LengthDir::Both);	Err	codemadness.org	70
i  2912

Err codemadness.org 70 i 2913 Err codemadness.org 70 i 2914

Note the use of a define_length_type! macro to generate code for Err codemadness.org 70 i 2915 those three newtypes.

Err codemadness.org 70 i 2916

Removing the `Data` associated type

Err codemadness.org 70 i 2917

And finally, this made it possible to remove the Data associated Err codemadness.org 70 i 2918 type from the Parse trait.

Err codemadness.org 70 i 2919

 pub trait Parse: Sized {	Err	codemadness.org	70
i  2920 -    type Data;	Err	codemadness.org	70
i  2921      type Err;	Err	codemadness.org	70
i  2922 	Err	codemadness.org	70
i  2923 -    fn parse(parser: &mut Parser<'_, '_>, data: Self::Data) -> Result<Self, Self::Err>;	Err	codemadness.org	70
i  2924 +    fn parse(parser: &mut Parser<'_, '_>) -> Result<Self, Self::Err>;	Err	codemadness.org	70
i  2925  }	Err	codemadness.org	70
i  2926

Err codemadness.org 70 i 2927 Err codemadness.org 70 i 2928

The resulting mega-commit removed a bunch of stray parentheses () Err codemadness.org 70 i 2929 from all calls to parse(), and the code ended up a lot easier to Err codemadness.org 70 i 2930 read.

Err codemadness.org 70 i 2931

Removing the newtypes

Err codemadness.org 70 i 2932

This was fine for a while. Recently, however, I figured out that it Err codemadness.org 70 i 2933 would be possible to embody the information for a length's direction Err codemadness.org 70 i 2934 in a different way.

Err codemadness.org 70 i 2935

But to get there, I first needed a temporary refactor.

Err codemadness.org 70 i 2936

Replacing the macro with a trait with a default implementation

Err codemadness.org 70 i 2937

Deep in the guts of length.rs, the key function that does something Err codemadness.org 70 i 2938 different based on LengthDir is its scaling_factor method:

Err codemadness.org 70 i 2939

enum LengthDir {	Err	codemadness.org	70
i  2940     Horizontal,	Err	codemadness.org	70
i  2941     Vertical,	Err	codemadness.org	70
i  2942     Both,	Err	codemadness.org	70
i  2943 }	Err	codemadness.org	70
i  2944 	Err	codemadness.org	70
i  2945 impl LengthDir {	Err	codemadness.org	70
i  2946     fn scaling_factor(self, x: f64, y: f64) -> f64 {	Err	codemadness.org	70
i  2947         match self {	Err	codemadness.org	70
i  2948             LengthDir::Horizontal => x,	Err	codemadness.org	70
i  2949             LengthDir::Vertical => y,	Err	codemadness.org	70
i  2950             LengthDir::Both => viewport_percentage(x, y),	Err	codemadness.org	70
i  2951         }	Err	codemadness.org	70
i  2952     }	Err	codemadness.org	70
i  2953 }	Err	codemadness.org	70
i  2954

Err codemadness.org 70 i 2955 Err codemadness.org 70 i 2956

That method gets passed, for example, the width/height of the Err codemadness.org 70 i 2957 current viewport for the x/y arguments. The method decides whether Err codemadness.org 70 i 2958 to use the width, height, or a combination of both.

Err codemadness.org 70 i 2959

And of course, the interesting part of the define_length_type! macro Err codemadness.org 70 i 2960 was to generate code for calling LengthDir::Horizontal::scaling_factor()/etc. as Err codemadness.org 70 i 2961 appropriate depending on the LengthDir in question.

Err codemadness.org 70 i 2962

First I made a trait called Orientation with a scaling_factor Err codemadness.org 70 i 2963 method, and three zero-sized types that implement that trait. Note Err codemadness.org 70 i 2964 how each of these three implementations corresponds to one of the Err codemadness.org 70 i 2965 match arms above:

Err codemadness.org 70 i 2966

pub trait Orientation {	Err	codemadness.org	70
i  2967     fn scaling_factor(x: f64, y: f64) -> f64;	Err	codemadness.org	70
i  2968 }	Err	codemadness.org	70
i  2969 	Err	codemadness.org	70
i  2970 pub struct Horizontal;	Err	codemadness.org	70
i  2971 pub struct Vertical;	Err	codemadness.org	70
i  2972 pub struct Both;	Err	codemadness.org	70
i  2973 	Err	codemadness.org	70
i  2974 impl Orientation for Horizontal {	Err	codemadness.org	70
i  2975     fn scaling_factor(x: f64, _y: f64) -> f64 {	Err	codemadness.org	70
i  2976         x	Err	codemadness.org	70
i  2977     }	Err	codemadness.org	70
i  2978 }	Err	codemadness.org	70
i  2979 	Err	codemadness.org	70
i  2980 impl Orientation for Vertical {	Err	codemadness.org	70
i  2981     fn scaling_factor(_x: f64, y: f64) -> f64 {	Err	codemadness.org	70
i  2982         y	Err	codemadness.org	70
i  2983     }	Err	codemadness.org	70
i  2984 }	Err	codemadness.org	70
i  2985 	Err	codemadness.org	70
i  2986 impl Orientation for Both {	Err	codemadness.org	70
i  2987     fn scaling_factor(x: f64, y: f64) -> f64 {	Err	codemadness.org	70
i  2988         viewport_percentage(x, y)	Err	codemadness.org	70
i  2989     }	Err	codemadness.org	70
i  2990 }	Err	codemadness.org	70
i  2991

Err codemadness.org 70 i 2992 Err codemadness.org 70 i 2993

Now most of the contents of the define_length_type! macro can go in Err codemadness.org 70 i 2994 the default implementation of a new trait Err codemadness.org 70 i 2995 LengthTrait. Crucially, this trait has an Err codemadness.org 70 i 2996 Orientation associated type, which it uses to call into the Err codemadness.org 70 i 2997 Orientation trait:

Err codemadness.org 70 i 2998

pub trait LengthTrait: Sized {	Err	codemadness.org	70
i  2999     type Orientation: Orientation;	Err	codemadness.org	70
i  3000 	Err	codemadness.org	70
i  3001     ...	Err	codemadness.org	70
i  3002 	Err	codemadness.org	70
i  3003     fn normalize(&self, values: &ComputedValues, params: &ViewParams) -> f64 {	Err	codemadness.org	70
i  3004         match self.unit() {	Err	codemadness.org	70
i  3005             LengthUnit::Px => self.length(),	Err	codemadness.org	70
i  3006 	Err	codemadness.org	70
i  3007             LengthUnit::Percent => {	Err	codemadness.org	70
i  3008                 self.length() *	Err	codemadness.org	70
i  3009                 <Self::Orientation>::scaling_factor(params.view_box_width, params.view_box_height)	Err	codemadness.org	70
i  3010             }	Err	codemadness.org	70
i  3011 	Err	codemadness.org	70
i  3012             ...	Err	codemadness.org	70
i  3013     }	Err	codemadness.org	70
i  3014 }	Err	codemadness.org	70
i  3015

Err codemadness.org 70 i 3016 Err codemadness.org 70 i 3017

Note that the incantation is Err codemadness.org 70 i 3018 <Self::Orientation>::scaling_factor(...) to call that method on the Err codemadness.org 70 i 3019 associated type.

Err codemadness.org 70 i 3020

Now the define_length_type! macro is shrunk a lot, with the Err codemadness.org 70 i 3021 interesting part being just this:

Err codemadness.org 70 i 3022

macro_rules! define_length_type {	Err	codemadness.org	70
i  3023     {$name:ident, $orient:ty} => {	Err	codemadness.org	70
i  3024         pub struct $name(Length);	Err	codemadness.org	70
i  3025 	Err	codemadness.org	70
i  3026         impl LengthTrait for $name {	Err	codemadness.org	70
i  3027             type Orientation = $orient;	Err	codemadness.org	70
i  3028         }	Err	codemadness.org	70
i  3029     }	Err	codemadness.org	70
i  3030 }	Err	codemadness.org	70
i  3031 	Err	codemadness.org	70
i  3032 define_length_type! { LengthHorizontal, Horizontal }	Err	codemadness.org	70
i  3033 define_length_type! { LengthVertical, Vertical }	Err	codemadness.org	70
i  3034 define_length_type! { LengthBoth, Both }	Err	codemadness.org	70
i  3035

Err codemadness.org 70 i 3036 Err codemadness.org 70 i 3037

We moved from having three newtypes of length-with-LengthDir to three Err codemadness.org 70 i 3038 newtypes with dir-as-associated-type.

Err codemadness.org 70 i 3039

Removing the newtypes and the macro

Err codemadness.org 70 i 3040

After that temporary refactoring, we had the Orientation trait and Err codemadness.org 70 i 3041 the three zero-sized types Horizontal, Vertical, Both.

Err codemadness.org 70 i 3042

I figured out that one can use PhantomData as a way to carry Err codemadness.org 70 i 3043 around the type that Length needs to normalize itself, instead of Err codemadness.org 70 i 3044 using an associated type in an extra LengthTrait. Behold!

Err codemadness.org 70 i 3045

pub struct Length<O: Orientation> {	Err	codemadness.org	70
i  3046     pub length: f64,	Err	codemadness.org	70
i  3047     pub unit: LengthUnit,	Err	codemadness.org	70
i  3048     orientation: PhantomData<O>,	Err	codemadness.org	70
i  3049 }	Err	codemadness.org	70
i  3050 	Err	codemadness.org	70
i  3051 impl<O: Orientation> Length<O> {	Err	codemadness.org	70
i  3052     pub fn normalize(&self, values: &ComputedValues, params: &ViewParams) -> f64 {	Err	codemadness.org	70
i  3053         match self.unit {	Err	codemadness.org	70
i  3054             LengthUnit::Px => self.length,	Err	codemadness.org	70
i  3055 	Err	codemadness.org	70
i  3056             LengthUnit::Percent => {	Err	codemadness.org	70
i  3057                 self.length 	Err	codemadness.org	70
i  3058                     * <O as Orientation>::scaling_factor(params.view_box_width, params.view_box_height)	Err	codemadness.org	70
i  3059             }	Err	codemadness.org	70
i  3060 	Err	codemadness.org	70
i  3061             ...	Err	codemadness.org	70
i  3062         }	Err	codemadness.org	70
i  3063     }	Err	codemadness.org	70
i  3064 }	Err	codemadness.org	70
i  3065

Err codemadness.org 70 i 3066 Err codemadness.org 70 i 3067

Now the incantation is <O as Orientation>::scaling_factor() to call Err codemadness.org 70 i 3068 the method on the generic type; it is no longer an associated type in Err codemadness.org 70 i 3069 a trait.

Err codemadness.org 70 i 3070

With that, users of lengths look like this; here, our <circle> Err codemadness.org 70 i 3071 element from before:

Err codemadness.org 70 i 3072

pub struct Circle {	Err	codemadness.org	70
i  3073     cx: Length<Horizontal>,	Err	codemadness.org	70
i  3074     cy: Length<Vertical>,	Err	codemadness.org	70
i  3075     r: Length<Both>,	Err	codemadness.org	70
i  3076 }	Err	codemadness.org	70
i  3077

Err codemadness.org 70 i 3078 Err codemadness.org 70 i 3079

I'm very happy with the readability of all the code now. I used to Err codemadness.org 70 i 3080 think of PhantomData as a way to deal with wrapping pointers from Err codemadness.org 70 i 3081 C, but it turns out that it is also useful to keep a generic Err codemadness.org 70 i 3082 type around should one need it.

Err codemadness.org 70 i 3083

The final Length struct is this:

Err codemadness.org 70 i 3084

pub struct Length<O: Orientation> {	Err	codemadness.org	70
i  3085     pub length: f64,	Err	codemadness.org	70
i  3086     pub unit: LengthUnit,	Err	codemadness.org	70
i  3087     orientation: PhantomData<O>,	Err	codemadness.org	70
i  3088 }	Err	codemadness.org	70
i  3089

Err codemadness.org 70 i 3090 Err codemadness.org 70 i 3091

And it only takes up as much space as its length and unit fields; Err codemadness.org 70 i 3092 PhantomData is zero-sized after all.

Err codemadness.org 70 i 3093

(Later, we renamed Orientation to Normalize, but the code's Err codemadness.org 70 i 3094 structure remained the same.)

Err codemadness.org 70 i 3095

Summary

Err codemadness.org 70 i 3096

Over a couple of years, librsvg's type that represents CSS lengths Err codemadness.org 70 i 3097 went from a C representation along the lines of "all data in the world Err codemadness.org 70 i 3098 is an int", to a Rust representation that uses some interesting type Err codemadness.org 70 i 3099 trickery:

Err codemadness.org 70 i 3100

Err codemadness.org 70 i 3102
C struct with char for units.
Err codemadness.org 70 i 3103
Err codemadness.org 70 i 3105
C struct with a LengthUnits enum.
Err codemadness.org 70 i 3106
Err codemadness.org 70 i 3108
C struct without an embodied direction; each place that needs to Err codemadness.org 70 i 3109 normalize needs to get the orientation right.
Err codemadness.org 70 i 3110
Err codemadness.org 70 i 3112
C struct with a built-in direction as an extra field, done at Err codemadness.org 70 i 3113 initialization time.
Err codemadness.org 70 i 3114
Err codemadness.org 70 i 3116
Same struct but in Rust.
Err codemadness.org 70 i 3117
Err codemadness.org 70 i 3119
An ugly but workable Parse trait so that the direction can be set Err codemadness.org 70 i 3120 at parse/initialization time.
Err codemadness.org 70 i 3121
Err codemadness.org 70 i 3123
Three newtypes LengthHorizontal, LengthVertical, Err codemadness.org 70 i 3124 LengthBoth with a common core. A cleaned-up Parse trait. A Err codemadness.org 70 i 3125 macro to generate those newtypes.
Err codemadness.org 70 i 3126
Err codemadness.org 70 i 3128
Replace the LengthDir enum with an Orientation Err codemadness.org 70 i 3129 trait, and three zero-sized types Horizontal/Vertical/Both that Err codemadness.org 70 i 3130 implement the trait.
Err codemadness.org 70 i 3131
Err codemadness.org 70 i 3133
Replace most of the macro with a helper trait LengthTrait that has Err codemadness.org 70 i 3134 an Orientation associated type.
Err codemadness.org 70 i 3135
Err codemadness.org 70 i 3137
Replace the helper trait with a single Length<T: Orientation> Err codemadness.org 70 i 3138 type, which puts the orientation as a generic parameter. The macro Err codemadness.org 70 i 3139 disappears and there is a single implementation for everything.
Err codemadness.org 70 i 3140

Err codemadness.org 70 i 3142

Refactoring never ends!

CSS in librsvg is now in Rust, courtesy of Mozilla Servo

2019-11-11T19:36:04-06:00

Summary: after an epic amount of Err codemadness.org 70 i 3149 refactoring, Err codemadness.org 70 i 3150 librsvg now does all CSS parsing and matching in Rust, without using Err codemadness.org 70 i 3151 libcroco. In addition, the CSS engine comes from Mozilla Servo, so Err codemadness.org 70 i 3152 it should be able to handle much more complex CSS than librsvg ever Err codemadness.org 70 i 3153 could before.

Err codemadness.org 70 i 3154

This is the story of CSS support in librsvg.

Err codemadness.org 70 i 3155

Introduction

Err codemadness.org 70 i 3156

The first commit to introduce CSS Err codemadness.org 70 i 3157 parsing Err codemadness.org 70 i 3158 in librsvg dates from 2002. It was as minimal as possible, written to Err codemadness.org 70 i 3159 support a small subset of what was then Err codemadness.org 70 i 3160 CSS2.

Err codemadness.org 70 i 3161

Librsvg handled CSS stylesheets more "piecing them apart" than Err codemadness.org 70 i 3162 "parsing them". You know, when g_strsplit() is your best friend. Err codemadness.org 70 i 3163 The basic parsing algorithm was to turn a stylesheet like this:

Err codemadness.org 70 i 3164

rect { fill: blue; }	Err	codemadness.org	70
i  3165 	Err	codemadness.org	70
i  3166 .classname {	Err	codemadness.org	70
i  3167     fill: green;	Err	codemadness.org	70
i  3168     stroke-width: 4;	Err	codemadness.org	70
i  3169 }	Err	codemadness.org	70
i  3170

Err codemadness.org 70 i 3171 Err codemadness.org 70 i 3172

Into a hash table whose keys are strings like rect and .classname, Err codemadness.org 70 i 3173 and whose values are everything inside curly braces.

Err codemadness.org 70 i 3174

The selector matching phase was equally simple. The code only handled Err codemadness.org 70 i 3175 a few possible match types as follows. If it wanted to match a Err codemadness.org 70 i 3176 certain kind of CSS selector, it would say, "what would this selector Err codemadness.org 70 i 3177 look like in CSS syntax", it would make up a string with that syntax, Err codemadness.org 70 i 3178 and compare it to the key strings it had stored in the hash table from Err codemadness.org 70 i 3179 above.

Err codemadness.org 70 i 3180

So, to match an element name selector, it would sprintf("%s", Err codemadness.org 70 i 3181 element->name), obtain something like rect and see if the hash Err codemadness.org 70 i 3182 table had such a key.

Err codemadness.org 70 i 3183

To match a class selector, it would sprintf(".%s", Err codemadness.org 70 i 3184 element->class), obtain something like .classname, and look it up Err codemadness.org 70 i 3185 in the hash table.

Err codemadness.org 70 i 3186

This scheme supported only a few combinations. It handled tag, Err codemadness.org 70 i 3187 .class, tag.class, and a few combinations with #id in them. Err codemadness.org 70 i 3188 This was enough to support very simple stylesheets.

Err codemadness.org 70 i 3189

The value corresponding to each key in the hash table was the stuff Err codemadness.org 70 i 3190 between curly braces in the stylesheet, so the second rule from the Err codemadness.org 70 i 3191 example above would contain fill: green; stroke-width: 4;. Once Err codemadness.org 70 i 3192 librsvg decided that an SVG element matched that CSS rule, it would Err codemadness.org 70 i 3193 re-parse the string with the CSS properties and apply them to the Err codemadness.org 70 i 3194 element's style.

Err codemadness.org 70 i 3195

I'm amazed that so little code was enough to deal with a good number Err codemadness.org 70 i 3196 of SVG files with stylesheets. I suspect that this was due to a few Err codemadness.org 70 i 3197 things:

Err codemadness.org 70 i 3198

Err codemadness.org 70 i 3200
While people were using complex CSS in HTML all the time, it was Err codemadness.org 70 i 3201 less common for SVG...
Err codemadness.org 70 i 3202
Err codemadness.org 70 i 3204
... because CSS2 was somewhat new, and the SVG spec was still being Err codemadness.org 70 i 3205 written...
Err codemadness.org 70 i 3206
Err codemadness.org 70 i 3208
... and SVGs created with illustration programs don't really use Err codemadness.org 70 i 3209 stylesheets; they include the full style information inside each Err codemadness.org 70 i 3210 element instead of symbolically referencing it from a stylesheet.
Err codemadness.org 70 i 3211

Err codemadness.org 70 i 3213

From the kinds of bugs that librsvg has gotten around "CSS support is Err codemadness.org 70 i 3214 too limited", it feels like SVGs which use CSS features are either Err codemadness.org 70 i 3215 hand-written, or machine-generated from custom programs like data Err codemadness.org 70 i 3216 plotting software. Illustration programs tend to list all style Err codemadness.org 70 i 3217 properties explicitly in each SVG element, and don't use CSS.

Err codemadness.org 70 i 3218

Libcroco appears

Err codemadness.org 70 i 3219

The first commit to introduce Err codemadness.org 70 i 3220 libcroco Err codemadness.org 70 i 3221 was to do CSS parsing, from March 2003.

Err codemadness.org 70 i 3222

At the same time, libcroco was introducing code to do CSS matching. Err codemadness.org 70 i 3223 However, this code never got used in librsvg; it still kept its simple Err codemadness.org 70 i 3224 string-based matcher. Maybe libcroco's API was not ready?

Err codemadness.org 70 i 3225

Libcroco fell out of maintainership around the first half of 2005, and Err codemadness.org 70 i 3226 volunteers have kept fixing it since then.

Err codemadness.org 70 i 3227

Problems with librsvg's string matcher for CSS

Err codemadness.org 70 i 3228

The C implementation of CSS matching in librsvg remained basically Err codemadness.org 70 i 3229 untouched until 2018, when Paolo Borelli and I started porting the Err codemadness.org 70 i 3230 surrounding code to Rust.

Err codemadness.org 70 i 3231

I had a lot of trouble figuring out the concepts from the code. I Err codemadness.org 70 i 3232 didn't know all the terminology of CSS Err codemadness.org 70 i 3233 implementations, Err codemadness.org 70 i 3234 and librsvg didn't use it, either.

Err codemadness.org 70 i 3235

I think that librsvg's code suffered from what the refactoring Err codemadness.org 70 i 3236 literature calls primitive Err codemadness.org 70 i 3237 obsession. Err codemadness.org 70 i 3238 Instead of having a parsed representation of CSS selectors, librsvg Err codemadness.org 70 i 3239 just stored a stringified version of them. So, a selector like Err codemadness.org 70 i 3240 rect#classname really was stored with a string like that, instead of Err codemadness.org 70 i 3241 an actual decomposition into structs.

Err codemadness.org 70 i 3242

Moreover, things were misnamed. This is the field that stored Err codemadness.org 70 i 3243 stylesheet data inside an RsvgHandle:

Err codemadness.org 70 i 3244

    GHashTable *css_props;	Err	codemadness.org	70
i  3245

Err codemadness.org 70 i 3246 Err codemadness.org 70 i 3247

From just looking at the field declaration, this doesn't tell me Err codemadness.org 70 i 3248 anything about what kind of data is stored there. One has to grep the Err codemadness.org 70 i 3249 source code for where that field is used:

Err codemadness.org 70 i 3250

static void	Err	codemadness.org	70
i  3251 rsvg_css_define_style (RsvgHandle * ctx,	Err	codemadness.org	70
i  3252                        const gchar * selector,	Err	codemadness.org	70
i  3253                        const gchar * style_name,	Err	codemadness.org	70
i  3254                        const gchar * style_value,	Err	codemadness.org	70
i  3255                        gboolean important)	Err	codemadness.org	70
i  3256 {	Err	codemadness.org	70
i  3257     GHashTable *styles;	Err	codemadness.org	70
i  3258 	Err	codemadness.org	70
i  3259     styles = g_hash_table_lookup (ctx->priv->css_props, selector);	Err	codemadness.org	70
i  3260

Err codemadness.org 70 i 3261 Err codemadness.org 70 i 3262

Okay, it looks up a selector by name in the css_props, and it Err codemadness.org 70 i 3263 gives back... another hash table styles? What's in there?

Err codemadness.org 70 i 3264

        g_hash_table_insert (styles,	Err	codemadness.org	70
i  3265                              g_strdup (style_name),	Err	codemadness.org	70
i  3266                              style_value_data_new (style_value, important));	Err	codemadness.org	70
i  3267

Err codemadness.org 70 i 3268 Err codemadness.org 70 i 3269

Another string key called style_name, whose key is a Err codemadness.org 70 i 3270 StyleValueData; what's in it?

Err codemadness.org 70 i 3271

typedef struct _StyleValueData {	Err	codemadness.org	70
i  3272     gchar *value;	Err	codemadness.org	70
i  3273     gboolean important;	Err	codemadness.org	70
i  3274 } StyleValueData;	Err	codemadness.org	70
i  3275

Err codemadness.org 70 i 3276 Err codemadness.org 70 i 3277

The value is another string. Strings all the way!

Err codemadness.org 70 i 3278

At the time, I didn't really figure out what each level of nested hash Err codemadness.org 70 i 3279 tables was supposed to mean. I didn't understand why we handled style Err codemadness.org 70 i 3280 properties in a completely different part of the code, and yet this Err codemadness.org 70 i 3281 part had a css_props field that didn't seem to store properties at all.

Err codemadness.org 70 i 3282

It took a while to realize that css_props was misnamed. It wasn't Err codemadness.org 70 i 3283 storing a mapping of selector names to properties; it was storing a Err codemadness.org 70 i 3284 mapping of selector names to declaration lists, which are lists of Err codemadness.org 70 i 3285 property/value pairs.

Err codemadness.org 70 i 3286

So, when I started porting the CSS parsing code to Err codemadness.org 70 i 3287 Rust, Err codemadness.org 70 i 3288 I started to create real types with for each concept.

Err codemadness.org 70 i 3289

// Maps property_name -> Declaration	Err	codemadness.org	70
i  3290 type DeclarationList = HashMap<String, Declaration>;	Err	codemadness.org	70
i  3291 	Err	codemadness.org	70
i  3292 pub struct CssStyles {	Err	codemadness.org	70
i  3293     selectors_to_declarations: HashMap<String, DeclarationList>,	Err	codemadness.org	70
i  3294 }	Err	codemadness.org	70
i  3295

Err codemadness.org 70 i 3296 Err codemadness.org 70 i 3297

Even though the keys of those HashMaps are still strings, because Err codemadness.org 70 i 3298 librsvg didn't have a better way to represent their corresponding Err codemadness.org 70 i 3299 concepts, at least those declarations let one see what the hell is Err codemadness.org 70 i 3300 being stored without grepping the rest of the code. This is a part of Err codemadness.org 70 i 3301 the code that I didn't really touch very much, so it was nice to have Err codemadness.org 70 i 3302 that reminder.

Err codemadness.org 70 i 3303

The first port of the CSS matching code to Err codemadness.org 70 i 3304 Rust Err codemadness.org 70 i 3305 kept the same algorithm as the C code, the one that created strings Err codemadness.org 70 i 3306 with element.class and compared them to the stored selector names. Err codemadness.org 70 i 3307 Ugly, but it still worked in the same limited fashion.

Err codemadness.org 70 i 3308

Rustifying the CSS parsers

Err codemadness.org 70 i 3309

It turns out that CSS parsing is divided in two parts. One can have a Err codemadness.org 70 i 3310 style attribute inside an element, for example

Err codemadness.org 70 i 3311

<rect x="0" y="0" width="100" height="100"	Err	codemadness.org	70
i  3312       style="fill: green; stroke: magenta; stroke-width: 4;"/>	Err	codemadness.org	70
i  3313

Err codemadness.org 70 i 3314 Err codemadness.org 70 i 3315

This is a plain declaration list which is not associated to any Err codemadness.org 70 i 3316 selectors, and which is applied directly to just the element in which it Err codemadness.org 70 i 3317 appears.

Err codemadness.org 70 i 3318

Then, there is the <style> element itself, with a normal-looking CSS stylesheet

Err codemadness.org 70 i 3319

<style type="text/css">	Err	codemadness.org	70
i  3320   rect {	Err	codemadness.org	70
i  3321     fill: green;	Err	codemadness.org	70
i  3322     stroke: magenta;	Err	codemadness.org	70
i  3323     stroke-width: 4;	Err	codemadness.org	70
i  3324   }	Err	codemadness.org	70
i  3325 </style>	Err	codemadness.org	70
i  3326

Err codemadness.org 70 i 3327 Err codemadness.org 70 i 3328

This means that all <rect> elements will get that style applied.

Err codemadness.org 70 i 3329

I started to look for existing Rust crates to parse and handle CSS Err codemadness.org 70 i 3330 data. The cssparser and Err codemadness.org 70 i 3331 selectors crates come from Mozilla, so Err codemadness.org 70 i 3332 I thought they should do a pretty good job of things.

Err codemadness.org 70 i 3333

And they do! Except that they are not a drop-in replacement for Err codemadness.org 70 i 3334 anything. They are what gets used in Mozilla's Servo browser engine, Err codemadness.org 70 i 3335 so they are optimized to hell, and the code can be pretty intimidating.

Err codemadness.org 70 i 3336

Out of the box, cssparser provides a CSS tokenizer, but it does Err codemadness.org 70 i 3337 not know how to handle any properties/values in particular. One must Err codemadness.org 70 i 3338 use the tokenizer to implement a parser for each kind of CSS property Err codemadness.org 70 i 3339 one wants to support — Servo has mountains of code for all of HTML's Err codemadness.org 70 i 3340 style properties, and librsvg had to provide a smaller mountain of code Err codemadness.org 70 i 3341 for SVG style properties.

Err codemadness.org 70 i 3342

Thus started the big task of porting librsvg's string-based parsers Err codemadness.org 70 i 3343 for CSS properties into ones based on cssparser tokens. Cssparser Err codemadness.org 70 i 3344 provides a Parser struct, which extracts tokens out of a CSS Err codemadness.org 70 i 3345 stream. Out of this, librsvg defines a Parse trait for parsable Err codemadness.org 70 i 3346 things:

Err codemadness.org 70 i 3347

use cssparser::Parser;	Err	codemadness.org	70
i  3348 	Err	codemadness.org	70
i  3349 pub trait Parse: Sized {	Err	codemadness.org	70
i  3350     type Err;	Err	codemadness.org	70
i  3351 	Err	codemadness.org	70
i  3352     fn parse(parser: &mut Parser<'_, '_>) -> Result<Self, Self::Err>;	Err	codemadness.org	70
i  3353 }	Err	codemadness.org	70
i  3354

Err codemadness.org 70 i 3355 Err codemadness.org 70 i 3356

What's with those two default lifetimes in Parser<'_, '_>? Err codemadness.org 70 i 3357 Cssparser tries very hard to be a zero-copy tokenizer. One of the Err codemadness.org 70 i 3358 lifetimes refers to the input string which is wrapped in a Err codemadness.org 70 i 3359 Tokenizer, which is wrapped in a ParserInput. The other lifetime Err codemadness.org 70 i 3360 is for the ParserInput itself.

Err codemadness.org 70 i 3361

In the actual implementation of that trait, the Err type also uses Err codemadness.org 70 i 3362 the lifetime that refers to the input string. For example, there is a Err codemadness.org 70 i 3363 BasicParseErrorKind::UnexpectedToken(Token<'i>), which one returns Err codemadness.org 70 i 3364 when there is an unexpected token. And to avoid copying the substring Err codemadness.org 70 i 3365 into the error, one returns a slice reference into the original Err codemadness.org 70 i 3366 string, thus the lifetime.

Err codemadness.org 70 i 3367

I was more of a Rust newbie back then, and it was very hard to make Err codemadness.org 70 i 3368 sense of how cssparser was meant to be used.

Err codemadness.org 70 i 3369

The process was more or less this:

Err codemadness.org 70 i 3370

Err codemadness.org 70 i 3372
Port the C parsers to Rust; implement types for each CSS property.
Err codemadness.org 70 i 3373
Err codemadness.org 70 i 3375
Port the &str-based parsers into ones that use cssparser.
Err codemadness.org 70 i 3376
Err codemadness.org 70 i 3378
Fix the error handling scheme to match what cssparser's high-level Err codemadness.org 70 i 3379 traits expect.
Err codemadness.org 70 i 3380

Err codemadness.org 70 i 3382

This last point was... hard. Again, I wasn't comfortable enough with Err codemadness.org 70 i 3383 Rust lifetimes and nested generics; in the end it was all right.

Err codemadness.org 70 i 3384

Moving declaration lists to Rust

Err codemadness.org 70 i 3385

With the individual parsers for CSS properties done, and with them Err codemadness.org 70 i 3386 already using a different type for each property, the next thing was Err codemadness.org 70 i 3387 to implement cssparser's traits to parse declaration lists.

Err codemadness.org 70 i 3388

Again, a declaration list looks like this:

Err codemadness.org 70 i 3389

fill: blue;	Err	codemadness.org	70
i  3390 stroke-width: 4;	Err	codemadness.org	70
i  3391

Err codemadness.org 70 i 3392 Err codemadness.org 70 i 3393

It's essentially a key/value list.

Err codemadness.org 70 i 3394

The trait that cssparser wants us to implement is this:

Err codemadness.org 70 i 3395

pub trait DeclarationParser<'i> {	Err	codemadness.org	70
i  3396     type Declaration;	Err	codemadness.org	70
i  3397     type Error: 'i;	Err	codemadness.org	70
i  3398 	Err	codemadness.org	70
i  3399     fn parse_value<'t>(	Err	codemadness.org	70
i  3400         &mut self,	Err	codemadness.org	70
i  3401         name: CowRcStr<'i>,	Err	codemadness.org	70
i  3402         input: &mut Parser<'i, 't>,	Err	codemadness.org	70
i  3403     ) -> Result<Self::Declaration, ParseError<'i, Self::Error>>;	Err	codemadness.org	70
i  3404 }	Err	codemadness.org	70
i  3405

Err codemadness.org 70 i 3406 Err codemadness.org 70 i 3407

That is, define a type for a Declaration, and implement a Err codemadness.org 70 i 3408 parse_value() method that takes a name and a Parser, and outputs Err codemadness.org 70 i 3409 a Declaration or an error.

Err codemadness.org 70 i 3410

What this really means is that the type you implement for Err codemadness.org 70 i 3411 Declaration needs to be able to represent all the CSS property types Err codemadness.org 70 i 3412 that you care about. Thus, a struct plus a big enum like this:

Err codemadness.org 70 i 3413

pub struct Declaration {	Err	codemadness.org	70
i  3414     pub prop_name: String,	Err	codemadness.org	70
i  3415     pub property: ParsedProperty,	Err	codemadness.org	70
i  3416     pub important: bool,	Err	codemadness.org	70
i  3417 }	Err	codemadness.org	70
i  3418 	Err	codemadness.org	70
i  3419 pub enum ParsedProperty {	Err	codemadness.org	70
i  3420     BaselineShift(SpecifiedValue<BaselineShift>),	Err	codemadness.org	70
i  3421     ClipPath(SpecifiedValue<ClipPath>),	Err	codemadness.org	70
i  3422     ClipRule(SpecifiedValue<ClipRule>),	Err	codemadness.org	70
i  3423     Color(SpecifiedValue<Color>),	Err	codemadness.org	70
i  3424     ColorInterpolationFilters(SpecifiedValue<ColorInterpolationFilters>),	Err	codemadness.org	70
i  3425     Direction(SpecifiedValue<Direction>),	Err	codemadness.org	70
i  3426     ...	Err	codemadness.org	70
i  3427 }	Err	codemadness.org	70
i  3428

Err codemadness.org 70 i 3429 Err codemadness.org 70 i 3430

This gives us declaration lists (the stuff inside curly braces in a Err codemadness.org 70 i 3431 CSS stylesheet), but it doesn't give us qualified rules, which are Err codemadness.org 70 i 3432 composed of selector names plus a declaration list.

Err codemadness.org 70 i 3433

Refactoring towards real CSS concepts

Err codemadness.org 70 i 3434

Paolo Borelli has been steadily refactoring librsvg and fixing things Err codemadness.org 70 i 3435 like the primitive obsession I mentioned above. We now have real Err codemadness.org 70 i 3436 concepts like a Document, Stylesheet, QualifiedRule, Rule, AtRule.

Err codemadness.org 70 i 3437

This refactoring took a long time, because it involved redoing the XML Err codemadness.org 70 i 3438 loading code and its interaction with the CSS parser a few times.

Err codemadness.org 70 i 3439

Implementing traits from the selectors crate

Err codemadness.org 70 i 3440

The selectors crate Err codemadness.org 70 i 3441 contains Servo's code for parsing CSS selectors and doing matching. Err codemadness.org 70 i 3442 However, it is extremely generic. Using it involves implementing a Err codemadness.org 70 i 3443 good number of concepts.

Err codemadness.org 70 i 3444

For example, this SelectorImpl trait has no methods, and is just a Err codemadness.org 70 i 3445 collection of types that refer to your implementation of an element Err codemadness.org 70 i 3446 tree. How do you represent an attribute/value? How do you represent Err codemadness.org 70 i 3447 an identifier? How do you represent a namespace and a local name?

Err codemadness.org 70 i 3448

pub trait SelectorImpl {	Err	codemadness.org	70
i  3449     type ExtraMatchingData: ...;	Err	codemadness.org	70
i  3450     type AttrValue: ...;	Err	codemadness.org	70
i  3451     type Identifier: ...;	Err	codemadness.org	70
i  3452     type ClassName: ...;	Err	codemadness.org	70
i  3453     type PartName: ...;	Err	codemadness.org	70
i  3454     type LocalName: ...;	Err	codemadness.org	70
i  3455     type NamespaceUrl: ...;	Err	codemadness.org	70
i  3456     type NamespacePrefix: ...;	Err	codemadness.org	70
i  3457     type BorrowedNamespaceUrl: ...;	Err	codemadness.org	70
i  3458     type BorrowedLocalName: ...;	Err	codemadness.org	70
i  3459     type NonTSPseudoClass: ...;	Err	codemadness.org	70
i  3460     type PseudoElement: ...;	Err	codemadness.org	70
i  3461 }	Err	codemadness.org	70
i  3462

Err codemadness.org 70 i 3463 Err codemadness.org 70 i 3464

A lot of those can be String, but Servo has smarter things in store. Err codemadness.org 70 i 3465 I ended up using the markup5ever crate, which provides a string Err codemadness.org 70 i 3466 interning framework for markup and XML concepts like a LocalName, a Err codemadness.org 70 i 3467 Namespace, etc. This reduces memory consumption, because instead of Err codemadness.org 70 i 3468 storing string copies of element names everywhere, one just stores Err codemadness.org 70 i 3469 tokens for interned strings.

Err codemadness.org 70 i 3470

(In the meantime I had to implement support for XML namespaces, which Err codemadness.org 70 i 3471 the selectors code really wants, but which librsvg never supported.)

Err codemadness.org 70 i 3472

Then, the selectors crate wants you to say how your code implements an Err codemadness.org 70 i 3473 element tree. It has a monster trait Element:

Err codemadness.org 70 i 3474

pub trait Element {	Err	codemadness.org	70
i  3475     type Impl: SelectorImpl;	Err	codemadness.org	70
i  3476 	Err	codemadness.org	70
i  3477     fn opaque(&self) -> OpaqueElement;	Err	codemadness.org	70
i  3478 	Err	codemadness.org	70
i  3479     fn parent_element(&self) -> Option<Self>;	Err	codemadness.org	70
i  3480 	Err	codemadness.org	70
i  3481     fn parent_node_is_shadow_root(&self) -> bool;	Err	codemadness.org	70
i  3482 	Err	codemadness.org	70
i  3483     ...	Err	codemadness.org	70
i  3484 	Err	codemadness.org	70
i  3485     fn prev_sibling_element(&self) -> Option<Self>;	Err	codemadness.org	70
i  3486     fn next_sibling_element(&self) -> Option<Self>;	Err	codemadness.org	70
i  3487 	Err	codemadness.org	70
i  3488     fn has_local_name(	Err	codemadness.org	70
i  3489         &self, 	Err	codemadness.org	70
i  3490         local_name: &<Self::Impl as SelectorImpl>::BorrowedLocalName	Err	codemadness.org	70
i  3491     ) -> bool;	Err	codemadness.org	70
i  3492 	Err	codemadness.org	70
i  3493     fn has_id(	Err	codemadness.org	70
i  3494         &self,	Err	codemadness.org	70
i  3495         id: &<Self::Impl as SelectorImpl>::Identifier,	Err	codemadness.org	70
i  3496         case_sensitivity: CaseSensitivity,	Err	codemadness.org	70
i  3497     ) -> bool;	Err	codemadness.org	70
i  3498 	Err	codemadness.org	70
i  3499     ...	Err	codemadness.org	70
i  3500 }	Err	codemadness.org	70
i  3501

Err codemadness.org 70 i 3502 Err codemadness.org 70 i 3503

That is, when you provide an implementation of Element and Err codemadness.org 70 i 3504 SelectorImpl, the selectors crate will know how to navigate your Err codemadness.org 70 i 3505 element tree and ask it questions like, "does this element have the id Err codemadness.org 70 i 3506 #foo?"; "does this element have the name rect?". It makes perfect Err codemadness.org 70 i 3507 sense in the end, but it is quite intimidating when you are not 100% Err codemadness.org 70 i 3508 comfortable with webs of traits and associated types and generics with Err codemadness.org 70 i 3509 a bunch of trait bounds!

Err codemadness.org 70 i 3510

I tried implementing that trait twice in the last year, and failed. Err codemadness.org 70 i 3511 It turns out that its API needed a key Err codemadness.org 70 i 3512 fix that landed last Err codemadness.org 70 i 3513 June, but I didn't notice until a couple of weeks ago.

Err codemadness.org 70 i 3514

So?

Err codemadness.org 70 i 3515

Two days ago, Paolo and I committed the last code to be able to Err codemadness.org 70 i 3516 completely replace Err codemadness.org 70 i 3517 libcroco.

Err codemadness.org 70 i 3518

And, after implementing CSS specificity (which was easy now that we Err codemadness.org 70 i 3519 have real CSS concepts and a good pipeline for the CSS cascade), a Err codemadness.org 70 i 3520 bunch of very old bugs started falling down Err codemadness.org 70 i 3521 (1 Err codemadness.org 70 i 3522 2 Err codemadness.org 70 i 3523 3 Err codemadness.org 70 i 3524 4 Err codemadness.org 70 i 3525 5 Err codemadness.org 70 i 3526 6).

Err codemadness.org 70 i 3527

Now it is going to be easy to implement things like letting the Err codemadness.org 70 i 3528 application specify a user Err codemadness.org 70 i 3529 stylesheet. In Err codemadness.org 70 i 3530 particular, this should let GTK remove the rather egregious Err codemadness.org 70 i 3531 hack Err codemadness.org 70 i 3532 it has to recolor SVG icons while using librsvg indirectly.

Err codemadness.org 70 i 3533

Conclusion

Err codemadness.org 70 i 3534

This will appear in librsvg 2.47.1 — that version will no longer Err codemadness.org 70 i 3535 require libcroco.

Err codemadness.org 70 i 3536

As far as I know, the only module that still depends on libcroco (in Err codemadness.org 70 i 3537 GNOME or otherwise) is gnome-shell. It uses libcroco to parse CSS Err codemadness.org 70 i 3538 and get the basic structure of selectors so it can implement matching Err codemadness.org 70 i 3539 by hand.

Err codemadness.org 70 i 3540

Gnome-shell has some code which looks awfully similar to what librsvg Err codemadness.org 70 i 3541 had when it was written in C:

Err codemadness.org 70 i 3542

Err codemadness.org 70 i 3544
StTheme Err codemadness.org 70 i 3545 has the high-level CSS stylesheet parser and the selector matching code.
Err codemadness.org 70 i 3546
Err codemadness.org 70 i 3548
StThemeNode Err codemadness.org 70 i 3549 has the low-level CSS property parsers.
Err codemadness.org 70 i 3550

Err codemadness.org 70 i 3552

... and it turns out that those files come all the way from Err codemadness.org 70 i 3553 HippoCanvas, Err codemadness.org 70 i 3554 the CSS-aware canvas that Mugshot used! Mugshot was a circa-2006 Err codemadness.org 70 i 3555 pre-Facebook aggregator for social media data like blogs, Flickr Err codemadness.org 70 i 3556 pictures, etc. HippoCanvas also got used in Sugar, the GUI for Err codemadness.org 70 i 3557 One Laptop Per Child. Yes, our code is that old.

Err codemadness.org 70 i 3558

Libcroco is unmaintained, and has outstanding CVEs. I would be very Err codemadness.org 70 i 3559 happy to assist someone in porting gnome-shell's CSS code to Rust :)

Gdk-pixbuf modules - call for help

2019-09-11T17:54:03-05:00

I've been doing a little refactoring of gdk-pixbuf's crufty code, to Err codemadness.org 70 i 3564 see if the gripes from my braindump can be solved. For Err codemadness.org 70 i 3565 things where it is not obvious how to proceed, I've started taking Err codemadness.org 70 i 3566 more detailed notes in a gdk-pixbuf survey.

Err codemadness.org 70 i 3567

Today I was looking at which gdk-pixbuf modules are Err codemadness.org 70 i 3568 implemented by third parties, that is, which external projects provide Err codemadness.org 70 i 3569 their own image codecs pluggable into gdk-pixbuf.

Err codemadness.org 70 i 3570

And there are not that many!

Err codemadness.org 70 i 3571

The only four that I found are libheif, libopenraw, libwmf, Err codemadness.org 70 i 3572 librsvg (this last one, of course).

Err codemadness.org 70 i 3573

Update 2019/Sep/12 - Added apng, exif-raw, psd, pvr, vtf, webp, xcf.

Err codemadness.org 70 i 3574

All of those use the gdk-pixbuf module API in a remarkably similar Err codemadness.org 70 i 3575 fashion. Did they cut&paste each other's code? Did they do the Err codemadness.org 70 i 3576 simplest thing that didn't crash in gdk-pixbuf's checks for buggy Err codemadness.org 70 i 3577 loaders, which happens to be exactly what they do? Who knows! Either Err codemadness.org 70 i 3578 way, this makes future API changes in the modules a lot easier, since Err codemadness.org 70 i 3579 they all do the same right now.

Err codemadness.org 70 i 3580

I'm trying to decide between these:

Err codemadness.org 70 i 3581

Err codemadness.org 70 i 3583
Keep modules as they are; find a way to sandbox them from gdk-pixbuf Err codemadness.org 70 i 3584 itself. This is hard because the API is "chatty"; modules and Err codemadness.org 70 i 3585 calling code go back and forth peeking at each other's structures.
Err codemadness.org 70 i 3586
Err codemadness.org 70 i 3588
Decide that third-party modules are only useful for thumbnailers; Err codemadness.org 70 i 3589 modify them to be thumbnailers instead of generic gdk-pixbuf Err codemadness.org 70 i 3590 modules. This would mean that those formats would stop working Err codemadness.org 70 i 3591 automatically in gdk-pixbuf based viewers like EOG.
Err codemadness.org 70 i 3592
Err codemadness.org 70 i 3594
Have "blessed" codecs inside gdk-pixbuf which are not modules so Err codemadness.org 70 i 3595 their no longer have API/ABI stability constraints. Keep Err codemadness.org 70 i 3596 third-party modules separate. Sandbox the internal ones with a Err codemadness.org 70 i 3597 non-chatty API.
Err codemadness.org 70 i 3598
Err codemadness.org 70 i 3600
If all third-party modules work indeed as I found, the Err codemadness.org 70 i 3601 module API can be simplified quite a lot since no third-party Err codemadness.org 70 i 3602 modules implement animations or saving. If so, simplify the module Err codemadness.org 70 i 3603 API and the gdk-pixbuf internals rather drastically.
Err codemadness.org 70 i 3604

Err codemadness.org 70 i 3606

Do you know any other image formats which provide gdk-pixbuf Err codemadness.org 70 i 3607 modules? Mail me, please!

On responsible vulnerability disclosure

2019-08-10T21:05:34-05:00

Recently KDE had an unfortunate event. Someone found a Err codemadness.org 70 i 3612 vulnerability in the code that processes .desktop and Err codemadness.org 70 i 3613 .directory files, through which an attacker could create a malicious Err codemadness.org 70 i 3614 file that causes shell command execution (analysis). They went for immediate, Err codemadness.org 70 i 3615 full disclosure, where KDE didn't even get a chance of fixing the bug Err codemadness.org 70 i 3616 before it was published.

Err codemadness.org 70 i 3617

There are many protocols for disclosing vulnerabilities in a Err codemadness.org 70 i 3618 coordinated, responsible fashion, but the gist of them is this:

Err codemadness.org 70 i 3619

Err codemadness.org 70 i 3621
Someone finds a vulnerability in some software through studying Err codemadness.org 70 i 3622 some code, or some other mechanism.
Err codemadness.org 70 i 3623
Err codemadness.org 70 i 3625
They report the vulnerability to the software's author through some Err codemadness.org 70 i 3626 private channel. For free softare in particular, researchers can Err codemadness.org 70 i 3627 use Openwall's recommended process for Err codemadness.org 70 i 3628 researchers, which includes notifying the Err codemadness.org 70 i 3629 author/maintainer and distros and security groups. Free Err codemadness.org 70 i 3630 software projects can follow a well-established process.
Err codemadness.org 70 i 3631
Err codemadness.org 70 i 3633
The author and reporter agree on a deadline for releasing a public Err codemadness.org 70 i 3634 report of the vulnerability, or in semi-automated systems like Err codemadness.org 70 i 3635 Google Zero, a deadline is automatically established.
Err codemadness.org 70 i 3636
Err codemadness.org 70 i 3638
The author works on fixing the vulnerability.
Err codemadness.org 70 i 3639
Err codemadness.org 70 i 3641
The deadline is reached; the patch has been publically released, Err codemadness.org 70 i 3642 the appropriate people have been notified, systems have been Err codemadness.org 70 i 3643 patched. If there is no patch, the author and reporter can agree Err codemadness.org 70 i 3644 on postponing the date, or the reporter can publish the Err codemadness.org 70 i 3645 vulnerability report, thus creating public pressure for a fix.
Err codemadness.org 70 i 3646

Err codemadness.org 70 i 3648

The steps above gloss over many practicalities and issues from the Err codemadness.org 70 i 3649 real world, but the idea is basically this: the author or maintainer Err codemadness.org 70 i 3650 of the software is given a chance to fix a security bug before Err codemadness.org 70 i 3651 information on the vulnerability is released to the hostile world. Err codemadness.org 70 i 3652 The idea is to keep harm from being done by not publishing Err codemadness.org 70 i 3653 unpatched vulnerabilities until there is a fix for them (... or until Err codemadness.org 70 i 3654 the deadline expires).

Err codemadness.org 70 i 3655

What happened instead

Err codemadness.org 70 i 3656

Around the beginning of July, the reporter posts about looking for Err codemadness.org 70 i 3657 bugs in KDE.

Err codemadness.org 70 i 3658

On July 30, he posts a video with the proof of concept.

Err codemadness.org 70 i 3659

On August 3, he makes a Twitter poll about what to do with the Err codemadness.org 70 i 3660 vulnerability.

Err codemadness.org 70 i 3661

On August 4, he publishes the vulnerability.

Err codemadness.org 70 i 3662

KDE is left with having to patch this in emergency mode. On August 7, Err codemadness.org 70 i 3663 KDE releases a security advisory in perfect form:

Err codemadness.org 70 i 3664

Err codemadness.org 70 i 3666
Description of exactly what causes the vulnerability.
Err codemadness.org 70 i 3667
Err codemadness.org 70 i 3669
Description of how it was solved.
Err codemadness.org 70 i 3670
Err codemadness.org 70 i 3672
Instructions on what to do for users of various versions of KDE Err codemadness.org 70 i 3673 libraries.
Err codemadness.org 70 i 3674
Err codemadness.org 70 i 3676
Links to easy-to-cherry-pick patches for distro vendors.
Err codemadness.org 70 i 3677

Err codemadness.org 70 i 3679

Now, distro vendors are, in turn, in emergency mode, as they must Err codemadness.org 70 i 3680 apply the patch, run it through QA, release their own advisories, Err codemadness.org 70 i 3681 etc.

Err codemadness.org 70 i 3682

What if this had been done with coordinated disclosure?

Err codemadness.org 70 i 3683

The bug would have been fixed, probably in the same way, but it would Err codemadness.org 70 i 3684 not be in emergency mode. KDE's advisory contains this:

Err codemadness.org 70 i 3685

Err codemadness.org 70 i 3686
Thanks to Dominik Penner for finding and documenting this issue (we wish however that he would Err codemadness.org 70 i 3687 have contacted us before making the issue public) and to David Faure for the fix.
Err codemadness.org 70 i 3688

Err codemadness.org 70 i 3689

This is an extremely gracious way of thanking the reporter.

Err codemadness.org 70 i 3690

I am not an infosec person...

Err codemadness.org 70 i 3691

... but some behaviors in the infosec sphere are deeply uncomfortable Err codemadness.org 70 i 3692 to me. I don't like it when security "research" is hard to tell from Err codemadness.org 70 i 3693 vandalism. "Excuse me, you left your car door unlocked" vs. "Hey Err codemadness.org 70 i 3694 everyone, this car is unlocked, have at it".

Err codemadness.org 70 i 3695

I don't know the details of the discourse in the infosec sphere around Err codemadness.org 70 i 3696 full disclosure against irresponsible vendors of proprietary software or Err codemadness.org 70 i 3697 services. However, KDE is free software! There is no need to be Err codemadness.org 70 i 3698 an asshole to them.

Constructors

2019-07-24T13:59:01-05:00

Have you ever had these annoyances with GObject-style constructors?

Err codemadness.org 70 i 3710

Err codemadness.org 70 i 3712
From a constructor, calling a method on a partially-constructed Err codemadness.org 70 i 3713 object is dangerous.
Err codemadness.org 70 i 3714
Err codemadness.org 70 i 3716
A constructor needs to set up "not quite initialized" values in the Err codemadness.org 70 i 3717 instance struct until a construct-time property is set.
Err codemadness.org 70 i 3718
Err codemadness.org 70 i 3720
You actually need to override GObjectClass::constructed (or was it Err codemadness.org 70 i 3721 ::constructor?) to take care of construct-only properties which Err codemadness.org 70 i 3722 need to be considered together, not individually.
Err codemadness.org 70 i 3723
Err codemadness.org 70 i 3725
Constructors can't report an error, unless you derive from Err codemadness.org 70 i 3726 GInitable, which is not in gobject, but in gio instead. (Also, Err codemadness.org 70 i 3727 why does that force the constructor to take a GCancellable...?)
Err codemadness.org 70 i 3728
Err codemadness.org 70 i 3730
You need more than one constructor, but that needs to be done with Err codemadness.org 70 i 3731 helper functions.
Err codemadness.org 70 i 3732

Err codemadness.org 70 i 3734

This article, Perils of Err codemadness.org 70 i 3735 Constructors, Err codemadness.org 70 i 3736 explains all of these problems very well. It is not centered on Err codemadness.org 70 i 3737 GObject, but rather on constructors in object-oriented languages in Err codemadness.org 70 i 3738 general.

Err codemadness.org 70 i 3739

(Spoiler: Rust does not have constructors or partially-initialized structs, so Err codemadness.org 70 i 3740 these problems don't really exist there.)

Err codemadness.org 70 i 3741

(Addendum: that makes it somewhat awkward to convert GObject code in Err codemadness.org 70 i 3742 C to Rust, but librsvg was able to solve it nicely with Err codemadness.org 70 i 3743 <buzzword>the typestate pattern</buzzword>.)

Gtk-rs tutorial

2019-07-08T18:36:11-05:00

Leonora Tindall has written a Err codemadness.org 70 i 3744 very nice tutorial on Speedy Desktop Apps With GTK and Err codemadness.org 70 i 3745 Rust. Err codemadness.org 70 i 3746 It covers prototyping a dice roller app with Glade, writing the code with Err codemadness.org 70 i 3747 Rust and the gtk-rs bindings, and integrating the app into the desktop with Err codemadness.org 70 i 3748 a .desktop file.

Removing rsvg-view

2019-07-02T11:36:59-05:00

I am preparing the 2.46.0 librsvg release. This will no longer have Err codemadness.org 70 i 3756 the rsvg-view-3 program.

Err codemadness.org 70 i 3757

History of rsvg-view

Err codemadness.org 70 i 3758

Rsvg-view started Err codemadness.org 70 i 3759 out Err codemadness.org 70 i 3760 as a 71-line C program to aid development of librsvg. It would just Err codemadness.org 70 i 3761 render an SVG file to a pixbuf, stick that pixbuf in a GtkImage Err codemadness.org 70 i 3762 widget, and show a window with that.

Err codemadness.org 70 i 3763

Over time, it slowly acquired most of the command-line options that Err codemadness.org 70 i 3764 rsvg-convert supports. And I suppose, as a way of testing the Err codemadness.org 70 i 3765 Cairo-ification of librsvg, it also got the ability to print SVG files Err codemadness.org 70 i 3766 to a GtkPrintContext. At last count, it was a 784-line C program Err codemadness.org 70 i 3767 that is not really the best code in the world.

Err codemadness.org 70 i 3768

What makes rsvg-view awkward?

Err codemadness.org 70 i 3769

Rsvg-view requires GTK. But GTK requires librsvg, indirectly, through Err codemadness.org 70 i 3770 gdk-pixbuf! There is not a hard circular dependency because GTK goes, Err codemadness.org 70 i 3771 "gdk-pixbuf, load me this SVG file" without knowing how it will be Err codemadness.org 70 i 3772 loaded. In turn, gdk-pixbuf initializes the SVG loader provided by Err codemadness.org 70 i 3773 librsvg, and that loader reads/renders the SVG file.

Err codemadness.org 70 i 3774

Ideally librsvg would only depend on gdk-pixbuf, so it would be able Err codemadness.org 70 i 3775 to provide the SVG loader.

Err codemadness.org 70 i 3776

The rsvg-view source code still has a few calls to GTK functions which Err codemadness.org 70 i 3777 are now deprecated. The program emits GTK warnings during normal use.

Err codemadness.org 70 i 3778

Rsvg-view is... not a very good SVG viewer. It doesn't even start up Err codemadness.org 70 i 3779 with the window scaled properly to the SVG's dimensions! If used for Err codemadness.org 70 i 3780 quick testing during development, it cannot even aid in viewing the Err codemadness.org 70 i 3781 transparent background regions which the SVG does not cover. It just Err codemadness.org 70 i 3782 sticks a lousy custom widget inside a GtkScrolledWindow, and does Err codemadness.org 70 i 3783 not have the conventional niceties to view images like zooming with Err codemadness.org 70 i 3784 the scroll wheel.

Err codemadness.org 70 i 3785

EOG is a much better SVG viewer than rsvg-view, and people actually Err codemadness.org 70 i 3786 invest effort in making it pleasant to use.

Err codemadness.org 70 i 3787

Removal of rsvg-view

Err codemadness.org 70 i 3788

So, the next version of librsvg will not provide the rsvg-view-3 Err codemadness.org 70 i 3789 binary. Please update your packages accordingly. Distros may be able to Err codemadness.org 70 i 3790 move the compilation of librsvg to a more sensible place in the Err codemadness.org 70 i 3791 platform stack, now that it doesn't depend on GTK being available.

Err codemadness.org 70 i 3792

What can you use instead? Any other image viewer. EOG works fine; Err codemadness.org 70 i 3793 there are dozens of other good viewers, too.

Bzip2 1.0.7 is released

2019-06-27T13:55:38-05:00

Bzip2 1.0.7 has been released by Mark Wielaard. We have a Err codemadness.org 70 i 3800 slight change of plans since my last post:

Err codemadness.org 70 i 3801

Err codemadness.org 70 i 3803
The 1.0.x series is in strict maintenance mode and will not change Err codemadness.org 70 i 3804 build systems. This is targeted towards embedded use, as in Err codemadness.org 70 i 3805 projects which already embed the bzip2-1.0.6 sources and undoubtedly Err codemadness.org 70 i 3806 patch the build system. Right now this series, and the tagged 1.0.7 Err codemadness.org 70 i 3807 release, live in the sourceware repository for bzip2.
Err codemadness.org 70 i 3808
Err codemadness.org 70 i 3810
The 1.1.x series has Meson and CMake build systems, and a couple of Err codemadness.org 70 i 3811 extra changes to modernize the C code but which were not fit for the Err codemadness.org 70 i 3812 1.0.7 release. This is targeted towards operating system Err codemadness.org 70 i 3813 distributions. This lives in the master branch of the gitlab Err codemadness.org 70 i 3814 repository for bzip2.
Err codemadness.org 70 i 3815

Err codemadness.org 70 i 3817

Distros and embedded users should start using bzip2-1.0.7 Err codemadness.org 70 i 3818 immediately. The patches they already have for the bzip2's Err codemadness.org 70 i 3819 traditional build system should still apply. The release includes bug Err codemadness.org 70 i 3820 fixes and security fixes that have accumulated over the years, Err codemadness.org 70 i 3821 including the new CVE-2019-12900.

Err codemadness.org 70 i 3822

Once 1.1.0 is released, distributions should be able to remove their Err codemadness.org 70 i 3823 patches to the build system and just start using Meson or CMake. You Err codemadness.org 70 i 3824 may want to monitor the 1.1.0 milestone — help is Err codemadness.org 70 i 3825 appreciated fixing the issues there so we can make the first release Err codemadness.org 70 i 3826 with the new build systems!

Preparing the bzip2-1.0.7 release

2019-06-20T11:47:26-05:00

ATTENTION ALL DISTRIBUTIONS: this is for you. THE SONAME MAY CHANGE!

Err codemadness.org 70 i 3836

I am preparing a bzip2-1.0.7 release. You can see the release Err codemadness.org 70 i 3837 notes, which should be of interest:

Err codemadness.org 70 i 3838

Err codemadness.org 70 i 3840
Many historical patches from various distributions are integrated Err codemadness.org 70 i 3841 now.
Err codemadness.org 70 i 3842
Err codemadness.org 70 i 3844
We have a new fix for the just-published CVE-2019-12900, courtesy of Err codemadness.org 70 i 3845 Albert Astals Cid.
Err codemadness.org 70 i 3846
Err codemadness.org 70 i 3848
Bzip2 has moved to Meson for its preferred build system, Err codemadness.org 70 i 3849 courtesy of Dylan Baker. For special situations, a CMake build Err codemadness.org 70 i 3850 system is also provided, courtesy of Micah Snyder.
Err codemadness.org 70 i 3851

Err codemadness.org 70 i 3853

What's with the soname?

Err codemadness.org 70 i 3854

From bzip2-1.0.1 (from the year 2000), until bzip2-1.0.6 (from 2010), Err codemadness.org 70 i 3855 release tarballs came with a special Err codemadness.org 70 i 3856 Makefile-libbz2_so to generate a shared library Err codemadness.org 70 i 3857 instead of a static one.

Err codemadness.org 70 i 3858

This never used libtool or anything; it specified linker flags by Err codemadness.org 70 i 3859 hand. Various distributions either patched this special makefile, or Err codemadness.org 70 i 3860 replaced it by another one, or outright replaced the complete build Err codemadness.org 70 i 3861 system for a different one.

Err codemadness.org 70 i 3862

Some things to note:

Err codemadness.org 70 i 3863

Err codemadness.org 70 i 3865
This hand-written Makefile-libbz2_so used a link Err codemadness.org 70 i 3866 line like $(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o Err codemadness.org 70 i 3867 libbz2.so.1.0.6. This means, make the DT_SONAME field inside the Err codemadness.org 70 i 3868 ELF file be libbz2.so.1.0 (note the two digits in 1.0), and make Err codemadness.org 70 i 3869 the filename of the shared library be libbz2.so.1.0.6.
Err codemadness.org 70 i 3870
Err codemadness.org 70 i 3872
Fedora patched the soname in a patch called "saneso" to just be Err codemadness.org 70 i 3873 libbz2.so.1.
Err codemadness.org 70 i 3874
Err codemadness.org 70 i 3876
Stanislav Brabec, from openSUSE, replaced the hand-written Err codemadness.org 70 i 3877 makefiles with autotools, which meant using libtool. It Err codemadness.org 70 i 3878 has this interesting note:
Err codemadness.org 70 i 3879

Err codemadness.org 70 i 3881

Err codemadness.org 70 i 3882
Incompatible changes:
Err codemadness.org 70 i 3883
soname change. Libtool has no support for two parts soname suffix (e. g. Err codemadness.org 70 i 3884 libbz2.so.1.0). It must be a single number (e. g. libbz2.so.1). That is Err codemadness.org 70 i 3885 why soname must change. But I see not a big problem with it. Several Err codemadness.org 70 i 3886 distributions already use the new number instead of the non-standard Err codemadness.org 70 i 3887 number from Makefile-libbz2_so.
Err codemadness.org 70 i 3888

Err codemadness.org 70 i 3889

(In fact, if I do objdump -x /usr/lib64/*.so | grep SONAME, I see Err codemadness.org 70 i 3890 that most libraries have single-digit sonames.)

Err codemadness.org 70 i 3891

In my experience, both Fedora and openSUSE are very strict, and Err codemadness.org 70 i 3892 correct, about obscure things like library sonames.

Err codemadness.org 70 i 3893

With the switch to Meson, bzip2 no longer uses libtool. It will have Err codemadness.org 70 i 3894 a single-digit soname — this is not in the meson.build yet, but Err codemadness.org 70 i 3895 expect it to be there within the next couple of days.

Err codemadness.org 70 i 3896

I don't know what distros which decided to preserve the 1.0 soname Err codemadness.org 70 i 3897 will need to do; maybe they will need to patch meson.build on their Err codemadness.org 70 i 3898 own.

Err codemadness.org 70 i 3899

Fortunately, the API/ABI are still exactly the same. You can Err codemadness.org 70 i 3900 preserve the old soname which your distro was using and linking libbz2 Err codemadness.org 70 i 3901 will probably keep working as usual.

Err codemadness.org 70 i 3902

(This is a C-only release as usual. The Rust branch is still Err codemadness.org 70 i 3903 experimental.)

Bzip2 in Rust: porting the randomization table

2019-06-11T14:30:17-05:00

Here is a straightforward port of some easy code.

Err codemadness.org 70 i 3918

randtable.c has a lookup table with seemingly-random Err codemadness.org 70 i 3919 numbers. This table is used by the following macros in Err codemadness.org 70 i 3920 bzlib_private.h:

Err codemadness.org 70 i 3921

extern Int32 BZ2_rNums[512];	Err	codemadness.org	70
i  3922 	Err	codemadness.org	70
i  3923 #define BZ_RAND_DECLS                          \	Err	codemadness.org	70
i  3924    Int32 rNToGo;                               \	Err	codemadness.org	70
i  3925    Int32 rTPos                                 \	Err	codemadness.org	70
i  3926 	Err	codemadness.org	70
i  3927 #define BZ_RAND_INIT_MASK                      \	Err	codemadness.org	70
i  3928    s->rNToGo = 0;                              \	Err	codemadness.org	70
i  3929    s->rTPos  = 0                               \	Err	codemadness.org	70
i  3930 	Err	codemadness.org	70
i  3931 #define BZ_RAND_MASK ((s->rNToGo == 1) ? 1 : 0)	Err	codemadness.org	70
i  3932 	Err	codemadness.org	70
i  3933 #define BZ_RAND_UPD_MASK                       \	Err	codemadness.org	70
i  3934    if (s->rNToGo == 0) {                       \	Err	codemadness.org	70
i  3935       s->rNToGo = BZ2_rNums[s->rTPos];         \	Err	codemadness.org	70
i  3936       s->rTPos++;                              \	Err	codemadness.org	70
i  3937       if (s->rTPos == 512) s->rTPos = 0;       \	Err	codemadness.org	70
i  3938    }                                           \	Err	codemadness.org	70
i  3939    s->rNToGo--;	Err	codemadness.org	70
i  3940

Err codemadness.org 70 i 3941 Err codemadness.org 70 i 3942

Here, BZ_RAND_DECLS is used to declare two fields, rNToGo and Err codemadness.org 70 i 3943 rTPos, into two structs (1, 2). Both are similar to this:

Err codemadness.org 70 i 3944

typedef struct {	Err	codemadness.org	70
i  3945    ...	Err	codemadness.org	70
i  3946    Bool     blockRandomised;	Err	codemadness.org	70
i  3947    BZ_RAND_DECLS	Err	codemadness.org	70
i  3948    ...	Err	codemadness.org	70
i  3949 } DState;	Err	codemadness.org	70
i  3950

Err codemadness.org 70 i 3951 Err codemadness.org 70 i 3952

Then, the code that needs to initialize those fields calls Err codemadness.org 70 i 3953 BZ_RAND_INIT_MASK, which expands into code to set the two fields to Err codemadness.org 70 i 3954 zero.

Err codemadness.org 70 i 3955

At several points in the code, BZ_RAND_UPD_MASK gets called, which Err codemadness.org 70 i 3956 expands into code that updates the randomization state, or something Err codemadness.org 70 i 3957 like that, and uses BZ_RAND_MASK to get a useful value out of the Err codemadness.org 70 i 3958 randomization state.

Err codemadness.org 70 i 3959

I have no idea yet what the state is about, but let's port it Err codemadness.org 70 i 3960 directly.

Err codemadness.org 70 i 3961

Give things a name

Err codemadness.org 70 i 3962

It's interesting to see that no code except for those macros uses Err codemadness.org 70 i 3963 the fields rNToGo and rTPos, which are declared via Err codemadness.org 70 i 3964 BZ_RAND_DECLS. So, let's make up a type with a name for that. Err codemadness.org 70 i 3965 Since I have no better name for it, I shall call it just Err codemadness.org 70 i 3966 RandState. I added that type definition in the C code, Err codemadness.org 70 i 3967 and replaced the macro-which-creates-struct-fields with a Err codemadness.org 70 i 3968 RandState-typed field:

Err codemadness.org 70 i 3969

-#define BZ_RAND_DECLS                          \	Err	codemadness.org	70
i  3970 -   Int32 rNToGo;                               \	Err	codemadness.org	70
i  3971 -   Int32 rTPos                                 \	Err	codemadness.org	70
i  3972 +typedef struct {	Err	codemadness.org	70
i  3973 +   Int32 rNToGo;	Err	codemadness.org	70
i  3974 +   Int32 rTPos;	Err	codemadness.org	70
i  3975 +} RandState;	Err	codemadness.org	70
i  3976 	Err	codemadness.org	70
i  3977 ...	Err	codemadness.org	70
i  3978 	Err	codemadness.org	70
i  3979 -      BZ_RAND_DECLS;	Err	codemadness.org	70
i  3980 +      RandState rand;	Err	codemadness.org	70
i  3981

Err codemadness.org 70 i 3982 Err codemadness.org 70 i 3983

Since the fields now live inside a sub-struct, I changed the other Err codemadness.org 70 i 3984 macros to use s->rand.rNToGo instead of s->rNToGo, and similarly Err codemadness.org 70 i 3985 for the other field.

Err codemadness.org 70 i 3986

Turn macros into functions

Err codemadness.org 70 i 3987

Now, three commits (1, 2, 3) to turn the Err codemadness.org 70 i 3988 macros BZ_RAND_INIT_MASK, BZ_RAND_MASK, and BZ_RAND_UPD_MASK Err codemadness.org 70 i 3989 into functions.

Err codemadness.org 70 i 3990

And now that the functions live in the same C source file as the Err codemadness.org 70 i 3991 lookup table they reference, the table can be made static const to Err codemadness.org 70 i 3992 avoid having it as read/write unshared data in the linked binary.

Err codemadness.org 70 i 3993

Premature optimization concern: doesn't de-inlining those macros Err codemadness.org 70 i 3994 cause performance problems? At first, we will get the added overhead Err codemadness.org 70 i 3995 from a function call. When the whole code is ported to Rust, the Rust Err codemadness.org 70 i 3996 compiler will probably be able to figure out that those tiny functions Err codemadness.org 70 i 3997 can be inlined (or we can #[inline] them by hand if we have proof, Err codemadness.org 70 i 3998 or if we have more hubris than faith in LLVM).

Err codemadness.org 70 i 3999

Port functions and table to Rust

Err codemadness.org 70 i 4000

The functions are so tiny, and the table so cut-and-pasteable, that Err codemadness.org 70 i 4001 it's easy to port them to Rust in a single shot:

Err codemadness.org 70 i 4002

#[no_mangle]	Err	codemadness.org	70
i  4003 pub unsafe extern "C" fn BZ2_rand_init() -> RandState {	Err	codemadness.org	70
i  4004     RandState {	Err	codemadness.org	70
i  4005         rNToGo: 0,	Err	codemadness.org	70
i  4006         rTPos: 0,	Err	codemadness.org	70
i  4007     }	Err	codemadness.org	70
i  4008 }	Err	codemadness.org	70
i  4009 	Err	codemadness.org	70
i  4010 #[no_mangle]	Err	codemadness.org	70
i  4011 pub unsafe extern "C" fn BZ2_rand_mask(r: &RandState) -> i32 {	Err	codemadness.org	70
i  4012     if r.rNToGo == 1 {	Err	codemadness.org	70
i  4013         1	Err	codemadness.org	70
i  4014     } else {	Err	codemadness.org	70
i  4015         0	Err	codemadness.org	70
i  4016     }	Err	codemadness.org	70
i  4017 }	Err	codemadness.org	70
i  4018 	Err	codemadness.org	70
i  4019 #[no_mangle]	Err	codemadness.org	70
i  4020 pub unsafe extern "C" fn BZ2_rand_update_mask(r: &mut RandState) {	Err	codemadness.org	70
i  4021     if r.rNToGo == 0 {	Err	codemadness.org	70
i  4022         r.rNToGo = RAND_TABLE[r.rTPos as usize];	Err	codemadness.org	70
i  4023         r.rTPos += 1;	Err	codemadness.org	70
i  4024         if r.rTPos == 512 {	Err	codemadness.org	70
i  4025             r.rTPos = 0;	Err	codemadness.org	70
i  4026         }	Err	codemadness.org	70
i  4027     }	Err	codemadness.org	70
i  4028     r.rNToGo -= 1;	Err	codemadness.org	70
i  4029 }	Err	codemadness.org	70
i  4030

Err codemadness.org 70 i 4031 Err codemadness.org 70 i 4032

Also, we define the RandState type as a Rust struct with a Err codemadness.org 70 i 4033 C-compatible representation, so it will have the same layout in memory Err codemadness.org 70 i 4034 as the C struct. This is what allows us to have a RandState in Err codemadness.org 70 i 4035 the C struct, while in reality the C code doesn't access it Err codemadness.org 70 i 4036 directly; it is just used as a struct field.

Err codemadness.org 70 i 4037

// Keep this in sync with bzlib_private.h:	Err	codemadness.org	70
i  4038 #[repr(C)]	Err	codemadness.org	70
i  4039 pub struct RandState {	Err	codemadness.org	70
i  4040     rNToGo: i32,	Err	codemadness.org	70
i  4041     rTPos: i32,	Err	codemadness.org	70
i  4042 }	Err	codemadness.org	70
i  4043

Err codemadness.org 70 i 4044 Err codemadness.org 70 i 4045

See the commit for the corresponding extern Err codemadness.org 70 i 4046 declarations in bzlib_private.h. With those functions and the table Err codemadness.org 70 i 4047 ported to Rust, we can remove randtable.c. Yay!

Err codemadness.org 70 i 4048

A few cleanups

Err codemadness.org 70 i 4049

After moving to another house one throws away useless boxes; we have Err codemadness.org 70 i 4050 to do some cleanup in the Rust code after the initial port, too.

Err codemadness.org 70 i 4051

Rust prefers snake_case fields rather than camelCase ones, and I Err codemadness.org 70 i 4052 agree. I renamed the fields to n_to_go and table_pos.

Err codemadness.org 70 i 4053

Then, I discovered that the EState struct doesn't actually use the Err codemadness.org 70 i 4054 fields for the randomization state. I just removed them.

Err codemadness.org 70 i 4055

Exegesis

Err codemadness.org 70 i 4056

What is that randomization state all about?

Err codemadness.org 70 i 4057

And why does DState (the struct used during decompression) need the Err codemadness.org 70 i 4058 randomization state, but EState (used during compression) doesn't Err codemadness.org 70 i 4059 need it?

Err codemadness.org 70 i 4060

I found this interesting comment:

Err codemadness.org 70 i 4061

      /*-- 	Err	codemadness.org	70
i  4062          Now a single bit indicating (non-)randomisation. 	Err	codemadness.org	70
i  4063          As of version 0.9.5, we use a better sorting algorithm	Err	codemadness.org	70
i  4064          which makes randomisation unnecessary.  So always set	Err	codemadness.org	70
i  4065          the randomised bit to 'no'.  Of course, the decoder	Err	codemadness.org	70
i  4066          still needs to be able to handle randomised blocks	Err	codemadness.org	70
i  4067          so as to maintain backwards compatibility with	Err	codemadness.org	70
i  4068          older versions of bzip2.	Err	codemadness.org	70
i  4069       --*/	Err	codemadness.org	70
i  4070       bsW(s,1,0);	Err	codemadness.org	70
i  4071

Err codemadness.org 70 i 4072 Err codemadness.org 70 i 4073

Okay! So compression no longer uses randomization, but Err codemadness.org 70 i 4074 decompression has to support files which were compressed with Err codemadness.org 70 i 4075 randomization. Here, bsW(s,1,0) always writes a 0 bit to the file.

Err codemadness.org 70 i 4076

However, the decompression code actually reads the blockRandomised Err codemadness.org 70 i 4077 bit from the file so that it can see whether it is Err codemadness.org 70 i 4078 dealing with an old-format file:

Err codemadness.org 70 i 4079

GET_BITS(BZ_X_RANDBIT, s->blockRandomised, 1);	Err	codemadness.org	70
i  4080

Err codemadness.org 70 i 4081 Err codemadness.org 70 i 4082

Later in the code, this s->blockRandomised field gets consulted; if Err codemadness.org 70 i 4083 the bit is on, the code calls BZ2_rand_update_mask() and friends as Err codemadness.org 70 i 4084 appropriate. If one is using files compressed with Bzip2 0.9.5 or Err codemadness.org 70 i 4085 later, those randomization functions are not even called.

Err codemadness.org 70 i 4086

Talk about preserving compatibility with the past.

Err codemadness.org 70 i 4087

Explanation, or building my headcanon

Err codemadness.org 70 i 4088

Bzip2's compression starts by running a Burrows-Wheeler Err codemadness.org 70 i 4089 Transform on a block of data to compress, which is a wonderful Err codemadness.org 70 i 4090 algorithm that I'm trying to fully understand. Part of the BWT Err codemadness.org 70 i 4091 involves sorting all the string rotations of the block in question.

Err codemadness.org 70 i 4092

Per the comment I cited, really old versions of bzip2 used a Err codemadness.org 70 i 4093 randomization helper to make sorting perform well in extreme cases, Err codemadness.org 70 i 4094 but not-so-old versions fixed this.

Err codemadness.org 70 i 4095

This explains why the decompression struct DState has a Err codemadness.org 70 i 4096 blockRandomised bit, but the compression struct EState doesn't Err codemadness.org 70 i 4097 need one. The fields that the original macro was pasting into Err codemadness.org 70 i 4098 EState were just a vestige from 1999, which is when Bzip2 0.9.5 was Err codemadness.org 70 i 4099 released.

Bzip2 uses Meson and Autotools now — and a plea for help

2019-06-07T11:01:44-05:00

There is a lot of activity in the bzip2 repository!

Err codemadness.org 70 i 4106

Perhaps the most exciting thing is that Dylan Baker made a merge Err codemadness.org 70 i 4107 request to add Meson as a build system for bzip2; this is Err codemadness.org 70 i 4108 merged now into the master branch.

Err codemadness.org 70 i 4109

The current status is this:

Err codemadness.org 70 i 4110

Both Meson and Autotools are supported.
We have CI runs for both build systems.

Err codemadness.org 70 i 4114

A plea for help: add CI runners for other platforms!

Err codemadness.org 70 i 4115

Do you use *BSD / Windows / Solaris / etc. and know how to make Err codemadness.org 70 i 4116 Gitlab's CI work for them?

Err codemadness.org 70 i 4117

The only runners we have now for bzip2 are for well-known Linux Err codemadness.org 70 i 4118 distros. I would really like to keep bzip2 working on non-Linux Err codemadness.org 70 i 4119 platforms. If you know how to make Gitlab CI runners for other Err codemadness.org 70 i 4120 systems, please send a merge request!

Err codemadness.org 70 i 4121

Why two build systems?

Err codemadness.org 70 i 4122

Mainly uncertainty on my part. I haven't used Meson extensively; Err codemadness.org 70 i 4123 people tell me that it works better than Autotools out of the box for Err codemadness.org 70 i 4124 Windows.

Err codemadness.org 70 i 4125

Bzip2 runs on all sorts of ancient systems, and I don't know whether Err codemadness.org 70 i 4126 Meson or Autotools will be a better fit for them. Time will tell. Err codemadness.org 70 i 4127 Hopefully in the future we can have only a single supported build Err codemadness.org 70 i 4128 system for bzip2.

Bzip2 repository reconstructed

2019-06-05T19:13:05-05:00

I have just done a git push --force-with-lease to bzip2's master Err codemadness.org 70 i 4134 branch, which means that if you had a previous clone of this Err codemadness.org 70 i 4135 repository, you'll have to re-fetch it and rebase any changes you may Err codemadness.org 70 i 4136 have on top.

Err codemadness.org 70 i 4137

I apologize for the inconvenience!

Err codemadness.org 70 i 4138

But I have a good excuse: Julian Seward pointed me to a repository Err codemadness.org 70 i 4139 at sourceware where Mark Wielaard reconstructed a commit Err codemadness.org 70 i 4140 history for bzip2, based on the historical tarballs starting from Err codemadness.org 70 i 4141 bzip2-0.1. Bzip2 was never maintained under revision control, so the Err codemadness.org 70 i 4142 reconstructed repository should be used mostly for historical Err codemadness.org 70 i 4143 reference (go look for bzip2.exe in the initial commit!).

Err codemadness.org 70 i 4144

I have rebased all the post-1.0.6 commits on top of Mark's repository; Err codemadness.org 70 i 4145 this is what is in the master branch now.

Err codemadness.org 70 i 4146

There is a new rustify branch as well, based on master, which is Err codemadness.org 70 i 4147 where I will do the gradual port to Rust.

Err codemadness.org 70 i 4148

I foresee no other force-pushes to the master branch in the future. Err codemadness.org 70 i 4149 Apologies again if this disrupts your workflow.

Err codemadness.org 70 i 4150

Update: Someone did another reconstruction. If they Err codemadness.org 70 i 4151 weave the histories together, I'll do another force-push, the very Err codemadness.org 70 i 4152 last one, I promise. If you send merge requests, I'll rebase them Err codemadness.org 70 i 4153 myself if that happens.

Maintaining bzip2

2019-06-04T19:41:57-05:00

Today I had a very pleasant conversation with Julian Seward, of bzip2 Err codemadness.org 70 i 4159 and Valgrind fame. Julian has kindly agreed Err codemadness.org 70 i 4160 to cede the maintainership of bzip2 Err codemadness.org 70 i 4161 to me.

Err codemadness.org 70 i 4162

Bzip2 has not had a release since 2010. In the meantime, Linux Err codemadness.org 70 i 4163 distros have accumulated a number of bug/security fixes for it. Err codemadness.org 70 i 4164 Seemingly every distributor of bzip2 patches its build system. The Err codemadness.org 70 i 4165 documentation generation step is a bit creaky. There is no source Err codemadness.org 70 i 4166 control repository, nor bug tracker. I hope to fix these things Err codemadness.org 70 i 4167 gradually.

Err codemadness.org 70 i 4168

This is the new repository for bzip2.

Err codemadness.org 70 i 4169

Ways in which you can immediately help by submitting merge requests:

Err codemadness.org 70 i 4170

Err codemadness.org 70 i 4172
Look at the issues; currently they are around auto-generating the Err codemadness.org 70 i 4173 version number.
Err codemadness.org 70 i 4174
Err codemadness.org 70 i 4176
Create a basic continuous integration pipeline that at least Err codemadness.org 70 i 4177 builds the code and runs the tests.
Err codemadness.org 70 i 4178
Err codemadness.org 70 i 4180
Test the autotools setup, courtesy of Stanislav Brabec, and improve Err codemadness.org 70 i 4181 it as you see fit.
Err codemadness.org 70 i 4182

Err codemadness.org 70 i 4184

The rustification will happen in a separate branch for now, at least Err codemadness.org 70 i 4185 until the Autotools setup settles down.

Err codemadness.org 70 i 4186

I hope to have a 1.0.7 release soon, but this really needs your Err codemadness.org 70 i 4187 help. Let's revive this awesome little project.

Bzip2 in Rust - Basic infrastructure and CRC32 computation

2019-05-30T10:36:19-05:00

I have started a little experiment in porting bits of the widely-used Err codemadness.org 70 i 4192 bzip2/bzlib to Rust. I hope this can Err codemadness.org 70 i 4193 serve to refresh bzip2, which had its last release in 2010 and has Err codemadness.org 70 i 4194 been nominally unmaintained for years.

Err codemadness.org 70 i 4195

I hope to make several posts detailing how this port is done. In this Err codemadness.org 70 i 4196 post, I'll talk about setting up a Rust infrastructure for bzip2 and Err codemadness.org 70 i 4197 my experiments in replacing the C code that does CRC32 computations.

Err codemadness.org 70 i 4198

Super-quick summary of how librsvg was ported to Rust

Err codemadness.org 70 i 4199

Err codemadness.org 70 i 4201
Add the necessary autotools infrastructure to build a Rust Err codemadness.org 70 i 4202 sub-library that gets linked into the main public library.
Err codemadness.org 70 i 4203
Err codemadness.org 70 i 4205
Port bit by bit to Rust. Add unit tests as appropriate. Refactor Err codemadness.org 70 i 4206 endlessly.
Err codemadness.org 70 i 4207
Err codemadness.org 70 i 4209
MAINTAIN THE PUBLIC API/ABI AT ALL COSTS so callers don't Err codemadness.org 70 i 4210 notice that the library is being rewritten under their feet.
Err codemadness.org 70 i 4211

Err codemadness.org 70 i 4213

I have no idea of how bzip2 works internally, but I do know how to Err codemadness.org 70 i 4214 maintain ABIs, so let's get started.

Err codemadness.org 70 i 4215

Bzip2's source tree

Err codemadness.org 70 i 4216

As a very small project that just builds a library and couple of Err codemadness.org 70 i 4217 executables, bzip2 was structured with all the source files directly Err codemadness.org 70 i 4218 under a toplevel directory.

Err codemadness.org 70 i 4219

The only tests in there are three reference files that get compressed, Err codemadness.org 70 i 4220 then uncompressed, and then compared to the original ones.

Err codemadness.org 70 i 4221

As the rustification proceeds, I'll move the files around to better Err codemadness.org 70 i 4222 places. The scheme from librsvg worked well in this respect, so I'll Err codemadness.org 70 i 4223 probably be copying many of the techniques and organization from Err codemadness.org 70 i 4224 there.

Err codemadness.org 70 i 4225

Deciding what to port first

Err codemadness.org 70 i 4226

I looked a bit at the bzip2 sources, and the code to do CRC32 Err codemadness.org 70 i 4227 computations seemed isolated enough from the rest of the code to port Err codemadness.org 70 i 4228 easily.

Err codemadness.org 70 i 4229

The CRC32 code was arranged like this. First, a lookup table in Err codemadness.org 70 i 4230 crc32table.c:

Err codemadness.org 70 i 4231

UInt32 BZ2_crc32Table[256] = {	Err	codemadness.org	70
i  4232    0x00000000L, 0x04c11db7L, 0x09823b6eL, 0x0d4326d9L,	Err	codemadness.org	70
i  4233    0x130476dcL, 0x17c56b6bL, 0x1a864db2L, 0x1e475005L,	Err	codemadness.org	70
i  4234    ...	Err	codemadness.org	70
i  4235 }	Err	codemadness.org	70
i  4236

Err codemadness.org 70 i 4237 Err codemadness.org 70 i 4238

And then, three macros in bzlib_private.h which make up all the Err codemadness.org 70 i 4239 CRC32 code in the library:

Err codemadness.org 70 i 4240

extern UInt32 BZ2_crc32Table[256];	Err	codemadness.org	70
i  4241 	Err	codemadness.org	70
i  4242 #define BZ_INITIALISE_CRC(crcVar)              \	Err	codemadness.org	70
i  4243 {                                              \	Err	codemadness.org	70
i  4244    crcVar = 0xffffffffL;                       \	Err	codemadness.org	70
i  4245 }	Err	codemadness.org	70
i  4246 	Err	codemadness.org	70
i  4247 #define BZ_FINALISE_CRC(crcVar)                \	Err	codemadness.org	70
i  4248 {                                              \	Err	codemadness.org	70
i  4249    crcVar = ~(crcVar);                         \	Err	codemadness.org	70
i  4250 }	Err	codemadness.org	70
i  4251 	Err	codemadness.org	70
i  4252 #define BZ_UPDATE_CRC(crcVar,cha)              \	Err	codemadness.org	70
i  4253 {                                              \	Err	codemadness.org	70
i  4254    crcVar = (crcVar << 8) ^                    \	Err	codemadness.org	70
i  4255             BZ2_crc32Table[(crcVar >> 24) ^    \	Err	codemadness.org	70
i  4256                            ((UChar)cha)];      \	Err	codemadness.org	70
i  4257 }	Err	codemadness.org	70
i  4258

Err codemadness.org 70 i 4259 Err codemadness.org 70 i 4260

Initially I wanted to just remove this code and replace it with one of Err codemadness.org 70 i 4261 the existing Rust crates to do CRC32 computations, but first I needed Err codemadness.org 70 i 4262 to know which variant of CRC32 this is.

Err codemadness.org 70 i 4263

Preparing the CRC32 port so it will not break

Err codemadness.org 70 i 4264

I needed to set up tests for the CRC32 code so the replacement code Err codemadness.org 70 i 4265 would compute exactly the same values as the original:

Err codemadness.org 70 i 4266

Rename crc32table.c to Err codemadness.org 70 i 4268 crc32.c - Err codemadness.org 70 i 4269 that file is going to hold all the CRC32 code, not only the lookup table.
Turn the CRC32 macros into Err codemadness.org 70 i 4271 functions - Err codemadness.org 70 i 4272 so I can move them to Rust and have the C code call them.

Err codemadness.org 70 i 4274

Then I needed a test that computed the CRC32 values of several Err codemadness.org 70 i 4275 strings, so I could capture the results and make them part of the Err codemadness.org 70 i 4276 test.

Err codemadness.org 70 i 4277

static const UChar buf1[] = "";	Err	codemadness.org	70
i  4278 static const UChar buf2[] = " ";	Err	codemadness.org	70
i  4279 static const UChar buf3[] = "hello world";	Err	codemadness.org	70
i  4280 static const UChar buf4[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, ";	Err	codemadness.org	70
i  4281 	Err	codemadness.org	70
i  4282 int	Err	codemadness.org	70
i  4283 main (void)	Err	codemadness.org	70
i  4284 {	Err	codemadness.org	70
i  4285     printf ("buf1: %x\n", crc32_buffer(buf1, strlen(buf1)));	Err	codemadness.org	70
i  4286     printf ("buf2: %x\n", crc32_buffer(buf2, strlen(buf2)));	Err	codemadness.org	70
i  4287     printf ("buf3: %x\n", crc32_buffer(buf3, strlen(buf3)));	Err	codemadness.org	70
i  4288     printf ("buf4: %x\n", crc32_buffer(buf4, strlen(buf4)));	Err	codemadness.org	70
i  4289     // ...	Err	codemadness.org	70
i  4290 }	Err	codemadness.org	70
i  4291

Err codemadness.org 70 i 4292 Err codemadness.org 70 i 4293

This computes the CRC32 values of some strings using the original Err codemadness.org 70 i 4294 algorithm, and prints their results. Then I could cut&paste those Err codemadness.org 70 i 4295 results, and turn the printf into assert — and that gives me a Err codemadness.org 70 i 4296 test.

Err codemadness.org 70 i 4297

int	Err	codemadness.org	70
i  4298 main (void)	Err	codemadness.org	70
i  4299 {	Err	codemadness.org	70
i  4300     assert (crc32_buffer (buf1, strlen (buf1)) == 0x00000000);	Err	codemadness.org	70
i  4301     assert (crc32_buffer (buf2, strlen (buf2)) == 0x29d4f6ab);	Err	codemadness.org	70
i  4302     assert (crc32_buffer (buf3, strlen (buf3)) == 0x44f71378);	Err	codemadness.org	70
i  4303     assert (crc32_buffer (buf4, strlen (buf4)) == 0xd31de6c9);	Err	codemadness.org	70
i  4304     // ...	Err	codemadness.org	70
i  4305 }	Err	codemadness.org	70
i  4306

Err codemadness.org 70 i 4307 Err codemadness.org 70 i 4308

Setting up a Rust infrastructure for bzip2

Err codemadness.org 70 i 4309

Two things made this reasonably easy:

Err codemadness.org 70 i 4310

A patch from Stanislav Brabec, from Suse, to add an autotools Err codemadness.org 70 i 4312 framework to bzip2
The existing Autotools + Rust machinery in librsvg.

Err codemadness.org 70 i 4315

I.e. "copy and paste from somewhere that I know works well". Err codemadness.org 70 i 4316 Wonderful!

Err codemadness.org 70 i 4317

This is the commit that adds a Rust infrastructure for Err codemadness.org 70 i 4318 bzip2. It does the following:

Err codemadness.org 70 i 4319

Create a Cargo workspace (a Cargo.toml in the toplevel) with a Err codemadness.org 70 i 4321 single member, a bzlib_rust directory where the Rustified parts Err codemadness.org 70 i 4322 of the code will live.
Create bzlib_rust/Cargo.toml and bzlib_rust/src for the Rust Err codemadness.org 70 i 4324 sources. This will generate a staticlib for libbzlib_rust.a, that Err codemadness.org 70 i 4325 can be linked into the main libbz2.la.
Puts in automake hooks so that make clean, make check, etc. all Err codemadness.org 70 i 4327 do what you expect for the Rust part.

Err codemadness.org 70 i 4329

As a side benefit, librsvg's Autotools+Rust infrastructure already Err codemadness.org 70 i 4330 handled things like cross-compilation correctly, so I have high hopes Err codemadness.org 70 i 4331 that this will be good enough for bzip2.

Err codemadness.org 70 i 4332

Can I use a Rust crate for CRC32?

Err codemadness.org 70 i 4333

There are many Rust crates to do CRC computations. I was Err codemadness.org 70 i 4334 hoping especially to be able to use crc32fast, which is Err codemadness.org 70 i 4335 SIMD-accelerated.

Err codemadness.org 70 i 4336

I wrote a Rust version of the "CRC me a buffer" test from above to see Err codemadness.org 70 i 4337 if crc32fast produced the same values as the C code, and of course it Err codemadness.org 70 i 4338 didn't. Eventually, after asking on Mastodon, Kepstin figured Err codemadness.org 70 i 4339 out what variant of CRC32 is being used in the original Err codemadness.org 70 i 4340 code.

Err codemadness.org 70 i 4341

It turns out that this is directly doable in Rust with the git version Err codemadness.org 70 i 4342 of the crc crate. This crate lets one configure the CRC32 Err codemadness.org 70 i 4343 polynomial and the mode of computation; there are many variants of Err codemadness.org 70 i 4344 CRC32 and I wasn't fully aware of them.

Err codemadness.org 70 i 4345

The magic incantation is this:

Err codemadness.org 70 i 4346

let mut digest = crc32::Digest::new_custom(crc32::IEEE, !0u32, !0u32, crc::CalcType::Normal);	Err	codemadness.org	70
i  4347

Err codemadness.org 70 i 4348 Err codemadness.org 70 i 4349

With that, the Rust Err codemadness.org 70 i 4350 test Err codemadness.org 70 i 4351 produces the same values as the C code. Yay!

Err codemadness.org 70 i 4352

But it can't be that easy

Err codemadness.org 70 i 4353

Bzlib stores its internal state in the EState Err codemadness.org 70 i 4354 struct, Err codemadness.org 70 i 4355 defined in Err codemadness.org 70 i 4356 bzlib_private.h.

Err codemadness.org 70 i 4357

That struct stores several running CRC32 computations, and the state Err codemadness.org 70 i 4358 for each one of those is a single UInt32 value. However, I cannot Err codemadness.org 70 i 4359 just replace those struct fields with something that comes from Rust, Err codemadness.org 70 i 4360 since the C code does not know the size of a crc32::Digest from Err codemadness.org 70 i 4361 Rust.

Err codemadness.org 70 i 4362

The normal way to do this (say, like in librsvg) would be to turn Err codemadness.org 70 i 4363 UInt32 some_crc into void *some_crc and heap-allocate that on the Err codemadness.org 70 i 4364 Rust side, with whatever size it needs.

Err codemadness.org 70 i 4365

However!

Err codemadness.org 70 i 4366

It turns out that bzlib lets the caller define a custom Err codemadness.org 70 i 4367 allocator Err codemadness.org 70 i 4368 so that bzlib doesn't use plain malloc() by default.

Err codemadness.org 70 i 4369

Rust lets one define a global, custom allocator. Err codemadness.org 70 i 4370 However, bzlib's concept of a custom allocator includes a bit of Err codemadness.org 70 i 4371 context:

Err codemadness.org 70 i 4372

typedef struct {	Err	codemadness.org	70
i  4373     // ...	Err	codemadness.org	70
i  4374 	Err	codemadness.org	70
i  4375     void *(*bzalloc)(void *opaque, int n, int m);	Err	codemadness.org	70
i  4376     void (*bzfree)(void *opaque, void *ptr);	Err	codemadness.org	70
i  4377     void *opaque;	Err	codemadness.org	70
i  4378 } bz_stream;	Err	codemadness.org	70
i  4379

Err codemadness.org 70 i 4380 Err codemadness.org 70 i 4381

The caller sets up bzalloc/bzfree callbacks and an optional opaque Err codemadness.org 70 i 4382 context for the allocator. However, Rust's GlobalAlloc is set up at Err codemadness.org 70 i 4383 compilation time, and we can't pass that context in a good, Err codemadness.org 70 i 4384 thread-safe fashion to it.

Err codemadness.org 70 i 4385

Who uses the bzlib custom allocator, anyway?

Err codemadness.org 70 i 4386

If one sets bzalloc/bzfree to NULL, bzlib will use the system's Err codemadness.org 70 i 4387 plain malloc()/free() by default. Most software does this.

Err codemadness.org 70 i 4388

I am looking in Debian's codesearch for where bzalloc Err codemadness.org 70 i 4389 gets set, hoping that I can figure out if that software really needs a Err codemadness.org 70 i 4390 custom allocator, or if they are just dressing up malloc() with Err codemadness.org 70 i 4391 logging code or similar (ImageMagick seems to do this; Python seems to Err codemadness.org 70 i 4392 have a genuine concern about the Global Interpreter Lock). Debian's Err codemadness.org 70 i 4393 codesearch is a fantastic tool!

Err codemadness.org 70 i 4394

The first rustified code

Err codemadness.org 70 i 4395

I cut&pasted the CRC32 lookup Err codemadness.org 70 i 4396 table Err codemadness.org 70 i 4397 and fixed it up for Rust's syntax, and also ported the CRC32 Err codemadness.org 70 i 4398 computation functions. I gave them the same names as the original C Err codemadness.org 70 i 4399 ones, and exported them, e.g.

Err codemadness.org 70 i 4400

const TABLE: [u32; 256] = [	Err	codemadness.org	70
i  4401    0x00000000, 0x04c11db7, 0x09823b6e, 0x0d4326d9,	Err	codemadness.org	70
i  4402    ...	Err	codemadness.org	70
i  4403 };	Err	codemadness.org	70
i  4404 	Err	codemadness.org	70
i  4405 #[no_mangle]	Err	codemadness.org	70
i  4406 pub unsafe extern "C" fn BZ2_update_crc(crc_var: &mut u32, cha: u8) {	Err	codemadness.org	70
i  4407     *crc_var = (*crc_var << 8) ^ TABLE[((*crc_var >> 24) ^ u32::from(cha)) as usize];	Err	codemadness.org	70
i  4408 }	Err	codemadness.org	70
i  4409

Err codemadness.org 70 i 4410 Err codemadness.org 70 i 4411

This is a straight port of the C code. Rust is very strict about Err codemadness.org 70 i 4412 integer sizes, and arrays can only be indexed with a usize, not any Err codemadness.org 70 i 4413 random integer — hence the explicit conversions.

Err codemadness.org 70 i 4414

And with this, and after fixing the Err codemadness.org 70 i 4415 linkage, Err codemadness.org 70 i 4416 the tests pass!

Err codemadness.org 70 i 4417

First pass at rustifying CRC32: done.

Err codemadness.org 70 i 4418

But that does one byte at a time

Err codemadness.org 70 i 4419

Indeed; the original C code to do CRC32 only handled one byte at a Err codemadness.org 70 i 4420 time. If I replace this with a SIMD-enabled Rust crate, it will want Err codemadness.org 70 i 4421 to process whole buffers at once. I hope the code in bzlib can be Err codemadness.org 70 i 4422 refactored to do that. We'll see!

Err codemadness.org 70 i 4423

How to use an existing Rust crate for this

Err codemadness.org 70 i 4424

I just found out that one does not in fact need to use a complete Err codemadness.org 70 i 4425 crc32::Digest to do equivalent computations; one can call Err codemadness.org 70 i 4426 crc32::update() Err codemadness.org 70 i 4427 by hand and maintain a single u32 state, just like the original Err codemadness.org 70 i 4428 UInt32 from the C code.

Err codemadness.org 70 i 4429

So, I may not need to mess around with a custom allocator just yet. Err codemadness.org 70 i 4430 Stay tuned.

Err codemadness.org 70 i 4431

In the meantime, I've filed a bug against Err codemadness.org 70 i 4432 crc32fast to make Err codemadness.org 70 i 4433 it possible to use a custom polynomial and order and still get the Err codemadness.org 70 i 4434 benefits of SIMD.

Containing mutability in GObjects

2019-04-16T17:04:33-05:00

Traditionally, GObject implementations in C are mutable: you Err codemadness.org 70 i 4440 instantiate a GObject and then change its state via method calls. Err codemadness.org 70 i 4441 Sometimes this is expected and desired; a GtkCheckButton widget Err codemadness.org 70 i 4442 certainly can change its internal state from pressed to not pressed, Err codemadness.org 70 i 4443 for example.

Err codemadness.org 70 i 4444

Other times, objects are mutable while they are being "assembled" or Err codemadness.org 70 i 4445 "configured", and only yield a final immutable result until later. Err codemadness.org 70 i 4446 This is the case for RsvgHandle from librsvg.

Err codemadness.org 70 i 4447

Please bear with me while I write about the history of the Err codemadness.org 70 i 4448 RsvgHandle API and why it ended up with different ways of doing the Err codemadness.org 70 i 4449 same thing.

Err codemadness.org 70 i 4450

The traditional RsvgHandle API

Err codemadness.org 70 i 4451

The final purpose of an RsvgHandle is to represent an SVG document Err codemadness.org 70 i 4452 loaded in memory. Once it is loaded, the SVG document does not Err codemadness.org 70 i 4453 change, as librsvg does not support animation or creating/removing SVG Err codemadness.org 70 i 4454 elements; it is a static renderer.

Err codemadness.org 70 i 4455

However, before an RsvgHandle achieves its immutable state, it has Err codemadness.org 70 i 4456 to be loaded first. Loading can be done in two ways:

Err codemadness.org 70 i 4457

The historical/deprecated way, using the rsvg_handle_write() and Err codemadness.org 70 i 4459 rsvg_handle_close() APIs. Plenty of code in GNOME used this Err codemadness.org 70 i 4460 write/close idiom before GLib got a good abstraction for Err codemadness.org 70 i 4461 streams; you can see another example in GdkPixbufLoader. Err codemadness.org 70 i 4462 The idea is that applications do this:

Err codemadness.org 70 i 4464

file = open a file...;	Err	codemadness.org	70
i  4465 handle = rsvg_handle_new ();	Err	codemadness.org	70
i  4466 	Err	codemadness.org	70
i  4467 while (file has more data) {	Err	codemadness.org	70
i  4468    rsvg_handle_write(handle, a bit of data);	Err	codemadness.org	70
i  4469 }	Err	codemadness.org	70
i  4470 	Err	codemadness.org	70
i  4471 rsvg_handle_close (handle);	Err	codemadness.org	70
i  4472 	Err	codemadness.org	70
i  4473 // now the handle is fully loaded and immutable	Err	codemadness.org	70
i  4474 	Err	codemadness.org	70
i  4475 rsvg_handle_render (handle, ...);	Err	codemadness.org	70
i  4476

Err codemadness.org 70 i 4477 Err codemadness.org 70 i 4478

The streaming way, with rsvg_handle_read_stream_sync(), Err codemadness.org 70 i 4480 which takes a GInputStream, or one of the convenience functions Err codemadness.org 70 i 4481 which take a GFile and produce a stream from it.

Err codemadness.org 70 i 4483

file = g_file_new_for_path ("/foo/bar.svg");	Err	codemadness.org	70
i  4484 stream = g_file_read (file, ...);	Err	codemadness.org	70
i  4485 handle = rsvg_handle_new ();	Err	codemadness.org	70
i  4486 	Err	codemadness.org	70
i  4487 rsvg_handle_read_stream_sync (handle, stream, ...);	Err	codemadness.org	70
i  4488 	Err	codemadness.org	70
i  4489 // now the handle is fully loaded and immutable	Err	codemadness.org	70
i  4490 	Err	codemadness.org	70
i  4491 rsvg_handle_render (handle, ...);	Err	codemadness.org	70
i  4492

Err codemadness.org 70 i 4493 Err codemadness.org 70 i 4494

A bit of history

Err codemadness.org 70 i 4495

Let's consider a few of RsvgHandle's functions.

Err codemadness.org 70 i 4496

Constructors:

Err codemadness.org 70 i 4497

rsvg_handle_new()
rsvg_handle_new_with_flags()

Err codemadness.org 70 i 4501

Configure the handle for loading:

Err codemadness.org 70 i 4502

rsvg_handle_set_base_uri()
rsvg_handle_set_base_gfile()

Err codemadness.org 70 i 4506

Deprecated loading API:

Err codemadness.org 70 i 4507

rsvg_handle_write()
rsvg_handle_close()

Err codemadness.org 70 i 4511

Streaming API:

Err codemadness.org 70 i 4512

rsvg_handle_read_stream_sync()

Err codemadness.org 70 i 4515

When librsvg first acquired the concept of an RsvgHandle, it just Err codemadness.org 70 i 4516 had rsvg_handle_new() with no arguments. About 9 years later, it Err codemadness.org 70 i 4517 got rsvg_handle_new_with_flags() to allow more options, but it took Err codemadness.org 70 i 4518 another 2 years to actually add some usable flags — the first one was Err codemadness.org 70 i 4519 to configure the parsing limits in the underlying calls to libxml2.

Err codemadness.org 70 i 4520

About 3 years after RsvgHandle appeared, it got Err codemadness.org 70 i 4521 rsvg_handle_set_base_uri() to configure the "base URI" against which Err codemadness.org 70 i 4522 relative references in the SVG document get resolved. For example, if Err codemadness.org 70 i 4523 you are reading /foo/bar.svg and it contains an element like <image Err codemadness.org 70 i 4524 xlink:ref="smiley.png"/>, then librsvg needs to be able to produce Err codemadness.org 70 i 4525 the path /foo/smiley.png and that is done relative to the base URI. Err codemadness.org 70 i 4526 (The base URI is implicit when reading from a specific SVG file, but Err codemadness.org 70 i 4527 it needs to be provided when reading from an arbitrary stream that may Err codemadness.org 70 i 4528 not even come from a file.)

Err codemadness.org 70 i 4529

Initially RsvgHandle had the write/close APIs, and 8 years later Err codemadness.org 70 i 4530 it got the streaming functions once GIO appeared. Eventually the Err codemadness.org 70 i 4531 streaming API would be the preferred one, instead of just being a Err codemadness.org 70 i 4532 convenience for those brave new apps that started using GIO.

Err codemadness.org 70 i 4533

A summary of librsvg's API may be something like:

Err codemadness.org 70 i 4534

Err codemadness.org 70 i 4536
librsvg gets written initially; it doesn't even have an Err codemadness.org 70 i 4537 RsvgHandle, and just provides a single function which takes a Err codemadness.org 70 i 4538 FILE * and renders it to a GdkPixbuf.
Err codemadness.org 70 i 4539
Err codemadness.org 70 i 4541
That gets replaced with RsvgHandle, its single rsvg_handle_new() Err codemadness.org 70 i 4542 constructor, and the write/close API to feed it data Err codemadness.org 70 i 4543 progressively.
Err codemadness.org 70 i 4544
Err codemadness.org 70 i 4546
GIO appears, we get the first widespread streaming APIs in GNOME, Err codemadness.org 70 i 4547 and RsvgHandle gets the ability to read from streams.
Err codemadness.org 70 i 4548
Err codemadness.org 70 i 4550
RsvgHandle gets rsvg_handle_new_with_flags() because now apps Err codemadness.org 70 i 4551 may want to configure extra stuff for libxml2.
Err codemadness.org 70 i 4552
Err codemadness.org 70 i 4554
When Cairo appears and librsvg is ported to it, RsvgHandle gets an Err codemadness.org 70 i 4555 extra flag so that SVGs rendered to PDF can embed image data Err codemadness.org 70 i 4556 efficiently.
Err codemadness.org 70 i 4557

Err codemadness.org 70 i 4559

It's a convoluted history, but git log -- rsvg.h makes it accessible.

Err codemadness.org 70 i 4560

Where is the mutability?

Err codemadness.org 70 i 4561

An RsvgHandle gets created, with flags or without. It's empty, and Err codemadness.org 70 i 4562 doesn't know if it will be given data with the write/close API or Err codemadness.org 70 i 4563 with the streaming API. Also, someone may call set_base_uri() on Err codemadness.org 70 i 4564 it. So, the handle must remain mutable while it is being populated Err codemadness.org 70 i 4565 with data. After that, it can say, "no more changes, I'm done".

Err codemadness.org 70 i 4566

In C, this doesn't even have a name. Everything is mutable by default Err codemadness.org 70 i 4567 all the time. This monster was the private data of RsvgHandle Err codemadness.org 70 i 4568 before it got ported to Rust:

Err codemadness.org 70 i 4569

struct RsvgHandlePrivate {	Err	codemadness.org	70
i  4570     // set during construction	Err	codemadness.org	70
i  4571     RsvgHandleFlags flags;	Err	codemadness.org	70
i  4572 	Err	codemadness.org	70
i  4573     // GObject-ism	Err	codemadness.org	70
i  4574     gboolean is_disposed;	Err	codemadness.org	70
i  4575 	Err	codemadness.org	70
i  4576     // Extra crap for a deprecated API	Err	codemadness.org	70
i  4577     RsvgSizeFunc size_func;	Err	codemadness.org	70
i  4578     gpointer user_data;	Err	codemadness.org	70
i  4579     GDestroyNotify user_data_destroy;	Err	codemadness.org	70
i  4580 	Err	codemadness.org	70
i  4581     // Data only used while parsing an SVG	Err	codemadness.org	70
i  4582     RsvgHandleState state;	Err	codemadness.org	70
i  4583     RsvgDefs *defs;	Err	codemadness.org	70
i  4584     guint nest_level;	Err	codemadness.org	70
i  4585     RsvgNode *currentnode;	Err	codemadness.org	70
i  4586     RsvgNode *treebase;	Err	codemadness.org	70
i  4587     GHashTable *css_props;	Err	codemadness.org	70
i  4588     RsvgSaxHandler *handler;	Err	codemadness.org	70
i  4589     int handler_nest;	Err	codemadness.org	70
i  4590     GHashTable *entities;	Err	codemadness.org	70
i  4591     xmlParserCtxtPtr ctxt;	Err	codemadness.org	70
i  4592     GError **error;	Err	codemadness.org	70
i  4593     GCancellable *cancellable;	Err	codemadness.org	70
i  4594     GInputStream *compressed_input_stream;	Err	codemadness.org	70
i  4595 	Err	codemadness.org	70
i  4596     // Data only used while rendering	Err	codemadness.org	70
i  4597     double dpi_x;	Err	codemadness.org	70
i  4598     double dpi_y;	Err	codemadness.org	70
i  4599 	Err	codemadness.org	70
i  4600     // The famous base URI, set before loading	Err	codemadness.org	70
i  4601     gchar *base_uri;	Err	codemadness.org	70
i  4602     GFile *base_gfile;	Err	codemadness.org	70
i  4603 	Err	codemadness.org	70
i  4604     // Some internal stuff	Err	codemadness.org	70
i  4605     gboolean in_loop;	Err	codemadness.org	70
i  4606     gboolean is_testing;	Err	codemadness.org	70
i  4607 };	Err	codemadness.org	70
i  4608

Err codemadness.org 70 i 4609 Err codemadness.org 70 i 4610

"Single responsibility principle"? This is a horror show. That Err codemadness.org 70 i 4611 RsvgHandlePrivate struct has all of these:

Err codemadness.org 70 i 4612

Data only settable during construction (flags)
Data set after construction, but which may only be set before Err codemadness.org 70 i 4615 loading (base URI)
Highly mutable data used only during the loading stage: state Err codemadness.org 70 i 4617 machines, XML parsers, a stack of XML elements, CSS properties...
The DPI (dots per inch) values only used during rendering.
Assorted fields used at various stages of the handle's life.

Err codemadness.org 70 i 4621

It took a lot of refactoring to get the code to a point where it was Err codemadness.org 70 i 4622 clear that an RsvgHandle in fact has distinct stages during its Err codemadness.org 70 i 4623 lifetime, and that some of that data should only live during a Err codemadness.org 70 i 4624 particular stage. Before, everything seemed a jumble of fields, used Err codemadness.org 70 i 4625 at various unclear points in the code (for the struct listing above, Err codemadness.org 70 i 4626 I've grouped related fields together — they were somewhat shuffled in Err codemadness.org 70 i 4627 the original code!).

Err codemadness.org 70 i 4628

What would a better separation look like?

Err codemadness.org 70 i 4629

In the master branch, now librsvg has this:

Err codemadness.org 70 i 4630

/// Contains all the interior mutability for a RsvgHandle to be called	Err	codemadness.org	70
i  4631 /// from the C API.	Err	codemadness.org	70
i  4632 pub struct CHandle {	Err	codemadness.org	70
i  4633     dpi: Cell<Dpi>,	Err	codemadness.org	70
i  4634     load_flags: Cell<LoadFlags>,	Err	codemadness.org	70
i  4635 	Err	codemadness.org	70
i  4636     base_url: RefCell<Option<Url>>,	Err	codemadness.org	70
i  4637     // needed because the C api returns *const char	Err	codemadness.org	70
i  4638     base_url_cstring: RefCell<Option<CString>>,	Err	codemadness.org	70
i  4639 	Err	codemadness.org	70
i  4640     size_callback: RefCell<SizeCallback>,	Err	codemadness.org	70
i  4641     is_testing: Cell<bool>,	Err	codemadness.org	70
i  4642     load_state: RefCell<LoadState>,	Err	codemadness.org	70
i  4643 }	Err	codemadness.org	70
i  4644

Err codemadness.org 70 i 4645 Err codemadness.org 70 i 4646

Internally, that CHandle struct is now the private data of the Err codemadness.org 70 i 4647 public RsvgHandle object. Note that all of CHandle's fields are a Err codemadness.org 70 i 4648 Cell<> or RefCell<>: in Rust terms, this means that those fields Err codemadness.org 70 i 4649 allow for "interior mutability" in the CHandle struct: they can be Err codemadness.org 70 i 4650 modified after intialization.

Err codemadness.org 70 i 4651

The last field's cell, load_state, contains this type:

Err codemadness.org 70 i 4652

enum LoadState {	Err	codemadness.org	70
i  4653     Start,	Err	codemadness.org	70
i  4654 	Err	codemadness.org	70
i  4655     // Being loaded using the legacy write()/close() API	Err	codemadness.org	70
i  4656     Loading { buffer: Vec<u8> },	Err	codemadness.org	70
i  4657 	Err	codemadness.org	70
i  4658     // Fully loaded, with a Handle to an SVG document	Err	codemadness.org	70
i  4659     ClosedOk { handle: Handle },	Err	codemadness.org	70
i  4660 	Err	codemadness.org	70
i  4661     ClosedError,	Err	codemadness.org	70
i  4662 }	Err	codemadness.org	70
i  4663

Err codemadness.org 70 i 4664 Err codemadness.org 70 i 4665

A CHandle starts in the Start state, where it doesn't know if it Err codemadness.org 70 i 4666 will be loaded with a stream, or with the legacy write/close API.

Err codemadness.org 70 i 4667

If the caller uses the write/close API, the handle moves to the Err codemadness.org 70 i 4668 Loading state, which has a buffer where it accumulates the data Err codemadness.org 70 i 4669 being fed to it.

Err codemadness.org 70 i 4670

But if the caller uses the stream API, the handle tries to parse an Err codemadness.org 70 i 4671 SVG document from the stream, and it moves either to the ClosedOk Err codemadness.org 70 i 4672 state, or to the ClosedError state if there is a parse error.

Err codemadness.org 70 i 4673

Correspondingly, when using the write/close API, when the caller Err codemadness.org 70 i 4674 finally calls rsvg_handle_close(), the handle creates a stream for Err codemadness.org 70 i 4675 the buffer, parses it, and also gets either into the ClosedOk or Err codemadness.org 70 i 4676 ClosedError state.

Err codemadness.org 70 i 4677

If you look at the variant ClosedOk { handle: Handle }, it contains Err codemadness.org 70 i 4678 a fully loaded Handle inside, which right now is just a wrapper Err codemadness.org 70 i 4679 around a reference-counted Svg object:

Err codemadness.org 70 i 4680

pub struct Handle {	Err	codemadness.org	70
i  4681     svg: Rc<Svg>,	Err	codemadness.org	70
i  4682 }	Err	codemadness.org	70
i  4683

Err codemadness.org 70 i 4684 Err codemadness.org 70 i 4685

The reason why LoadState::ClosedOk does not contain an Rc<Svg> Err codemadness.org 70 i 4686 directly, and instead wraps it with a Handle, is that this is just Err codemadness.org 70 i 4687 the first pass at refactoring. Also, Handle contains some Err codemadness.org 70 i 4688 API-level logic which I'm not completely sure makes sense as a Err codemadness.org 70 i 4689 lower-level Svg object. We'll see.

Err codemadness.org 70 i 4690

Couldn't you move more of `CHandle`'s fields into `LoadState`?

Err codemadness.org 70 i 4691

Sort of, kind of, but the public API still lets one do things like Err codemadness.org 70 i 4692 call rsvg_handle_get_base_uri() after the handle is fully loaded, Err codemadness.org 70 i 4693 even though its result will be of little value. So, the fields that Err codemadness.org 70 i 4694 hold the base_uri information are kept in the longer-lived Err codemadness.org 70 i 4695 CHandle, not in the individual variants of LoadState.

Err codemadness.org 70 i 4696

How does this look from the Rust API?

Err codemadness.org 70 i 4697

CHandle implements the public C API of librsvg. Internally, Err codemadness.org 70 i 4698 Handle implements the basic "load from stream", "get the geometry of Err codemadness.org 70 i 4699 an SVG element", and "render to a Cairo context" functionality.

Err codemadness.org 70 i 4700

This basic functionality gets exported in a cleaner way through the Err codemadness.org 70 i 4701 Rust API, discussed previously. There is no Err codemadness.org 70 i 4702 interior mutability in there at all; that API uses a builder pattern Err codemadness.org 70 i 4703 to gradually configure an SVG loader, which returns a fully loaded Err codemadness.org 70 i 4704 SvgHandle, out of which you can create a CairoRenderer.

Err codemadness.org 70 i 4705

In fact, it may be possible to refactor all of this a bit and Err codemadness.org 70 i 4706 implement CHandle directly in terms of the new Rust API: in effect, Err codemadness.org 70 i 4707 use CHandle as the "holding space" while the SVG loader gets Err codemadness.org 70 i 4708 configured, and later turned into a fully loaded SvgHandle Err codemadness.org 70 i 4709 internally.

Err codemadness.org 70 i 4710

Conclusion

Err codemadness.org 70 i 4711

The C version of RsvgHandle's private structure used to have a bunch Err codemadness.org 70 i 4712 of fields. Without knowing the code, it was hard to know that they Err codemadness.org 70 i 4713 belonged in groups, and each group corresponded roughtly to a stage in Err codemadness.org 70 i 4714 the handle's lifetime.

Err codemadness.org 70 i 4715

It took plenty of refactoring to get the fields split up cleanly in Err codemadness.org 70 i 4716 librsvg's internals. The process of refactoring RsvgHandle's fields, Err codemadness.org 70 i 4717 and ensuring that the various states of a handle are consistent, in Err codemadness.org 70 i 4718 fact exposed a few bugs where the state was not being checked Err codemadness.org 70 i 4719 appropriately. The public C API remains the same as always, but has Err codemadness.org 70 i 4720 better internal checks now.

Err codemadness.org 70 i 4721

GObject APIs tend to allow for a lot of mutability via methods that Err codemadness.org 70 i 4722 change the internal state of objects. For RsvgHandle, it was possible Err codemadness.org 70 i 4723 to change this into a single CHandle that maintains the mutable data Err codemadness.org 70 i 4724 in a contained fashion, and later translates it internally into an Err codemadness.org 70 i 4725 immutable Handle that represents a fully-loaded SVG document. This Err codemadness.org 70 i 4726 scheme ties in well with the new Rust API for librsvg, which keeps Err codemadness.org 70 i 4727 everything immutable after creation.

A Rust API for librsvg

2019-03-15T13:36:47-06:00

After the librsvg team finished the rustification of Err codemadness.org 70 i 4732 librsvg's main library, I wanted to start porting the high-level test Err codemadness.org 70 i 4733 suite to Rust. This is mainly to be able to run tests in parallel, Err codemadness.org 70 i 4734 which cargo test does automatically in order to reduce test times. Err codemadness.org 70 i 4735 However, this meant that librsvg needed a Rust API that would exercise Err codemadness.org 70 i 4736 the same code paths as the C entry points.

Err codemadness.org 70 i 4737

At the same time, I wanted the Rust API to make it impossible to Err codemadness.org 70 i 4738 misuse the library. From the viewpoint of the C API, an RsvgHandle Err codemadness.org 70 i 4739 has different stages:

Err codemadness.org 70 i 4740

Just initialized
Loading
Loaded, or in an error state after a failed load
Ready to render

Err codemadness.org 70 i 4746

To ensure consistency, the public API checks that you cannot render an Err codemadness.org 70 i 4747 RsvgHandle that is not completely loaded yet, or one that resulted Err codemadness.org 70 i 4748 in a loading error. But wouldn't it be nice if it were impossible to Err codemadness.org 70 i 4749 call the API functions in the wrong order?

Err codemadness.org 70 i 4750

This is exactly what the Rust API does. There is a Loader, Err codemadness.org 70 i 4751 to which you give a filename or a stream, and it will return a Err codemadness.org 70 i 4752 fully-loaded SvgHandle or an error. Then, you can only create a Err codemadness.org 70 i 4753 CairoRenderer if you have an SvgHandle.

Err codemadness.org 70 i 4754

For historical reasons, the C API in librsvg is not perfectly Err codemadness.org 70 i 4755 consistent. For example, some functions which return an error will Err codemadness.org 70 i 4756 actually return a proper GError, but some others will just Err codemadness.org 70 i 4757 return a gboolean with no further explanation of what went wrong. Err codemadness.org 70 i 4758 In contrast, all the Rust API functions that can fail will actually Err codemadness.org 70 i 4759 return a Result, and the error case will have a meaningful Err codemadness.org 70 i 4760 error value. In the Rust API, there is no "wrong order" in which the Err codemadness.org 70 i 4761 various API functions and methods can be called; it tries to do the Err codemadness.org 70 i 4762 whole "make invalid states unrepresentable".

Err codemadness.org 70 i 4763

To implement the Rust API, I had to do some refactoring of the Err codemadness.org 70 i 4764 internals that hook to the public entry points. This made me Err codemadness.org 70 i 4765 realize that librsvg could be a lot easier to use. The C API has Err codemadness.org 70 i 4766 always forced you to call it in this fashion:

Err codemadness.org 70 i 4767

Ask the SVG for its dimensions, or how big it is.
Based on that, scale your Cairo context to the size you actually Err codemadness.org 70 i 4770 want.
Render the SVG to that context's current transformation matrix.

Err codemadness.org 70 i 4773

But first, (1) gives you inadequate information because Err codemadness.org 70 i 4774 rsvg_handle_get_dimensions() returns a Err codemadness.org 70 i 4775 structure with int fields for the width and Err codemadness.org 70 i 4776 height. The API is similar to gdk-pixbuf's in that it always wants to Err codemadness.org 70 i 4777 think in whole pixels. However, an SVG is not necessarily Err codemadness.org 70 i 4778 integer-sized.

Err codemadness.org 70 i 4779

Then, (2) forces you to calculate some geometry in almost all cases, Err codemadness.org 70 i 4780 as most apps want to render SVG content scaled proportionally to a Err codemadness.org 70 i 4781 certain size. This is not hard to do, but it's an inconvenience.

Err codemadness.org 70 i 4782

SVG dimensions

Err codemadness.org 70 i 4783

Let's look at (1) again. The question, "how big is the SVG" is a bit Err codemadness.org 70 i 4784 meaningless when we consider that SVGs can be scaled to any size; Err codemadness.org 70 i 4785 that's the whole point of them!

Err codemadness.org 70 i 4786

When you ask RsvgHandle how big it is, in reality it should look at Err codemadness.org 70 i 4787 you and whisper in your ear, "how big do you want it to be?".

Err codemadness.org 70 i 4788

And that's the thing. The HTML/CSS/SVG model is that one embeds Err codemadness.org 70 i 4789 content into viewports of a given size. The software is Err codemadness.org 70 i 4790 responsible for scaling the content to fit into that viewport.

Err codemadness.org 70 i 4791

In the end, what we want is a rendering function that takes a Cairo Err codemadness.org 70 i 4792 context and a Rectangle for a viewport, and that's it. The function Err codemadness.org 70 i 4793 should take care of fitting the SVG's contents within that viewport.

Err codemadness.org 70 i 4794

There is now an open bug about exactly this sort of API. In Err codemadness.org 70 i 4795 the end, programs should just have to load their SVG handle, and Err codemadness.org 70 i 4796 directly ask it to render at whatever size they need, instead of doing Err codemadness.org 70 i 4797 the size computations by hand.

Err codemadness.org 70 i 4798

When will this be available?

Err codemadness.org 70 i 4799

I'm in the middle of a rather large refactor Err codemadness.org 70 i 4800 to make this viewport concept really work. So far this involves:

Err codemadness.org 70 i 4801

Err codemadness.org 70 i 4803
Defining APIs that take a viewport.
Err codemadness.org 70 i 4804
Err codemadness.org 70 i 4806
Refactoring all the geometry computation to support the semantics of the Err codemadness.org 70 i 4807 C API, plus the new with_viewport semantics.
Err codemadness.org 70 i 4808
Err codemadness.org 70 i 4810
Fixing the code that kept track of an internal offset for all Err codemadness.org 70 i 4811 temporary images.
Err codemadness.org 70 i 4812
Err codemadness.org 70 i 4814
Refactoring all the code that mucks around with the Cairo context's Err codemadness.org 70 i 4815 affine transformation matrix, which is a big mutable mess.
Err codemadness.org 70 i 4816
Err codemadness.org 70 i 4818
Tests, examples, documentation.
Err codemadness.org 70 i 4819

Err codemadness.org 70 i 4821

I want to make the Rust API available for the 2.46 release, which is Err codemadness.org 70 i 4822 hopefully not too far off. It should be ready for the next GNOME Err codemadness.org 70 i 4823 release. In the meantime, you can check out the open bugs for the Err codemadness.org 70 i 4824 2.46.0 milestone. Help is appreciated; the deadline Err codemadness.org 70 i 4825 for the first 3.33 tarballs is approximately one month from now!

Rust build scripts vs. Meson

2019-02-27T12:14:12-06:00

One of the pain points in trying to make the Meson build system work Err codemadness.org 70 i 4829 with Rust and Cargo is Cargo's use of build scripts, i.e. the Err codemadness.org 70 i 4830 build.rs that many Rust programs use for doing things before the Err codemadness.org 70 i 4831 main build. This post is about my exploration of what build.rs Err codemadness.org 70 i 4832 does.

Err codemadness.org 70 i 4833

Thanks to Nirbheek Chauhan for his comments Err codemadness.org 70 i 4834 and additions to a draft of this article!

Err codemadness.org 70 i 4835

TL;DR: build.rs is pretty ad-hoc and somewhat primitive, when Err codemadness.org 70 i 4836 compared to Meson's very nice, high-level patterns for build-time Err codemadness.org 70 i 4837 things.

Err codemadness.org 70 i 4838

I have the intuition that giving names to the things that are Err codemadness.org 70 i 4839 usually done in build.rs scripts, and creating abstractions for Err codemadness.org 70 i 4840 them, can make it easier later to implement those abstractions in Err codemadness.org 70 i 4841 terms of Meson. Maybe we can eliminate build.rs in most cases? Err codemadness.org 70 i 4842 Maybe Cargo can acquire higher-level concepts that plug well to Meson?

Err codemadness.org 70 i 4843

(That is... I think we can refactor our way out of this mess.)

Err codemadness.org 70 i 4844

What does `build.rs` do?

Err codemadness.org 70 i 4845

The first paragraph in the documentation for Cargo build Err codemadness.org 70 i 4846 scripts tells us this:

Err codemadness.org 70 i 4847

Err codemadness.org 70 i 4848
Some packages need to compile third-party non-Rust code, for example Err codemadness.org 70 i 4849 C libraries. Other packages need to link to C libraries which can Err codemadness.org 70 i 4850 either be located on the system or possibly need to be built from Err codemadness.org 70 i 4851 source. Others still need facilities for functionality such as code Err codemadness.org 70 i 4852 generation before building (think parser generators).
Err codemadness.org 70 i 4853

Err codemadness.org 70 i 4854

That is,

Err codemadness.org 70 i 4855

Err codemadness.org 70 i 4857
Compiling third-party non-Rust code. For example, maybe there is a Err codemadness.org 70 i 4858 C sub-library that the Rust crate needs.
Err codemadness.org 70 i 4859
Err codemadness.org 70 i 4861
Link to C libraries... located on the system... or built from Err codemadness.org 70 i 4862 source. For example, in gtk-rs, the sys crates link to Err codemadness.org 70 i 4863 libgtk-3.so, libcairo.so, etc. and need to find a way to locate Err codemadness.org 70 i 4864 those libraries with pkg-config.
Err codemadness.org 70 i 4865
Err codemadness.org 70 i 4867
Code generation. In the C world this could be generating a parser Err codemadness.org 70 i 4868 with yacc; in the Rust world there are many utilities to generate Err codemadness.org 70 i 4869 code that is later used in your actual program.
Err codemadness.org 70 i 4870

Err codemadness.org 70 i 4872

In the next sections I'll look briefly at each of these cases, but in Err codemadness.org 70 i 4873 a different order.

Err codemadness.org 70 i 4874

Code generation

Err codemadness.org 70 i 4875

Here is an example, in how librsvg generates code Err codemadness.org 70 i 4876 for a couple of things that get autogenerated before compiling the Err codemadness.org 70 i 4877 main library:

Err codemadness.org 70 i 4878

A perfect hash function (PHF) of attributes and CSS property names.
A pair of lookup tables for SRGB linearization and un-linearization.

Err codemadness.org 70 i 4882

For example, this is main() in build.rs:

Err codemadness.org 70 i 4883

fn main() {	Err	codemadness.org	70
i  4884     generate_phf_of_svg_attributes();	Err	codemadness.org	70
i  4885     generate_srgb_tables();	Err	codemadness.org	70
i  4886 }	Err	codemadness.org	70
i  4887

Err codemadness.org 70 i 4888 Err codemadness.org 70 i 4889

And this is the first few lines of of the first function:

Err codemadness.org 70 i 4890

fn generate_phf_of_svg_attributes() {	Err	codemadness.org	70
i  4891     let path = Path::new(&env::var("OUT_DIR").unwrap()).join("attributes-codegen.rs");	Err	codemadness.org	70
i  4892     let mut file = BufWriter::new(File::create(&path).unwrap());	Err	codemadness.org	70
i  4893 	Err	codemadness.org	70
i  4894     writeln!(&mut file, "#[repr(C)]").unwrap();	Err	codemadness.org	70
i  4895 	Err	codemadness.org	70
i  4896     // ... etc	Err	codemadness.org	70
i  4897 }	Err	codemadness.org	70
i  4898

Err codemadness.org 70 i 4899 Err codemadness.org 70 i 4900

Generate a path like $OUT_DIR/attributes-codegen.rs, create a file Err codemadness.org 70 i 4901 with that name, a BufWriter for the file, and start outputting code Err codemadness.org 70 i 4902 to it.

Err codemadness.org 70 i 4903

Similarly, the second function:

Err codemadness.org 70 i 4904

fn generate_srgb_tables() {	Err	codemadness.org	70
i  4905     let linearize_table = compute_table(linearize);	Err	codemadness.org	70
i  4906     let unlinearize_table = compute_table(unlinearize);	Err	codemadness.org	70
i  4907 	Err	codemadness.org	70
i  4908     let path = Path::new(&env::var("OUT_DIR").unwrap()).join("srgb-codegen.rs");	Err	codemadness.org	70
i  4909     let mut file = BufWriter::new(File::create(&path).unwrap());	Err	codemadness.org	70
i  4910 	Err	codemadness.org	70
i  4911     // ...	Err	codemadness.org	70
i  4912 	Err	codemadness.org	70
i  4913     print_table(&mut file, "LINEARIZE", &linearize_table);	Err	codemadness.org	70
i  4914     print_table(&mut file, "UNLINEARIZE", &unlinearize_table);	Err	codemadness.org	70
i  4915 }	Err	codemadness.org	70
i  4916

Err codemadness.org 70 i 4917 Err codemadness.org 70 i 4918

Compute two lookup tables, create a file named Err codemadness.org 70 i 4919 $OUT_DIR/srgb-codegen.rs, and write the lookup tables to the file.

Err codemadness.org 70 i 4920

Later in the actual librsvg code, the generated files get included Err codemadness.org 70 i 4921 into the source code using the include! macro. For example, here is Err codemadness.org 70 i 4922 where attributes-codegen.rs gets included:

Err codemadness.org 70 i 4923

// attributes.rs	Err	codemadness.org	70
i  4924 	Err	codemadness.org	70
i  4925 extern crate phf;  // crate for perfect hash function	Err	codemadness.org	70
i  4926 	Err	codemadness.org	70
i  4927 // the generated file has the declaration for enum Attribute	Err	codemadness.org	70
i  4928 include!(concat!(env!("OUT_DIR"), "/attributes-codegen.rs"));	Err	codemadness.org	70
i  4929

Err codemadness.org 70 i 4930 Err codemadness.org 70 i 4931

One thing to note here is that the generated source files Err codemadness.org 70 i 4932 (attributes-codegen.rs, srgb-codegen.rs) get put in $OUT_DIR, a Err codemadness.org 70 i 4933 directory that Cargo creates for the compilation artifacts. The files do Err codemadness.org 70 i 4934 not get put into the original source directories with the rest of Err codemadness.org 70 i 4935 the library's code; the idea is to keep the source directories Err codemadness.org 70 i 4936 pristine.

Err codemadness.org 70 i 4937

At least in those terms, Meson and Cargo agree that source directories Err codemadness.org 70 i 4938 should be kept clean of autogenerated files.

Err codemadness.org 70 i 4939

The Code Generation section of Cargo's documentation agrees:

Err codemadness.org 70 i 4940

Err codemadness.org 70 i 4941
In general, build scripts should not modify any files outside of Err codemadness.org 70 i 4942 OUT_DIR. It may seem fine on the first blush, but it does cause Err codemadness.org 70 i 4943 problems when you use such crate as a dependency, because there's an Err codemadness.org 70 i 4944 implicit invariant that sources in .cargo/registry should be Err codemadness.org 70 i 4945 immutable. cargo won't allow such scripts when packaging.
Err codemadness.org 70 i 4946

Err codemadness.org 70 i 4947

Now, some things to note here:

Err codemadness.org 70 i 4948

Err codemadness.org 70 i 4950
Both the build.rs program and the actual library sources look at Err codemadness.org 70 i 4951 the $OUT_DIR environment variable for the location of the Err codemadness.org 70 i 4952 generated sources.
Err codemadness.org 70 i 4953
Err codemadness.org 70 i 4955
The Cargo docs say that if the code generator needs input files, it Err codemadness.org 70 i 4956 can look for them based on its current directory, which will be the Err codemadness.org 70 i 4957 toplevel of your source package (i.e. your toplevel Cargo.toml).
Err codemadness.org 70 i 4958

Err codemadness.org 70 i 4960

Meson hates this scheme of things. In particular, Meson is very Err codemadness.org 70 i 4961 systematic about where it finds input files and sources, and where Err codemadness.org 70 i 4962 things like code generators are allowed to place their output.

Err codemadness.org 70 i 4963

The way Meson communicates these paths to code generators is via Err codemadness.org 70 i 4964 command-line arguments to "custom targets". Here is Err codemadness.org 70 i 4965 an example that is easier to read than the Err codemadness.org 70 i 4966 documentation:

Err codemadness.org 70 i 4967

gen = find_program('generator.py')	Err	codemadness.org	70
i  4968 	Err	codemadness.org	70
i  4969 outputs = custom_target('generated',	Err	codemadness.org	70
i  4970   output : ['foo.h', 'foo.c'],	Err	codemadness.org	70
i  4971   command : [gen, '@OUTDIR@'],	Err	codemadness.org	70
i  4972   ...	Err	codemadness.org	70
i  4973 )	Err	codemadness.org	70
i  4974

Err codemadness.org 70 i 4975 Err codemadness.org 70 i 4976

This defines a target named 'generated', which will use the Err codemadness.org 70 i 4977 generator.py program to output two files, foo.h and foo.c. That Err codemadness.org 70 i 4978 Python program will get called with @OUTDIR@ as a command-line Err codemadness.org 70 i 4979 argument; in effect, meson will call Err codemadness.org 70 i 4980 /full/path/to/generator.py @OUTDIR@ explicitly, without any magic Err codemadness.org 70 i 4981 passed through environment variables.

Err codemadness.org 70 i 4982

If this looks similar to what Cargo does above with build.rs, it's Err codemadness.org 70 i 4983 because it is similar. It's just that Meson gives a name to Err codemadness.org 70 i 4984 the concept of generating code at build time (Meson's name for this is Err codemadness.org 70 i 4985 a custom target), and provides a mechanism to say which program is Err codemadness.org 70 i 4986 the generator, which files it is expected to generate, and how to call Err codemadness.org 70 i 4987 the program with appropriate arguments to put files in the right Err codemadness.org 70 i 4988 place.

Err codemadness.org 70 i 4989

In contrast, Cargo assumes that all of that information can be Err codemadness.org 70 i 4990 inferred from an environment variable.

Err codemadness.org 70 i 4991

In addition, if the custom target takes other files as input (say, so Err codemadness.org 70 i 4992 it can call yacc my-grammar.y), the custom_target() command can Err codemadness.org 70 i 4993 take an input: argument. This way, Meson can add a dependency on Err codemadness.org 70 i 4994 those input files, so that the appropriate things will be rebuilt if Err codemadness.org 70 i 4995 the input files change.

Err codemadness.org 70 i 4996

Now, Cargo could very well provide a small utility crate that build Err codemadness.org 70 i 4997 scripts could use to figure out all that information. Meson would Err codemadness.org 70 i 4998 tell Cargo to use its scheme of things, and pass it down to build Err codemadness.org 70 i 4999 scripts via that utility crate. I.e. to have

Err codemadness.org 70 i 5000

// build.rs	Err	codemadness.org	70
i  5001 	Err	codemadness.org	70
i  5002 extern crate cargo_high_level;	Err	codemadness.org	70
i  5003 	Err	codemadness.org	70
i  5004 let output = Path::new(cargo_high_level::get_output_path()).join("codegen.rs");	Err	codemadness.org	70
i  5005 //                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this, instead of:	Err	codemadness.org	70
i  5006 	Err	codemadness.org	70
i  5007 let output = Path::new(&env::var("OUT_DIR").unwrap()).join("codegen.rs");	Err	codemadness.org	70
i  5008 	Err	codemadness.org	70
i  5009 // let the build system know about generated dependencies	Err	codemadness.org	70
i  5010 cargo_high_level::add_output(output);	Err	codemadness.org	70
i  5011

Err codemadness.org 70 i 5012 Err codemadness.org 70 i 5013

A similar mechanism could be used for the way Meson likes to pass Err codemadness.org 70 i 5014 command-line arguments to the programs that deal with custom targets.

Err codemadness.org 70 i 5015

Linking to C libraries on the system

Err codemadness.org 70 i 5016

Some Rust crates need to link to lower-level C libraries that actually Err codemadness.org 70 i 5017 do the work. For example, in gtk-rs, there are high-level binding Err codemadness.org 70 i 5018 crates called gtk, gdk, cairo, etc. These use low-level crates Err codemadness.org 70 i 5019 called gtk-sys, gdk-sys, cairo-sys. Those -sys crates are Err codemadness.org 70 i 5020 just direct wrappers on top of the C functions of the respective Err codemadness.org 70 i 5021 system libraries: gtk-sys makes almost every function in Err codemadness.org 70 i 5022 libgtk-3.so available as a Rust-callable function.

Err codemadness.org 70 i 5023

System libraries sometimes live in a well-known part of the filesystem Err codemadness.org 70 i 5024 (/usr/lib64, for example); other times, like in Windows and MacOS, Err codemadness.org 70 i 5025 they could be anywhere. To find that location plus other related Err codemadness.org 70 i 5026 metadata (include paths for C header files, library version), many Err codemadness.org 70 i 5027 system libraries use pkg-config. At the highest Err codemadness.org 70 i 5028 level, one can run pkg-config on the command line, or from build Err codemadness.org 70 i 5029 scripts, to query some things about libraries. For example:

Err codemadness.org 70 i 5030

# what's the system's installed version of GTK?	Err	codemadness.org	70
i  5031 $ pkg-config --modversion gtk+-3.0	Err	codemadness.org	70
i  5032 3.24.4	Err	codemadness.org	70
i  5033 	Err	codemadness.org	70
i  5034 # what compiler flags would a C compiler need for GTK?	Err	codemadness.org	70
i  5035 $ pkg-config --cflags gtk+-3.0	Err	codemadness.org	70
i  5036 -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/libxkbcommon -I/usr/include/wayland -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/fribidi -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libdrm -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include	Err	codemadness.org	70
i  5037 	Err	codemadness.org	70
i  5038 # and which libraries?	Err	codemadness.org	70
i  5039 $ pkg-config --libs gtk+-3.0	Err	codemadness.org	70
i  5040 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0	Err	codemadness.org	70
i  5041

Err codemadness.org 70 i 5042 Err codemadness.org 70 i 5043

There is a pkg-config crate which build.rs can Err codemadness.org 70 i 5044 use to call this, and communicate that information to Cargo. The Err codemadness.org 70 i 5045 example in the crate's documentation is for asking pkg-config for the Err codemadness.org 70 i 5046 foo package, with version at least 1.2.3:

Err codemadness.org 70 i 5047

extern crate pkg_config;	Err	codemadness.org	70
i  5048 	Err	codemadness.org	70
i  5049 fn main() {	Err	codemadness.org	70
i  5050     pkg_config::Config::new().atleast_version("1.2.3").probe("foo").unwrap();	Err	codemadness.org	70
i  5051 }	Err	codemadness.org	70
i  5052

Err codemadness.org 70 i 5053 Err codemadness.org 70 i 5054

And the documentation says,

Err codemadness.org 70 i 5055

Err codemadness.org 70 i 5056
After running pkg-config all appropriate Cargo metadata will be Err codemadness.org 70 i 5057 printed on stdout if the search was successful.
Err codemadness.org 70 i 5058

Err codemadness.org 70 i 5059

Wait, what?

Err codemadness.org 70 i 5060

Indeed, printing specially-formated stuff on stdout is how build.rs Err codemadness.org 70 i 5061 scripts communicate back to Cargo about their findings. To quote Cargo's docs Err codemadness.org 70 i 5062 on build scripts; the following is talking Err codemadness.org 70 i 5063 about the stdout of build.rs:

Err codemadness.org 70 i 5064

Err codemadness.org 70 i 5065
Any line that starts with cargo: is interpreted directly by Err codemadness.org 70 i 5066 Cargo. This line must be of the form cargo:key=value, like the Err codemadness.org 70 i 5067 examples below:
Err codemadness.org 70 i 5068

Err codemadness.org 70 i 5069

# specially recognized by Cargo	Err	codemadness.org	70
i  5070 cargo:rustc-link-lib=static=foo	Err	codemadness.org	70
i  5071 cargo:rustc-link-search=native=/path/to/foo	Err	codemadness.org	70
i  5072 cargo:rustc-cfg=foo	Err	codemadness.org	70
i  5073 cargo:rustc-env=FOO=bar	Err	codemadness.org	70
i  5074 # arbitrary user-defined metadata	Err	codemadness.org	70
i  5075 cargo:root=/path/to/foo	Err	codemadness.org	70
i  5076 cargo:libdir=/path/to/foo/lib	Err	codemadness.org	70
i  5077 cargo:include=/path/to/foo/include	Err	codemadness.org	70
i  5078

Err codemadness.org 70 i 5079 Err codemadness.org 70 i 5080

One can use the stdout of a build.rs program to add additional Err codemadness.org 70 i 5081 command-line options for rustc, or set environment variables for it, Err codemadness.org 70 i 5082 or add library paths, or specific libraries.

Err codemadness.org 70 i 5083

Meson hates this scheme of things. I suppose it would prefer to Err codemadness.org 70 i 5084 do the pkg-config calls itself, and then pass that information down to Err codemadness.org 70 i 5085 Cargo, you guessed it, via command-line options or something Err codemadness.org 70 i 5086 well-defined like that. Again, the example cargo_high_level crate I Err codemadness.org 70 i 5087 proposed above could be used to communicate this information from Err codemadness.org 70 i 5088 Meson to Cargo scripts. Meson also doesn't like this because it would Err codemadness.org 70 i 5089 prefer to know about pkg-config-based libraries in a declarative Err codemadness.org 70 i 5090 fashion, without having to run a random script like build.rs.

Err codemadness.org 70 i 5091

Building C code from Rust

Err codemadness.org 70 i 5092

Finally, some Rust crates build a bit of C code and then link that Err codemadness.org 70 i 5093 into the compiled Rust code. I have no experience with that, but Err codemadness.org 70 i 5094 the respective build scripts generally use the cc crate to Err codemadness.org 70 i 5095 call a C compiler and pass options to it conveniently. I suppose Err codemadness.org 70 i 5096 Meson would prefer to do this instead, or at least to have a Err codemadness.org 70 i 5097 high-level way of passing down information to Cargo.

Err codemadness.org 70 i 5098

In effect, Meson has to be in charge of picking the C compiler. Err codemadness.org 70 i 5099 Having the thing-to-be-built pick on its own has caused big problems Err codemadness.org 70 i 5100 in the past: GObject-Introspection made the same mistake years ago Err codemadness.org 70 i 5101 when it decided to use distutils to detect the C compiler; gtk-doc did Err codemadness.org 70 i 5102 as well. When those tools are used, we still deal with problems with Err codemadness.org 70 i 5103 cross-compilation and when the system has more than one C compiler in Err codemadness.org 70 i 5104 it.

Err codemadness.org 70 i 5105

Snarky comments about the Unix philosophy

Err codemadness.org 70 i 5106

If part of the Unix philosophy is that shit can be glued together with Err codemadness.org 70 i 5107 environment variables and stringly-typed stdout... it's a pretty bad Err codemadness.org 70 i 5108 philosophy. All the cases above boil down to having a well-defined, Err codemadness.org 70 i 5109 more or less strongly-typed way to pass information between programs Err codemadness.org 70 i 5110 instead of shaking proverbial tree of the filesystem and the Err codemadness.org 70 i 5111 environment and seeing if something usable falls down.

Err codemadness.org 70 i 5112

Would we really have to modify all `build.rs` scripts for this?

Err codemadness.org 70 i 5113

Probably. Why not? Meson already has a lot of very well-structured Err codemadness.org 70 i 5114 knowledge of how to deal with multi-platform compilation and Err codemadness.org 70 i 5115 installation. Re-creating this knowledge in ad-hoc ways in build.rs Err codemadness.org 70 i 5116 is not very pleasant or maintainable.

Err codemadness.org 70 i 5117

Related work

Err codemadness.org 70 i 5118

Err codemadness.org 70 i 5120
Cargo internals Err codemadness.org 70 i 5121 thread on using a declarative format to specify external dependencies
Err codemadness.org 70 i 5122
Err codemadness.org 70 i 5124
Run pkg-config from declarative dependencies in Cargo.toml
Err codemadness.org 70 i 5125

Who wrote librsvg?

2019-02-15T13:12:19-06:00

Authors by lines of code, each year:

Err codemadness.org 70 i 5134

Err codemadness.org 70 i 5135

Authors by percentage of lines of code, each year:

Err codemadness.org 70 i 5136

Err codemadness.org 70 i 5137

Which lines of code remain each year?

Err codemadness.org 70 i 5138

Err codemadness.org 70 i 5139

The shitty thing about a gradual rewrite is that a few people end up Err codemadness.org 70 i 5140 "owning" all the lines of source code. Hopefully this post is a little Err codemadness.org 70 i 5141 acknowledgment of the people that made librsvg possible.

Err codemadness.org 70 i 5142

The charts are made with the incredible tool Err codemadness.org 70 i 5143 git-of-theseus — thanks Err codemadness.org 70 i 5144 to @norwin@mastodon.art for digging it Err codemadness.org 70 i 5145 up! Its README also points to a Err codemadness.org 70 i 5146 Hercules plotter with awesome Err codemadness.org 70 i 5147 graphs. You know, for if you needed something to keep your computer Err codemadness.org 70 i 5148 busy during the weekend.

Librsvg's GObject boilerplate is in Rust now

2019-01-23T18:12:44-06:00

The other day I wrote about how most of librsvg's library code is in Err codemadness.org 70 i 5154 Rust now.

Err codemadness.org 70 i 5155

Today I finished porting the GObject boilerplate for the main Err codemadness.org 70 i 5156 RsvgHandle object into Rust. This means that the C code no longer Err codemadness.org 70 i 5157 calls things like g_type_register_static(), nor implements Err codemadness.org 70 i 5158 rsvg_handle_class_init() and such; all those are in Rust now. How Err codemadness.org 70 i 5159 is this done?

Err codemadness.org 70 i 5160

The life-changing magic of glib::subclass

Err codemadness.org 70 i 5161

Sebastian Dröge has been working for many months on refining Err codemadness.org 70 i 5162 utilities to make it possible to subclass GObjects in Rust, with Err codemadness.org 70 i 5163 little or no unsafe code. This subclass module is now part of Err codemadness.org 70 i 5164 glib-rs, the Rust bindings to GLib.

Err codemadness.org 70 i 5165

Librsvg now uses the subclassing functionality in glib-rs, which takes Err codemadness.org 70 i 5166 care of some things automatically:

Err codemadness.org 70 i 5167

Registering your GObject types at runtime.
Creating safe traits on which you can implement class_init, Err codemadness.org 70 i 5170 instance_init, set_property, get_property, and all the usual Err codemadness.org 70 i 5171 GObject paraphernalia.

Err codemadness.org 70 i 5173

Check this out:

Err codemadness.org 70 i 5174

use glib::subclass::prelude::*;	Err	codemadness.org	70
i  5175 	Err	codemadness.org	70
i  5176 impl ObjectSubclass for Handle {	Err	codemadness.org	70
i  5177     const NAME: &'static str = "RsvgHandle";	Err	codemadness.org	70
i  5178 	Err	codemadness.org	70
i  5179     type ParentType = glib::Object;	Err	codemadness.org	70
i  5180 	Err	codemadness.org	70
i  5181     type Instance = RsvgHandle;	Err	codemadness.org	70
i  5182     type Class = RsvgHandleClass;	Err	codemadness.org	70
i  5183 	Err	codemadness.org	70
i  5184     glib_object_subclass!();	Err	codemadness.org	70
i  5185 	Err	codemadness.org	70
i  5186     fn class_init(klass: &mut RsvgHandleClass) {	Err	codemadness.org	70
i  5187         klass.install_properties(&PROPERTIES);	Err	codemadness.org	70
i  5188     }	Err	codemadness.org	70
i  5189 	Err	codemadness.org	70
i  5190     fn new() -> Self {	Err	codemadness.org	70
i  5191         Handle::new()	Err	codemadness.org	70
i  5192     }	Err	codemadness.org	70
i  5193 }	Err	codemadness.org	70
i  5194

Err codemadness.org 70 i 5195 Err codemadness.org 70 i 5196

In the impl line, Handle is librsvg's internals object — what used Err codemadness.org 70 i 5197 to be RsvgHandlePrivate in the C code.

Err codemadness.org 70 i 5198

The following lines say this:

Err codemadness.org 70 i 5199

Err codemadness.org 70 i 5201
const NAME: &'static str = "RsvgHandle"; - the name of the type, Err codemadness.org 70 i 5202 for GType's perusal.
Err codemadness.org 70 i 5203
Err codemadness.org 70 i 5205
type ParentType = glib::Object; - Parent class.
Err codemadness.org 70 i 5206
Err codemadness.org 70 i 5208
type Instance, type Class - Structs with #[repr(C)], Err codemadness.org 70 i 5209 equivalent to GObject's class and instance structs.
Err codemadness.org 70 i 5210
Err codemadness.org 70 i 5212
glib_object_subclass!(); - All the boilerplate happens here Err codemadness.org 70 i 5213 automatically.
Err codemadness.org 70 i 5214
Err codemadness.org 70 i 5216
fn class_init - Should be familiar to anyone who implements Err codemadness.org 70 i 5217 GObjects!
Err codemadness.org 70 i 5218

Err codemadness.org 70 i 5220

And then, a couple of the property declarations:

Err codemadness.org 70 i 5221

static PROPERTIES: [subclass::Property; 11] = [	Err	codemadness.org	70
i  5222     subclass::Property("flags", |name| {	Err	codemadness.org	70
i  5223         ParamSpec::flags(	Err	codemadness.org	70
i  5224             name,	Err	codemadness.org	70
i  5225             "Flags",	Err	codemadness.org	70
i  5226             "Loading flags",	Err	codemadness.org	70
i  5227             HandleFlags::static_type(),	Err	codemadness.org	70
i  5228             0,	Err	codemadness.org	70
i  5229             ParamFlags::READWRITE | ParamFlags::CONSTRUCT_ONLY,	Err	codemadness.org	70
i  5230         )	Err	codemadness.org	70
i  5231     }),	Err	codemadness.org	70
i  5232     subclass::Property("dpi-x", |name| {	Err	codemadness.org	70
i  5233         ParamSpec::double(	Err	codemadness.org	70
i  5234             name,	Err	codemadness.org	70
i  5235             "Horizontal DPI",	Err	codemadness.org	70
i  5236             "Horizontal resolution in dots per inch",	Err	codemadness.org	70
i  5237             0.0,	Err	codemadness.org	70
i  5238             f64::MAX,	Err	codemadness.org	70
i  5239             0.0,	Err	codemadness.org	70
i  5240             ParamFlags::READWRITE | ParamFlags::CONSTRUCT,	Err	codemadness.org	70
i  5241         )	Err	codemadness.org	70
i  5242     }),	Err	codemadness.org	70
i  5243     // ... etcetera	Err	codemadness.org	70
i  5244 ];	Err	codemadness.org	70
i  5245

Err codemadness.org 70 i 5246 Err codemadness.org 70 i 5247

This is quite similar to the way C code usually registers properties Err codemadness.org 70 i 5248 for new GObject subclasses.

Err codemadness.org 70 i 5249

The moment at which a new GObject subclass gets registered against the Err codemadness.org 70 i 5250 GType system is in the foo_get_type() call. This is the C code in Err codemadness.org 70 i 5251 librsvg for that:

Err codemadness.org 70 i 5252

extern GType rsvg_handle_rust_get_type (void);	Err	codemadness.org	70
i  5253 	Err	codemadness.org	70
i  5254 GType	Err	codemadness.org	70
i  5255 rsvg_handle_get_type (void)	Err	codemadness.org	70
i  5256 {	Err	codemadness.org	70
i  5257     return rsvg_handle_rust_get_type ();	Err	codemadness.org	70
i  5258 }	Err	codemadness.org	70
i  5259

Err codemadness.org 70 i 5260 Err codemadness.org 70 i 5261

And the Rust function that actually implements this:

Err codemadness.org 70 i 5262

#[no_mangle]	Err	codemadness.org	70
i  5263 pub unsafe extern "C" fn rsvg_handle_rust_get_type() -> glib_sys::GType {	Err	codemadness.org	70
i  5264     Handle::get_type().to_glib()	Err	codemadness.org	70
i  5265 }	Err	codemadness.org	70
i  5266

Err codemadness.org 70 i 5267 Err codemadness.org 70 i 5268

Here, Handle::get_type() gets implemented automatically by Err codemadness.org 70 i 5269 Sebastian's subclass traits. It gets things like the type name and Err codemadness.org 70 i 5270 the parent class from the impl ObjectSubclass for Handle we saw Err codemadness.org 70 i 5271 above, and calls g_type_register_static() internally.

Err codemadness.org 70 i 5272

I can confirm now that implementing GObjects in Rust in this way, and Err codemadness.org 70 i 5273 exposing them to C, really works and is actually quite pleasant to Err codemadness.org 70 i 5274 do. You can look at librsvg's Rust code for GObject here.

Err codemadness.org 70 i 5275

Further work

Err codemadness.org 70 i 5276

There is some auto-generated C code to register librsvg's error enum Err codemadness.org 70 i 5277 and a flags type against GType; I'll move those to Rust over the next Err codemadness.org 70 i 5278 few days.

Err codemadness.org 70 i 5279

Then, I think I'll try to actually remove all of the library's entry Err codemadness.org 70 i 5280 points from the C code and implement them in Rust. Right now each C Err codemadness.org 70 i 5281 function is really just a single call to a Rust function, so this Err codemadness.org 70 i 5282 should be trivial-ish to do.

Err codemadness.org 70 i 5283

I'm waiting for a glib-rs release, the first one that will have the Err codemadness.org 70 i 5284 glib::subclass code in it, before merging all of the above into Err codemadness.org 70 i 5285 librsvg's master branch.

Err codemadness.org 70 i 5286

A new Rust API for librsvg?

Err codemadness.org 70 i 5287

Finally, this got me thinking about what to do about the Rust bindings Err codemadness.org 70 i 5288 to librsvg itself. The rsvg crate uses the gtk-rs Err codemadness.org 70 i 5289 machinery to generate the binding: it reads the GObject Err codemadness.org 70 i 5290 Introspection data from Rsvg.gir and generates a Rust binding Err codemadness.org 70 i 5291 for it.

Err codemadness.org 70 i 5292

However, the resulting API is mostly identical to the C API. There is Err codemadness.org 70 i 5293 an rsvg::Handle with the same methods as the ones from C's Err codemadness.org 70 i 5294 RsvgHandle... and that API is not particularly Rusty.

Err codemadness.org 70 i 5295

At some point I had an unfinished branch to merge rsvg-rs into Err codemadness.org 70 i 5296 librsvg. The intention was that librsvg's build procedure Err codemadness.org 70 i 5297 would first build librsvg.so itself, then generate Rsvg.gir as Err codemadness.org 70 i 5298 usual, and then generate rsvg-rs from that. But I got tired of Err codemadness.org 70 i 5299 fucking with Autotools, and didn't finish integrating the projects.

Err codemadness.org 70 i 5300

Rsvg-rs is an okay Rust API for using librsvg. It still works Err codemadness.org 70 i 5301 perfectly well from the standalone crate. However, now Err codemadness.org 70 i 5302 that all the functionality of librsvg is in Rust, I would like to take Err codemadness.org 70 i 5303 this opportunity to experiment with a better API for loading and Err codemadness.org 70 i 5304 rendering SVGs from Rust. This may make it more clear how to refactor Err codemadness.org 70 i 5305 the toplevel of the library. Maybe the librsvg project can provide Err codemadness.org 70 i 5306 its own Rust crate for public consumption, in addition to the usual Err codemadness.org 70 i 5307 librsvg.so and Rsvg.gir which need to remain with a stable API and Err codemadness.org 70 i 5308 ABI.

Librsvg is almost rustified now

2019-01-10T12:28:11-06:00

Since a few days ago, librsvg's library implementation is almost 100% Err codemadness.org 70 i 5316 Rust code. Paolo Borelli's and Carlos Martín Nieto's latest commits Err codemadness.org 70 i 5317 made it possible.

Err codemadness.org 70 i 5318

What does "almost 100% Rust code" mean here?

Err codemadness.org 70 i 5319

Err codemadness.org 70 i 5321
The C code no longer has struct fields that refer to the library's Err codemadness.org 70 i 5322 real work. The only field in RsvgHandlePrivate is an opaque Err codemadness.org 70 i 5323 pointer to a Rust-side structure. All the rest of the library's Err codemadness.org 70 i 5324 data lives in Rust structs.
Err codemadness.org 70 i 5325
Err codemadness.org 70 i 5327
The public API is implemented in C, but it is just stubs that Err codemadness.org 70 i 5328 immediately call into Rust functions. For example:
Err codemadness.org 70 i 5329

Err codemadness.org 70 i 5331

gboolean	Err	codemadness.org	70
i  5332 rsvg_handle_render_cairo_sub (RsvgHandle * handle, cairo_t * cr, const char *id)	Err	codemadness.org	70
i  5333 {	Err	codemadness.org	70
i  5334     g_return_val_if_fail (RSVG_IS_HANDLE (handle), FALSE);	Err	codemadness.org	70
i  5335     g_return_val_if_fail (cr != NULL, FALSE);	Err	codemadness.org	70
i  5336 	Err	codemadness.org	70
i  5337     return rsvg_handle_rust_render_cairo_sub (handle, cr, id);	Err	codemadness.org	70
i  5338 }	Err	codemadness.org	70
i  5339

Err codemadness.org 70 i 5340 Err codemadness.org 70 i 5341

Err codemadness.org 70 i 5343
The GObject boilerplate and supporting code is still in C: Err codemadness.org 70 i 5344 rsvg_handle_class_init and set_property and friends.
Err codemadness.org 70 i 5345
Err codemadness.org 70 i 5347
All the high-level tests are still done in C.
Err codemadness.org 70 i 5348
Err codemadness.org 70 i 5350
The gdk-pixbuf loader for SVG files is done in C.
Err codemadness.org 70 i 5351

Err codemadness.org 70 i 5353

Someone posted a chart on Reddit about the rustification of librsvg, Err codemadness.org 70 i 5354 comparing lines of code in each language vs. time.

Err codemadness.org 70 i 5355

Rustifying the remaining C code

Err codemadness.org 70 i 5356

There is only a handful of very small functions from the public API Err codemadness.org 70 i 5357 still implemented in C, and I am converting them one by one to Rust. Err codemadness.org 70 i 5358 These are just helper functions built on top of other public API that Err codemadness.org 70 i 5359 does the real work.

Err codemadness.org 70 i 5360

Converting the gdk-pixbuf loader to Rust seems like writing a little Err codemadness.org 70 i 5361 glue code for the loadable module; the actual loading is just a couple Err codemadness.org 70 i 5362 of calls to librsvg's API.

Err codemadness.org 70 i 5363

Rsvg-rs in rsvg?

Err codemadness.org 70 i 5364

Converting the tests to Rust... ideally this would use the rsvg-rs Err codemadness.org 70 i 5365 bindings; for example, it is what I already use for rsvg-bench, a Err codemadness.org 70 i 5366 benchmarking program for librsvg.

Err codemadness.org 70 i 5367

I have an unfinished branch to merge the rsvg-rs repository Err codemadness.org 70 i 5368 into librsvg's own repository. This is because...

Err codemadness.org 70 i 5369

Librsvg builds its library, librsvg.so
Gobject-introspection runs on librsvg.so and the source code, and Err codemadness.org 70 i 5372 produces librsvg.gir
Rsvg-rs's build system calls gir on librsvg.gir to generate the Err codemadness.org 70 i 5374 Rust binding's code.

Err codemadness.org 70 i 5376

As you can imagine, doing all of this with Autotools is... rather Err codemadness.org 70 i 5377 convoluted. It gives me a lot of anxiety to think that there is also Err codemadness.org 70 i 5378 an unfinished branch to port the build system to Meson, where Err codemadness.org 70 i 5379 probably doing the .so→.gir→rs chain would be easier, but who Err codemadness.org 70 i 5380 knows. Help in this area is much appreciated!

Err codemadness.org 70 i 5381

An alternative?

Err codemadness.org 70 i 5382

Rustified tests could, of course, call the C API of librsvg by hand, Err codemadness.org 70 i 5383 in unsafe code. This may not be idiomatic, but sounds like it could Err codemadness.org 70 i 5384 be done relatively quickly.

Err codemadness.org 70 i 5385

Future work

Err codemadness.org 70 i 5386

There are two options to get rid of all the C code in the library, and Err codemadness.org 70 i 5387 just leave C header files for public consumption:

Err codemadness.org 70 i 5388

Err codemadness.org 70 i 5390
Do the GObject implementation in Rust, using Sebastian Dröge's work Err codemadness.org 70 i 5391 from GStreamer to do this easily.
Err codemadness.org 70 i 5392
Err codemadness.org 70 i 5394
Work on making gnome-class powerful enough to implement the librsvg Err codemadness.org 70 i 5395 API directly, and in an ABI-compatible fashion to what there is Err codemadness.org 70 i 5396 right now.
Err codemadness.org 70 i 5397

Err codemadness.org 70 i 5399

The second case will probably build upon the first one, since one of Err codemadness.org 70 i 5400 my plans for gnome-class is to make it generate code that uses Err codemadness.org 70 i 5401 Sebastian's, instead of generating all the GObject boilerplate by Err codemadness.org 70 i 5402 hand.

In support of Coraline Ada Ehmke

2018-12-07T14:00:06-06:00

Last night, the linux.org DNS was hijacked and redirected to a page Err codemadness.org 70 i 5408 that doxed her. Coraline is doing extremely valuable work with the Err codemadness.org 70 i 5409 Contributor Covenant code of conduct, which many free software Err codemadness.org 70 i 5410 projects have adopted already.

Err codemadness.org 70 i 5411

Coraline has been working for years in making free software, and Err codemadness.org 70 i 5412 computer technology circles in general, a welcome place for Err codemadness.org 70 i 5413 underrepresented groups.

Err codemadness.org 70 i 5414

I hope Coraline stays safe and strong. You can support her directly Err codemadness.org 70 i 5415 on Patreon.

My GUADEC 2018 presentation

2018-12-04T18:57:00-06:00

I just realized that I forgot to publish my presentation from this Err codemadness.org 70 i 5422 year's GUADEC. Sorry, here it is!

Err codemadness.org 70 i 5423

Err codemadness.org 70 i 5424

You can also get the ODP file for the presentation. This is Err codemadness.org 70 i 5425 released under a CC-BY-SA license.

Err codemadness.org 70 i 5426

This is the video of the presentation.

Err codemadness.org 70 i 5427

Update Dec/06: Keen readers spotted an incorrect use of opaque Err codemadness.org 70 i 5428 pointers; I've updated the example code in the presentation to Err codemadness.org 70 i 5429 match Jordan's fix with the recommended usage. That merge Err codemadness.org 70 i 5430 request has an interesting conversation on FFI esoterica, too.

Refactoring allowed URLs in librsvg

2018-11-29T11:31:37-06:00

While in the middle of converting librsvg's code that processes XML from C Err codemadness.org 70 i 5435 to Rust, I went into a digression that has to do with the way librsvg Err codemadness.org 70 i 5436 decides which files are allowed to be referenced from within an SVG.

Err codemadness.org 70 i 5437

Resource references in SVG

Err codemadness.org 70 i 5438

SVG files can reference other files, i.e. they are not Err codemadness.org 70 i 5439 self-contained. For example, there can be an element like <image Err codemadness.org 70 i 5440 xlink:href="foo.png">, or one can request that a sub-element of Err codemadness.org 70 i 5441 another SVG be included with <use xlink:href="secondary.svg#foo">. Err codemadness.org 70 i 5442 Finally, there is the xi:include mechanism to include chunks of text Err codemadness.org 70 i 5443 or XML into another XML file.

Err codemadness.org 70 i 5444

Since librsvg is sometimes used to render untrusted files that come from Err codemadness.org 70 i 5445 the internet, it needs to be careful not to allow those files to Err codemadness.org 70 i 5446 reference any random resource on the filesystem. We don't want Err codemadness.org 70 i 5447 something like Err codemadness.org 70 i 5448 <text><xi:include href="/etc/passwd" parse="text"/></text> Err codemadness.org 70 i 5449 or something equally nefarious that would exfiltrate a random file Err codemadness.org 70 i 5450 into the rendered output.

Err codemadness.org 70 i 5451

Also, want to catch malicious SVGs that want to "phone home" by Err codemadness.org 70 i 5452 referencing a network resource like Err codemadness.org 70 i 5453 <image xlink:href="http://evil.com/pingback.jpg">.

Err codemadness.org 70 i 5454

So, librsvg is careful to have a single place where it can load Err codemadness.org 70 i 5455 secondary resources, and first it validates the resource's URL to see Err codemadness.org 70 i 5456 if it is allowed.

Err codemadness.org 70 i 5457

The actual validation rules are not very important for this Err codemadness.org 70 i 5458 discussion; they are something like "no absolute URLs allowed" (so you Err codemadness.org 70 i 5459 can't request /etc/passwd, "only siblings or (grand)children of Err codemadness.org 70 i 5460 siblings allowed" (so foo.svg can request bar.svg and Err codemadness.org 70 i 5461 subdir/bar.svg, but not ../../bar.svg).

Err codemadness.org 70 i 5462

The code

Err codemadness.org 70 i 5463

There was a central function rsvg_io_acquire_stream() which took a Err codemadness.org 70 i 5464 URL as a string. The code assumed that that URL had been first Err codemadness.org 70 i 5465 validated with a function called allow_load(url). While the code's Err codemadness.org 70 i 5466 structure guaranteed that all the places that may acquire a stream Err codemadness.org 70 i 5467 would actually go through allow_load() first, the structure of the Err codemadness.org 70 i 5468 code in Rust made it possible to actually make it impossible to Err codemadness.org 70 i 5469 acquire a disallowed URL.

Err codemadness.org 70 i 5470

Before:

Err codemadness.org 70 i 5471

pub fn allow_load(url: &str) -> bool;	Err	codemadness.org	70
i  5472 	Err	codemadness.org	70
i  5473 pub fn acquire_stream(url: &str, ...) -> Result<gio::InputStream, glib::Error>;	Err	codemadness.org	70
i  5474 	Err	codemadness.org	70
i  5475 pub fn rsvg_acquire_stream(url: &str, ...) -> Result<gio::InputStream, LoadingError> {	Err	codemadness.org	70
i  5476     if allow_load(url) {	Err	codemadness.org	70
i  5477         acquire_stream(url, ...)?	Err	codemadness.org	70
i  5478     } else {	Err	codemadness.org	70
i  5479         Err(LoadingError::NotAllowed)	Err	codemadness.org	70
i  5480     }	Err	codemadness.org	70
i  5481 }	Err	codemadness.org	70
i  5482

Err codemadness.org 70 i 5483 Err codemadness.org 70 i 5484

The refactored code now has an AllowedUrl type that encapsulates a Err codemadness.org 70 i 5485 URL, plus the promise that it has gone through these steps:

Err codemadness.org 70 i 5486

The URL has been run through a URL well-formedness parser.
The resource is allowed to be loaded following librsvg's rules.

Err codemadness.org 70 i 5490

pub struct AllowedUrl(Url);  // from the Url parsing crate	Err	codemadness.org	70
i  5491 	Err	codemadness.org	70
i  5492 impl AllowedUrl {	Err	codemadness.org	70
i  5493     pub fn from_href(href: &str) -> Result<AllowedUrl, ...> {	Err	codemadness.org	70
i  5494         let parsed = Url::parse(href)?; // may return LoadingError::InvalidUrl	Err	codemadness.org	70
i  5495 	Err	codemadness.org	70
i  5496         if allow_load(parsed) {	Err	codemadness.org	70
i  5497             Ok(AllowedUrl(parsed))	Err	codemadness.org	70
i  5498         } else {	Err	codemadness.org	70
i  5499             Err(LoadingError::NotAllowed)	Err	codemadness.org	70
i  5500         }	Err	codemadness.org	70
i  5501     }	Err	codemadness.org	70
i  5502 }	Err	codemadness.org	70
i  5503 	Err	codemadness.org	70
i  5504 // new prototype	Err	codemadness.org	70
i  5505 pub fn acquire_stream(url: &AllowedUrl, ...) -> Result<gio::InputStream, glib::Error>;	Err	codemadness.org	70
i  5506

Err codemadness.org 70 i 5507 Err codemadness.org 70 i 5508

This forces callers to validate the URLs as soon as possible, right Err codemadness.org 70 i 5509 after they get them from the SVG file. Now it is not possible to Err codemadness.org 70 i 5510 request a stream unless the URL has been validated first.

Err codemadness.org 70 i 5511

Plain URIs vs. fragment identifiers

Err codemadness.org 70 i 5512

Some of the elements in SVG that reference other data require full Err codemadness.org 70 i 5513 files:

Err codemadness.org 70 i 5514

<image xlink:href="foo.png" ...>      <!-- no fragments allowed -->	Err	codemadness.org	70
i  5515

Err codemadness.org 70 i 5516 Err codemadness.org 70 i 5517

And some others, that reference particular elements in secondary SVGs, Err codemadness.org 70 i 5518 require a fragment ID:

Err codemadness.org 70 i 5519

<use xlink:href="icons.svg#app_name" ...>   <!-- fragment id required -->	Err	codemadness.org	70
i  5520

Err codemadness.org 70 i 5521 Err codemadness.org 70 i 5522

And finally, the feImage element, used to paste an image as part of Err codemadness.org 70 i 5523 a filter effects pipeline, allows either:

Err codemadness.org 70 i 5524

<!-- will use that image -->	Err	codemadness.org	70
i  5525 <feImage xlink:href="foo.png" ...>	Err	codemadness.org	70
i  5526 	Err	codemadness.org	70
i  5527 <!-- will render just this element from an SVG and use it as an image -->	Err	codemadness.org	70
i  5528 <feImage xlink:href="foo.svg#element">	Err	codemadness.org	70
i  5529

Err codemadness.org 70 i 5530 Err codemadness.org 70 i 5531

So, I introduced a general Href parser :

Err codemadness.org 70 i 5532

pub enum Href {	Err	codemadness.org	70
i  5533     PlainUri(String),	Err	codemadness.org	70
i  5534     WithFragment(Fragment),	Err	codemadness.org	70
i  5535 }	Err	codemadness.org	70
i  5536 	Err	codemadness.org	70
i  5537 /// Optional URI, mandatory fragment id	Err	codemadness.org	70
i  5538 pub struct Fragment(Option<String>, String);	Err	codemadness.org	70
i  5539

Err codemadness.org 70 i 5540 Err codemadness.org 70 i 5541

The parts of the code that absolutely require a fragment id now take a Err codemadness.org 70 i 5542 Fragment. Parts which require a PlainUri can unwrap that case.

Err codemadness.org 70 i 5543

The next step is making those structs contain an AllowedUrl Err codemadness.org 70 i 5544 directly, instead of just strings, so that for callers, obtaining a Err codemadness.org 70 i 5545 fully validated name is a one-step operation.

Err codemadness.org 70 i 5546

In general, the code is moving towards a scheme where all file I/O is Err codemadness.org 70 i 5547 done at loading time. Right now, some of those external references Err codemadness.org 70 i 5548 get resolved at rendering time, which is somewhat awkward (for Err codemadness.org 70 i 5549 example, at rendering time the caller has no chance to use a Err codemadness.org 70 i 5550 GCancellable to cancel loading). This refactoring to do early Err codemadness.org 70 i 5551 validation is leaving the code in a very nice state.

Thessaloniki GNOME+Rust Hackfest 2018

2018-11-27T17:37:31-06:00

A couple of weeks ago we had the fourth GNOME+Rust hackfest, this time Err codemadness.org 70 i 5556 in Thessaloniki, Greece. This is the beautiful city that will host Err codemadness.org 70 i 5557 next year's GUADEC, but fortunately GUADEC will be in summertime!

Err codemadness.org 70 i 5558

We held the hackfest at the CoHo coworking space, a small, cozy Err codemadness.org 70 i 5559 office between the University and the sea.

Err codemadness.org 70 i 5560

Every such hackfest I am overwhelmed by the kind hackers who work on Err codemadness.org 70 i 5561 [gnome-class], the code generator for GObject implementations in Err codemadness.org 70 i 5562 Rust.

Err codemadness.org 70 i 5563

Mredlek has been working on generalizing the code generators in Err codemadness.org 70 i 5564 gnome-class, so that we can have the following from the same run:

Err codemadness.org 70 i 5565

Err codemadness.org 70 i 5567
Rust code generation, for the GObject implementations themselves. Err codemadness.org 70 i 5568 Thanks to mredlek, this is much cleaner than it was before; now both Err codemadness.org 70 i 5569 classes and interfaces share the same code for most of the Err codemadness.org 70 i 5570 boilerplate.
Err codemadness.org 70 i 5571
Err codemadness.org 70 i 5573
GObject Introspection (.gir) generation, so that language bindings Err codemadness.org 70 i 5574 can be generated automatically.
Err codemadness.org 70 i 5575
Err codemadness.org 70 i 5577
C header files (.h), so the generated GObjects can be called from Err codemadness.org 70 i 5578 C code as usual.
Err codemadness.org 70 i 5579

Err codemadness.org 70 i 5581

So far, Rust and GIR work; C header files are not generated yet.

Err codemadness.org 70 i 5582

Mredlek is a new contributor to gnome-class, but unfortunately was not Err codemadness.org 70 i 5583 able to attend the hackfest. Not only did he rewrite the gnome-class Err codemadness.org 70 i 5584 parser using the new version of syn; he also added support for Err codemadness.org 70 i 5585 passing owned types to GObject methods, such as String and Err codemadness.org 70 i 5586 Variant. But the biggest thing is probably that mredlek made it a Err codemadness.org 70 i 5587 lot easier to debug the generated Rust source; see the documentation Err codemadness.org 70 i 5588 on debugging for details.

Err codemadness.org 70 i 5589

Speaking of which, thanks to Jordan Petridis for making the Err codemadness.org 70 i 5590 documentation be published automatically from Gitlab's Continuous Err codemadness.org 70 i 5591 Integration pipelines.

Err codemadness.org 70 i 5592

Alex Crichton kindly refactored our error propagation code, and even Err codemadness.org 70 i 5593 wrote docs on it! Along with Jordan, they updated the Err codemadness.org 70 i 5594 code for the Rust 2018 edition, and generally wrangled the build Err codemadness.org 70 i 5595 process to conform with the lastest Rust nightlies. Alex also made Err codemadness.org 70 i 5596 code generation a lot faster, by offloading auto-indentation to an Err codemadness.org 70 i 5597 external rustfmt process, instead of using it as a crate: using the Err codemadness.org 70 i 5598 rustfmt crate meant that the compiler had a lot more work to do. Err codemadness.org 70 i 5599 During the whole hackfest, Alex was very helpful with Rust questions Err codemadness.org 70 i 5600 in general. While my strategy to see what the compiler does is to Err codemadness.org 70 i 5601 examine the disassembly in gdb, his strategy seems to be to look at Err codemadness.org 70 i 5602 the LLVM intermediate representation instead... OMG.

Err codemadness.org 70 i 5603

And we can derive very simple GtkWidgets now!

Err codemadness.org 70 i 5604

Saving the best for last... Antoni Boucher, the author of relm, has Err codemadness.org 70 i 5605 been working on making it possible to derive from gtk::Widget. Once Err codemadness.org 70 i 5606 this merge request is done, we'll have an example of Err codemadness.org 70 i 5607 deriving from gtk::DrawingArea from Rust with very little code.

Err codemadness.org 70 i 5608

Normally, the gtk-rs bindings work as a statically-generated binding Err codemadness.org 70 i 5609 for GObject, which really is a type hierarchy defined at runtime. The Err codemadness.org 70 i 5610 static binding really wants to know what is a subclass of what: it Err codemadness.org 70 i 5611 needs to know in advance that Button's hierarchy is Button → Bin → Err codemadness.org 70 i 5612 Container → Widget → Object, plus all the GTypeInterfaces supported Err codemadness.org 70 i 5613 by any of those classes. Antoni has been working on making Err codemadness.org 70 i 5614 gnome-class extract that information automatically from GIR files, so Err codemadness.org 70 i 5615 that the gtk-rs macros that define new types will get all the Err codemadness.org 70 i 5616 necessary information.

Err codemadness.org 70 i 5617

Future work

Err codemadness.org 70 i 5618

There are still bugs in the GIR pipeline that prevent us Err codemadness.org 70 i 5619 from deriving, say, from gtk::Container, but hopefully these will be Err codemadness.org 70 i 5620 resolved soon.

Err codemadness.org 70 i 5621

Sebastian Dröge has been refactoring his Rust tools to create GObject Err codemadness.org 70 i 5622 subclasses with very idiomatic and refined Rust code. This is now at Err codemadness.org 70 i 5623 a state where gnome-class itself could generate that sort of code, Err codemadness.org 70 i 5624 instead of generating all the boilerplate from scratch. So, we'll Err codemadness.org 70 i 5625 start doing that, and integrating the necessary bits into gtk-rs as Err codemadness.org 70 i 5626 well.

Err codemadness.org 70 i 5627

Finally, during the last day I took a little break from gnome-class to Err codemadness.org 70 i 5628 work on librsvg. Julian Sparber has been updating the code to use new Err codemadness.org 70 i 5629 bindings in cairo-rs, and is also adding a new API to Err codemadness.org 70 i 5630 fetch an SVG element's geometry precisely.

Err codemadness.org 70 i 5631

Thessaloniki

Err codemadness.org 70 i 5632

Oh, boy, I wish the weather had been warmer. The city looks Err codemadness.org 70 i 5633 delightful to walk around, especially in the narrow streets on the Err codemadness.org 70 i 5634 hills. Can't wait to see it in summer during GUADEC.

Err codemadness.org 70 i 5635

Thanks

Err codemadness.org 70 i 5636

Finally, thanks to CoHo for hosting the hackfest, and to the GNOME Err codemadness.org 70 i 5637 Foundation for sponsoring my travel and accomodation. And to Err codemadness.org 70 i 5638 Centricular for taking us all to dinner!

Err codemadness.org 70 i 5639

Special thanks to Jordan Petridis for being on top of everything Err codemadness.org 70 i 5640 build-wise all the time.

Err codemadness.org 70 i 5641

Propagating Errors

2018-11-21T13:58:12-06:00

Lately, I have been converting the code in librsvg that handles XML Err codemadness.org 70 i 5646 from C to Rust. For many technical reasons, the library still uses Err codemadness.org 70 i 5647 libxml2, GNOME's historic XML parsing library, but some of the Err codemadness.org 70 i 5648 callbacks to handle XML events like start_element, end_element, Err codemadness.org 70 i 5649 characters, are now implemented in Rust. This has meant that I'm Err codemadness.org 70 i 5650 running into all the cases where the original C code in librsvg failed Err codemadness.org 70 i 5651 to handle errors properly; Rust really makes it obvious when that Err codemadness.org 70 i 5652 happens.

Err codemadness.org 70 i 5653

In this post I want to talk a bit about propagating errors. You call Err codemadness.org 70 i 5654 a function, it returns an error, and then what?

Err codemadness.org 70 i 5655

What can fail?

Err codemadness.org 70 i 5656

It turns out that this question is highly context-dependent. Let's Err codemadness.org 70 i 5657 say a program is starting up and tries to read a configuration file. Err codemadness.org 70 i 5658 What could go wrong?

Err codemadness.org 70 i 5659

Err codemadness.org 70 i 5661
The file doesn't exist. Maybe it is the very first time the program Err codemadness.org 70 i 5662 is run, and so there isn't a configuration file at all? Can the Err codemadness.org 70 i 5663 program provide a default configuration in this case? Or does it Err codemadness.org 70 i 5664 absolutely need a pre-written configuration file to be somewhere?
Err codemadness.org 70 i 5665
Err codemadness.org 70 i 5667
The file can't be parsed. Should the program warn the user and Err codemadness.org 70 i 5668 exit, or should it revert to a default configuration (should it Err codemadness.org 70 i 5669 overwrite the file with valid, default values)? Can Err codemadness.org 70 i 5670 the program warn the user, or is it a user-less program that at best Err codemadness.org 70 i 5671 can just shout into the void of a server-side log file?
Err codemadness.org 70 i 5672
Err codemadness.org 70 i 5674
The file can be parsed, but the values are invalid. Same questions Err codemadness.org 70 i 5675 as the case above.
Err codemadness.org 70 i 5676
Err codemadness.org 70 i 5678
Etcetera.
Err codemadness.org 70 i 5679

Err codemadness.org 70 i 5681

At each stage, the code will probably see very low-level errors ("file Err codemadness.org 70 i 5682 not found", "I/O error", "parsing failed", "value is out of range"). Err codemadness.org 70 i 5683 What the code decides to do, or what it is able to do at any Err codemadness.org 70 i 5684 particular stage, depends both on the semantics you want from the Err codemadness.org 70 i 5685 program, and from the code structure itself.

Err codemadness.org 70 i 5686

Structuring the problem

Err codemadness.org 70 i 5687

This is an easy, but very coarse way of handling things:

Err codemadness.org 70 i 5688

gboolean	Err	codemadness.org	70
i  5689 read_configuration (const char *config_file_name)	Err	codemadness.org	70
i  5690 {	Err	codemadness.org	70
i  5691     /* open the file */	Err	codemadness.org	70
i  5692 	Err	codemadness.org	70
i  5693     /* parse it */	Err	codemadness.org	70
i  5694 	Err	codemadness.org	70
i  5695     /* set global variables to the configuration values */	Err	codemadness.org	70
i  5696 	Err	codemadness.org	70
i  5697     /* return true if success, or false if failure */	Err	codemadness.org	70
i  5698 }	Err	codemadness.org	70
i  5699

Err codemadness.org 70 i 5700 Err codemadness.org 70 i 5701

What is bad about this? Let's see:

Err codemadness.org 70 i 5702

Err codemadness.org 70 i 5704
The calling code just gets a success/failure condition. In the case Err codemadness.org 70 i 5705 of failure, it doesn't get to know why things failed.
Err codemadness.org 70 i 5706
Err codemadness.org 70 i 5708
If the function sets global variables with configuration values as Err codemadness.org 70 i 5709 they get read... and something goes wrong and the function returns Err codemadness.org 70 i 5710 an error... the caller ends up possibly in an inconsistent state, Err codemadness.org 70 i 5711 with a set of configuration variables that are only halfway-set.
Err codemadness.org 70 i 5712
Err codemadness.org 70 i 5714
If the function finds parse errors, well, do you really want to call Err codemadness.org 70 i 5715 UI code from inside it? The caller might be a better place to make Err codemadness.org 70 i 5716 that decision.
Err codemadness.org 70 i 5717

Err codemadness.org 70 i 5719

A slightly better structure

Err codemadness.org 70 i 5720

Let's add an enumeration to indicate the possible errors, and a Err codemadness.org 70 i 5721 structure of configuration values.

Err codemadness.org 70 i 5722

enum ConfigError {	Err	codemadness.org	70
i  5723     ConfigFileDoesntExist,	Err	codemadness.org	70
i  5724     ParseError, // config file has bad syntax or something	Err	codemadness.org	70
i  5725     ValueError, // config file has an invalid value	Err	codemadness.org	70
i  5726 }	Err	codemadness.org	70
i  5727 	Err	codemadness.org	70
i  5728 struct ConfigValues {	Err	codemadness.org	70
i  5729     // a bunch of fields here with the program's configuration	Err	codemadness.org	70
i  5730 }	Err	codemadness.org	70
i  5731 	Err	codemadness.org	70
i  5732 fn read_configuration(filename: &Path) -> Result<ConfigValues, ConfigError> {	Err	codemadness.org	70
i  5733     // open the file, or return Err(ConfigError::ConfigFileDoesntExist)	Err	codemadness.org	70
i  5734 	Err	codemadness.org	70
i  5735     // parse the file; or return Err(ConfigError::ParseError)	Err	codemadness.org	70
i  5736 	Err	codemadness.org	70
i  5737     // validate the values, or return Err(ConfigError::ValueError)	Err	codemadness.org	70
i  5738 	Err	codemadness.org	70
i  5739     // if everything succeeds, return Ok(ConfigValues)	Err	codemadness.org	70
i  5740 }	Err	codemadness.org	70
i  5741

Err codemadness.org 70 i 5742 Err codemadness.org 70 i 5743

This is better, in that the caller decides what to do with the Err codemadness.org 70 i 5744 validated ConfigValues: maybe it can just copy them to the Err codemadness.org 70 i 5745 program's global variables for configuration.

Err codemadness.org 70 i 5746

However, this scheme doesn't give the caller all the information it Err codemadness.org 70 i 5747 would like to present a really good error message. For example, the Err codemadness.org 70 i 5748 caller will get to know if there is a parse error, but it doesn't know Err codemadness.org 70 i 5749 specifically what failed during parsing. Similarly, it will just get Err codemadness.org 70 i 5750 to know if there was an invalid value, but not which one.

Err codemadness.org 70 i 5751

Ah, so the problem is fractal

Err codemadness.org 70 i 5752

We could have new structs to represent the little errors, and then Err codemadness.org 70 i 5753 make them part of the original error enum:

Err codemadness.org 70 i 5754

struct ParseError {	Err	codemadness.org	70
i  5755     line: usize,	Err	codemadness.org	70
i  5756     column: usize,	Err	codemadness.org	70
i  5757     error_reason: String,	Err	codemadness.org	70
i  5758 }	Err	codemadness.org	70
i  5759 	Err	codemadness.org	70
i  5760 struct ValueError {	Err	codemadness.org	70
i  5761     config_key: String,	Err	codemadness.org	70
i  5762     error_reason: String,	Err	codemadness.org	70
i  5763 }	Err	codemadness.org	70
i  5764 	Err	codemadness.org	70
i  5765 enum ConfigError {	Err	codemadness.org	70
i  5766     ConfigFileDoesntExist,	Err	codemadness.org	70
i  5767     ParseError(ParseError), // we put those structs in here	Err	codemadness.org	70
i  5768     ValueError(ValueError),	Err	codemadness.org	70
i  5769 }	Err	codemadness.org	70
i  5770

Err codemadness.org 70 i 5771 Err codemadness.org 70 i 5772

Is that enough? It depends.

Err codemadness.org 70 i 5773

The ParseError and ValueError structs have individual Err codemadness.org 70 i 5774 error_reason fields, which are strings. Presumably, one could have Err codemadness.org 70 i 5775 a ParseError with error_reason = "unexpected token", or a Err codemadness.org 70 i 5776 ValueError with error_reason = "cannot be a negative number".

Err codemadness.org 70 i 5777

One problem with this is that if the low-level errors come with error Err codemadness.org 70 i 5778 messages in English, then the caller has to know how to localize them Err codemadness.org 70 i 5779 to the user's language. Also, if they don't have a machine-readable Err codemadness.org 70 i 5780 error code, then the calling code may not have enough information to Err codemadness.org 70 i 5781 decide what do do with the error.

Err codemadness.org 70 i 5782

Let's say we had a ParseErrorKind enum with variants like Err codemadness.org 70 i 5783 UnexpectedToken, EndOfFile, etc. This is fine; it lets the Err codemadness.org 70 i 5784 calling code know the reason for the error. Also, there can be a Err codemadness.org 70 i 5785 gimme_localized_error_message() method for that particular type of Err codemadness.org 70 i 5786 error.

Err codemadness.org 70 i 5787

enum ParseErrorKind {	Err	codemadness.org	70
i  5788     UnexpectedToken,	Err	codemadness.org	70
i  5789     EndOfFile,	Err	codemadness.org	70
i  5790     MissingComma,	Err	codemadness.org	70
i  5791     // ... etc.	Err	codemadness.org	70
i  5792 }	Err	codemadness.org	70
i  5793 	Err	codemadness.org	70
i  5794 struct ParseError {	Err	codemadness.org	70
i  5795     line: usize,	Err	codemadness.org	70
i  5796     column: usize,	Err	codemadness.org	70
i  5797     kind: ParseErrorKind,	Err	codemadness.org	70
i  5798 }	Err	codemadness.org	70
i  5799

Err codemadness.org 70 i 5800 Err codemadness.org 70 i 5801

How can we expand this? Maybe the ParseErrorKind::UnexpectedToken Err codemadness.org 70 i 5802 variant wants to contain data that indicates which token it got that Err codemadness.org 70 i 5803 was wrong, so it would be UnexpectedToken(String) or something Err codemadness.org 70 i 5804 similar.

Err codemadness.org 70 i 5805

But is that useful to the calling code? For our example program, Err codemadness.org 70 i 5806 which is reading a configuration file... it probably only needs to Err codemadness.org 70 i 5807 know if it could parse the file, but maybe it doesn't really need any Err codemadness.org 70 i 5808 additional details on the reason for the parse error, other than Err codemadness.org 70 i 5809 having something useful to present to the user. Whether it is Err codemadness.org 70 i 5810 appropriate to burden the user with the actual details... does the app Err codemadness.org 70 i 5811 expect to make it the user's job to fix broken configuration files? Err codemadness.org 70 i 5812 Yes for a web server, where the user is a sysadmin; probably not for a Err codemadness.org 70 i 5813 random end-user graphical app, where people shouldn't need to write Err codemadness.org 70 i 5814 configuration files by hand in the first place (should those have a Err codemadness.org 70 i 5815 "Details" section in the error message window? I don't know!).

Err codemadness.org 70 i 5816

Maybe the low-level parsing/validation code can emit those detailed Err codemadness.org 70 i 5817 errors. But how can we propagate them to something more useful to the Err codemadness.org 70 i 5818 upper layers of the code?

Err codemadness.org 70 i 5819

Translation and propagation

Err codemadness.org 70 i 5820

Maybe our original read_configuration() function can translate the Err codemadness.org 70 i 5821 low-level errors into high-level ones:

Err codemadness.org 70 i 5822

fn read_configuration(filename: &Path) -> Result<ConfigValues, ConfigError> {	Err	codemadness.org	70
i  5823     // open file	Err	codemadness.org	70
i  5824 	Err	codemadness.org	70
i  5825     if cannot_open_file {	Err	codemadness.org	70
i  5826         return Err(ConfigError::ConfigFileDoesntExist);	Err	codemadness.org	70
i  5827     }	Err	codemadness.org	70
i  5828 	Err	codemadness.org	70
i  5829     let contents = read_the_file().map_err(|e| ... oops, maybe we need an IoError case, too)?;	Err	codemadness.org	70
i  5830 	Err	codemadness.org	70
i  5831     // parse file	Err	codemadness.org	70
i  5832 	Err	codemadness.org	70
i  5833     let parsed = parse(contents).map_err(|e| ... translate to a higher-level error)?	Err	codemadness.org	70
i  5834 	Err	codemadness.org	70
i  5835     // validate	Err	codemadness.org	70
i  5836 	Err	codemadness.org	70
i  5837     let validated = validate(parsed).map_err(|e| ... translate to a higher-level error)?;	Err	codemadness.org	70
i  5838 	Err	codemadness.org	70
i  5839     // yay!	Err	codemadness.org	70
i  5840     Ok(ConfigValues::from(validated))	Err	codemadness.org	70
i  5841 }	Err	codemadness.org	70
i  5842

Err codemadness.org 70 i 5843 Err codemadness.org 70 i 5844

Etcetera. It is up to each part of the code to decide what do do with Err codemadness.org 70 i 5845 lower-level errors. Can it recover from them? Should it fail the Err codemadness.org 70 i 5846 whole operation and return a higher-level error? Should it warn the Err codemadness.org 70 i 5847 user right there?

Err codemadness.org 70 i 5848

Language facilities

Err codemadness.org 70 i 5849

C makes it really easy to ignore errors, and pretty hard to present Err codemadness.org 70 i 5850 detailed errors like the above. One could mimic what Rust is actually Err codemadness.org 70 i 5851 doing with a collection of union and struct and enum, but this Err codemadness.org 70 i 5852 gets very awkward very fast.

Err codemadness.org 70 i 5853

Rust provides these facilities at the language level, and the idioms Err codemadness.org 70 i 5854 around Result and error handling are very nice to use. There are Err codemadness.org 70 i 5855 even crates like failure that go a long way towards Err codemadness.org 70 i 5856 automating error translation, propagation, and conversion to strings Err codemadness.org 70 i 5857 for presenting to users.

Err codemadness.org 70 i 5858

Infinite details

Err codemadness.org 70 i 5859

I've been recommending The Error Model to anyone who Err codemadness.org 70 i 5860 comes into a discussion of error handling in programming languages. Err codemadness.org 70 i 5861 It's a long, detailed, but very enlightening read on recoverable Err codemadness.org 70 i 5862 vs. unrecoverable errors, simple error codes vs. exceptions Err codemadness.org 70 i 5863 vs. monadic results, the performance/reliability/ease of use of each Err codemadness.org 70 i 5864 model... Definitely worth a read.

My gdk-pixbuf braindump

2018-09-05T21:35:41-05:00

I want to write a braindump on the stuff that I remember from Err codemadness.org 70 i 5870 gdk-pixbuf's history. There is some talk about replacing it with Err codemadness.org 70 i 5871 something newer; hopefully this history will show some things that Err codemadness.org 70 i 5872 worked, some that didn't, and why.

Err codemadness.org 70 i 5873

The beginnings

Err codemadness.org 70 i 5874

Gdk-pixbuf started as a replacement for Imlib, the image loading and Err codemadness.org 70 i 5875 rendering library that GNOME used in its earliest versions. Imlib Err codemadness.org 70 i 5876 came from the Enlightenment project; it provided an easy API around Err codemadness.org 70 i 5877 the idiosyncratic libungif, libjpeg, libpng, etc., and it maintained Err codemadness.org 70 i 5878 decoded images in memory with a uniform representation. Imlib also Err codemadness.org 70 i 5879 worked as an image cache for the Enlightenment window manager, which Err codemadness.org 70 i 5880 made memory management very inconvenient for GNOME.

Err codemadness.org 70 i 5881

Imlib worked well as a "just load me an image" library. It showed Err codemadness.org 70 i 5882 that a small, uniform API to load various image formats into a common Err codemadness.org 70 i 5883 representation was desirable. And in those days, hiding all the Err codemadness.org 70 i 5884 complexities of displaying images in X was very important indeed.

Err codemadness.org 70 i 5885

The initial API

Err codemadness.org 70 i 5886

Gdk-pixbuf replaced Imlib, and added two important features: Err codemadness.org 70 i 5887 reference counting for image data, and support for an alpha channel.

Err codemadness.org 70 i 5888

Gdk-pixbuf appeared with support for RGB(A) images. And although in Err codemadness.org 70 i 5889 theory it was possible to grow the API to support other Err codemadness.org 70 i 5890 representations, GdkColorspace never acquired anything other than Err codemadness.org 70 i 5891 GDK_COLORSPACE_RGB, and the bits_per_sample argument to some Err codemadness.org 70 i 5892 functions only ever supported being 8. The presence or absence of an alpha Err codemadness.org 70 i 5893 channel was done with a gboolean argument in conjunction with that Err codemadness.org 70 i 5894 single GDK_COLORSPACE_RGB value; we didn't have something like Err codemadness.org 70 i 5895 cairo_format_t which actually specifies the pixel format in single Err codemadness.org 70 i 5896 enum values.

Err codemadness.org 70 i 5897

While all the code in gdk-pixbuf carefully checks that those Err codemadness.org 70 i 5898 conditions are met — RGBA at 8 bits per channel —, some applications Err codemadness.org 70 i 5899 inadvertently assume that that is the only possible case, and would get Err codemadness.org 70 i 5900 into trouble really fast if gdk-pixbuf ever started returning pixbufs Err codemadness.org 70 i 5901 with different color spaces or depths.

Err codemadness.org 70 i 5902

One can still see the battle between bilevel-alpha Err codemadness.org 70 i 5903 vs. continuous-alpha in this enum:

Err codemadness.org 70 i 5904

typedef enum	Err	codemadness.org	70
i  5905 {	Err	codemadness.org	70
i  5906         GDK_PIXBUF_ALPHA_BILEVEL,	Err	codemadness.org	70
i  5907         GDK_PIXBUF_ALPHA_FULL	Err	codemadness.org	70
i  5908 } GdkPixbufAlphaMode;	Err	codemadness.org	70
i  5909

Err codemadness.org 70 i 5910 Err codemadness.org 70 i 5911

Fortunately, only the "render this pixbuf with alpha to an Xlib Err codemadness.org 70 i 5912 drawable" functions take values of this type: before the Xrender Err codemadness.org 70 i 5913 days, it was a Big Deal to draw an image with alpha to an X window, Err codemadness.org 70 i 5914 and applications often opted to use a bitmask instead, even if they Err codemadness.org 70 i 5915 had jagged edges as a result.

Err codemadness.org 70 i 5916

Pixel formats

Err codemadness.org 70 i 5917

The only pixel format that ever got implemented was unpremultiplied Err codemadness.org 70 i 5918 RGBA on all platforms. Back then I didn't understand premultiplied Err codemadness.org 70 i 5919 alpha! Also, the GIMP followed that scheme, and copying Err codemadness.org 70 i 5920 it seemed like the easiest thing.

Err codemadness.org 70 i 5921

After gdk-pixbuf, libart also copied that pixel format, I think.

Err codemadness.org 70 i 5922

But later we got Cairo, Pixman, and all the Xrender stack. These Err codemadness.org 70 i 5923 prefer premultiplied ARGB. Moreover, Cairo prefers it if each pixel Err codemadness.org 70 i 5924 is actually a 32-bit value, with the ARGB values inside it in Err codemadness.org 70 i 5925 platform-endian order. So if you look at a memory dump, a Cairo pixel Err codemadness.org 70 i 5926 looks like BGRA on a little-endian box, while it looks like ARGB on a Err codemadness.org 70 i 5927 big-endian box.

Err codemadness.org 70 i 5928

Every time we paint a GdkPixbuf to a cairo_t, there is a Err codemadness.org 70 i 5929 conversion from unpremultiplied RGBA to premultiplied, platform-endian Err codemadness.org 70 i 5930 ARGB. I talked a bit about this in Reducing the number of image Err codemadness.org 70 i 5931 copies in GNOME.

Err codemadness.org 70 i 5932

The loading API

Err codemadness.org 70 i 5933

The public loading API in gdk-pixbuf, and its relationship to loader Err codemadness.org 70 i 5934 plug-ins, evolved in interesting ways.

Err codemadness.org 70 i 5935

At first the public API and loaders only implemented load_from_file: Err codemadness.org 70 i 5936 you gave the library a FILE * and it gave you back a GdkPixbuf. Err codemadness.org 70 i 5937 Back then we didn't have a robust MIME sniffing framework in the form Err codemadness.org 70 i 5938 of a library, so gdk-pixbuf got its own. This lives in the Err codemadness.org 70 i 5939 mostly-obsolete GdkPixbufFormat machinery; it Err codemadness.org 70 i 5940 even has its own little language for sniffing file headers! Err codemadness.org 70 i 5941 Nowadays we do most MIME sniffing with GIO.

Err codemadness.org 70 i 5942

After the intial load_from_file API... I think we got progressive Err codemadness.org 70 i 5943 loading first, and animation support aftewards.

Err codemadness.org 70 i 5944

Progressive loading

Err codemadness.org 70 i 5945

This where the calling program feeds chunks of bytes to the library, Err codemadness.org 70 i 5946 and at the end a fully-formed GdkPixbuf comes out, instead of having Err codemadness.org 70 i 5947 a single "read a whole file" operation.

Err codemadness.org 70 i 5948

We conflated this with a way to get updates on how the image area gets Err codemadness.org 70 i 5949 modified as the data gets parsed. I think we wanted to support the Err codemadness.org 70 i 5950 case of a web browser, which downloads images slowly over the network, Err codemadness.org 70 i 5951 and gradually displays them as they are downloaded. In 1998, images Err codemadness.org 70 i 5952 downloading slowly over the network was a real concern!

Err codemadness.org 70 i 5953

It took a lot of very careful work to convert the image loaders, which Err codemadness.org 70 i 5954 parsed a whole file at a time, into loaders that could maintain some Err codemadness.org 70 i 5955 state between each time that they got handed an extra bit of buffer.

Err codemadness.org 70 i 5956

It also sounded easy to implement the progressive updating API by Err codemadness.org 70 i 5957 simply emitting a signal that said, "this rectangular area got updated Err codemadness.org 70 i 5958 from the last read". It could handle the case of reading whole Err codemadness.org 70 i 5959 scanlines, or a few pixels, or even area-based updates for progressive Err codemadness.org 70 i 5960 JPEGs and PNGs.

Err codemadness.org 70 i 5961

The internal API for the image format loaders still keeps a Err codemadness.org 70 i 5962 distinction between the "load a whole file" API and the "load an image Err codemadness.org 70 i 5963 in chunks". Not all loaders got redone to simply just use the second Err codemadness.org 70 i 5964 one: io-jpeg.c still implements loading whole files by calling the Err codemadness.org 70 i 5965 corresponding libjpeg functions. I think it could remove that code Err codemadness.org 70 i 5966 and use the progressive loading functions instead.

Err codemadness.org 70 i 5967

Animations

Err codemadness.org 70 i 5968

Animations: we followed the GIF model for animations, in which each Err codemadness.org 70 i 5969 frame overlays the previous one, and there's a delay set between each Err codemadness.org 70 i 5970 frame. This is not a video file; it's a hacky flipbook.

Err codemadness.org 70 i 5971

However, animations presented the problem that the whole gdk-pixbuf Err codemadness.org 70 i 5972 API was meant for static images, and now we needed to support Err codemadness.org 70 i 5973 multi-frame images as well.

Err codemadness.org 70 i 5974

We defined the "correct" way to use the gdk-pixbuf library as to Err codemadness.org 70 i 5975 actually try to load an animation, and then see if it is a Err codemadness.org 70 i 5976 single-frame image, in which case you can just get a GdkPixbuf for Err codemadness.org 70 i 5977 the only frame and use it.

Err codemadness.org 70 i 5978

Or, if you got an animation, that would be a GdkPixbufAnimation Err codemadness.org 70 i 5979 object, from which you could ask for an iterator to get each frame as Err codemadness.org 70 i 5980 a separate GdkPixbuf.

Err codemadness.org 70 i 5981

However, the progressive updating API never got extended to really Err codemadness.org 70 i 5982 support animations. So, we have awkward functions like Err codemadness.org 70 i 5983 gdk_pixbuf_animation_iter_on_currently_loading_frame() instead.

Err codemadness.org 70 i 5984

Necessary accretion

Err codemadness.org 70 i 5985

Gdk-pixbuf got support for saving just a few formats: JPEG, PNG, Err codemadness.org 70 i 5986 TIFF, ICO, and some of the formats that are implemented with the Err codemadness.org 70 i 5987 Windows-native loaders.

Err codemadness.org 70 i 5988

Over time gdk-pixbuf got support for preserving some metadata-ish Err codemadness.org 70 i 5989 chunks from formats that provide it: DPI, color profiles, image Err codemadness.org 70 i 5990 comments, hotspots for cursors/icons...

Err codemadness.org 70 i 5991

While an image is being loaded with the progressive loaders, there is Err codemadness.org 70 i 5992 a clunky way to specify that one doesn't want the actual size of the Err codemadness.org 70 i 5993 image, but another size instead. The loader can handle that situation Err codemadness.org 70 i 5994 itself, hopefully if an image format actually embeds different sizes Err codemadness.org 70 i 5995 in it. Or if not, the main loading code will rescale the full loaded Err codemadness.org 70 i 5996 image into the size specified by the application.

Err codemadness.org 70 i 5997

Historical cruft

Err codemadness.org 70 i 5998

GdkPixdata - a way to embed binary image data in executables, with a Err codemadness.org 70 i 5999 funky encoding. Nowadays it's just easier to directly store a PNG or Err codemadness.org 70 i 6000 JPEG or whatever in a GResource.

Err codemadness.org 70 i 6001

contrib/gdk-pixbuf-xlib - to deal with old-style X drawables. Err codemadness.org 70 i 6002 Hopefully mostly unused now, but there's a good number of mostly old, Err codemadness.org 70 i 6003 third-party software that still uses gdk-pixbuf as an image loader and Err codemadness.org 70 i 6004 renderer to X drawables.

Err codemadness.org 70 i 6005

gdk-pixbuf-transform.h - Gdk-pixbuf had some very high-quality Err codemadness.org 70 i 6006 scaling functions, which the original versions of EOG used for the Err codemadness.org 70 i 6007 core of the image viewer. Nowadays Cairo is the preferred way of Err codemadness.org 70 i 6008 doing this, since it not only does scaling, but general affine Err codemadness.org 70 i 6009 transformations as well. Did you know that Err codemadness.org 70 i 6010 gdk_pixbuf_composite_color takes 17 arguments, and it can composite Err codemadness.org 70 i 6011 an image with alpha on top of a checkerboard? Yes, that used to be Err codemadness.org 70 i 6012 the core of EOG.

Err codemadness.org 70 i 6013

Debatable historical cruft

Err codemadness.org 70 i 6014

gdk_pixbuf_get_pixels(). This lets the program look into the actual Err codemadness.org 70 i 6015 pixels of a loaded pixbuf, and modify them. Gdk-pixbuf just did not Err codemadness.org 70 i 6016 have a concept of immutability.

Err codemadness.org 70 i 6017

Back in GNOME 1.x / 2.x, when it was fashionable to put icons beside Err codemadness.org 70 i 6018 menu items, or in toolbar buttons, applications would load their icon Err codemadness.org 70 i 6019 images, and modify them in various ways before setting them onto the Err codemadness.org 70 i 6020 corresponding widgets. Some things they did: load a colorful icon, Err codemadness.org 70 i 6021 desaturate it for "insensitive" command buttons or menu items, or Err codemadness.org 70 i 6022 simulate desaturation by compositing a 1x1-pixel checkerboard on the Err codemadness.org 70 i 6023 icon image. Or lighten the icon and set it as the "prelight" one onto Err codemadness.org 70 i 6024 widgets.

Err codemadness.org 70 i 6025

The concept of "decode an image and just give me the pixels" is of Err codemadness.org 70 i 6026 course useful. Image viewers, image processing programs, and all Err codemadness.org 70 i 6027 those, of course need this functionality.

Err codemadness.org 70 i 6028

However, these days GTK would prefer to have a way to decode an image, Err codemadness.org 70 i 6029 and ship it as fast as possible ot the GPU, without intermediaries. Err codemadness.org 70 i 6030 There is all sorts of awkward machinery in the GTK widgets that Err codemadness.org 70 i 6031 can consume either an icon from an icon theme, or a user-supplied Err codemadness.org 70 i 6032 image, or one of the various schemes for providing icons that GTK has Err codemadness.org 70 i 6033 acquired over the years.

Err codemadness.org 70 i 6034

It is interesting to note that gdk_pixbuf_get_pixels() was available Err codemadness.org 70 i 6035 pretty much since the beginning, but it was only until much later that Err codemadness.org 70 i 6036 we got gdk_pixbuf_get_pixels_with_length(), the "give me the guchar Err codemadness.org 70 i 6037 * buffer and also its length" function, so that calling code has a Err codemadness.org 70 i 6038 chance of actually checking for buffer overruns. (... and it is one Err codemadness.org 70 i 6039 of the broken "give me a length" functions that returns a guint Err codemadness.org 70 i 6040 rather than a gsize. There is a better Err codemadness.org 70 i 6041 gdk_pixbuf_get_byte_length() which actually returns a gsize, Err codemadness.org 70 i 6042 though.)

Err codemadness.org 70 i 6043

Problems with mutable pixbufs

Err codemadness.org 70 i 6044

The main problem is that as things are right now, we have no Err codemadness.org 70 i 6045 flexibility in changing the internal representation of image data to Err codemadness.org 70 i 6046 make it better for current idioms: GPU-specific pixel formats may not Err codemadness.org 70 i 6047 be unpremultiplied RGBA data.

Err codemadness.org 70 i 6048

We have no API to say, "this pixbuf has been modified", akin to Err codemadness.org 70 i 6049 cairo_surface_mark_dirty(): once an application calls Err codemadness.org 70 i 6050 gdk_pixbuf_get_pixels(), gdk-pixbuf or GTK have to assume that the Err codemadness.org 70 i 6051 data will be changed and they have to re-run the pipeline to send Err codemadness.org 70 i 6052 the image to the GPU (format conversions? caching? creating a Err codemadness.org 70 i 6053 texture?).

Err codemadness.org 70 i 6054

Also, ever since the beginnings of the gdk-pixbuf API, we had a way to Err codemadness.org 70 i 6055 create pixbufs from arbitrary user-supplied RGBA buffers: the Err codemadness.org 70 i 6056 gdk_pixbuf_new_from_data functions. One problem with this scheme is Err codemadness.org 70 i 6057 that memory management of the buffer is up to the calling application, Err codemadness.org 70 i 6058 so the resulting pixbuf isn't free to handle those resources as it Err codemadness.org 70 i 6059 pleases.

Err codemadness.org 70 i 6060

A relatively recent addition is gdk_pixbuf_new_from_bytes(), which Err codemadness.org 70 i 6061 takes a GBytes buffer instead of a random guchar *. When a pixbuf Err codemadness.org 70 i 6062 is created that way, it is assumed to be immutable, since a GBytes Err codemadness.org 70 i 6063 is basically a shared reference into a byte buffer, and it's just Err codemadness.org 70 i 6064 easier to think of it as immutable. (Nothing in C actually enforces Err codemadness.org 70 i 6065 immutability, but the API indicates that convention.)

Err codemadness.org 70 i 6066

Internally, GdkPixbuf actually prefers to be created from a Err codemadness.org 70 i 6067 GBytes. It will downgrade itself to a guchar * buffer if Err codemadness.org 70 i 6068 something calls the old gdk_pixbuf_get_pixels(); in the best case, Err codemadness.org 70 i 6069 that will just take ownership of the internal buffer from the Err codemadness.org 70 i 6070 GBytes (if the GBytes has a single reference count); in the worst Err codemadness.org 70 i 6071 case, it will copy the buffer from the GBytes and retain ownership Err codemadness.org 70 i 6072 of that copy. In either case, when the pixbuf downgrades itself to Err codemadness.org 70 i 6073 pixels, it is assumed that the calling application will modify the Err codemadness.org 70 i 6074 pixel data.

Err codemadness.org 70 i 6075

What would immutable pixbufs look like?

Err codemadness.org 70 i 6076

I mentioned this a bit in "Reducing Copies". The Err codemadness.org 70 i 6077 loaders in gdk-pixbuf would create immutable pixbufs, with an internal Err codemadness.org 70 i 6078 representation that is friendly to GPUs. In the proposed scheme, that Err codemadness.org 70 i 6079 internal representation would be a Cairo image surface; it can be Err codemadness.org 70 i 6080 something else if GTK/GDK eventually prefer a different way of Err codemadness.org 70 i 6081 shipping image data into the toolkit.

Err codemadness.org 70 i 6082

Those pixbufs would be immutable. In true C fashion we can call it Err codemadness.org 70 i 6083 undefined behavior to change the pixel data (say, an app could request Err codemadness.org 70 i 6084 gimme_the_cairo_surface and tweak it, but that would not be Err codemadness.org 70 i 6085 supported).

Err codemadness.org 70 i 6086

I think we could also have a "just give me the pixels" API, and a Err codemadness.org 70 i 6087 "create a pixbuf from these pixels" one, but those would be one-time Err codemadness.org 70 i 6088 conversions at the edge of the API. Internally, the pixel data that Err codemadness.org 70 i 6089 actually lives inside a GdkPixbuf would remain immutable, in some Err codemadness.org 70 i 6090 preferred representation, which is not necessarily what the Err codemadness.org 70 i 6091 application sees.

Err codemadness.org 70 i 6092

What worked well

Err codemadness.org 70 i 6093

A small API to load multiple image formats, and paint the images Err codemadness.org 70 i 6094 easily to the screen, while handling most of the X awkwardness Err codemadness.org 70 i 6095 semi-automatically, was very useful!

Err codemadness.org 70 i 6096

A way to get and modify pixel data: applications clearly like doing Err codemadness.org 70 i 6097 this. We can formalize it as an application-side thing only, and keep Err codemadness.org 70 i 6098 the internal representation immutable and in a format that can evolve Err codemadness.org 70 i 6099 according to the needs of the internal API.

Err codemadness.org 70 i 6100

Pluggable loaders, up to a point. Gdk-pixbuf doesn't support all the Err codemadness.org 70 i 6101 image formats in the world out of the box, but it is relatively easy Err codemadness.org 70 i 6102 for third-parties to provide loaders that, once installed, are Err codemadness.org 70 i 6103 automatically usable for all applications.

Err codemadness.org 70 i 6104

What didn't work well

Err codemadness.org 70 i 6105

Having effectively two pixel formats supported, and nothing else: Err codemadness.org 70 i 6106 gdk-pixbuf does packed RGB and unpremultiplied RGBA, and that's it. Err codemadness.org 70 i 6107 This isn't completely terrible: applications which really want to Err codemadness.org 70 i 6108 know about indexed or grayscale images, or high bit-depth ones, are Err codemadness.org 70 i 6109 probably specialized enough that they can afford to have their own Err codemadness.org 70 i 6110 custom loaders with all the functionality they need.

Err codemadness.org 70 i 6111

Pluggable loaders, up to a point. While it is relatively easy to Err codemadness.org 70 i 6112 create third-party loaders, installation is awkward from a system's Err codemadness.org 70 i 6113 perspective: one has to run the script to regenerate the loader cache, Err codemadness.org 70 i 6114 there are more shared libraries running around, and the loaders are Err codemadness.org 70 i 6115 not sandboxed by default.

Err codemadness.org 70 i 6116

I'm not sure if it's worthwhile to let any application read "any" Err codemadness.org 70 i 6117 image format if gdk-pixbuf supports it. If your word processor lets Err codemadness.org 70 i 6118 you paste an image into the document... do you want it to use Err codemadness.org 70 i 6119 gdk-pixbuf's limited view of things and include a high bit-depth image Err codemadness.org 70 i 6120 with its probably inadequate conversions? Or would you rather do some Err codemadness.org 70 i 6121 processing by hand to ensure that the image looks as good as it can, Err codemadness.org 70 i 6122 in the format that your word processor actually supports? I don't Err codemadness.org 70 i 6123 know.

Err codemadness.org 70 i 6124

The API for animations is very awkward. We don't even support Err codemadness.org 70 i 6125 APNG... but honestly I don't recall actually seeing one of those in Err codemadness.org 70 i 6126 the wild.

Err codemadness.org 70 i 6127

The progressive loading API is awkward. The "feed some bytes into the Err codemadness.org 70 i 6128 loader" part is mostly okay; the "notify me about changes to the pixel Err codemadness.org 70 i 6129 data" is questionable nowadays. Web browsers don't use it; they Err codemadness.org 70 i 6130 implement their own loaders. Even EOG doesn't use it.

Err codemadness.org 70 i 6131

I think most code that actually connects to GdkPixbufLoader's Err codemadness.org 70 i 6132 signals only uses the size-prepared signal — the one that gets Err codemadness.org 70 i 6133 emitted soon after reading the image headers, when the loader gets to Err codemadness.org 70 i 6134 know the dimensions of the image. Apps sometimes use this to say, Err codemadness.org 70 i 6135 "this image is W*H pixels in size", but don't actually decode the Err codemadness.org 70 i 6136 rest of the image.

Err codemadness.org 70 i 6137

The gdk-pixbuf model of static images, or GIF animations, doesn't work Err codemadness.org 70 i 6138 well for multi-page TIFFs. I'm not sure if this is actualy a problem. Err codemadness.org 70 i 6139 Again, applications with actual needs for multi-page TIFFs are Err codemadness.org 70 i 6140 probably specialized enough that they will want a full-featured TIFF Err codemadness.org 70 i 6141 loader of their own.

Err codemadness.org 70 i 6142

Awkward architectures

Err codemadness.org 70 i 6143

Thumbnailers

Err codemadness.org 70 i 6144

The thumbnailing system has slowly been moving towards a model where Err codemadness.org 70 i 6145 we actually have thumbnailers specific to each file format, instead of Err codemadness.org 70 i 6146 just assuming that we can dump any image into a gdk-pixbuf loader.

Err codemadness.org 70 i 6147

If we take this all the way, we would be able to remove some weird Err codemadness.org 70 i 6148 code in, for example, the JPEG pixbuf loader. Right now it supports Err codemadness.org 70 i 6149 loading images at a size that the calling code requests, not only at Err codemadness.org 70 i 6150 the "natural" size of the JPEG. The thumbnailer can say, "I want to Err codemadness.org 70 i 6151 load this JPEG at 128x128 pixels" or whatever, and in theory the Err codemadness.org 70 i 6152 JPEG loader will do the minimal amount of work required to do that. Err codemadness.org 70 i 6153 It's not 100% clear to me if this is actually working as intended, or Err codemadness.org 70 i 6154 if we downscale the whole image anyway.

Err codemadness.org 70 i 6155

We had a distinction between in-process and out-of-process Err codemadness.org 70 i 6156 thumbnailers, and it had to do with the way pixbuf loaders are used; Err codemadness.org 70 i 6157 I'm not sure if they are all out-of-process and sandboxed now.

Err codemadness.org 70 i 6158

Non-raster data

Err codemadness.org 70 i 6159

There is a gdk-pixbuf loader for SVG images which uses librsvg Err codemadness.org 70 i 6160 internally, but only in a very basic way: it simply loads the SVG at Err codemadness.org 70 i 6161 its preferred size. Librsvg jumps through some hoops to compute a Err codemadness.org 70 i 6162 "preferred size" for SVGs, as not all of them actually indicate one. Err codemadness.org 70 i 6163 The SVG model would rather have the renderer say that the SVG is to be Err codemadness.org 70 i 6164 inserted into a rectangle of certain width/height, and Err codemadness.org 70 i 6165 scaled/positioned inside the rectangle according to some other Err codemadness.org 70 i 6166 parameters (i.e. like one would put it inside an HTML document, with a Err codemadness.org 70 i 6167 preserveAspectRatio attribute and all that). GNOME applications Err codemadness.org 70 i 6168 historically operated with a different model, one of "load me an Err codemadness.org 70 i 6169 image, I'll scale it to whatever size, and paint it".

Err codemadness.org 70 i 6170

This gdk-pixbuf loader for SVG files gets used for the SVG Err codemadness.org 70 i 6171 thumbnailer, or more accurately, the "throw random images into a Err codemadness.org 70 i 6172 gdk-pixbuf loader" thumbnailer. It may be better/cleaner to have a Err codemadness.org 70 i 6173 specific thumbnailer for SVGs instead.

Err codemadness.org 70 i 6174

Even EOG, our by-default image viewer, doesn't use the gdk-pixbuf Err codemadness.org 70 i 6175 loader for SVGs: it actually special-cases them and uses librsvg Err codemadness.org 70 i 6176 directly, to be able to load an SVG once and re-render it at different Err codemadness.org 70 i 6177 sizes if one changes the zoom factor, for example.

Err codemadness.org 70 i 6178

GTK reads its SVG icons... without using librsvg... by assuming that Err codemadness.org 70 i 6179 librsvg installed its gdk-pixbuf loader, so it loads them as any Err codemadness.org 70 i 6180 normal raster image. This kind of dirty, but I can't quite pinpoint Err codemadness.org 70 i 6181 why. I'm sure it would be convenient for icon themes to ship a single Err codemadness.org 70 i 6182 SVG with tons of icons, and some metadata on their ids, so that GTK Err codemadness.org 70 i 6183 could pick them out of the SVG file with rsvg_render_cairo_sub() or Err codemadness.org 70 i 6184 something. Right now icon theme authors are responsible for splitting Err codemadness.org 70 i 6185 out those huge SVGs into many little ones, one for each icon, and I Err codemadness.org 70 i 6186 don't think that's their favorite thing in the world to do :)

Err codemadness.org 70 i 6187

Exotic raster data

Err codemadness.org 70 i 6188

High bit-depth images... would you expect EOG to be able to load them? Err codemadness.org 70 i 6189 Certainly; maybe not with all the fancy conversions from a real RAW Err codemadness.org 70 i 6190 photo editor. But maybe this can be done as EOG-specific plugins, Err codemadness.org 70 i 6191 rather than as low in the platform as the gdk-pixbuf loaders?

Err codemadness.org 70 i 6192

(Same thing for thumbnailing high bit-depth images: the loading code Err codemadness.org 70 i 6193 should just provide its own thumbnailer program for those.)

Err codemadness.org 70 i 6194

Non-image metadata

Err codemadness.org 70 i 6195

The gdk_pixbuf_set_option / gdk_pixbuf_get_option family of Err codemadness.org 70 i 6196 functions is so that pixbuf loaders can set key/value pairs of strings Err codemadness.org 70 i 6197 onto a pixbuf. Loaders use this for comment blocks, or ICC profiles Err codemadness.org 70 i 6198 for color calibration, or DPI information for images that have it, or Err codemadness.org 70 i 6199 EXIF data from photos. It is up to applications to actually use this Err codemadness.org 70 i 6200 information.

Err codemadness.org 70 i 6201

It's a bit uncomfortable that gdk-pixbuf makes no promises about the Err codemadness.org 70 i 6202 kind of raster data it gives to the caller: right now it is raw Err codemadness.org 70 i 6203 RGB(A) data that is not gamma-corrected nor in any particular color Err codemadness.org 70 i 6204 space. It is up to the caller to see if the pixbuf has an ICC profile Err codemadness.org 70 i 6205 attached to it as an option. Effectively, this means that Err codemadness.org 70 i 6206 applications don't know if they are getting SRGB, or linear RGB, or Err codemadness.org 70 i 6207 what... unless they specifically care to look.

Err codemadness.org 70 i 6208

The gdk-pixbuf API could probably make promises: if you call this Err codemadness.org 70 i 6209 function you will get SRGB data; if you call this other function, Err codemadness.org 70 i 6210 you'll get the raw RGBA data and we'll tell you its Err codemadness.org 70 i 6211 colorspace/gamma/etc.

Err codemadness.org 70 i 6212

The various set_option / get_option pairs are also usable by the Err codemadness.org 70 i 6213 gdk-pixbuf saving code (up to now we have just talked about Err codemadness.org 70 i 6214 loaders). I don't know enough about how applications use the saving Err codemadness.org 70 i 6215 code in gdk-pixbuf... the thumbnailers use it to save PNGs or JPEGs, Err codemadness.org 70 i 6216 but other apps? No idea.

Err codemadness.org 70 i 6217

What I would like to see

Err codemadness.org 70 i 6218

Immutable pixbufs in a useful format. I've started work on Err codemadness.org 70 i 6219 this in a merge request; the internal code is now ready Err codemadness.org 70 i 6220 to take in different internal representations of pixel data. My goal Err codemadness.org 70 i 6221 is to make Cairo image surfaces the preferred, immutable, internal Err codemadness.org 70 i 6222 representation. This would give us a Err codemadness.org 70 i 6223 gdk_pixbuf_get_cairo_surface(), which pretty much everything that Err codemadness.org 70 i 6224 needs one reimplements by hand.

Err codemadness.org 70 i 6225

Find places that assume mutable pixbufs. To gradually deprecate Err codemadness.org 70 i 6226 mutable pixbufs, I think we would need to audit applications and Err codemadness.org 70 i 6227 libraries to find places that cause GdkPixbuf structures to degrade Err codemadness.org 70 i 6228 into mutable ones: basically, find callers of Err codemadness.org 70 i 6229 gdk_pixbuf_get_pixels() and related functions, see what they do, and Err codemadness.org 70 i 6230 reimplement them differently. Maybe they don't need to tint icons by Err codemadness.org 70 i 6231 hand anymore? Maybe they don't need icons anymore, given our Err codemadness.org 70 i 6232 changing UI paradigms? Maybe they are using gdk-pixbuf as an image Err codemadness.org 70 i 6233 loader only?

Err codemadness.org 70 i 6234

Reconsider the loading-updates API. Do we need the Err codemadness.org 70 i 6235 GdkPixbufLoader::area-updated signal at all? Does anything break Err codemadness.org 70 i 6236 if we just... not emit it, or just emit it once at the end of the Err codemadness.org 70 i 6237 loading process? (Caveat: keeping it unchanged more or less means Err codemadness.org 70 i 6238 that "immutable pixbufs" as loaded by gdk-pixbuf actually mutate while Err codemadness.org 70 i 6239 being loaded, and this mutation is exposed to applications.)

Err codemadness.org 70 i 6240

Sandboxed loaders. While these days gdk-pixbuf loaders prefer the Err codemadness.org 70 i 6241 progressive feed-it-bytes API, sandboxed loaders would maybe prefer a Err codemadness.org 70 i 6242 read-a-whole-file approach. I don't know enough about memfd or how Err codemadness.org 70 i 6243 sandboxes pass data around to know how either would work.

Err codemadness.org 70 i 6244

Move loaders to Rust. Yes, really. Loaders are Err codemadness.org 70 i 6245 security-sensitive, and while we do need to sandbox them, it would Err codemadness.org 70 i 6246 certainly be better to do them in a memory-safe language. There are Err codemadness.org 70 i 6247 already pure Rust-based image loaders: JPEG, Err codemadness.org 70 i 6248 PNG, TIFF, GIF, ICO. Err codemadness.org 70 i 6249 I have no idea how featureful they are. We can certainly try them Err codemadness.org 70 i 6250 with gdk-pixbuf's own suite of test images. We can modify them to add Err codemadness.org 70 i 6251 hooks for things like a size-prepared notification, if they don't Err codemadness.org 70 i 6252 already have a way to read "just the image headers".

Err codemadness.org 70 i 6253

Rust makes it very easy to plug in micro-benchmarks, Err codemadness.org 70 i 6254 fuzz testing, and other modern amenities. These would be Err codemadness.org 70 i 6255 perfect for improving the loaders.

Err codemadness.org 70 i 6256

I started sketching a Rust backend for gdk-pixbuf Err codemadness.org 70 i 6257 loaders some months ago, but there's nothing useful Err codemadness.org 70 i 6258 yet. One mismatch between gdk-pixbuf's model for loaders, and the Err codemadness.org 70 i 6259 existing Rust codecs, is that Rust codecs generally take something Err codemadness.org 70 i 6260 that implements the Read trait: a blocking API to read bytes from Err codemadness.org 70 i 6261 abstract sources; it's a pull API. The gdk-pixbuf model is a push Err codemadness.org 70 i 6262 API: the calling code creates a loader object, and then pushes bytes Err codemadness.org 70 i 6263 into it. The gdk-pixbuf convenience functions that take a Err codemadness.org 70 i 6264 GInputStream basically do this:

Err codemadness.org 70 i 6265

loader = gdk_pixbuf_loader_new (...);	Err	codemadness.org	70
i  6266 	Err	codemadness.org	70
i  6267 while (more_bytes) {	Err	codemadness.org	70
i  6268     n_read = g_input_stream_read (stream, buffer, ...);	Err	codemadness.org	70
i  6269     gdk_pixbuf_loader_write(loader, buffer, n_read, ...);	Err	codemadness.org	70
i  6270 }	Err	codemadness.org	70
i  6271 	Err	codemadness.org	70
i  6272 gdk_pixbuf_loader_close (loader);	Err	codemadness.org	70
i  6273

Err codemadness.org 70 i 6274 Err codemadness.org 70 i 6275

However, this cannot be flipped around easily. We could probably use Err codemadness.org 70 i 6276 a second thread (easy, safe to do in Rust) to make the reader/decoder Err codemadness.org 70 i 6277 thread block while the main thread pushes bytes into it.

Err codemadness.org 70 i 6278

Also, I don't know how the Rust bindings for GIO present things like Err codemadness.org 70 i 6279 GInputStream and friends, with our nice async cancellables and all Err codemadness.org 70 i 6280 that.

Err codemadness.org 70 i 6281

Deprecate animations? Move that code to EOG, just so one can look Err codemadness.org 70 i 6282 at memes in it? Do any "real apps" actually use GIF animations for Err codemadness.org 70 i 6283 their UI?

Err codemadness.org 70 i 6284

Formalize promises around returned color profiles, gamma, etc. As Err codemadness.org 70 i 6285 mentioned above: have an "easy API" that returns SRGB, and a "raw API" Err codemadness.org 70 i 6286 that returns the ARGB data from the image, plus info on its ICC Err codemadness.org 70 i 6287 profile, gamma, or any other info needed to turn this into a Err codemadness.org 70 i 6288 "good enough to be universal" representation. (I think all the Err codemadness.org 70 i 6289 Apple APIs that pass colors around do so with an ICC profile attached, Err codemadness.org 70 i 6290 which seems... pretty much necessary for correctness.)

Err codemadness.org 70 i 6291

Remove the internal MIME-sniffing machinery. And just use GIO's.

Err codemadness.org 70 i 6292

Deprecate the crufty/old APIs in gdk-pixbuf. Err codemadness.org 70 i 6293 Scaling/transformation, compositing, GdkPixdata, Err codemadness.org 70 i 6294 gdk-pixbuf-csource, all those. Pixel crunching can be done by Err codemadness.org 70 i 6295 Cairo; the others are better done with GResource these days.

Err codemadness.org 70 i 6296

Figure out if we want blessed codecs; fix thumbnailers. Link those Err codemadness.org 70 i 6297 loaders statically, unconditionally. Exotic formats can go in their Err codemadness.org 70 i 6298 own custom thumbnailers. Figure out if we want sandboxed loaders for Err codemadness.org 70 i 6299 everything, or just for user-side images (not ones read from the Err codemadness.org 70 i 6300 trusted system installation).

Err codemadness.org 70 i 6301

Have GTK4 communicate clearly about its drawing model. I think we Err codemadness.org 70 i 6302 are having a disconnect between the GUI chrome, which is CSS/GPU Err codemadness.org 70 i 6303 friendly, and graphical content generated by applications, which by Err codemadness.org 70 i 6304 default right now is done via Cairo. And having Cairo as a to-screen Err codemadness.org 70 i 6305 and to-printer API is certainly very convenient! You Wouldn't Print a Err codemadness.org 70 i 6306 GUI, but certainly you would print a displayed document.

Err codemadness.org 70 i 6307

It would also be useful for GTK4 to actually define what its preferred Err codemadness.org 70 i 6308 image format is if it wants to ship it to the GPU with as little work Err codemadness.org 70 i 6309 as possible. Maybe it's a Cairo image surface? Maybe something else?

Err codemadness.org 70 i 6310

Conclusion

Err codemadness.org 70 i 6311

We seem to change imaging models every ten years or so. Xlib, then Err codemadness.org 70 i 6312 Xrender with Cairo, then GPUs and CSS-based drawing for widgets. Err codemadness.org 70 i 6313 We've gone from trusted data on your local machine, to potentially malicious data that Err codemadness.org 70 i 6314 rains from the Internet. Gdk-pixbuf has spanned all of these periods Err codemadness.org 70 i 6315 so far, and it is due for a big change.

Debugging an Rc reference leak in Rust

2018-08-29T16:47:13-05:00

The bug that caused two brown-paper-bag released in librsvg — Err codemadness.org 70 i 6320 because it was leaking all the SVG nodes — has been interesting.

Err codemadness.org 70 i 6321

Memory leaks in Rust? Isn't it supposed to prevent that?

Err codemadness.org 70 i 6322

Well, yeah, but the leaks were caused by the C side of things, and by Err codemadness.org 70 i 6323 unsafe code in Rust, which does not prevent leaks.

Err codemadness.org 70 i 6324

The first part of the bug was easy: C code started calling a Err codemadness.org 70 i 6325 function implemented in Rust, which returns a newly-acquired reference Err codemadness.org 70 i 6326 to an SVG node. The old code simply got a pointer to the node, Err codemadness.org 70 i 6327 without acquiring a reference. The new code was forgetting to Err codemadness.org 70 i 6328 rsvg_node_unref(). No biggie.

Err codemadness.org 70 i 6329

The second part of the bug was trickier to find. The C code Err codemadness.org 70 i 6330 was apparently calling all the functions to unref nodes as Err codemadness.org 70 i 6331 appropriate, and even calling the rsvg_tree_free() function in the Err codemadness.org 70 i 6332 end; this is the "free the whole SVG tree" function.

Err codemadness.org 70 i 6333

There are these types:

Err codemadness.org 70 i 6334

// We take a pointer to this and expose it as an opaque pointer to C	Err	codemadness.org	70
i  6335 pub enum RsvgTree {}	Err	codemadness.org	70
i  6336 	Err	codemadness.org	70
i  6337 // This is the real structure we care about	Err	codemadness.org	70
i  6338 pub struct Tree {	Err	codemadness.org	70
i  6339     // This is the Rc that was getting leaked	Err	codemadness.org	70
i  6340     pub root: Rc<Node>,	Err	codemadness.org	70
i  6341     ...	Err	codemadness.org	70
i  6342 }	Err	codemadness.org	70
i  6343

Err codemadness.org 70 i 6344 Err codemadness.org 70 i 6345

Tree is the real struct that holds the root of the SVG tree and some Err codemadness.org 70 i 6346 other data. Each node is an Rc<Node>; the root node was getting Err codemadness.org 70 i 6347 leaked (... and all the children, recursively) because its reference Err codemadness.org 70 i 6348 count never went down from 1.

Err codemadness.org 70 i 6349

RsvgTree is just an empty type. The code does an unsafe cast of Err codemadness.org 70 i 6350 *const Tree as *const RsvgTree in order to expose a raw pointer to Err codemadness.org 70 i 6351 the C code.

Err codemadness.org 70 i 6352

The rsvg_tree_free() function, callable from C, looked like this:

Err codemadness.org 70 i 6353

#[no_mangle]	Err	codemadness.org	70
i  6354 pub extern "C" fn rsvg_tree_free(tree: *mut RsvgTree) {	Err	codemadness.org	70
i  6355     if !tree.is_null() {	Err	codemadness.org	70
i  6356         let _ = unsafe { Box::from_raw(tree) };	Err	codemadness.org	70
i  6357                          // ^ this returns a Box<RsvgTree> which is an empty type!	Err	codemadness.org	70
i  6358     }	Err	codemadness.org	70
i  6359 }	Err	codemadness.org	70
i  6360

Err codemadness.org 70 i 6361 Err codemadness.org 70 i 6362

When we call Box::from_raw() on a *mut RsvgTree, it gives us back Err codemadness.org 70 i 6363 a Box<RsvgTree>... which is a box of a zero-sized type. So, the program Err codemadness.org 70 i 6364 frees zero memory when the box gets dropped.

Err codemadness.org 70 i 6365

The code was missing this cast:

Err codemadness.org 70 i 6366

    let tree = unsafe { &mut *(tree as *mut Tree) };	Err	codemadness.org	70
i  6367                                  // ^ this cast to the actual type inside the Box	Err	codemadness.org	70
i  6368     let _ = unsafe { Box::from_raw(tree) };	Err	codemadness.org	70
i  6369

Err codemadness.org 70 i 6370 Err codemadness.org 70 i 6371

So, tree as *mut Tree gives us a value which will cause Err codemadness.org 70 i 6372 Box::from_raw() to return a Box<Tree>, which is what we intended. Err codemadness.org 70 i 6373 Dropping the box will drop the Tree, reduce the last reference count Err codemadness.org 70 i 6374 on the root node, and free all the nodes recursively.

Err codemadness.org 70 i 6375

Monitoring an `Rc<T>`'s reference count in gdb

Err codemadness.org 70 i 6376

So, how does one set a gdb watchpoint on the reference count?

Err codemadness.org 70 i 6377

First I set a breakpoint on a function which I knew would get passed Err codemadness.org 70 i 6378 the Rc<Node> I care about:

Err codemadness.org 70 i 6379

(gdb) b <rsvg_internals::structure::NodeSvg as rsvg_internals::node::NodeTrait>::set_atts	Err	codemadness.org	70
i  6380 Breakpoint 3 at 0x7ffff71f3aaa: file rsvg_internals/src/structure.rs, line 131.	Err	codemadness.org	70
i  6381 	Err	codemadness.org	70
i  6382 (gdb) c	Err	codemadness.org	70
i  6383 Continuing.	Err	codemadness.org	70
i  6384 	Err	codemadness.org	70
i  6385 Thread 1 "rsvg-convert" hit Breakpoint 3, <rsvg_internals::structure::NodeSvg as rsvg_internals::node::NodeTrait>::set_atts (self=0x646c60, node=0x64c890, pbag=0x64c820) at rsvg_internals/src/structure.rs:131	Err	codemadness.org	70
i  6386 	Err	codemadness.org	70
i  6387 (gdb) p node	Err	codemadness.org	70
i  6388 $5 = (alloc::rc::Rc<rsvg_internals::node::Node> *) 0x64c890	Err	codemadness.org	70
i  6389

Err codemadness.org 70 i 6390 Err codemadness.org 70 i 6391

Okay, node is a reference to an Rc<Node>. What's inside?

Err codemadness.org 70 i 6392

(gdb) p *node	Err	codemadness.org	70
i  6393 $6 = {ptr = {pointer = {__0 = 0x625800}}, phantom = {<No data fields>}}	Err	codemadness.org	70
i  6394

Err codemadness.org 70 i 6395 Err codemadness.org 70 i 6396

Why, a pointer to the actual contents of the Rc. Look inside Err codemadness.org 70 i 6397 again:

Err codemadness.org 70 i 6398

(gdb) p *node.ptr.pointer.__0	Err	codemadness.org	70
i  6399 $9 = {strong = {value = {value = 3}}, weak = {value = {value = 1}},  ... and lots of extra crap ...	Err	codemadness.org	70
i  6400

Err codemadness.org 70 i 6401 Err codemadness.org 70 i 6402

Aha! There are the strong and weak reference counts. So, set a Err codemadness.org 70 i 6403 watchpoint on the strong reference count:

Err codemadness.org 70 i 6404

(gdb) set $ptr = &node.ptr.pointer.__0.strong.value.value	Err	codemadness.org	70
i  6405 (gdb) watch *$ptr	Err	codemadness.org	70
i  6406 Hardware watchpoint 4: *$ptr	Err	codemadness.org	70
i  6407

Err codemadness.org 70 i 6408 Err codemadness.org 70 i 6409

Continue running the program until the reference count changes:

Err codemadness.org 70 i 6410

(gdb) continue	Err	codemadness.org	70
i  6411 Thread 1 "rsvg-convert" hit Hardware watchpoint 4: *$ptr	Err	codemadness.org	70
i  6412 	Err	codemadness.org	70
i  6413 Old value = 3	Err	codemadness.org	70
i  6414 New value = 2	Err	codemadness.org	70
i  6415

Err codemadness.org 70 i 6416 Err codemadness.org 70 i 6417

At this point I can print a stack trace and see if it makes sense, Err codemadness.org 70 i 6418 check that the refs/unrefs are matched, etc.

Err codemadness.org 70 i 6419

TL;DR: dig into the Rc<T> until you find the reference count, and Err codemadness.org 70 i 6420 watch it. It's wrapped in several layers of Rust-y types; NonNull Err codemadness.org 70 i 6421 pointers, an RcBox for the actual container of the refcount plus the Err codemadness.org 70 i 6422 object it's wrapping, and Cells for the refcount values. Just dig Err codemadness.org 70 i 6423 until you reach the refcount values and they are there.

Err codemadness.org 70 i 6424

So, how did I find the missing cast?

Err codemadness.org 70 i 6425

Using that gdb recipe, I watched the reference count of the toplevel Err codemadness.org 70 i 6426 SVG node change until the program exited. When the program Err codemadness.org 70 i 6427 terminated, the reference count was 1 — it should have dropped to 0 if Err codemadness.org 70 i 6428 there was no memory leak.

Err codemadness.org 70 i 6429

The last place where the toplevel node loses a reference is in Err codemadness.org 70 i 6430 rsvg_tree_free(). I ran the program again and checked if that Err codemadness.org 70 i 6431 function was being called; it was being called correctly. So I knew Err codemadness.org 70 i 6432 that the problem must lie in that function. After a little Err codemadness.org 70 i 6433 head-scratching, I found the missing cast. Other functions of the Err codemadness.org 70 i 6434 form rsvg_tree_whatever() had that cast, but rsvg_tree_free() was Err codemadness.org 70 i 6435 missing it.

Err codemadness.org 70 i 6436

I think Rust now has better facilities to tag structs that are exposed Err codemadness.org 70 i 6437 as raw pointers to extern code, to avoid this kind of perilous Err codemadness.org 70 i 6438 casting. We'll see.

Err codemadness.org 70 i 6439

In the meantime, apologies for the buggy releases!

Logging from Rust in librsvg

2018-08-03T19:29:43-05:00

Over in this issue we are discussing how to add debug logging Err codemadness.org 70 i 6448 for librsvg.

Err codemadness.org 70 i 6449

A popular way to add logging to Rust code is to use the log crate. Err codemadness.org 70 i 6450 This lets you sprinkle simple messages in your code:

Err codemadness.org 70 i 6451

error!("something bad happened: {}", foo);	Err	codemadness.org	70
i  6452 debug!("a debug message");	Err	codemadness.org	70
i  6453

Err codemadness.org 70 i 6454 Err codemadness.org 70 i 6455

However, the log create is just a facade, and by default the Err codemadness.org 70 i 6456 messages do not get emitted anywhere. The calling code has to set up Err codemadness.org 70 i 6457 a logger. Crates like env_logger let one set up a logger, during Err codemadness.org 70 i 6458 program initialization, that gets configured through an environment Err codemadness.org 70 i 6459 variable.

Err codemadness.org 70 i 6460

And this is a problem for librsvg: we are not the program's Err codemadness.org 70 i 6461 initialization! Librsvg is a library; it doesn't have a main() Err codemadness.org 70 i 6462 function. And since most of the calling code is not Rust, we can't Err codemadness.org 70 i 6463 assume that they can call code that can initialize the logging Err codemadness.org 70 i 6464 framework.

Err codemadness.org 70 i 6465

Why not use glib's logging stuff?

Err codemadness.org 70 i 6466

Currently this is a bit clunky to use from Rust, since glib's Err codemadness.org 70 i 6467 structured logging functions are not bound yet in glib-rs. Maybe it Err codemadness.org 70 i 6468 would be good to bind them and get this over with.

Err codemadness.org 70 i 6469

What user experience do we want?

Err codemadness.org 70 i 6470

In the past, what has worked well for me to do logging from libraries Err codemadness.org 70 i 6471 is to allow the user to set an environment variable to control the Err codemadness.org 70 i 6472 logging, or to drop a log configuration file in their $HOME. The Err codemadness.org 70 i 6473 former works well when the user is in control of running the program Err codemadness.org 70 i 6474 that will print the logs; the latter is useful when the user is not Err codemadness.org 70 i 6475 directly in control, like for gnome-shell, which gets launched through Err codemadness.org 70 i 6476 a lot of magic during session startup.

Err codemadness.org 70 i 6477

For librsvg, it's probably enough to just use an environment Err codemadness.org 70 i 6478 variable. Set RSVG_LOG=parse_errors, run your program, and get Err codemadness.org 70 i 6479 useful output. Push button, receive bacon.

Err codemadness.org 70 i 6480

Other options in Rust?

Err codemadness.org 70 i 6481

There is a slog crate which looks promising. Instead of using Err codemadness.org 70 i 6482 context-less macros which depend on a single global logger, it Err codemadness.org 70 i 6483 provides logging macros to which you pass a logger object.

Err codemadness.org 70 i 6484

For librsvg, this means that the basic RsvgHandle could create its Err codemadness.org 70 i 6485 own logger, based on an environment variable or whatever, and pass it Err codemadness.org 70 i 6486 around to all its child functions for when they need to log something.

Err codemadness.org 70 i 6487

Slog supports structured logging, and seems to have some fancy output Err codemadness.org 70 i 6488 modes. We'll see.

Three big things happening in librsvg

2018-05-21T19:21:08-05:00

I am incredibly happy because of three big things that are going Err codemadness.org 70 i 6495 on in librsvg right now:

Err codemadness.org 70 i 6496

Err codemadness.org 70 i 6498
Paolo Borelli finished porting all the CSS properties to Rust. Err codemadness.org 70 i 6499 What was once a gigantic RsvgState struct in C is totally gone, Err codemadness.org 70 i 6500 along with all the janky C code to parse individual properties. Err codemadness.org 70 i 6501 The process of porting RsvgState to Rust has been going on since Err codemadness.org 70 i 6502 about two months ago, and has involved many multi-commit Err codemadness.org 70 i 6503 merge requests and refactorings. This is a tremendous amount of Err codemadness.org 70 i 6504 really good work! The result is all in Rust now in a State Err codemadness.org 70 i 6505 struct, which is opaque from C's viewpoint. The only places in C Err codemadness.org 70 i 6506 that still require accessors to the State are in the filter Err codemadness.org 70 i 6507 effects code. Which brings me to...
Err codemadness.org 70 i 6508
Err codemadness.org 70 i 6510
Ivan Molodetskikh, my Summer of Code student, submitted his first Err codemadness.org 70 i 6511 merge request and it's merged to master now. This ports Err codemadness.org 70 i 6512 the bookkeeping infrastructure for SVG filters to Rust, and also Err codemadness.org 70 i 6513 the feOffset filter is ported now. Right now the code doesn't do Err codemadness.org 70 i 6514 anything fancy to iterate over the pixels of Cairo image surfaces; Err codemadness.org 70 i 6515 that will come later. I am very happy that filters, which were a Err codemadness.org 70 i 6516 huge barrier, are now starting to get chipped away into nicer code.
Err codemadness.org 70 i 6517
Err codemadness.org 70 i 6519
I have started to move librsvg's old representation of CSS Err codemadness.org 70 i 6520 properties into something that can really represent properties that Err codemadness.org 70 i 6521 are not specified, or explicitly set to inherit from an SVG Err codemadness.org 70 i 6522 element's parent, or set to a normal value. Librsvg never had a Err codemadness.org 70 i 6523 representation of property values that actually matched the SVG/CSS Err codemadness.org 70 i 6524 specs; it just knew whether a property was specified or not for an Err codemadness.org 70 i 6525 element. This worked fine for properties which the spec mandates Err codemadness.org 70 i 6526 that they should inherit automatically, but those that don't, Err codemadness.org 70 i 6527 were handled through special hacks. The new code makes this a lot Err codemadness.org 70 i 6528 cleaner. It should also make it easier to copy Servo's idioms for Err codemadness.org 70 i 6529 property inheritance.
Err codemadness.org 70 i 6530

Reducing the number of image copies in GNOME

2018-05-14T07:45:11-05:00

Our graphics stack that deals with images has evolved a lot Err codemadness.org 70 i 6537 over the years.

Err codemadness.org 70 i 6538

In ye olden days

Err codemadness.org 70 i 6539

In the context of GIMP/GNOME, the only thing that knew how to draw RGB Err codemadness.org 70 i 6540 images to X11 windows (doing palette mapping for 256-color graphics Err codemadness.org 70 i 6541 cards and dithering if necessary) was the GIMP. Later, when GTK+ was Err codemadness.org 70 i 6542 written, it exported a GtkPreview widget, which could take an RGB Err codemadness.org 70 i 6543 image buffer supplied by the application and render it to an X window Err codemadness.org 70 i 6544 — this was what GIMP plug-ins could use in their user interface to Err codemadness.org 70 i 6545 show, well, previews of what they were about to do with the user's Err codemadness.org 70 i 6546 images. Later we got some obscure magic in a GdkColorContext Err codemadness.org 70 i 6547 object, which helped allocate X11 colors for the X drawing primitives. Err codemadness.org 70 i 6548 In turn, GdkColorContext came from the port that Miguel and I did of Err codemadness.org 70 i 6549 XmHTML's color context object (and for those that remember, XmHTML Err codemadness.org 70 i 6550 became the first version of GtkHtml; later it was rewritten as a port Err codemadness.org 70 i 6551 of KDE's HTML widget). Thankfully all that stuff is gone now; we can Err codemadness.org 70 i 6552 now assume that video cards are 24-bit RGB or better everywhere, and Err codemadness.org 70 i 6553 there is no need to worry about limited color palettes and color Err codemadness.org 70 i 6554 allocation.

Err codemadness.org 70 i 6555

Later, we started using the Imlib library, from the Enlightenment Err codemadness.org 70 i 6556 project, as an easy API to load images — the APIs from libungif, Err codemadness.org 70 i 6557 libjpeg, libpng, etc. were not something one really wanted to use Err codemadness.org 70 i 6558 directly — and also to keep images in memory with a uniform Err codemadness.org 70 i 6559 representation. Unfortunately, Imlib's memory management was Err codemadness.org 70 i 6560 peculiar, as it was tied to Enlightenment's model for caching and Err codemadness.org 70 i 6561 rendering loaded/scaled images.

Err codemadness.org 70 i 6562

A bunch of people worked to write GdkPixbuf: it kept Imlib's concepts Err codemadness.org 70 i 6563 of a unified representation for image data, and an easy API to load Err codemadness.org 70 i 6564 various image formats. It added support for an alpha channel (we only Err codemadness.org 70 i 6565 had 1-bit masks before), and it put memory management in the hands of Err codemadness.org 70 i 6566 the calling application, in the form of reference counting. GdkPixbuf Err codemadness.org 70 i 6567 obtained some high-quality scaling functions, mainly for use by Eye Of Err codemadness.org 70 i 6568 Gnome (our image viewer) and by applications that just needed scaling Err codemadness.org 70 i 6569 instead of arbitrary transformations.

Err codemadness.org 70 i 6570

Later, we got libart, the first library in GNOME to do antialiased Err codemadness.org 70 i 6571 vector rendering and affine transformations. Libart was more or less Err codemadness.org 70 i 6572 compatible with GdkPixbuf: they both had the same internal Err codemadness.org 70 i 6573 representation for pixel data, but one had to pass the Err codemadness.org 70 i 6574 pixels/width/height/rowstride around by hand.

Err codemadness.org 70 i 6575

Mea culpa

Err codemadness.org 70 i 6576

Back then I didn't understand premultiplied alpha, Err codemadness.org 70 i 6577 which is now ubiquitous. The GIMP made the decision to use Err codemadness.org 70 i 6578 non-premultiplied alpha when it introduced layers with transparency, Err codemadness.org 70 i 6579 probably to "avoid losing data" from transparent pixels. GdkPixbuf Err codemadness.org 70 i 6580 follows the same scheme.

Err codemadness.org 70 i 6581

(Now that the GIMP uses GEGL for its internal representation of Err codemadness.org 70 i 6582 images... I have no idea what it does with respect to alpha.)

Err codemadness.org 70 i 6583

Cairo and afterwards

Err codemadness.org 70 i 6584

Some time after the libart days, we got Cairo and pixman. Cairo had a Err codemadness.org 70 i 6585 different representation of images than GdkPixbuf's, and it supported Err codemadness.org 70 i 6586 more pixel formats and color models.

Err codemadness.org 70 i 6587

GTK2 got patched to use Cairo in the toplevel API. We still had a Err codemadness.org 70 i 6588 dichotomy between Cairo's image surfaces, which are ARGB premultiplied Err codemadness.org 70 i 6589 data in memory, and GdkPixbufs, which are RGBA non-premultiplied. Err codemadness.org 70 i 6590 There are utilities in GTK+ to do these translations, but they are Err codemadness.org 70 i 6591 inconvenient: every time a program loads an image with GdkPixbuf's Err codemadness.org 70 i 6592 easy API, a translation has to happen from non-premul RGBA to premul Err codemadness.org 70 i 6593 ARGB.

Err codemadness.org 70 i 6594

Having two formats means that we inevitably do translations back and Err codemadness.org 70 i 6595 forth of practically the same data. For example, when one embeds a Err codemadness.org 70 i 6596 JPEG inside an SVG, librsvg will read that JPEG using GdkPixbuf, Err codemadness.org 70 i 6597 translate it to Cairo's representation, composite it with Cairo onto Err codemadness.org 70 i 6598 the final result, and finally translate the whole thing back to a Err codemadness.org 70 i 6599 GdkPixbuf... if someone uses librsvg's legacy APIs to output pixbufs Err codemadness.org 70 i 6600 instead of rendering directly to a Cairo surface.

Err codemadness.org 70 i 6601

Who uses that legacy API? GTK+, of course! GTK+ loads scalable SVG Err codemadness.org 70 i 6602 icons with GdkPixbuf's loader API, which dynamically links librsvg at Err codemadness.org 70 i 6603 runtime: in effect, GTK+ doesn't use librsvg directly. And the SVG Err codemadness.org 70 i 6604 pixbuf loader uses the "gimme a pixbuf" API in librsvg.

Err codemadness.org 70 i 6605

GPUs

Err codemadness.org 70 i 6606

Then, we got GPUs everywhere. Each GPU has its own preferred pixel Err codemadness.org 70 i 6607 format. Image data has to be copied to the GPU at some point. Err codemadness.org 70 i 6608 Cairo's ARGB needs to be translated to the GPU's preferred format and Err codemadness.org 70 i 6609 alignment.

Err codemadness.org 70 i 6610

Summary so far

Err codemadness.org 70 i 6611

Err codemadness.org 70 i 6613
Libraries that load images from standard formats have different Err codemadness.org 70 i 6614 output formats. Generally they can be coaxed into spitting ARGB or Err codemadness.org 70 i 6615 RGBA, but we don't expect them to support any random representation Err codemadness.org 70 i 6616 that a GPU may want.
Err codemadness.org 70 i 6617
Err codemadness.org 70 i 6619
GdkPixbuf uses non-premultiplied RGBA data, always in that order.
Err codemadness.org 70 i 6620
Err codemadness.org 70 i 6622
Cairo uses premultiplied ARGB in platform-endian 32-bit chunks: if Err codemadness.org 70 i 6623 each pixel is 0xaarrggbb, then the bytes are shuffled around Err codemadness.org 70 i 6624 depending on whether the platform is little-endian or big-endian.
Err codemadness.org 70 i 6625
Err codemadness.org 70 i 6627
Cairo internally uses a subset of the formats supported by pixman.
Err codemadness.org 70 i 6628
Err codemadness.org 70 i 6630
GPUs use whatever they damn well please.
Err codemadness.org 70 i 6631
Err codemadness.org 70 i 6633
Hilarity ensues.
Err codemadness.org 70 i 6634

Err codemadness.org 70 i 6636

What would we like to do?

Err codemadness.org 70 i 6637

We would like to reduce the number of translations between image Err codemadness.org 70 i 6638 formats along the loading-processing-display pipeline. Here is a Err codemadness.org 70 i 6639 plan:

Err codemadness.org 70 i 6640

Err codemadness.org 70 i 6642
Make sure Cairo/pixman support the image formats that GPUs generally Err codemadness.org 70 i 6643 prefer. Have them do the necessary conversions if the rest of the Err codemadness.org 70 i 6644 program passes an unsupported format. Ensure that a Cairo image Err codemadness.org 70 i 6645 surface can be created with the GPU's preferred format.
Err codemadness.org 70 i 6646
Err codemadness.org 70 i 6648
Make GdkPixbuf just be a wrapper around a Cairo image surface. Err codemadness.org 70 i 6649 GdkPixbuf is already an opaque structure, and it already knows how Err codemadness.org 70 i 6650 to copy pixel data in case the calling code requests it, or wants to Err codemadness.org 70 i 6651 turn a pixbuf from immutable to mutable.
Err codemadness.org 70 i 6652
Err codemadness.org 70 i 6654
Provide GdkPixbuf APIs that deal with Cairo image surfaces. For Err codemadness.org 70 i 6655 example, deprecate gdk_pixbuf_new() and Err codemadness.org 70 i 6656 gdk_pixbuf_new_from_data(), in favor of a new Err codemadness.org 70 i 6657 gdk_pixbuf_new_from_cairo_image_surface(). Instead of Err codemadness.org 70 i 6658 gdk_pixbuf_get_pixels() and related functions, have Err codemadness.org 70 i 6659 gdk_pixbuf_get_cairo_image_surface(). Mark the "give me the pixel Err codemadness.org 70 i 6660 data" functions as highly discouraged, and only for use really by Err codemadness.org 70 i 6661 applications that want to use GdkPixbuf as an image loader and Err codemadness.org 70 i 6662 little else.
Err codemadness.org 70 i 6663
Err codemadness.org 70 i 6665
Remove calls in GTK+ that cause image conversions; make them use Err codemadness.org 70 i 6666 Cairo image surfaces directly, from GdkTexture up.
Err codemadness.org 70 i 6667
Err codemadness.org 70 i 6669
Audit applications to remove calls that cause image conversions. Err codemadness.org 70 i 6670 Generally, look for where they use GdkPixbuf's deprecated APIs and Err codemadness.org 70 i 6671 update them.
Err codemadness.org 70 i 6672

Err codemadness.org 70 i 6674

Is this really a performance problem?

Err codemadness.org 70 i 6675

This is in the "excess work" category of performance Err codemadness.org 70 i 6676 issues. All those conversions are not really slow (they don't make up Err codemadness.org 70 i 6677 for the biggest part of profiles), but they are nevertheless things Err codemadness.org 70 i 6678 that we could avoid doing. We may get some speedups, but it's Err codemadness.org 70 i 6679 probably more interesting to look at things like power consumption.

Err codemadness.org 70 i 6680

Right now I'm seeing this as a cool, minor optimization, but more as Err codemadness.org 70 i 6681 a way to gradually modernize our image API.

Err codemadness.org 70 i 6682

We seem to change imaging models every N years (X11 -> libart Err codemadness.org 70 i 6683 -> Cairo -> render trees in GPUs -> ???). It is very hard to change Err codemadness.org 70 i 6684 applications to use different APIs. In the meantime, we can provide a Err codemadness.org 70 i 6685 more linear path for image data, instead of doing unnecessary Err codemadness.org 70 i 6686 conversions everywhere.

Err codemadness.org 70 i 6687

Code

Err codemadness.org 70 i 6688

I have a use-cairo-surface-internally branch in Err codemadness.org 70 i 6689 gdk-pixbuf, Err codemadness.org 70 i 6690 which I'll be working on this week. Meanwhile, you may be interested Err codemadness.org 70 i 6691 in the ongoing Performance Hackfest in Cambridge!

Madrid GNOME+Rust Hackfest, part 3 (conclusion)

2018-04-23T15:04:32-05:00

The last code I wrote during the hackfest was the start of code Err codemadness.org 70 i 6703 generation for GObject interfaces. This is so that you can do

Err codemadness.org 70 i 6704

gobject_gen! {	Err	codemadness.org	70
i  6705     interface Foo {	Err	codemadness.org	70
i  6706         virtual fn frob(&self);	Err	codemadness.org	70
i  6707     }	Err	codemadness.org	70
i  6708 }	Err	codemadness.org	70
i  6709

Err codemadness.org 70 i 6710 Err codemadness.org 70 i 6711

and it will generate the appropriate FooIface like one would expect Err codemadness.org 70 i 6712 with the C versions of interfaces.

Err codemadness.org 70 i 6713

It turns out that this can share a lot of code from the existing code Err codemadness.org 70 i 6714 generator for classes: both classes and interfaces are "just virtual Err codemadness.org 70 i 6715 method tables", plus signals and properties, and classes can actually Err codemadness.org 70 i 6716 have per-instance fields and such. I started refactoring the code Err codemadness.org 70 i 6717 generator to allow this.

Err codemadness.org 70 i 6718

I also took a second look at how to present good error messages when Err codemadness.org 70 i 6719 the syn crate encounters a parse error. I need to sit down at home Err codemadness.org 70 i 6720 and experiment with this carefully.

Err codemadness.org 70 i 6721

Back home

Err codemadness.org 70 i 6722

I'm back home now, jetlagged but very happy that gnome-class is in a much Err codemadness.org 70 i 6723 more advanced a state than it was before the hackfest. I'm very Err codemadness.org 70 i 6724 thankful that practically everyone worked on it!

Err codemadness.org 70 i 6725

Also, thanks to Alberto and Natalia for hosting me at their Err codemadness.org 70 i 6726 apartment and showing me around Madrid, all while wrangling their Err codemadness.org 70 i 6727 adorable baby Mario. We had a lovely time on Saturday, and ate Err codemadness.org 70 i 6728 excellent food downtown.

Err codemadness.org 70 i 6729

Err codemadness.org 70 i 6730

Madrid GNOME+Rust Hackfest, part 2

2018-04-20T08:59:23+02:00

Hacking on gnome-class continues apace!

Err codemadness.org 70 i 6738

Philippe updated our dependencies.

Err codemadness.org 70 i 6739

Alberto made the syntax for per-instance private structs Err codemadness.org 70 i 6740 more ergonomic, and then made that code nice and compact.

Err codemadness.org 70 i 6741

Martin improved our conversion from CamelCase to Err codemadness.org 70 i 6742 snake_case for code generation.

Err codemadness.org 70 i 6743

Daniel added initial support for GObject properties. Err codemadness.org 70 i 6744 This is not finished yet, but the initial parser and code generation Err codemadness.org 70 i 6745 is done.

Err codemadness.org 70 i 6746

Guillaume turned gir, the binding generator in Err codemadness.org 70 i 6747 gtk-rs, from a binary into a library crate. This will let Err codemadness.org 70 i 6748 us have all the GObject Introspection information for parent classes Err codemadness.org 70 i 6749 at compilation time.

Err codemadness.org 70 i 6750

Antoni has been working on a tricky problem. GTK+ structs that have Err codemadness.org 70 i 6751 bitfields do not get reconstructed correctly from the Err codemadness.org 70 i 6752 GObject Introspection information — Rust does not handle C Err codemadness.org 70 i 6753 bitfields yet. This has two implications. Err codemadness.org 70 i 6754 First, we lose some of the original struct fields in the generated Err codemadness.org 70 i 6755 bindings. Second, the sizes of the generated structs are not the Err codemadness.org 70 i 6756 same as the original C structs, so g_type_register_static() Err codemadness.org 70 i 6757 complains that one is trying to register an invalid class.

Err codemadness.org 70 i 6758

Yesterday we got as far as reading the amd64 and [ARM][arm] Err codemadness.org 70 i 6759 ABI manuals to see what the hell C compilers are supposed to do for Err codemadness.org 70 i 6760 laying out structs with bitfields. Most likely, we will have a Err codemadness.org 70 i 6761 temporary fix in gir's code generator so that it generates Err codemadness.org 70 i 6762 structs with the same layout as the C ones, with padding in place of Err codemadness.org 70 i 6763 the space for bitfields. Later we can remove this when rustc gets Err codemadness.org 70 i 6764 support for C bitfields.

Err codemadness.org 70 i 6765

I've been working on support for GObject interfaces. The basic Err codemadness.org 70 i 6766 parsing is done; I'm about to refactor the code generation so I can Err codemadness.org 70 i 6767 reuse the parts that fill vtables from classes.

Err codemadness.org 70 i 6768

Yesterday we went to the Madrid Rust Meetup, a regular meeting of Err codemadness.org 70 i 6769 rustaceans here. Martin talked about WebRender; I talked about Err codemadness.org 70 i 6770 refactoring C to port it to Rust, and then Alex talked about Rust's Err codemadness.org 70 i 6771 plans for 2018. Fun times.

Err codemadness.org 70 i 6772

Err codemadness.org 70 i 6773

Err codemadness.org 70 i 6774

[arm]:

Madrid GNOME+Rust Hackfest, part 1

2018-04-18T02:55:12-05:00

I'm in Madrid since Monday, at the third GNOME+Rust hackfest! Err codemadness.org 70 i 6780 The OpenShine folks are kindly letting us use their offices, on the Err codemadness.org 70 i 6781 seventh floor of a building by the Cuatro Caminos Err codemadness.org 70 i 6782 roundabout.

Err codemadness.org 70 i 6783

I am very, very thankful that this time everyone seems to be working Err codemadness.org 70 i 6784 on developing gnome-class. It's a difficult project for me, and Err codemadness.org 70 i 6785 more brainpower is definitely welcome — all the indirection, type Err codemadness.org 70 i 6786 conversion, GObject obscurity, and procedural macro shenanigans Err codemadness.org 70 i 6787 definitely take a toll on oneself.

Err codemadness.org 70 i 6788

Gnome-class internals

Err codemadness.org 70 i 6789

Err codemadness.org 70 i 6790

I explained how gnome-class works to the rest of the hackfest Err codemadness.org 70 i 6791 attendees. I've been writing a document on gnome-class's Err codemadness.org 70 i 6792 internals, so the whiteboard was a whirlwind tour through Err codemadness.org 70 i 6793 it.

Err codemadness.org 70 i 6794

Error messages from the compiler

Err codemadness.org 70 i 6795

Antoni Boucher, the author of relm (a Rust crate to write GTK+ Err codemadness.org 70 i 6796 asynchronous widgets with an Elm-like model), explained to me how relm Err codemadness.org 70 i 6797 manages to present good error messages from the Rust compiler, when Err codemadness.org 70 i 6798 the user's code has mistakes. Right now this is in a very bad state Err codemadness.org 70 i 6799 in gnome-class: user errors within the invocation of the procedural Err codemadness.org 70 i 6800 macro get shown by the compiler as errors at the macro call, so you Err codemadness.org 70 i 6801 don't get line number information that is meaningful.

Err codemadness.org 70 i 6802

For a large part of the day we tried to refactor bits of gnome-class Err codemadness.org 70 i 6803 to do something similar. It is very slightly better now, but this Err codemadness.org 70 i 6804 really requires me to sit down calmly, at home, and to fully Err codemadness.org 70 i 6805 understand how relm does it and what changes are needed in the syn Err codemadness.org 70 i 6806 parser crate to make it easy to present good errors.

Err codemadness.org 70 i 6807

I think I'll continue this work at home, as there is a lot of source Err codemadness.org 70 i 6808 code to understand: the combinator parsers in syn, the error Err codemadness.org 70 i 6809 handling scheme in relm, and the peculiarities of gnome-class.

Err codemadness.org 70 i 6810

Further work during the hackfest

Err codemadness.org 70 i 6811

Other people working on gnome-class are adding support for GObject Err codemadness.org 70 i 6812 properties, inheritance from non-Rust classes, and improving the Err codemadness.org 70 i 6813 ergonomics of class-private structures.

Err codemadness.org 70 i 6814

I think I'll stop working on error messages for now, and focus instead Err codemadness.org 70 i 6815 on either supporting GTypeInterfaces, or completing support for type Err codemadness.org 70 i 6816 conversions for methods and signals.

Err codemadness.org 70 i 6817

Other happenings in Rust

Err codemadness.org 70 i 6818

Paolo Borelli has been porting RsvgState to Rust in librsvg. Err codemadness.org 70 i 6819 This is the big structure that holds all the CSS state for SVG Err codemadness.org 70 i 6820 elements. This is very meticulous work, and I'm thankful that Paolo Err codemadness.org 70 i 6821 is paying good attention to it. Soon we will have all the style Err codemadness.org 70 i 6822 machinery for librsvg in Rust, which will make it easier to use the Err codemadness.org 70 i 6823 selectors crate from Servo instead of libcroco, as the Err codemadness.org 70 i 6824 latter is unmaintained.

Err codemadness.org 70 i 6825

Food

Err codemadness.org 70 i 6826

Err codemadness.org 70 i 6827

Ah, Spanish food. We have been enjoying cheese, jamón, tortilla, Err codemadness.org 70 i 6828 pimientos, oxtail stews, natillas, café con leche...

Err codemadness.org 70 i 6829

Thanks

Err codemadness.org 70 i 6830

Thanks to OpenShine for hosting the hackfest, and to the GNOME Err codemadness.org 70 i 6831 Foundation for sponsoring my travel. And thanks for Alberto Ruiz for Err codemadness.org 70 i 6832 putting me up in his house!

Err codemadness.org 70 i 6833

Refactoring some repetitive code to a Rust macro

2018-03-23T11:01:30-06:00

I have started porting the code in librsvg that parses SVG's CSS Err codemadness.org 70 i 6844 properties from C to Rust. Many properties have symbolic values:

Err codemadness.org 70 i 6845

stroke-linejoin: miter | round | bevel | inherit	Err	codemadness.org	70
i  6846 	Err	codemadness.org	70
i  6847 stroke-linecap: butt | round | square | inherit	Err	codemadness.org	70
i  6848 	Err	codemadness.org	70
i  6849 fill-rule: nonzero | evenodd | inherit	Err	codemadness.org	70
i  6850

Err codemadness.org 70 i 6851 Err codemadness.org 70 i 6852

StrokeLinejoin is the first property that I ported. First I had to Err codemadness.org 70 i 6853 write a little bunch of machinery to allow CSS properties to be kept Err codemadness.org 70 i 6854 in Rust-space instead of the main C structure that holds them Err codemadness.org 70 i 6855 (upcoming blog post about that). But for now, I just want to show how Err codemadness.org 70 i 6856 this boiled down to a macro after refactoring.

Err codemadness.org 70 i 6857

First cut at the code

Err codemadness.org 70 i 6858

The stroke-linejoin property can have the values miter, round, Err codemadness.org 70 i 6859 bevel, or inherit. Here is an enum definition for those values, Err codemadness.org 70 i 6860 and the conventional machinery which librsvg uses to parse property values:

Err codemadness.org 70 i 6861

#[derive(Debug, Copy, Clone)]	Err	codemadness.org	70
i  6862 pub enum StrokeLinejoin {	Err	codemadness.org	70
i  6863     Miter,	Err	codemadness.org	70
i  6864     Round,	Err	codemadness.org	70
i  6865     Bevel,	Err	codemadness.org	70
i  6866     Inherit,	Err	codemadness.org	70
i  6867 }	Err	codemadness.org	70
i  6868 	Err	codemadness.org	70
i  6869 impl Parse for StrokeLinejoin {	Err	codemadness.org	70
i  6870     type Data = ();	Err	codemadness.org	70
i  6871     type Err = AttributeError;	Err	codemadness.org	70
i  6872 	Err	codemadness.org	70
i  6873     fn parse(s: &str, _: Self::Data) -> Result<StrokeLinejoin, AttributeError> {	Err	codemadness.org	70
i  6874         match s.trim() {	Err	codemadness.org	70
i  6875             "miter" => Ok(StrokeLinejoin::Miter),	Err	codemadness.org	70
i  6876             "round" => Ok(StrokeLinejoin::Round),	Err	codemadness.org	70
i  6877             "bevel" => Ok(StrokeLinejoin::Bevel),	Err	codemadness.org	70
i  6878             "inherit" => Ok(StrokeLinejoin::Inherit),	Err	codemadness.org	70
i  6879             _ => Err(AttributeError::from(ParseError::new("invalid value"))),	Err	codemadness.org	70
i  6880         }	Err	codemadness.org	70
i  6881     }	Err	codemadness.org	70
i  6882 }	Err	codemadness.org	70
i  6883

Err codemadness.org 70 i 6884 Err codemadness.org 70 i 6885

We match the allowed string values and map them to enum values. No Err codemadness.org 70 i 6886 big deal, right?

Err codemadness.org 70 i 6887

Properties also have a default value. For example, the SVG spec says Err codemadness.org 70 i 6888 that if a shape doesn't have a stroke-linejoin property specified, Err codemadness.org 70 i 6889 it will use miter by default. Let's implement that:

Err codemadness.org 70 i 6890

impl Default for StrokeLinejoin {	Err	codemadness.org	70
i  6891     fn default() -> StrokeLinejoin {	Err	codemadness.org	70
i  6892         StrokeLinejoin::Miter	Err	codemadness.org	70
i  6893     }	Err	codemadness.org	70
i  6894 }	Err	codemadness.org	70
i  6895

Err codemadness.org 70 i 6896 Err codemadness.org 70 i 6897

So far, we have three things:

Err codemadness.org 70 i 6898

An enum definition for the property's possible values.
impl Parse so we can parse the property from a string.
impl Default so the property knows its default value.

Err codemadness.org 70 i 6903

Where things got repetitive

Err codemadness.org 70 i 6904

The next property I ported was stroke-linecap, which can take the Err codemadness.org 70 i 6905 following values:

Err codemadness.org 70 i 6906

#[derive(Debug, Copy, Clone)]	Err	codemadness.org	70
i  6907 pub enum StrokeLinecap {	Err	codemadness.org	70
i  6908     Butt,	Err	codemadness.org	70
i  6909     Round,	Err	codemadness.org	70
i  6910     Square,	Err	codemadness.org	70
i  6911     Inherit,	Err	codemadness.org	70
i  6912 }	Err	codemadness.org	70
i  6913

Err codemadness.org 70 i 6914 Err codemadness.org 70 i 6915

This is similar in shape to the StrokeLinejoin enum above; Err codemadness.org 70 i 6916 it's just different names.

Err codemadness.org 70 i 6917

The parsing has exactly the same shape, and just different values:

Err codemadness.org 70 i 6918

impl Parse for StrokeLinecap {	Err	codemadness.org	70
i  6919     type Data = ();	Err	codemadness.org	70
i  6920     type Err = AttributeError;	Err	codemadness.org	70
i  6921 	Err	codemadness.org	70
i  6922     fn parse(s: &str, _: Self::Data) -> Result<StrokeLinecap, AttributeError> {	Err	codemadness.org	70
i  6923         match s.trim() {	Err	codemadness.org	70
i  6924             "butt" => Ok(StrokeLinecap::Butt),	Err	codemadness.org	70
i  6925             "round" => Ok(StrokeLinecap::Round),	Err	codemadness.org	70
i  6926             "square" => Ok(StrokeLinecap::Square),	Err	codemadness.org	70
i  6927             "inherit" => Ok(StrokeLinecap::Inherit),	Err	codemadness.org	70
i  6928 	Err	codemadness.org	70
i  6929             _ => Err(AttributeError::from(ParseError::new("invalid value"))),	Err	codemadness.org	70
i  6930         }	Err	codemadness.org	70
i  6931     }	Err	codemadness.org	70
i  6932 }	Err	codemadness.org	70
i  6933

Err codemadness.org 70 i 6934 Err codemadness.org 70 i 6935

Same thing with the default:

Err codemadness.org 70 i 6936

impl Default for StrokeLinecap {	Err	codemadness.org	70
i  6937     fn default() -> StrokeLinecap {	Err	codemadness.org	70
i  6938         StrokeLinecap::Butt	Err	codemadness.org	70
i  6939     }	Err	codemadness.org	70
i  6940 }	Err	codemadness.org	70
i  6941

Err codemadness.org 70 i 6942 Err codemadness.org 70 i 6943

Yes, the SVG spec has

Err codemadness.org 70 i 6944

default: butt	Err	codemadness.org	70
i  6945

Err codemadness.org 70 i 6946 Err codemadness.org 70 i 6947

somewhere in it, much to the delight of the 12-year old in me.

Err codemadness.org 70 i 6948

Refactoring to a macro

Err codemadness.org 70 i 6949

Here I wanted to define a make_ident_property!() macro that would Err codemadness.org 70 i 6950 get invoked like this:

Err codemadness.org 70 i 6951

make_ident_property!(	Err	codemadness.org	70
i  6952     StrokeLinejoin,	Err	codemadness.org	70
i  6953     default: Miter,	Err	codemadness.org	70
i  6954 	Err	codemadness.org	70
i  6955     "miter" => Miter,	Err	codemadness.org	70
i  6956     "round" => Round,	Err	codemadness.org	70
i  6957     "bevel" => Bevel,	Err	codemadness.org	70
i  6958     "inherit" => Inherit,	Err	codemadness.org	70
i  6959 );	Err	codemadness.org	70
i  6960

Err codemadness.org 70 i 6961 Err codemadness.org 70 i 6962

It's called make_ident_property because it makes a property Err codemadness.org 70 i 6963 definition from simple string identifiers. It has the name of the Err codemadness.org 70 i 6964 property (StrokeLinejoin), a default value, and a few repeating Err codemadness.org 70 i 6965 elements, one for each possible value.

Err codemadness.org 70 i 6966

In Rust-speak, the macro's basic pattern is like this:

Err codemadness.org 70 i 6967

macro_rules! make_ident_property {	Err	codemadness.org	70
i  6968     ($name: ident,	Err	codemadness.org	70
i  6969      default: $default: ident,	Err	codemadness.org	70
i  6970      $($str_prop: expr => $variant: ident,)+	Err	codemadness.org	70
i  6971     ) => {	Err	codemadness.org	70
i  6972         ... macro body will go here ...	Err	codemadness.org	70
i  6973     };	Err	codemadness.org	70
i  6974 }	Err	codemadness.org	70
i  6975

Err codemadness.org 70 i 6976 Err codemadness.org 70 i 6977

Let's dissect that pattern:

Err codemadness.org 70 i 6978

macro_rules! make_ident_property {	Err	codemadness.org	70
i  6979     ($name: ident,	Err	codemadness.org	70
i  6980 //   ^^^^^^^^^^^^ will match an identifier and put it in $name	Err	codemadness.org	70
i  6981 	Err	codemadness.org	70
i  6982      default: $default: ident,	Err	codemadness.org	70
i  6983 //            ^^^^^^^^^^^^^^^ will match an identifier and put it in $default	Err	codemadness.org	70
i  6984 //   ^^^^^^^^ arbitrary text	Err	codemadness.org	70
i  6985 	Err	codemadness.org	70
i  6986      $($str_prop: expr => $variant: ident,)+	Err	codemadness.org	70
i  6987                        ^^ arbitrary text	Err	codemadness.org	70
i  6988 //   ^^ start of repetition               ^^ end of repetition, repeats one or more times	Err	codemadness.org	70
i  6989 	Err	codemadness.org	70
i  6990     ) => {	Err	codemadness.org	70
i  6991         ...	Err	codemadness.org	70
i  6992     };	Err	codemadness.org	70
i  6993 }	Err	codemadness.org	70
i  6994

Err codemadness.org 70 i 6995 Err codemadness.org 70 i 6996

For example, saying "$foo: ident" in a macro's pattern means that the Err codemadness.org 70 i 6997 compiler will expect an identifier, and bind it to $foo within the Err codemadness.org 70 i 6998 macro's definition.

Err codemadness.org 70 i 6999

Similarly, an expr means that the compiler will Err codemadness.org 70 i 7000 look for an expression — in this case, we want one of the string Err codemadness.org 70 i 7001 values.

Err codemadness.org 70 i 7002

In a macro pattern, anything that is not a binding is just arbitrary Err codemadness.org 70 i 7003 text which must appear in the macro's invocation. This is how we can Err codemadness.org 70 i 7004 create a little syntax of our own within the macro: the "default:" Err codemadness.org 70 i 7005 part, and the "=>" inside each string/symbol pair.

Err codemadness.org 70 i 7006

Finally, macro patterns allow repetition. Anything within $(...) Err codemadness.org 70 i 7007 indicates repetition. Here, $(...)+ indicates that the Err codemadness.org 70 i 7008 compiler must match one or more of the repeating elements.

Err codemadness.org 70 i 7009

I pasted the duplicated code, and substituted the actual symbol names Err codemadness.org 70 i 7010 for the macro's bindings:

Err codemadness.org 70 i 7011

macro_rules! make_ident_property {	Err	codemadness.org	70
i  7012     ($name: ident,	Err	codemadness.org	70
i  7013      default: $default: ident,	Err	codemadness.org	70
i  7014      $($str_prop: expr => $variant: ident,)+	Err	codemadness.org	70
i  7015     ) => {	Err	codemadness.org	70
i  7016         #[derive(Debug, Copy, Clone)]	Err	codemadness.org	70
i  7017         pub enum $name {	Err	codemadness.org	70
i  7018             $($variant),+	Err	codemadness.org	70
i  7019 //          ^^^^^^^^^^^^^ this is how we invoke a repeated element	Err	codemadness.org	70
i  7020 	Err	codemadness.org	70
i  7021         }	Err	codemadness.org	70
i  7022 	Err	codemadness.org	70
i  7023         impl Default for $name {	Err	codemadness.org	70
i  7024             fn default() -> $name {	Err	codemadness.org	70
i  7025                 $name::$default	Err	codemadness.org	70
i  7026 //              ^^^^^^^^^^^^^^^ construct an enum::variant	Err	codemadness.org	70
i  7027 	Err	codemadness.org	70
i  7028             }	Err	codemadness.org	70
i  7029         }	Err	codemadness.org	70
i  7030 	Err	codemadness.org	70
i  7031         impl Parse for $name {	Err	codemadness.org	70
i  7032             type Data = ();	Err	codemadness.org	70
i  7033             type Err = AttributeError;	Err	codemadness.org	70
i  7034 	Err	codemadness.org	70
i  7035             fn parse(s: &str, _: Self::Data) -> Result<$name, AttributeError> {	Err	codemadness.org	70
i  7036                 match s.trim() {	Err	codemadness.org	70
i  7037                     $($str_prop => Ok($name::$variant),)+	Err	codemadness.org	70
i  7038 //                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expand repeated elements	Err	codemadness.org	70
i  7039 	Err	codemadness.org	70
i  7040                     _ => Err(AttributeError::from(ParseError::new("invalid value"))),	Err	codemadness.org	70
i  7041                 }	Err	codemadness.org	70
i  7042             }	Err	codemadness.org	70
i  7043         }	Err	codemadness.org	70
i  7044     };	Err	codemadness.org	70
i  7045 }	Err	codemadness.org	70
i  7046

Err codemadness.org 70 i 7047 Err codemadness.org 70 i 7048

Getting rid of duplicated code

Err codemadness.org 70 i 7049

Now we have a macro that we can call to define new properties. Err codemadness.org 70 i 7050 Librsvg now has this, which is much more readable than all the code Err codemadness.org 70 i 7051 written by hand:

Err codemadness.org 70 i 7052

make_ident_property!(	Err	codemadness.org	70
i  7053     StrokeLinejoin,	Err	codemadness.org	70
i  7054     default: Miter,	Err	codemadness.org	70
i  7055 	Err	codemadness.org	70
i  7056     "miter" => Miter,	Err	codemadness.org	70
i  7057     "round" => Round,	Err	codemadness.org	70
i  7058     "bevel" => Bevel,	Err	codemadness.org	70
i  7059     "inherit" => Inherit,	Err	codemadness.org	70
i  7060 );	Err	codemadness.org	70
i  7061 	Err	codemadness.org	70
i  7062 make_ident_property!(	Err	codemadness.org	70
i  7063     StrokeLinecap,	Err	codemadness.org	70
i  7064     default: Butt,   // :)	Err	codemadness.org	70
i  7065 	Err	codemadness.org	70
i  7066     "butt" => Butt,	Err	codemadness.org	70
i  7067     "round" => Round,	Err	codemadness.org	70
i  7068     "square" => Square,	Err	codemadness.org	70
i  7069     "inherit" => Inherit,	Err	codemadness.org	70
i  7070 );	Err	codemadness.org	70
i  7071 	Err	codemadness.org	70
i  7072 make_ident_property!(	Err	codemadness.org	70
i  7073     FillRule,	Err	codemadness.org	70
i  7074     default: NonZero,	Err	codemadness.org	70
i  7075 	Err	codemadness.org	70
i  7076     "nonzero" => NonZero,	Err	codemadness.org	70
i  7077     "evenodd" => EvenOdd,	Err	codemadness.org	70
i  7078     "inherit" => Inherit,	Err	codemadness.org	70
i  7079 );	Err	codemadness.org	70
i  7080

Err codemadness.org 70 i 7081 Err codemadness.org 70 i 7082

Etcetera. It's now easy to port similar symbol-based properties from Err codemadness.org 70 i 7083 C to Rust.

Err codemadness.org 70 i 7084

Eventually I'll need to refactor all the crap that deals with Err codemadness.org 70 i 7085 inheritable properties, but that's for another time.

Err codemadness.org 70 i 7086

Conclusion and references

Err codemadness.org 70 i 7087

Rust macros are very powerful to refactor repetitive code like this.

Err codemadness.org 70 i 7088

The Rust book Err codemadness.org 70 i 7089 has an introductory appendix to macros, and The Little Book of Rust Err codemadness.org 70 i 7090 Macros is a Err codemadness.org 70 i 7091 fantastic resource that really dives into what you can do.

Making sure the repository doesn't break, automatically

2018-03-20T19:37:02-06:00

Gitlab has a fairly conventional Continuous Integration system: you Err codemadness.org 70 i 7097 push some commits, the CI pipelines build the code and presumably run Err codemadness.org 70 i 7098 the test suite, and later you can know if this succeeded of failed.

Err codemadness.org 70 i 7099

But by the time something fails, the broken code is already in the Err codemadness.org 70 i 7100 public repository.

Err codemadness.org 70 i 7101

The Rust community uses Bors, a bot that prevents this from happening:

Err codemadness.org 70 i 7102

Err codemadness.org 70 i 7104
You push some commits and submit a merge request.
Err codemadness.org 70 i 7105
Err codemadness.org 70 i 7107
A human looks at your merge request; they may tell you to make Err codemadness.org 70 i 7108 changes, or they may tell Bors that your request is approved for Err codemadness.org 70 i 7109 merging.
Err codemadness.org 70 i 7110
Err codemadness.org 70 i 7112
Bors looks for approved merge requests. It merges each into a Err codemadness.org 70 i 7113 temporary branch and waits for the CI pipeline to run there. If Err codemadness.org 70 i 7114 CI passes, Bors automatically merges to master. If CI fails, Bors Err codemadness.org 70 i 7115 annotates the merge request with the failure, and the main Err codemadness.org 70 i 7116 repository stays working.
Err codemadness.org 70 i 7117

Err codemadness.org 70 i 7119

Bors also tells you if the mainline has moved forward and there's a Err codemadness.org 70 i 7120 merge conflict. In that case you need to do a rebase yourself; the Err codemadness.org 70 i 7121 repository stays working in the meantime.

Err codemadness.org 70 i 7122

This leads to a very fair, very transparent process for contributors Err codemadness.org 70 i 7123 and for maintainers. For all the details, watch Emily Dunham's Err codemadness.org 70 i 7124 presentation on Rust's community Err codemadness.org 70 i 7125 automation Err codemadness.org 70 i 7126 (transcript).

Err codemadness.org 70 i 7127

For a description of where Bors came from, read Graydon Hoare's Err codemadness.org 70 i 7128 blog.

Err codemadness.org 70 i 7129

Bors evolved into Err codemadness.org 70 i 7130 Homu and it is what Rust and Servo Err codemadness.org 70 i 7131 use currently. However, Homu depends on Github.

Err codemadness.org 70 i 7132

I just found out that there is a port of Homu for Err codemadness.org 70 i 7133 Gitlab. Would anyone care Err codemadness.org 70 i 7134 to set it up?

Err codemadness.org 70 i 7135

Update: Two Err codemadness.org 70 i 7136 people have Err codemadness.org 70 i 7137 suggested porting Bors-ng to Gitlab instead, Err codemadness.org 70 i 7138 for scalability Err codemadness.org 70 i 7139 reasons.

Librsvg and Gnome-class accepting interns

2018-03-12T19:00:08-06:00

I would like to mentor people for librsvg and gnome-class this Err codemadness.org 70 i 7146 Summer, both for Outreachy and Summer of Code.

Err codemadness.org 70 i 7147

Librsvg projects

Err codemadness.org 70 i 7148

Project: port filter effects from C to Rust

Err codemadness.org 70 i 7149

Currently librsvg implements SVG filter effects in C. These are basic Err codemadness.org 70 i 7150 image processing filters like Gaussian blur, matrix convolution, Err codemadness.org 70 i 7151 Porter-Duff alpha compositing, etc.

Err codemadness.org 70 i 7152

There are some things that need to be done:

Err codemadness.org 70 i 7153

Err codemadness.org 70 i 7155
Split the single rsvg-filter.c into multiple source files, so it's Err codemadness.org 70 i 7156 easier to port each one individually.
Err codemadness.org 70 i 7157
Err codemadness.org 70 i 7159
Figure out the common infrasctructure: RsvgFilter, Err codemadness.org 70 i 7160 RsvgFilterPrimitive. All the filter use these to store Err codemadness.org 70 i 7161 intermediate results when processing SVG elements.
Err codemadness.org 70 i 7162
Err codemadness.org 70 i 7164
Experiment with the correct Rust abstractions to process images Err codemadness.org 70 i 7165 pixel-by-pixel. We would like to omit per-pixel bounds checks on Err codemadness.org 70 i 7166 array accesses. The image crate has some nice iterator Err codemadness.org 70 i 7167 traits for pixels. WebKit's implementation of SVG filters also has Err codemadness.org 70 i 7168 interesting abstractions for things like the need for a sliding Err codemadness.org 70 i 7169 window with edge handling for Gaussian blurs.
Err codemadness.org 70 i 7170
Err codemadness.org 70 i 7172
Ensure that our current filters code is actually working. Not all Err codemadness.org 70 i 7173 of the official SVG test suite's tests are in place right now for Err codemadness.org 70 i 7174 the filter effects; it is likely that some of our implementation is Err codemadness.org 70 i 7175 broken.
Err codemadness.org 70 i 7176

Err codemadness.org 70 i 7178

For this project, it will be especially helpful to have a little Err codemadness.org 70 i 7179 background in image processing. You don't need to be an expert; just Err codemadness.org 70 i 7180 to have done some pixel crunching at some point. You need to be able Err codemadness.org 70 i 7181 to read C and write Rust.

Err codemadness.org 70 i 7182

Project: CSS styling with rust-selectors

Err codemadness.org 70 i 7183

Librsvg uses an very simplistic algorithm for CSS cascading. It uses Err codemadness.org 70 i 7184 libcroco to parse CSS style data; libcroco is unmaintained and rather Err codemadness.org 70 i 7185 prone to exploits. I want to use Servo's selectors crate to do the Err codemadness.org 70 i 7186 cascading; we already use the rust-cssparser crate as a tokenizer for Err codemadness.org 70 i 7187 basic CSS properties.

Err codemadness.org 70 i 7188

Err codemadness.org 70 i 7190
For each node in its DOM tree, librsvg's Node structure keeps a Err codemadness.org 70 i 7191 Vec<> of children. We need to move this to store the next Err codemadness.org 70 i 7192 sibling and the first/last children instead. This is the Err codemadness.org 70 i 7193 data structure that rust-selectors prefers. The Kuchiki crate has Err codemadness.org 70 i 7194 an example implementation; borrowing some patterns from there could Err codemadness.org 70 i 7195 also help us simplify our reference counting for nodes.
Err codemadness.org 70 i 7196
Err codemadness.org 70 i 7198
Our styling machinery needs porting to Rust. We have a Err codemadness.org 70 i 7199 big RsvgState struct which holds the CSS state for each node. It Err codemadness.org 70 i 7200 is easy to port this to Rust; it's more interesting to gradually Err codemadness.org 70 i 7201 move it to a scheme like Servo's, with a distinction between Err codemadness.org 70 i 7202 specified/computed/used values for each CSS property.
Err codemadness.org 70 i 7203

Err codemadness.org 70 i 7205

For this project, it will be helpful to know a bit of how CSS works. Err codemadness.org 70 i 7206 Definitely be comfortable with Rust concepts like ownership and Err codemadness.org 70 i 7207 borrowing. You don't need to be an expert, but if you are going Err codemadness.org 70 i 7208 through the "fighting the borrow checker" stage, you'll have a harder Err codemadness.org 70 i 7209 time with this. Or it may be what lets you grow out of it! You need Err codemadness.org 70 i 7210 to be able to read C and write Rust.

Err codemadness.org 70 i 7211

Bugs for newcomers: We have a number of easy bugs for newcomers Err codemadness.org 70 i 7212 to librsvg. Some of these are in the Rust part, some Err codemadness.org 70 i 7213 in the C part, some in both — take your pick!

Err codemadness.org 70 i 7214

Projects for gnome-class

Err codemadness.org 70 i 7215

Gnome-class is the code generator that lets you write Err codemadness.org 70 i 7216 GObject implementations in Rust. Or at least that's the intention Err codemadness.org 70 i 7217 — the project is in early development. The code is so new that Err codemadness.org 70 i 7218 practically all of our bugs are of an exploratory Err codemadness.org 70 i 7219 nature.

Err codemadness.org 70 i 7220

Gnome-class works like a little compiler. This is from one of the Err codemadness.org 70 i 7221 examples; note the call to gobject_gen! in there:

Err codemadness.org 70 i 7222

struct SignalerPrivate {	Err	codemadness.org	70
i  7223     val: Cell<u32>	Err	codemadness.org	70
i  7224 }	Err	codemadness.org	70
i  7225 	Err	codemadness.org	70
i  7226 impl Default for SignalerPrivate {	Err	codemadness.org	70
i  7227     fn default() -> Self {	Err	codemadness.org	70
i  7228         SignalerPrivate {	Err	codemadness.org	70
i  7229             val: Cell::new(0)	Err	codemadness.org	70
i  7230         }	Err	codemadness.org	70
i  7231     }	Err	codemadness.org	70
i  7232 }	Err	codemadness.org	70
i  7233 	Err	codemadness.org	70
i  7234 gobject_gen! {	Err	codemadness.org	70
i  7235     class Signaler {	Err	codemadness.org	70
i  7236         type InstancePrivate = SignalerPrivate;	Err	codemadness.org	70
i  7237     }	Err	codemadness.org	70
i  7238 	Err	codemadness.org	70
i  7239     impl Signaler {	Err	codemadness.org	70
i  7240         signal fn value_changed(&self);	Err	codemadness.org	70
i  7241 	Err	codemadness.org	70
i  7242         fn set_value(&self, v: u32) {	Err	codemadness.org	70
i  7243             let private = self.get_priv();	Err	codemadness.org	70
i  7244             private.val.set(v);	Err	codemadness.org	70
i  7245             self.emit_value_changed();	Err	codemadness.org	70
i  7246         }	Err	codemadness.org	70
i  7247     }	Err	codemadness.org	70
i  7248 }	Err	codemadness.org	70
i  7249

Err codemadness.org 70 i 7250 Err codemadness.org 70 i 7251

Gnome-class implements this gobject_gen! macro as follows:

Err codemadness.org 70 i 7252

Err codemadness.org 70 i 7254
First we parse the code inside the macro using the syn crate. Err codemadness.org 70 i 7255 This is a crate that lets you parse Rust source code from the Err codemadness.org 70 i 7256 TokenStream that the compiler hands to implementations of procedural Err codemadness.org 70 i 7257 macros. You give a TokenStream to syn, and it gives you back Err codemadness.org 70 i 7258 structs that represent function definitions, impl blocks, Err codemadness.org 70 i 7259 expressions, etc. From this parsing stage we build an Abstract Syntax Err codemadness.org 70 i 7260 Tree (AST) that closely matches the structure of the code that the Err codemadness.org 70 i 7261 user wrote.
Err codemadness.org 70 i 7262
Err codemadness.org 70 i 7264
Second, we take the AST and convert it to higher-level concepts, Err codemadness.org 70 i 7265 while verifying that the code is semantically valid. For example, we Err codemadness.org 70 i 7266 build up a Class structure for each defined GObject class, and Err codemadness.org 70 i 7267 annotate it with the methods and signals that the user defined for it. Err codemadness.org 70 i 7268 This stage is the High-level Internal Representation (HIR).
Err codemadness.org 70 i 7269
Err codemadness.org 70 i 7271
Third, we generate Rust code from the validated HIR. For each Err codemadness.org 70 i 7272 class, we write out the boilerplate needed to register it against the Err codemadness.org 70 i 7273 GObject type system. For each virtual method we write a trampoline to Err codemadness.org 70 i 7274 let the C code call into the Rust implementation, and then write out Err codemadness.org 70 i 7275 the actual Rust impl that the user wrote. For each signal, we Err codemadness.org 70 i 7276 register it against the GObjectClass, and write the appropriate Err codemadness.org 70 i 7277 trampolines both to invoke the signal's default handler and any Rust Err codemadness.org 70 i 7278 callbacks for signal handlers.
Err codemadness.org 70 i 7279

Err codemadness.org 70 i 7281

For this project, you definitely need to have written GObject code in Err codemadness.org 70 i 7282 C in the past. You don't need to know the GObject internals; just Err codemadness.org 70 i 7283 know that there are things like type registration, signal creation, Err codemadness.org 70 i 7284 argument marshalling, etc.

Err codemadness.org 70 i 7285

You don't need to know about compiler internals.

Err codemadness.org 70 i 7286

You don't need to have written Rust procedural macros; you can learn Err codemadness.org 70 i 7287 as you go. The code has enough infrastructure right now that you can Err codemadness.org 70 i 7288 cut&paste useful bits to get started with new features. You should Err codemadness.org 70 i 7289 definitely be comfortable with the Rust borrow checker and simple Err codemadness.org 70 i 7290 lifetimes — again, you can cut&paste useful code already, and Err codemadness.org 70 i 7291 I'm happy to help with those.

Err codemadness.org 70 i 7292

This project demands a little patience. Working on the implementation Err codemadness.org 70 i 7293 of procedural macros is not the smoothest experience right now (one Err codemadness.org 70 i 7294 needs to examine generated code carefully, and play some tricks with Err codemadness.org 70 i 7295 the compiler to debug things), but it's getting better very fast.

Err codemadness.org 70 i 7296

How to apply as an intern

Err codemadness.org 70 i 7297

Details for Outreachy

Err codemadness.org 70 i 7298

Details for Summer of Code

Helping Cairo

2018-03-06T18:22:52-06:00

Cairo needs help. It is the main 2D rendering library we use Err codemadness.org 70 i 7304 in GNOME, and in particular, it's what librsvg uses to render all Err codemadness.org 70 i 7305 SVGs.

Err codemadness.org 70 i 7306

My immediate problem with Cairo is that it explodes when called with Err codemadness.org 70 i 7307 floating-point coordinates that fall outside the range that its Err codemadness.org 70 i 7308 internal fixed-point numbers can represent. There is no validation of Err codemadness.org 70 i 7309 incoming data, so the polygon intersector ends up with data that makes Err codemadness.org 70 i 7310 no sense, and it crashes.

Err codemadness.org 70 i 7311

I've been studying how Cairo converts from floating-point to its Err codemadness.org 70 i 7312 fixed-point representation, and it's a nifty little algorithm. So I Err codemadness.org 70 i 7313 thought, no problem, I'll add validation, see how to represent the Err codemadness.org 70 i 7314 error state internally in Cairo, and see if clients are happy with Err codemadness.org 70 i 7315 getting back a cairo_t in an error state.

Err codemadness.org 70 i 7316

Cairo has a very thorough test suite... that doesn't pass. It Err codemadness.org 70 i 7317 is documented to be very hard to pass fully for all rendering Err codemadness.org 70 i 7318 backends. This is understandable, as there may be bugs in X servers Err codemadness.org 70 i 7319 or OpenGL implementations and such. But for the basic, software-only, Err codemadness.org 70 i 7320 in-memory image backend, Cairo should 100% pass its test suite all the Err codemadness.org 70 i 7321 time. This is not the case right now; in my tree, for all the tests Err codemadness.org 70 i 7322 of the image backend I get

Err codemadness.org 70 i 7323

497 Passed, 54 Failed [0 crashed, 14 expected], 27 Skipped	Err	codemadness.org	70
i  7324

Err codemadness.org 70 i 7325 Err codemadness.org 70 i 7326

I have been looking at test failures to see what needs fixing. Some Err codemadness.org 70 i 7327 reference images just need to be regenerated: there have been minor Err codemadness.org 70 i 7328 changes in font rendering that broke the reference tests. Some Err codemadness.org 70 i 7329 others have small differences in rendering gradients - not noticeable Err codemadness.org 70 i 7330 by eye, just by diff tools.

Err codemadness.org 70 i 7331

But some tests, I have no idea what changed that made them break.

Err codemadness.org 70 i 7332

Cairo's git repository is accessible through [cgit.freedesktop.org]. Err codemadness.org 70 i 7333 As far as I know there is no continuous integration infrastructure to Err codemadness.org 70 i 7334 ensure that tests keep passing.

Err codemadness.org 70 i 7335

Adding minimal continuous testing

Err codemadness.org 70 i 7336

I've set up a Cairo repository at gitlab.com. That branch Err codemadness.org 70 i 7337 already has a fix for an uninitialized-memory bug which leads to an Err codemadness.org 70 i 7338 invalid free(), and some regenerated test files.

Err codemadness.org 70 i 7339

The repository is configured to run a continuous integration Err codemadness.org 70 i 7340 pipeline on every commit. The test artifacts can then be Err codemadness.org 70 i 7341 downloaded when the test suite fails. Right now it is only testing Err codemadness.org 70 i 7342 the image backend, for in-memory software rendering.

Err codemadness.org 70 i 7343

Initial bugs

Err codemadness.org 70 i 7344

I've started reporting a few bugs against that repository for Err codemadness.org 70 i 7345 tests that fail. These should really be in Cairo's Bugzilla, but for Err codemadness.org 70 i 7346 now Gitlab makes it much easier to include test images directly in the Err codemadness.org 70 i 7347 bug descriptions, so that they are easier to browse. Read on.

Err codemadness.org 70 i 7348

Would you like to help?

Err codemadness.org 70 i 7349

A lot of projects use Cairo. We owe it to ourselves to have a library Err codemadness.org 70 i 7350 with a test suite that doesn't break. Getting to that point requires Err codemadness.org 70 i 7351 several things:

Err codemadness.org 70 i 7352

Fixing current failures in the image backend.
Setting up the CI infrastructure to be able to test other backends.
Fixing failures in the other backends.

Err codemadness.org 70 i 7357

If you have experience with Cairo, please take a look at the bugs. Err codemadness.org 70 i 7358 You can see the CI configuration to see how to run the test Err codemadness.org 70 i 7359 suite in the same fashion on your machine.

Err codemadness.org 70 i 7360

I think we can make use of modern infrastructure like gitlab and Err codemadness.org 70 i 7361 continuous integration to improve Cairo quickly. Currently it suffers Err codemadness.org 70 i 7362 from lack of attention and hostile tools. Help us out if you can!

Quick and dirty checklist to update syn 0.11.x to syn 0.12

2018-02-26T19:20:42-06:00

Today I ported gnome-class from version 0.11 of the syn crate to Err codemadness.org 70 i 7367 version 0.12. syn is a somewhat esoteric crate that you use to Err codemadness.org 70 i 7368 parse Rust code... from a stream of tokens... from within the Err codemadness.org 70 i 7369 implementation of a procedural macro. Gnome-class implements a Err codemadness.org 70 i 7370 mini-language inside your own Rust code, and so it needs to parse Err codemadness.org 70 i 7371 Rust!

Err codemadness.org 70 i 7372

The API of syn has changed a lot, which is kind of a pain in the Err codemadness.org 70 i 7373 ass — but the new API seems on the road to stabilization, and is nicer Err codemadness.org 70 i 7374 indeed.

Err codemadness.org 70 i 7375

Here is a quick list of things I had to change in gnome-class to Err codemadness.org 70 i 7376 upgrade its version of syn.

Err codemadness.org 70 i 7377

There is no extern crate synom anymore. You can use syn::synom now.

Err codemadness.org 70 i 7378

extern crate synom;    ->   use syn::synom;	Err	codemadness.org	70
i  7379

Err codemadness.org 70 i 7380 Err codemadness.org 70 i 7381

SynomBuffer is now TokenBuffer:

Err codemadness.org 70 i 7382

synom::SynomBuffer  ->  syn::buffer:TokenBuffer	Err	codemadness.org	70
i  7383

Err codemadness.org 70 i 7384 Err codemadness.org 70 i 7385

PResult, the result of Synom::parse(), now has the tuple's Err codemadness.org 70 i 7386 arguments reversed:

Err codemadness.org 70 i 7387

- pub type PResult<'a, O> = Result<(Cursor<'a>, O), ParseError>;	Err	codemadness.org	70
i  7388 + pub type PResult<'a, O> = Result<(O, Cursor<'a>), ParseError>;	Err	codemadness.org	70
i  7389 	Err	codemadness.org	70
i  7390 // therefore:	Err	codemadness.org	70
i  7391 	Err	codemadness.org	70
i  7392 impl Synom for MyThing { ... }	Err	codemadness.org	70
i  7393 	Err	codemadness.org	70
i  7394 let x = MyThing::parse(...).unwrap().1;   ->  let x = MyThing::parse(...).unwrap().0;	Err	codemadness.org	70
i  7395

Err codemadness.org 70 i 7396 Err codemadness.org 70 i 7397

The language tokens like synom::tokens::Amp, and keywords like Err codemadness.org 70 i 7398 synom::tokens::Type, are easier to use now. There is a Token! Err codemadness.org 70 i 7399 macro which you can use in type definitions, instead of having to Err codemadness.org 70 i 7400 remember the particular name of each token type:

Err codemadness.org 70 i 7401

synom::tokens::Amp  ->  Token!(&)	Err	codemadness.org	70
i  7402 	Err	codemadness.org	70
i  7403 synom::tokens::For  ->  Token!(for)	Err	codemadness.org	70
i  7404

Err codemadness.org 70 i 7405 Err codemadness.org 70 i 7406

And for the corresponding values when matching:

Err codemadness.org 70 i 7407

syn!(tokens::Colon)  ->  punct!(:)	Err	codemadness.org	70
i  7408 	Err	codemadness.org	70
i  7409 syn!(tokens::Type)   ->  keyword!(type)	Err	codemadness.org	70
i  7410

Err codemadness.org 70 i 7411 Err codemadness.org 70 i 7412

And to instantiate them for quoting/spanning:

Err codemadness.org 70 i 7413

-     tokens::Comma::default().to_tokens(tokens);	Err	codemadness.org	70
i  7414 +     Token!(,)([Span::def_site()]).to_tokens(tokens);	Err	codemadness.org	70
i  7415

Err codemadness.org 70 i 7416 Err codemadness.org 70 i 7417

(OK, that one wasn't nicer after all.)

Err codemadness.org 70 i 7418

To the get string for an Ident:

Err codemadness.org 70 i 7419

ident.sym.as_str()  ->  ident.as_ref()	Err	codemadness.org	70
i  7420

Err codemadness.org 70 i 7421 Err codemadness.org 70 i 7422

There is no Delimited anymore; instead there is a Punctuated Err codemadness.org 70 i 7423 struct. My diff has this:

Err codemadness.org 70 i 7424

-  inputs: parens!(call!(Delimited::<MyThing, tokens::Comma>::parse_terminated)) >>	Err	codemadness.org	70
i  7425 +  inputs: parens!(syn!(Punctuated<MyThing, Token!(,)>)) >>	Err	codemadness.org	70
i  7426

Err codemadness.org 70 i 7427 Err codemadness.org 70 i 7428

There is no syn::Mutability anymore; now it's an Option<token>, so Err codemadness.org 70 i 7429 basically

Err codemadness.org 70 i 7430

syn::Mutability  ->  Option<Token![mut]>	Err	codemadness.org	70
i  7431

Err codemadness.org 70 i 7432 Err codemadness.org 70 i 7433

which I guess lets you refer to the span of the original mut token Err codemadness.org 70 i 7434 if you need.

Err codemadness.org 70 i 7435

Some things changed names:

Err codemadness.org 70 i 7436

TypeTup { tys, .. }  ->  TypeTuple { elems, .. }	Err	codemadness.org	70
i  7437 	Err	codemadness.org	70
i  7438 PatIdent {                          ->  PatIdent {	Err	codemadness.org	70
i  7439     mode: BindingMode(Mutability)           by_ref: Option<Token!(ref)>,	Err	codemadness.org	70
i  7440                                             mutability: Option<Token![mut]>,	Err	codemadness.org	70
i  7441     ident: Ident,                           ident: Ident,	Err	codemadness.org	70
i  7442     subpat: ...,                            subpat: Option<(Token![@], Box<Pat>)>,	Err	codemadness.org	70
i  7443     at_token: ...,                      }	Err	codemadness.org	70
i  7444 }	Err	codemadness.org	70
i  7445 	Err	codemadness.org	70
i  7446 TypeParen.ty  ->  TypeParen.elem   (and others like this, too)	Err	codemadness.org	70
i  7447

Err codemadness.org 70 i 7448 Err codemadness.org 70 i 7449

(I don't know everything that changed names; gnome-class doesn't use Err codemadness.org 70 i 7450 all the syn types yet; these are just the ones I've run into.)

Err codemadness.org 70 i 7451

This new syn is much better at acknowledging the fine points of Err codemadness.org 70 i 7452 macro hygiene. The examples directory is particularly instructive; Err codemadness.org 70 i 7453 it shows how to properly span generated code vs. original code, so Err codemadness.org 70 i 7454 compiler error messages are nice. I need to write something about Err codemadness.org 70 i 7455 macro hygiene at some point.

Librsvg's continuous integration pipeline

2018-02-23T16:25:13-06:00

Jordan Petridis has been kicking ass by overhauling Err codemadness.org 70 i 7463 librsvg's continous integration (CI) pipeline. Take a look at this Err codemadness.org 70 i 7464 beauty:

Err codemadness.org 70 i 7465

Err codemadness.org 70 i 7466

On every push, we run the Test stage. This is a quick compilation Err codemadness.org 70 i 7467 on a Fedora container that runs "make check" and ensures that the Err codemadness.org 70 i 7468 test suite passes.

Err codemadness.org 70 i 7469

We have a Lint stage which can be run manually. This runs cargo Err codemadness.org 70 i 7470 clippy to get Rust lints (check the style of Rust idioms), and cargo Err codemadness.org 70 i 7471 fmt to check indentation and code style and such.

Err codemadness.org 70 i 7472

We have a Distro_test stage which I think will be scheduled Err codemadness.org 70 i 7473 weekly, using Gitlab's Schedules feature, to check that the tests Err codemadness.org 70 i 7474 pass on three major Linux distros. Recently we had trouble with Err codemadness.org 70 i 7475 different rendering due to differences in Freetype versions, which Err codemadness.org 70 i 7476 broke the tests (ahem, likely because I hadn't updated my Err codemadness.org 70 i 7477 Freetype in a while and distros were already using a newer one); these Err codemadness.org 70 i 7478 distro tests are intended to catch that.

Err codemadness.org 70 i 7479

Finally, we have a Rustc_test stage. The various crates that Err codemadness.org 70 i 7480 librsvg depends on have different minimum versions for the Rust Err codemadness.org 70 i 7481 compiler. These tests are intended to show when updating a dependency Err codemadness.org 70 i 7482 changes the minimum Rust version on which librsvg would compile. We Err codemadness.org 70 i 7483 don't have a policy yet for "how far from $newest" we should always Err codemadness.org 70 i 7484 work on, and it would be good to get input from distros on this. I Err codemadness.org 70 i 7485 think these Rust tests will be scheduled weekly as well.

Err codemadness.org 70 i 7486

Jordan has been experimenting with the pipeline's stages and the Err codemadness.org 70 i 7487 distro-specific idiosyncrasies for each build. This pipeline depends Err codemadness.org 70 i 7488 on some custom-built container images that already have Err codemadness.org 70 i 7489 librsvg's dependencies installed. These images are built weekly in Err codemadness.org 70 i 7490 gitlab.com, so every week gitlab.gnome.org gets fresh images for Err codemadness.org 70 i 7491 librsvg's CI pipelines. Once image registries are enabled in Err codemadness.org 70 i 7492 gitlab.gnome.org, we should be able to regenerate the container Err codemadness.org 70 i 7493 images locally without depending on an external service.

Err codemadness.org 70 i 7494

With the pre-built images, and caching of Rust artifacts, Jordan was Err codemadness.org 70 i 7495 able to reduce the time for the "test on every commit" builds from Err codemadness.org 70 i 7496 around 20 minutes, to little under 4 minutes in the current Err codemadness.org 70 i 7497 iteration. This will get even faster if the builds start using ccache Err codemadness.org 70 i 7498 and parallel builds from GNU make.

Err codemadness.org 70 i 7499

Currently we have a problem in that tests are failing on 32-bit Err codemadness.org 70 i 7500 builds, and haven't had a chance to investigate the root Err codemadness.org 70 i 7501 cause. Hopefully we can add 32-bit jobs to the CI pipeline to catch Err codemadness.org 70 i 7502 this breakage as soon as possible.

Err codemadness.org 70 i 7503

Having all these container images built for the CI infrastructure also Err codemadness.org 70 i 7504 means that it will be easy for people to set up a development Err codemadness.org 70 i 7505 environment for librsvg, even though we have better instructions Err codemadness.org 70 i 7506 now thanks to Jordan. I haven't investigated setting up a Err codemadness.org 70 i 7507 Flatpak-based environment; this would be nice to have as well.

RFC: Integrating rsvg-rs into librsvg

2018-02-22T09:57:52-06:00

I have started an RFC to integrate rsvg-rs into librsvg. Err codemadness.org 70 i 7512 rsvg-rs is the Rust binding to librsvg. Like the gtk-rs bindings, Err codemadness.org 70 i 7513 it gets generated from a pre-built GIR file.

Err codemadness.org 70 i 7514

It would be nice for librsvg to provide the Rust binding by itself, so Err codemadness.org 70 i 7515 that librsvg's own internal tools can be implemented in Rust — Err codemadness.org 70 i 7516 currently all the tests are done in C, as are the rsvg-convert(1) and Err codemadness.org 70 i 7517 rsvg-view-3(1) programs.

Err codemadness.org 70 i 7518

There are some implications for how rsvg-rs would get built then. Err codemadness.org 70 i 7519 For librsvg's internal consumption, the binding can be built from the Err codemadness.org 70 i 7520 Rsvg-2.0.gir file that gets built out of the main librsvg.so. But Err codemadness.org 70 i 7521 for public consumption of rsvg-rs, when it is being used as a normal Err codemadness.org 70 i 7522 crate and built by Cargo, that Rsvg-2.0.gir needs to be already Err codemadness.org 70 i 7523 built and available: it wouldn't be appropriate for Cargo to build Err codemadness.org 70 i 7524 librsvg and the .gir file itself.

Err codemadness.org 70 i 7525

If this sort of thing interests you, take a look at the RFC!

Rust things I miss in C

2018-02-18T21:26:04-06:00

Librsvg feels like it is reaching a tipping point, where suddenly it Err codemadness.org 70 i 7530 seems like it would be easier to just port some major parts from C to Err codemadness.org 70 i 7531 Rust than to just add accessors for them. Also, more and more of the Err codemadness.org 70 i 7532 meat of the library is in Rust now.

Err codemadness.org 70 i 7533

I'm switching back and forth a lot between C and Rust these days, and Err codemadness.org 70 i 7534 C feels very, very primitive these days.

Err codemadness.org 70 i 7535

A sort of elegy to C

Err codemadness.org 70 i 7536

I fell in love with the C language about 24 years ago. I learned the Err codemadness.org 70 i 7537 basics of it by reading a Spanish translation of The C Programming Err codemadness.org 70 i 7538 Language by K&R second edition. I had been using Turbo Pascal Err codemadness.org 70 i 7539 before in a reasonably low-level fashion, with pointers and manual Err codemadness.org 70 i 7540 memory allocation, and C felt refreshing and empowering.

Err codemadness.org 70 i 7541

K&R is a great book for its style of writing and its conciseness of Err codemadness.org 70 i 7542 programming. This little book even taught you how to implement a Err codemadness.org 70 i 7543 simple malloc()/free(), which was completely enlightening. Even Err codemadness.org 70 i 7544 low-level constructs that seemed part of the language could be Err codemadness.org 70 i 7545 implemented in the language itself!

Err codemadness.org 70 i 7546

I got good at C over the following years. It is a small language, Err codemadness.org 70 i 7547 with a small standard library. It was probably the perfect language Err codemadness.org 70 i 7548 to implement Unix kernels in 20,000 lines of code or so.

Err codemadness.org 70 i 7549

The GIMP and GTK+ taught me how to do fancy object orientation in C. Err codemadness.org 70 i 7550 GNOME taught me how to maintain large-scale software in C. 20,000 Err codemadness.org 70 i 7551 lines of C code started to seem like a project one could more or less Err codemadness.org 70 i 7552 fully understand in a few weeks.

Err codemadness.org 70 i 7553

But our code bases are not that small anymore. Our software now has Err codemadness.org 70 i 7554 huge expectations on the features that are available in the Err codemadness.org 70 i 7555 language's standard library.

Err codemadness.org 70 i 7556

Some good experiences with C

Err codemadness.org 70 i 7557

Reading the POV-Ray code source code for the first time and learning Err codemadness.org 70 i 7558 how to do object orientation and inheritance in C.

Err codemadness.org 70 i 7559

Reading the GTK+ source code and learning a C style that was legible, Err codemadness.org 70 i 7560 maintainable, and clean.

Err codemadness.org 70 i 7561

Reading SIOD's source code, then the early Guile sources, and seeing Err codemadness.org 70 i 7562 how a Scheme interpreter can be written in C.

Err codemadness.org 70 i 7563

Writing the initial versions of Eye of Gnome and fine-tuning the Err codemadness.org 70 i 7564 microtile rendering.

Err codemadness.org 70 i 7565

Some bad experiences with C

Err codemadness.org 70 i 7566

In the Evolution team, when everything was crashing. We had to buy a Err codemadness.org 70 i 7567 Solaris machine just to be able to buy Purify; there was no Valgrind Err codemadness.org 70 i 7568 back then.

Err codemadness.org 70 i 7569

Debugging gnome-vfs threading deadlocks.

Err codemadness.org 70 i 7570

Debugging Mesa and getting nowhere.

Err codemadness.org 70 i 7571

Taking over the intial versions of Nautilus-share and seeing that it Err codemadness.org 70 i 7572 never free()d anything.

Err codemadness.org 70 i 7573

Trying to refactor code where I had no idea about the memory Err codemadness.org 70 i 7574 management strategy.

Err codemadness.org 70 i 7575

Trying to turn code into a library when it is full of global variables Err codemadness.org 70 i 7576 and no functions are static.

Err codemadness.org 70 i 7577

But anyway — let's get on with things in Rust I miss in C.

Err codemadness.org 70 i 7578

Automatic resource management

Err codemadness.org 70 i 7579

One of the first blog posts I read about Rust was "Rust means never Err codemadness.org 70 i 7580 having to close a socket". Rust borrows C++'s ideas about Err codemadness.org 70 i 7581 Resource Acquisition Is Initialization (RAII), Smart Pointers, Err codemadness.org 70 i 7582 adds in the single-ownership principle for values, and gives you Err codemadness.org 70 i 7583 automatic, deterministic resource management in a very neat package.

Err codemadness.org 70 i 7584

Err codemadness.org 70 i 7586
Automatic: you don't free() by hand. Memory gets deallocated, Err codemadness.org 70 i 7587 files get closed, mutexes get unlocked when they go out of scope. Err codemadness.org 70 i 7588 If you are wrapping an external resource, you just implement the Err codemadness.org 70 i 7589 Drop trait and that's basically it. The wrapped resource feels Err codemadness.org 70 i 7590 like part of the language since you don't have to babysit its Err codemadness.org 70 i 7591 lifetime by hand.
Err codemadness.org 70 i 7592
Err codemadness.org 70 i 7594
Deterministic: resources get created (memory allocated, initialized, Err codemadness.org 70 i 7595 files opened, etc.), and they get destroyed when they go out of Err codemadness.org 70 i 7596 scope. There is no garbage collection: things really get terminated Err codemadness.org 70 i 7597 when you close a brace. You start to see your program's data Err codemadness.org 70 i 7598 lifetimes as a tree of function calls.
Err codemadness.org 70 i 7599

Err codemadness.org 70 i 7601

After forgetting to free/close/destroy C objects all the time, or Err codemadness.org 70 i 7602 worse, figuring out where code that I didn't write forgot to do those Err codemadness.org 70 i 7603 things (or did them twice, incorrectly)... I don't want to do it Err codemadness.org 70 i 7604 again.

Err codemadness.org 70 i 7605

Generics

Err codemadness.org 70 i 7606

Vec<T> really is a vector of whose elements are the size of T. Err codemadness.org 70 i 7607 It's not an array of pointers to individually allocated objects. It Err codemadness.org 70 i 7608 gets compiled specifically to code that can only handle objects of Err codemadness.org 70 i 7609 type T.

Err codemadness.org 70 i 7610

After writing many janky macros in C to do similar things... I don't Err codemadness.org 70 i 7611 want to do it again.

Err codemadness.org 70 i 7612

Traits are not just interfaces

Err codemadness.org 70 i 7613

Rust is not a Java-like object-oriented language. Instead it Err codemadness.org 70 i 7614 has traits, which at first seem like Java interfaces — an easy way to Err codemadness.org 70 i 7615 do dynamic dispatch, so that if an object implements Drawable then Err codemadness.org 70 i 7616 you can assume it has a draw() method.

Err codemadness.org 70 i 7617

However, traits are more powerful than that.

Err codemadness.org 70 i 7618

Associated types

Err codemadness.org 70 i 7619

Traits can have associated types. As an example, Rust Err codemadness.org 70 i 7620 provies the Iterator trait which you can implement:

Err codemadness.org 70 i 7621

pub trait Iterator {	Err	codemadness.org	70
i  7622     type Item;	Err	codemadness.org	70
i  7623     fn next(&mut self) -> Option<Self::Item>;	Err	codemadness.org	70
i  7624 }	Err	codemadness.org	70
i  7625

Err codemadness.org 70 i 7626 Err codemadness.org 70 i 7627

This means that whenever you implement Iterator for some iterable Err codemadness.org 70 i 7628 object, you also have to specify an Item type for the things that Err codemadness.org 70 i 7629 will be produced. If you call next() and there are more elements, Err codemadness.org 70 i 7630 you'll get back a Some(YourElementType). When your iterator runs Err codemadness.org 70 i 7631 out of items, it will return None.

Err codemadness.org 70 i 7632

Associated types can refer to other traits.

Err codemadness.org 70 i 7633

For example, in Rust, you can use for loops on anything that Err codemadness.org 70 i 7634 implements the IntoIterator trait:

Err codemadness.org 70 i 7635

pub trait IntoIterator {	Err	codemadness.org	70
i  7636     /// The type of the elements being iterated over.	Err	codemadness.org	70
i  7637     type Item;	Err	codemadness.org	70
i  7638 	Err	codemadness.org	70
i  7639     /// Which kind of iterator are we turning this into?	Err	codemadness.org	70
i  7640     type IntoIter: Iterator<Item=Self::Item>;	Err	codemadness.org	70
i  7641 	Err	codemadness.org	70
i  7642     fn into_iter(self) -> Self::IntoIter;	Err	codemadness.org	70
i  7643 }	Err	codemadness.org	70
i  7644

Err codemadness.org 70 i 7645 Err codemadness.org 70 i 7646

When implementing this trait, you must provide both the type of the Err codemadness.org 70 i 7647 Item which your iterator will produce, and IntoIter, the actual Err codemadness.org 70 i 7648 type that implements Iterator and that holds your iterator's state.

Err codemadness.org 70 i 7649

This way you can build webs of types that refer to each other. You Err codemadness.org 70 i 7650 can have a trait that says, "I can do foo and bar, but only if you Err codemadness.org 70 i 7651 give me a type that can do this and that".

Err codemadness.org 70 i 7652

Slices

Err codemadness.org 70 i 7653

I already posted about the lack of string slices in C and Err codemadness.org 70 i 7654 how this is a pain in the ass once you get used to having them.

Err codemadness.org 70 i 7655

Modern tooling for dependency management

Err codemadness.org 70 i 7656

Instead of

Err codemadness.org 70 i 7657

Having to invoke pkg-config by hand or with Autotools macros
Wrangling include paths for header files...
... and library files.
And basically depending on the user to ensure that the correct Err codemadness.org 70 i 7662 versions of libraries are installed,

Err codemadness.org 70 i 7664

You write a Cargo.toml file which lists the names and versions of Err codemadness.org 70 i 7665 your dependencies. These get downloaded from a well-known location, Err codemadness.org 70 i 7666 or from elsewhere if you specify.

Err codemadness.org 70 i 7667

You don't have to fight dependencies. It just works when you cargo build.

Err codemadness.org 70 i 7668

Tests

Err codemadness.org 70 i 7669

C makes it very hard to have unit tests for several reasons:

Err codemadness.org 70 i 7670

Err codemadness.org 70 i 7672
Internal functions are often static. This means they can't be Err codemadness.org 70 i 7673 called outside of the source file that defined them. A test program Err codemadness.org 70 i 7674 either has to #include the source file where the static functions Err codemadness.org 70 i 7675 live, or use #ifdefs to remove the statics only during testing.
Err codemadness.org 70 i 7676
Err codemadness.org 70 i 7678
You have to write Makefile-related hackery to link the test program Err codemadness.org 70 i 7679 to only part of your code's dependencies, or to only part of the Err codemadness.org 70 i 7680 rest of your code.
Err codemadness.org 70 i 7681
Err codemadness.org 70 i 7683
You have to pick a testing framework. You have to register tests Err codemadness.org 70 i 7684 against the testing framework. You have to learn the testing Err codemadness.org 70 i 7685 framework.
Err codemadness.org 70 i 7686

Err codemadness.org 70 i 7688

In Rust you write

Err codemadness.org 70 i 7689

#[test]	Err	codemadness.org	70
i  7690 fn test_that_foo_works() {	Err	codemadness.org	70
i  7691     assert!(foo() == expected_result);	Err	codemadness.org	70
i  7692 }	Err	codemadness.org	70
i  7693

Err codemadness.org 70 i 7694 Err codemadness.org 70 i 7695

anywhere in your program or library, and when you type cargo test, Err codemadness.org 70 i 7696 IT JUST FUCKING WORKS. That code only gets linked into the Err codemadness.org 70 i 7697 test binary. You don't have to compile anything twice by hand, or Err codemadness.org 70 i 7698 write Makefile hackery, or figure out how to extract internal Err codemadness.org 70 i 7699 functions for testing.

Err codemadness.org 70 i 7700

This is a very killer feature for me.

Err codemadness.org 70 i 7701

Documentation, with tests

Err codemadness.org 70 i 7702

Rust generates documentation from comments in Markdown syntax. Code Err codemadness.org 70 i 7703 in the docs gets run as tests. You can illustrate how a function is Err codemadness.org 70 i 7704 used and test it at the same time:

Err codemadness.org 70 i 7705

/// Multiples the specified number by two	Err	codemadness.org	70
i  7706 ///	Err	codemadness.org	70
i  7707 /// ```	Err	codemadness.org	70
i  7708 /// assert_eq!(multiply_by_two(5), 10);	Err	codemadness.org	70
i  7709 /// ```	Err	codemadness.org	70
i  7710 fn multiply_by_two(x: i32) -> i32 {	Err	codemadness.org	70
i  7711     x * 2	Err	codemadness.org	70
i  7712 }	Err	codemadness.org	70
i  7713

Err codemadness.org 70 i 7714 Err codemadness.org 70 i 7715

Your example code gets run as tests to ensure that your Err codemadness.org 70 i 7716 documentation stays up to date with the actual code.

Err codemadness.org 70 i 7717

Update 2018/Feb/23: QuietMisdreavus has posted how rustdoc turns Err codemadness.org 70 i 7718 doctests into runnable code Err codemadness.org 70 i 7719 internally. Err codemadness.org 70 i 7720 This is high-grade magic and thoroughly interesting.

Err codemadness.org 70 i 7721

Hygienic macros

Err codemadness.org 70 i 7722

Rust has hygienic macros that avoid all of C's problems with things in Err codemadness.org 70 i 7723 macros that inadvertently shadow identifiers in the code. You don't Err codemadness.org 70 i 7724 need to write macros where every symbol has to be in parentheses for Err codemadness.org 70 i 7725 max(5 + 3, 4) to work correctly.

Err codemadness.org 70 i 7726

No automatic coercions

Err codemadness.org 70 i 7727

All the bugs in C that result from inadvertently converting an int Err codemadness.org 70 i 7728 to a short or char or whatever — Rust doesn't do them. You have Err codemadness.org 70 i 7729 to explicitly convert.

Err codemadness.org 70 i 7730

No integer overflow

Err codemadness.org 70 i 7731

Enough said.

Err codemadness.org 70 i 7732

Generally, no undefined behavior in safe Rust

Err codemadness.org 70 i 7733

In Rust, it is considered a bug in the language if something written Err codemadness.org 70 i 7734 in "safe Rust" (what you would be allowed to write outside unsafe {} Err codemadness.org 70 i 7735 blocks) results in undefined behavior. You can shift-right a negative Err codemadness.org 70 i 7736 integer and it will do exactly what you expect.

Err codemadness.org 70 i 7737

Pattern matching

Err codemadness.org 70 i 7738

You know how gcc warns you if you switch() on an enum but don't Err codemadness.org 70 i 7739 handle all values? That's like a little baby.

Err codemadness.org 70 i 7740

Rust has pattern matching in various places. It can do that Err codemadness.org 70 i 7741 trick for enums inside a match() expression. It can do Err codemadness.org 70 i 7742 destructuring so you can return multiple values from a function:

Err codemadness.org 70 i 7743

impl f64 {	Err	codemadness.org	70
i  7744     pub fn sin_cos(self) -> (f64, f64);	Err	codemadness.org	70
i  7745 }	Err	codemadness.org	70
i  7746 	Err	codemadness.org	70
i  7747 let angle: f64 = 42.0;	Err	codemadness.org	70
i  7748 let (sin_angle, cos_angle) = angle.sin_cos();	Err	codemadness.org	70
i  7749

Err codemadness.org 70 i 7750 Err codemadness.org 70 i 7751

You can match() on strings. YOU CAN MATCH ON FUCKING STRINGS.

Err codemadness.org 70 i 7752

let color = "green";	Err	codemadness.org	70
i  7753 	Err	codemadness.org	70
i  7754 match color {	Err	codemadness.org	70
i  7755     "red"   => println!("it's red"),	Err	codemadness.org	70
i  7756     "green" => println!("it's green"),	Err	codemadness.org	70
i  7757     _       => println!("it's something else"),	Err	codemadness.org	70
i  7758 }	Err	codemadness.org	70
i  7759

Err codemadness.org 70 i 7760 Err codemadness.org 70 i 7761

You know how this is illegible?

Err codemadness.org 70 i 7762

my_func(true, false, false)	Err	codemadness.org	70
i  7763

Err codemadness.org 70 i 7764 Err codemadness.org 70 i 7765

How about this instead, with pattern matching on function arguments:

Err codemadness.org 70 i 7766

pub struct Fubarize(pub bool);	Err	codemadness.org	70
i  7767 pub struct Frobnify(pub bool);	Err	codemadness.org	70
i  7768 pub struct Bazificate(pub bool);	Err	codemadness.org	70
i  7769 	Err	codemadness.org	70
i  7770 fn my_func(Fubarize(fub): Fubarize, 	Err	codemadness.org	70
i  7771            Frobnify(frob): Frobnify, 	Err	codemadness.org	70
i  7772            Bazificate(baz): Bazificate) {	Err	codemadness.org	70
i  7773     if fub {	Err	codemadness.org	70
i  7774         ...;	Err	codemadness.org	70
i  7775     }	Err	codemadness.org	70
i  7776 	Err	codemadness.org	70
i  7777     if frob && baz {	Err	codemadness.org	70
i  7778         ...;	Err	codemadness.org	70
i  7779     }	Err	codemadness.org	70
i  7780 }	Err	codemadness.org	70
i  7781 	Err	codemadness.org	70
i  7782 ...	Err	codemadness.org	70
i  7783 	Err	codemadness.org	70
i  7784 my_func(Fubarize(true), Frobnify(false), Bazificate(true));	Err	codemadness.org	70
i  7785

Err codemadness.org 70 i 7786 Err codemadness.org 70 i 7787

Standard, useful error handling

Err codemadness.org 70 i 7788

I've talked at length about this. No more returning a boolean with no Err codemadness.org 70 i 7789 extra explanation for an error, no ignoring errors inadvertently, no Err codemadness.org 70 i 7790 exception handling with nonlocal jumps.

Err codemadness.org 70 i 7791

#[derive(Debug)]

Err codemadness.org 70 i 7792

If you write a new type (say, a struct with a ton of fields), you can Err codemadness.org 70 i 7793 #[derive(Debug)] and Rust will know how to automatically print that Err codemadness.org 70 i 7794 type's contents for debug output. You no longer have to write a Err codemadness.org 70 i 7795 special function that you must call in gdb by hand just to examine a Err codemadness.org 70 i 7796 custom type.

Err codemadness.org 70 i 7797

Closures

Err codemadness.org 70 i 7798

No more passing function pointers and a user_data by hand.

Err codemadness.org 70 i 7799

Conclusion

Err codemadness.org 70 i 7800

I haven't done the "fearless concurrency" bit yet, where the Err codemadness.org 70 i 7801 compiler is able to prevent data races in threaded code. I imagine it Err codemadness.org 70 i 7802 being a game-changer for people who write concurrent code on an Err codemadness.org 70 i 7803 everyday basis.

Err codemadness.org 70 i 7804

C is an old language with primitive constructs and primitive tooling. Err codemadness.org 70 i 7805 It was a good language for small uniprocessor Unix kernels that ran in Err codemadness.org 70 i 7806 trusted, academic environments. It's no longer a good language for Err codemadness.org 70 i 7807 the software of today.

Err codemadness.org 70 i 7808

Rust is not easy to learn, but I think it is completely worth it. Err codemadness.org 70 i 7809 It's hard because it demands a lot from your understanding of the code Err codemadness.org 70 i 7810 you want to write. I think it's one of those languages that make you Err codemadness.org 70 i 7811 a better programmer and that let you tackle more ambitious problems.

Writing a command-line program in Rust

2018-02-03T11:41:20-06:00

As a library writer, it feels a bit strange, but refreshing, to write Err codemadness.org 70 i 7818 a program that actually has a main() function.

Err codemadness.org 70 i 7819

My experience with Rust so far has been threefold:

Err codemadness.org 70 i 7820

Err codemadness.org 70 i 7822
Porting chunks of C to Rust for librsvg - this is all work on Err codemadness.org 70 i 7823 librsvg's internals and no users are exposed to it directly.
Err codemadness.org 70 i 7824
Err codemadness.org 70 i 7826
Working on gnome-class, the procedural macro ("a little compiler") Err codemadness.org 70 i 7827 to generate GObject boilerplate from Rust. This feels like working Err codemadness.org 70 i 7828 on the edge of the exotic; it is something that runs in the Rust Err codemadness.org 70 i 7829 compiler and spits code on behalf of the programmer.
Err codemadness.org 70 i 7830
Err codemadness.org 70 i 7832
A few patches to the gtk-rs ecosystem. Again, work on the Err codemadness.org 70 i 7833 internals, or something that feels library-like.
Err codemadness.org 70 i 7834

Err codemadness.org 70 i 7836

But other than toy programs to test things, I haven't written a Err codemadness.org 70 i 7837 stand-alone tool until rsvg-bench. It's quite a thrill to be able Err codemadness.org 70 i 7838 to just run the thing instead of waiting for other people to write Err codemadness.org 70 i 7839 code to use it!

Err codemadness.org 70 i 7840

Parsing command-line arguments

Err codemadness.org 70 i 7841

There are quite a few Rust crates ("libraries") to parse command-line Err codemadness.org 70 i 7842 arguments. I read about structopt via Robert O'Callahan's Err codemadness.org 70 i 7843 blog; structopt lets you define a struct to hold the values of Err codemadness.org 70 i 7844 your command-line options, and then you annotate the fields in that Err codemadness.org 70 i 7845 struct to indicate how they should be parsed from the command line. Err codemadness.org 70 i 7846 It works via Rust's procedural macros. Internally it generates stuff Err codemadness.org 70 i 7847 for the clap crate, a well-established mechanism for dealing with Err codemadness.org 70 i 7848 command-line options.

Err codemadness.org 70 i 7849

And it is quite pleasant! This is basically all I needed to do:

Err codemadness.org 70 i 7850

#[derive(StructOpt, Debug)]	Err	codemadness.org	70
i  7851 #[structopt(name = "rsvg-bench", about = "Benchmarking utility for librsvg.")]	Err	codemadness.org	70
i  7852 struct Opt {	Err	codemadness.org	70
i  7853     #[structopt(short = "s",	Err	codemadness.org	70
i  7854                 long  = "sleep",	Err	codemadness.org	70
i  7855                 help  = "Number of seconds to sleep before starting to process SVGs",	Err	codemadness.org	70
i  7856                 default_value = "0")]	Err	codemadness.org	70
i  7857     sleep_secs: usize,	Err	codemadness.org	70
i  7858 	Err	codemadness.org	70
i  7859     #[structopt(short = "p",	Err	codemadness.org	70
i  7860                 long  = "num-parse",	Err	codemadness.org	70
i  7861                 help  = "Number of times to parse each file",	Err	codemadness.org	70
i  7862                 default_value = "100")]	Err	codemadness.org	70
i  7863     num_parse: usize,	Err	codemadness.org	70
i  7864 	Err	codemadness.org	70
i  7865     #[structopt(short = "r",	Err	codemadness.org	70
i  7866                 long  = "num-render",	Err	codemadness.org	70
i  7867                 help  = "Number of times to render each file",	Err	codemadness.org	70
i  7868                 default_value = "100")]	Err	codemadness.org	70
i  7869     num_render: usize,	Err	codemadness.org	70
i  7870 	Err	codemadness.org	70
i  7871     #[structopt(long = "pixbuf",	Err	codemadness.org	70
i  7872                 help = "Render to a GdkPixbuf instead of a Cairo image surface")]	Err	codemadness.org	70
i  7873     render_to_pixbuf: bool,	Err	codemadness.org	70
i  7874 	Err	codemadness.org	70
i  7875     #[structopt(help = "Input files or directories",	Err	codemadness.org	70
i  7876                 parse(from_os_str))]	Err	codemadness.org	70
i  7877     inputs: Vec<PathBuf>	Err	codemadness.org	70
i  7878 }	Err	codemadness.org	70
i  7879 	Err	codemadness.org	70
i  7880 fn main() {	Err	codemadness.org	70
i  7881     let opt = Opt::from_args();	Err	codemadness.org	70
i  7882 	Err	codemadness.org	70
i  7883     if opt.inputs.len() == 0 {	Err	codemadness.org	70
i  7884         eprintln!("No input files or directories specified\n");	Err	codemadness.org	70
i  7885         process.exit(1);	Err	codemadness.org	70
i  7886     }	Err	codemadness.org	70
i  7887 	Err	codemadness.org	70
i  7888     ...	Err	codemadness.org	70
i  7889 }	Err	codemadness.org	70
i  7890

Err codemadness.org 70 i 7891 Err codemadness.org 70 i 7892

Each field in the Opt struct above corresponds to one command-line Err codemadness.org 70 i 7893 argument; each field has annotations for structopt to generate the Err codemadness.org 70 i 7894 appropriate code to parse each option. For example, the Err codemadness.org 70 i 7895 render_to_pixbuf field has a long option name called "pixbuf"; Err codemadness.org 70 i 7896 that field will be set to true if the --pixbuf option gets passed Err codemadness.org 70 i 7897 to rsvg-bench.

Err codemadness.org 70 i 7898

Handling errors

Err codemadness.org 70 i 7899

Command-line programs generally have the luxury of being able to just Err codemadness.org 70 i 7900 exit as soon as they encounter an error.

Err codemadness.org 70 i 7901

In C this is a bit cumbersome since you need to deal with every Err codemadness.org 70 i 7902 place that may return an error, find out what to print, and call Err codemadness.org 70 i 7903 exit(1) by hand or something. If you miss a single place where an Err codemadness.org 70 i 7904 error is returned, your program will keep running with an inconsistent Err codemadness.org 70 i 7905 state.

Err codemadness.org 70 i 7906

In languages with exception handling, it's a bit easier - a small Err codemadness.org 70 i 7907 script can just let exceptions be thrown wherever, and if it catches Err codemadness.org 70 i 7908 them at the toplevel, it can just print the exception and abort Err codemadness.org 70 i 7909 gracefully. However, these nonlocal jumps make me uncomfortable; I Err codemadness.org 70 i 7910 think exceptions are hard to reason about.

Err codemadness.org 70 i 7911

Rust makes this easy: it forces you to handle every call that may Err codemadness.org 70 i 7912 return an error, but it lets you bubble errors up easily, or handle Err codemadness.org 70 i 7913 them in-place, or translate them to a higher-level error.

Err codemadness.org 70 i 7914

In the Rust world the [failure] crate is getting a lot of traction Err codemadness.org 70 i 7915 as a convenient, modern way to handle errors.

Err codemadness.org 70 i 7916

In rsvg-bench, errors can come from several places:

Err codemadness.org 70 i 7917

Err codemadness.org 70 i 7919
I/O errors when reading files and directories.
Err codemadness.org 70 i 7920
Err codemadness.org 70 i 7922
Errors from librsvg's parsing stage; you get a GError.
Err codemadness.org 70 i 7923
Err codemadness.org 70 i 7925
Errors from the rendering stage. This can be a Cairo error (a Err codemadness.org 70 i 7926 cairo_status_t), or a simple "something bad happened; can't Err codemadness.org 70 i 7927 render" from librsvg's old convenience api in C. Don't you hate it Err codemadness.org 70 i 7928 when C code just gives up and returns NULL or a boolean false, Err codemadness.org 70 i 7929 without any further details on what went wrong?
Err codemadness.org 70 i 7930

Err codemadness.org 70 i 7932

For rsvg-bench, I just needed to be able to represent Cairo errors and Err codemadness.org 70 i 7933 generic rendering errors. Everything else, like an io::Error, is Err codemadness.org 70 i 7934 automatically wrapped by the failure crate's mechanism. I just Err codemadness.org 70 i 7935 needed to do this:

Err codemadness.org 70 i 7936

extern crate failure;	Err	codemadness.org	70
i  7937 #[macro_use]	Err	codemadness.org	70
i  7938 extern crate failure_derive;	Err	codemadness.org	70
i  7939 	Err	codemadness.org	70
i  7940 #[derive(Debug, Fail)]	Err	codemadness.org	70
i  7941 enum ProcessingError {	Err	codemadness.org	70
i  7942     #[fail(display = "Cairo error: {:?}", status)]	Err	codemadness.org	70
i  7943     CairoError {	Err	codemadness.org	70
i  7944         status: cairo::Status	Err	codemadness.org	70
i  7945     },	Err	codemadness.org	70
i  7946 	Err	codemadness.org	70
i  7947     #[fail(display = "Rendering error")]	Err	codemadness.org	70
i  7948     RenderingError	Err	codemadness.org	70
i  7949 }	Err	codemadness.org	70
i  7950

Err codemadness.org 70 i 7951 Err codemadness.org 70 i 7952

Whenever the code gets a Cairo error, I can translate it to a Err codemadness.org 70 i 7953 ProcessingError::CairoError and bubble it up:

Err codemadness.org 70 i 7954

fn render_to_cairo(handle: &rsvg::Handle) -> Result<(), Error> {	Err	codemadness.org	70
i  7955     let dim = handle.get_dimensions();	Err	codemadness.org	70
i  7956     let surface = cairo::ImageSurface::create(cairo::Format::ARgb32,	Err	codemadness.org	70
i  7957                                               dim.width,	Err	codemadness.org	70
i  7958                                               dim.height)	Err	codemadness.org	70
i  7959         .map_err(|e| ProcessingError::CairoError { status: e })?;	Err	codemadness.org	70
i  7960 	Err	codemadness.org	70
i  7961     ...	Err	codemadness.org	70
i  7962 }	Err	codemadness.org	70
i  7963

Err codemadness.org 70 i 7964 Err codemadness.org 70 i 7965

And when librsvg returns a "couldn't render" error, I translate that Err codemadness.org 70 i 7966 to a ProcessingError::RenderingError:

Err codemadness.org 70 i 7967

fn render_to_cairo(handle: &rsvg::Handle) -> Result<(), Error> {	Err	codemadness.org	70
i  7968     ...	Err	codemadness.org	70
i  7969 	Err	codemadness.org	70
i  7970     let cr = cairo::Context::new(&surface);	Err	codemadness.org	70
i  7971 	Err	codemadness.org	70
i  7972     if handle.render_cairo(&cr) {	Err	codemadness.org	70
i  7973         Ok(())	Err	codemadness.org	70
i  7974     } else {	Err	codemadness.org	70
i  7975         Err(Error::from(ProcessingError::RenderingError))	Err	codemadness.org	70
i  7976     }	Err	codemadness.org	70
i  7977 }	Err	codemadness.org	70
i  7978

Err codemadness.org 70 i 7979 Err codemadness.org 70 i 7980

Here, the Ok() case of the Result does not contain any value — Err codemadness.org 70 i 7981 it's just (), as the generated images are not stored anywhere: they Err codemadness.org 70 i 7982 are just rendered to get some timings, not to be saved or anything.

Err codemadness.org 70 i 7983

Up to where do errors bubble?

Err codemadness.org 70 i 7984

This is the "do everything" function:

Err codemadness.org 70 i 7985

fn run(opt: &Opt) -> Result<(), Error> {	Err	codemadness.org	70
i  7986     ...	Err	codemadness.org	70
i  7987 	Err	codemadness.org	70
i  7988     for path in &opt.inputs {	Err	codemadness.org	70
i  7989         process_path(opt, &path)?;	Err	codemadness.org	70
i  7990     }	Err	codemadness.org	70
i  7991 	Err	codemadness.org	70
i  7992     Ok(())	Err	codemadness.org	70
i  7993 }	Err	codemadness.org	70
i  7994

Err codemadness.org 70 i 7995 Err codemadness.org 70 i 7996

For each path passed in the command line, process it. The program Err codemadness.org 70 i 7997 sees if the path corresponds to a directory, and it will scan it Err codemadness.org 70 i 7998 recursively. Or if the path is an SVG file, the program will load the Err codemadness.org 70 i 7999 file and render it.

Err codemadness.org 70 i 8000

Finally, main() just has this:

Err codemadness.org 70 i 8001

fn main() {	Err	codemadness.org	70
i  8002     let opt = Opt::from_args();	Err	codemadness.org	70
i  8003 	Err	codemadness.org	70
i  8004     ...	Err	codemadness.org	70
i  8005 	Err	codemadness.org	70
i  8006     match run(&opt) {	Err	codemadness.org	70
i  8007         Ok(_) => (),	Err	codemadness.org	70
i  8008         Err(e) => {	Err	codemadness.org	70
i  8009             eprintln!("{}", e);	Err	codemadness.org	70
i  8010             process::exit(1);	Err	codemadness.org	70
i  8011         }	Err	codemadness.org	70
i  8012     }	Err	codemadness.org	70
i  8013 }	Err	codemadness.org	70
i  8014

Err codemadness.org 70 i 8015 Err codemadness.org 70 i 8016

I.e. process command line arguments, run the whole thing, and print an Err codemadness.org 70 i 8017 error if there was one.

Err codemadness.org 70 i 8018

I really appreciate that most places that can return an error an just Err codemadness.org 70 i 8019 put a ? for the error to bubble up. This is much more legible than Err codemadness.org 70 i 8020 in C, where every call must have an if (something_bad_happened) { Err codemadness.org 70 i 8021 deal_with_it; } after it... and Rust won't let me get away with Err codemadness.org 70 i 8022 ignoring an error, but it makes it easy to actually deal with it properly.

Err codemadness.org 70 i 8023

Reading an SVG file quickly

Err codemadness.org 70 i 8024

Why, just mmap() it and feed it to librsvg, to avoid buffer copies. Err codemadness.org 70 i 8025 This is easy in Rust:

Err codemadness.org 70 i 8026

fn process_file<P: AsRef<Path>>(opt: &Opt, path: P) -> Result<(), Error> {	Err	codemadness.org	70
i  8027     let file = File::open(path)?;	Err	codemadness.org	70
i  8028     let mmap = unsafe { MmapOptions::new().map(&file)? };	Err	codemadness.org	70
i  8029 	Err	codemadness.org	70
i  8030     let bytes = &mmap;	Err	codemadness.org	70
i  8031 	Err	codemadness.org	70
i  8032     let handle = rsvg::Handle::new_from_data(bytes)?;	Err	codemadness.org	70
i  8033     ...	Err	codemadness.org	70
i  8034 }	Err	codemadness.org	70
i  8035

Err codemadness.org 70 i 8036 Err codemadness.org 70 i 8037

Many things can go wrong here:

Err codemadness.org 70 i 8038

File::open() can return an io::Error.
MmapOptions::map() can return an io::Error from the mmap(2) Err codemadness.org 70 i 8041 system call, or from the fstat(2) to read the file's size to map Err codemadness.org 70 i 8042 it.
rsvg::Handle::new_from_data() can return a GError from parsing the Err codemadness.org 70 i 8044 file.

Err codemadness.org 70 i 8046

The little ? characters after each call that can return an error Err codemadness.org 70 i 8047 mean, just give me back the result, or convert the error to a Err codemadness.org 70 i 8048 failure::Error that can be examined later. This is beautifully Err codemadness.org 70 i 8049 legible to me.

Err codemadness.org 70 i 8050

Summary

Err codemadness.org 70 i 8051

Writing command-line programs in Rust is fun! It's nice to have Err codemadness.org 70 i 8052 neurotically-safe scripts that one can trust in the future.

Err codemadness.org 70 i 8053

Rsvg-bench is available here.

rsvg-bench - a benchmark for librsvg

2018-02-02T16:10:34-06:00

Librsvg 2.42.0 came out with a rather major performance regression Err codemadness.org 70 i 8058 compared to 2.40.20: SVGs with many transform Err codemadness.org 70 i 8059 attributes would slow it down. It was fixed in 2.42.1. We changed Err codemadness.org 70 i 8060 from using a parser that would recompile regexes each time it was Err codemadness.org 70 i 8061 called, to one that does simple string-based matching and Err codemadness.org 70 i 8062 parsing.

Err codemadness.org 70 i 8063

When I rewrote librsvg's parser for the transform attribute from C Err codemadness.org 70 i 8064 to Rust, I was just learning about writing parsers in Rust. Err codemadness.org 70 i 8065 I chose lalrpop, an excellent, Yacc-like parser generator for Rust. Err codemadness.org 70 i 8066 It generates big, fast parsers, like what you would need for a Err codemadness.org 70 i 8067 compiler — but it compiles the tokenizer's regexes each time you call Err codemadness.org 70 i 8068 the parser. This is not a problem for a compiler, where you basically Err codemadness.org 70 i 8069 call the parser only once, but in librsvg, we may call it thousands of Err codemadness.org 70 i 8070 times for an SVG file with thousands of objects with transform Err codemadness.org 70 i 8071 attributes.

Err codemadness.org 70 i 8072

So, for 2.42.1 I rewrote that parser using Err codemadness.org 70 i 8073 rust-cssparser. This is what Servo uses to Err codemadness.org 70 i 8074 parse CSS data; it's a simple tokenizer with an API that knows about Err codemadness.org 70 i 8075 CSS's particular constructs. This is exactly the kind of data that Err codemadness.org 70 i 8076 librsvg cares about. Today all of librsvg's internal parsers work Err codemadness.org 70 i 8077 using rust-cssparser, or they are so simple that they can be done with Err codemadness.org 70 i 8078 Rust's normal functions to split strings and such.

Err codemadness.org 70 i 8079

Getting good timings

Err codemadness.org 70 i 8080

Librsvg ships with rsvg-convert, a command-line utility that can Err codemadness.org 70 i 8081 render an SVG file and write the output to a PNG. While it would be Err codemadness.org 70 i 8082 possible to get timings for SVG rendering by timing how long Err codemadness.org 70 i 8083 rsvg-convert takes to run, it's a bit clunky for that. The process Err codemadness.org 70 i 8084 startup adds noise to the timings, and it only handles one file at a Err codemadness.org 70 i 8085 time.

Err codemadness.org 70 i 8086

So, I've written rsvg-bench, a small utility to get timings out of Err codemadness.org 70 i 8087 librsvg. I wanted a tool that:

Err codemadness.org 70 i 8088

Err codemadness.org 70 i 8090
Is able to process many SVG images with a single command. For Err codemadness.org 70 i 8091 example, this lets us answer a question like, "how long does version Err codemadness.org 70 i 8092 N of librsvg take to render a directory full of SVG icons?" — which Err codemadness.org 70 i 8093 is important for the performance of an application chooser.
Err codemadness.org 70 i 8094
Err codemadness.org 70 i 8096
Is able to repeatedly process SVG files, for example, "render this Err codemadness.org 70 i 8097 SVG 1000 times in a row". This is useful to get accurate timings, Err codemadness.org 70 i 8098 as a single render may only take a few microseconds and may be hard Err codemadness.org 70 i 8099 to measure. It also helps with running profilers, as they will be Err codemadness.org 70 i 8100 able to get more useful samples if the SVG rendering process runs Err codemadness.org 70 i 8101 repeatedly for a long time.
Err codemadness.org 70 i 8102
Err codemadness.org 70 i 8104
Exercises librsvg's major code paths for parsing and rendering Err codemadness.org 70 i 8105 separately. For example, librsvg uses different parts of the XML Err codemadness.org 70 i 8106 parser depending on whether it is being pushed data, vs. being asked Err codemadness.org 70 i 8107 to pull data from a stream. Also, we may only want to benchmark the Err codemadness.org 70 i 8108 parser but not the renderer; or we may want to parse SVGs only once Err codemadness.org 70 i 8109 but render them many times after that.
Err codemadness.org 70 i 8110
Err codemadness.org 70 i 8112
Is aware of librsvg's peculiarities, such as the extra pass to Err codemadness.org 70 i 8113 convert a Cairo image surface to a GdkPixbuf when one uses the Err codemadness.org 70 i 8114 convenience function rsvg_handle_get_pixbuf().
Err codemadness.org 70 i 8115

Err codemadness.org 70 i 8117

Currently rsvg-bench supports all of that.

Err codemadness.org 70 i 8118

An initial benchmark

Err codemadness.org 70 i 8119

I ran this

Err codemadness.org 70 i 8120

/usr/bin/time rsvg-bench -p 1 -r 1 /usr/share/icons

Err codemadness.org 70 i 8121

to cause every SVG icon in /usr/share/icons to be parsed once, and Err codemadness.org 70 i 8122 rendered once (i.e. just render every file sequentially). I did this Err codemadness.org 70 i 8123 for librsvg 2.40.20 (C only), and 2.42.{0, 1, 2} (C and Rust). There Err codemadness.org 70 i 8124 are 5522 SVG files in there. The timings look like this:

Err codemadness.org 70 i 8125 Err codemadness.org 70 i 8126 Err codemadness.org 70 i 8127 Err codemadness.org 70 i 8128 Err codemadness.org 70 i 8129 Err codemadness.org 70 i 8130 Err codemadness.org 70 i 8131 Err codemadness.org 70 i 8132 Err codemadness.org 70 i 8133 Err codemadness.org 70 i 8134 Err codemadness.org 70 i 8135 Err codemadness.org 70 i 8136 Err codemadness.org 70 i 8137 Err codemadness.org 70 i 8138 Err codemadness.org 70 i 8139 Err codemadness.org 70 i 8140 Err codemadness.org 70 i 8141 Err codemadness.org 70 i 8142 Err codemadness.org 70 i 8143 Err codemadness.org 70 i 8144 Err codemadness.org 70 i 8145 Err codemadness.org 70 i 8146 Err codemadness.org 70 i 8147 Err codemadness.org 70 i 8148 Err codemadness.org 70 i 8149 Err codemadness.org 70 i 8150

version	time (sec)
2.40.20	95.54
2.42.0	209.50
2.42.1	97.18
2.42.2	95.89

Err codemadness.org 70 i 8151

Err codemadness.org 70 i 8152

So, 2.42.0 was over twice as slow as the C-only version, due to the Err codemadness.org 70 i 8153 parsing problems. But now, 2.42.2 is practically just as fast as the Err codemadness.org 70 i 8154 C only version. What made this possible?

Err codemadness.org 70 i 8155

2.40.20 - the old C-only version
2.42.0 - C + Rust, with a lalrpop parser for the transform attribute
2.42.1 - Servo's cssparser for the transform attribute
2.42.2 - removed most C-to-Rust string copies during parsing

Err codemadness.org 70 i 8161

I have started taking profiles of rsvg-bench runs with sysprof, and Err codemadness.org 70 i 8162 there are some improvements worth making. Expect news soon!

Err codemadness.org 70 i 8163

Rsvg-bench is available in Gnome's gitlab instance.

Help needed for librsvg 2.42.1

2018-01-16T11:01:05-06:00

Would you like to help fix a couple of bugs in librsvg, in Err codemadness.org 70 i 8168 preparation for the 2.42.1 release?

Err codemadness.org 70 i 8169

I have prepared a list of bugs which I'd like to be fixed in the Err codemadness.org 70 i 8170 2.42.1 milestone. Two of them are assigned to myself, as Err codemadness.org 70 i 8171 I'm already working on them.

Err codemadness.org 70 i 8172

There are two other bugs which I'd love someone to look at. Neither Err codemadness.org 70 i 8173 of these requires deep knowledge of librsvg, just some debugging and Err codemadness.org 70 i 8174 code-writing:

Err codemadness.org 70 i 8175

Err codemadness.org 70 i 8177
Bug 141 - GNOME's thumbnailing machinery creates an icon which has Err codemadness.org 70 i 8178 the wrong fill: it's an image of a builder's trowel, and the inside Err codemadness.org 70 i 8179 is filled black instead of with a nice gradient. This is the only Err codemadness.org 70 i 8180 place in librsvg where a cairo_surface_t is converted to a Err codemadness.org 70 i 8181 GdkPixbuf; this involves unpremultiplying the alpha channel. Err codemadness.org 70 i 8182 Maybe the relevant function is buggy?
Err codemadness.org 70 i 8183
Err codemadness.org 70 i 8185
Bug 136: The stroke-dasharray attribute in SVG elements is Err codemadness.org 70 i 8186 parsed incorrectly. It is a list of CSS length values, separated by Err codemadness.org 70 i 8187 commas or spaces. Currently librsvg uses a shitty parser based on Err codemadness.org 70 i 8188 g_strsplit() only for commas; it doesn't allow just a Err codemadness.org 70 i 8189 space-separated list. Then, it uses g_ascii_strtod() to parse Err codemadness.org 70 i 8190 plain numbers; it doesn't support CSS lengths generically. This Err codemadness.org 70 i 8191 parser needs to be rewritten in Rust; we already have machinery Err codemadness.org 70 i 8192 there to parse CSS length values properly.
Err codemadness.org 70 i 8193

Err codemadness.org 70 i 8195

Feel free to contact me by mail, or write something in the Err codemadness.org 70 i 8196 bugs themselves, if you would like to work on them. I'll happily Err codemadness.org 70 i 8197 guide you through the code :)

Librsvg gets Continuous Integration

2018-01-12T14:04:20-06:00

One nice thing about gitlab.gnome.org is that we can now have Err codemadness.org 70 i 8203 Continuous Integration (CI) enabled for projects there. After every Err codemadness.org 70 i 8204 commit, the CI machinery can build the project, run the tests, and Err codemadness.org 70 i 8205 tell you if something goes wrong.

Err codemadness.org 70 i 8206

Carlos Soriano posted a "tips of the Err codemadness.org 70 i 8207 week" mail to desktop-devel-list, and a link to how Nautilus Err codemadness.org 70 i 8208 implements CI in Gitlab. It turns out that it's reasonably easy to Err codemadness.org 70 i 8209 set up: you just create a .gitlab-ci.yml file in the Err codemadness.org 70 i 8210 toplevel of your project, and that has the configuration for what to Err codemadness.org 70 i 8211 run on every commit.

Err codemadness.org 70 i 8212

Of course instead of reading the manual, I copied-and-pasted the file Err codemadness.org 70 i 8213 from Nautilus and just changed some things in it. There is a .yml Err codemadness.org 70 i 8214 linter so you can at least check the syntax before pushing a Err codemadness.org 70 i 8215 full job.

Err codemadness.org 70 i 8216

Then I read Robert Ancell's reply about how simple-scan Err codemadness.org 70 i 8217 builds its CI jobs on both Fedora and Ubuntu... and then the Err codemadness.org 70 i 8218 realization hit me:

Err codemadness.org 70 i 8219

This lets me CI librsvg on multiple distros at once. I've had Err codemadness.org 70 i 8220 trouble with slight differences in fontconfig/freetype in the past, Err codemadness.org 70 i 8221 and this would let me catch them early.

Err codemadness.org 70 i 8222

However, people on IRC advised against this, as we need more Err codemadness.org 70 i 8223 hardware to run CI on a large scale.

Err codemadness.org 70 i 8224

Linux distros have a vested interest in getting code out of gnome.org Err codemadness.org 70 i 8225 that works well. Surely they can give us some hardware?

Loving Gitlab.gnome.org, and getting notifications

2018-01-08T11:45:01-06:00

I'm loving gitlab.gnome.org. It has been only a couple of Err codemadness.org 70 i 8231 weeks since librsvg moved to gitlab, and I've Err codemadness.org 70 i 8232 already received and merged two merge requests. (Isn't it a bit Err codemadness.org 70 i 8233 weird that Github uses "pull request" and Everyone(tm) knows the PR Err codemadness.org 70 i 8234 acronym, but Gitlab uses "merge request"?)

Err codemadness.org 70 i 8235

Notifications about merge requests

Err codemadness.org 70 i 8236

One thing to note if your GNOME project has moved to Gitlab: if you Err codemadness.org 70 i 8237 want to get notified of incoming merge requests, you need Err codemadness.org 70 i 8238 to tell Gitlab that you want to "Watch" that project, instead of Err codemadness.org 70 i 8239 using one of the default notification settings. Thanks to Carlos Err codemadness.org 70 i 8240 Soriano for making me aware of this.

Err codemadness.org 70 i 8241

Notifications from Github's mirror

Err codemadness.org 70 i 8242

The github mirror of git.gnome.org is configured so that pull Err codemadness.org 70 i 8243 requests are automatically closed, since currently there Err codemadness.org 70 i 8244 is no way to notify the upstream maintainers when someone creates a Err codemadness.org 70 i 8245 pull request in the mirror (this is super-unfriendly by default, but Err codemadness.org 70 i 8246 at least submitters get notified that their PR would not be looked at Err codemadness.org 70 i 8247 by anyone, by default).

Err codemadness.org 70 i 8248

If you have a Github account, you can Watch the project in question to Err codemadness.org 70 i 8249 get notified — the bot will close the pull request, but you will get Err codemadness.org 70 i 8250 notified, and then you can check it by hand, review it as appropriate, Err codemadness.org 70 i 8251 or redirect the submitter to gitlab.gnome.org instead.

Librsvg 2.40.20 is released

2017-12-15T18:31:50-06:00

Today I released librsvg 2.40.20. This will be the Err codemadness.org 70 i 8257 last release in the 2.40.x series, which is deprecated effectively Err codemadness.org 70 i 8258 immediately.

Err codemadness.org 70 i 8259

People and distros are strongly encouraged to switch to Err codemadness.org 70 i 8260 librsvg 2.41.x as soon as possible. This is the version that is Err codemadness.org 70 i 8261 implemented in a mixture of C and Rust. It is 100% API and ABI Err codemadness.org 70 i 8262 compatible with 2.40.x, so it is a drop-in replacement for it. If you Err codemadness.org 70 i 8263 or your distro can compile Firefox 57, you can probably build Err codemadness.org 70 i 8264 librsvg-2.41.x without problems.

Err codemadness.org 70 i 8265

Some statistics

Err codemadness.org 70 i 8266

Here are a few runs of loc — a tool to count lines of code — Err codemadness.org 70 i 8267 when run on librsvg. The output is trimmed by hand to only include C Err codemadness.org 70 i 8268 and Rust files.

Err codemadness.org 70 i 8269

This is 2.40.20:	Err	codemadness.org	70
i  8270 -------------------------------------------------------	Err	codemadness.org	70
i  8271  Language      Files   Lines   Blank   Comment    Code	Err	codemadness.org	70
i  8272 -------------------------------------------------------	Err	codemadness.org	70
i  8273  C                41   20972    3438      2100   15434	Err	codemadness.org	70
i  8274  C/C++ Header     27    2377     452       625    1300	Err	codemadness.org	70
i  8275

Err codemadness.org 70 i 8276 Err codemadness.org 70 i 8277

This is 2.41.latest (the master branch):	Err	codemadness.org	70
i  8278 -------------------------------------------------------	Err	codemadness.org	70
i  8279  Language      Files   Lines   Blank   Comment    Code	Err	codemadness.org	70
i  8280 -------------------------------------------------------	Err	codemadness.org	70
i  8281  C                34   17253    3024      1892   12337	Err	codemadness.org	70
i  8282  C/C++ Header     23    2327     501       624    1202	Err	codemadness.org	70
i  8283  Rust             38   11254    1873       675    8706	Err	codemadness.org	70
i  8284

Err codemadness.org 70 i 8285 Err codemadness.org 70 i 8286

And this is 2.41.latest *without unit tests*, 	Err	codemadness.org	70
i  8287 just "real source code":	Err	codemadness.org	70
i  8288 -------------------------------------------------------	Err	codemadness.org	70
i  8289  Language      Files   Lines   Blank   Comment    Code	Err	codemadness.org	70
i  8290 -------------------------------------------------------	Err	codemadness.org	70
i  8291  C                34   17253    3024      1892   12337	Err	codemadness.org	70
i  8292  C/C++ Header     23    2327     501       624    1202	Err	codemadness.org	70
i  8293  Rust             38    9340    1513       610    7217	Err	codemadness.org	70
i  8294

Err codemadness.org 70 i 8295 Err codemadness.org 70 i 8296

Summary

Err codemadness.org 70 i 8297

Not counting blank lines nor comments:

Err codemadness.org 70 i 8298

Err codemadness.org 70 i 8300
The C-only version has 16734 lines of C code.
Err codemadness.org 70 i 8301
Err codemadness.org 70 i 8303
The C-only version has no unit tests, just some integration tests.
Err codemadness.org 70 i 8304
Err codemadness.org 70 i 8306
The Rust-and-C version has 13539 lines of C code, 7217 lines of Rust Err codemadness.org 70 i 8307 code, and 1489 lines of unit tests in Rust.
Err codemadness.org 70 i 8308

Err codemadness.org 70 i 8310

As for the integration tests:

Err codemadness.org 70 i 8311

Err codemadness.org 70 i 8313
The C-only version has 64 integration tests.
Err codemadness.org 70 i 8314
Err codemadness.org 70 i 8316
The Rust-and-C version has 130 integration tests.
Err codemadness.org 70 i 8317

Err codemadness.org 70 i 8319

The Rust-and-C version supports a few more SVG features, and it is A Err codemadness.org 70 i 8320 LOT more robust and spec-compliant with the SVG features that were Err codemadness.org 70 i 8321 supported in the C-only version.

Err codemadness.org 70 i 8322

The C sources in librsvg are shrinking steadily. It would be Err codemadness.org 70 i 8323 incredibly awesome if someone could run some git filter-branch magic Err codemadness.org 70 i 8324 with the loc tool and generate some pretty graphs of source Err codemadness.org 70 i 8325 lines vs. commits over time.

Librsvg moves to Gitlab

2017-12-13T14:09:32-06:00

Librsvg now lives in GNOME's Gitlab instance. You can Err codemadness.org 70 i 8331 access it here.

Err codemadness.org 70 i 8332

Gitlab allows workflows similar to Github: you can create an account Err codemadness.org 70 i 8333 there, fork the librsvg repository, file bug reports, create merge Err codemadness.org 70 i 8334 requests... Hopefully this will make it nicer for contributors.

Err codemadness.org 70 i 8335

In the meantime, feel free to take a look!

Err codemadness.org 70 i 8336

This is a huge improvement for GNOME's development infrastructure. Err codemadness.org 70 i 8337 Thanks to Carlos Soriano, Andrea Veri, Philip Chimento, Alberto Ruiz, Err codemadness.org 70 i 8338 and all the people that made the move to Gitlab possible.

A mini-rant on the lack of string slices in C

2017-12-07T09:54:14-06:00

Porting of librsvg to Rust goes on. Yesterday I started porting the Err codemadness.org 70 i 8344 C code that implements SVG's <text> family of elements. I have also Err codemadness.org 70 i 8345 been replacing the little parsers in librsvg with Rust code.

Err codemadness.org 70 i 8346

And these days, the lack of string slices in C is bothering me a Err codemadness.org 70 i 8347 lot.

Err codemadness.org 70 i 8348

What if...

Err codemadness.org 70 i 8349

It feels like it should be easy to just write something like

Err codemadness.org 70 i 8350

typedef struct {	Err	codemadness.org	70
i  8351     const char *ptr;	Err	codemadness.org	70
i  8352     size_t len;	Err	codemadness.org	70
i  8353 } StringSlice;	Err	codemadness.org	70
i  8354

Err codemadness.org 70 i 8355 Err codemadness.org 70 i 8356

And then a whole family of functions. The starting point, where you Err codemadness.org 70 i 8357 slice a whole string:

Err codemadness.org 70 i 8358

StringSlice	Err	codemadness.org	70
i  8359 make_slice_from_string (const char *s)	Err	codemadness.org	70
i  8360 {	Err	codemadness.org	70
i  8361     StringSlice slice;	Err	codemadness.org	70
i  8362 	Err	codemadness.org	70
i  8363     assert (s != NULL);	Err	codemadness.org	70
i  8364 	Err	codemadness.org	70
i  8365     slice.ptr = s;	Err	codemadness.org	70
i  8366     slice.len = strlen (s);	Err	codemadness.org	70
i  8367     return slice;	Err	codemadness.org	70
i  8368 }	Err	codemadness.org	70
i  8369

Err codemadness.org 70 i 8370 Err codemadness.org 70 i 8371

But that wouldn't keep track of the lifetime of the original string. Err codemadness.org 70 i 8372 Okay, this is C, so you are used to keeping track of that yourself.

Err codemadness.org 70 i 8373

Onwards. Substrings?

Err codemadness.org 70 i 8374

StringSlice	Err	codemadness.org	70
i  8375 make_sub_slice(StringSlice slice, size_t start, size_t len)	Err	codemadness.org	70
i  8376 {	Err	codemadness.org	70
i  8377     StringSlice sub;	Err	codemadness.org	70
i  8378 	Err	codemadness.org	70
i  8379     assert (len <= slice.len);	Err	codemadness.org	70
i  8380     assert (start <= slice.len - len);  /* Not "start + len <= slice.len" or it can overflow. */	Err	codemadness.org	70
i  8381                                         /* The subtraction can't underflow because of the previous assert */	Err	codemadness.org	70
i  8382     sub.ptr = slice.ptr + start;	Err	codemadness.org	70
i  8383     sub.len = len;	Err	codemadness.org	70
i  8384     return sub;	Err	codemadness.org	70
i  8385 }	Err	codemadness.org	70
i  8386

Err codemadness.org 70 i 8387 Err codemadness.org 70 i 8388

Then you could write a million wrappers for g_strsplit() and Err codemadness.org 70 i 8389 friends, or equivalents to them, to give you slices instead of C Err codemadness.org 70 i 8390 strings. But then:

Err codemadness.org 70 i 8391

Err codemadness.org 70 i 8393
You have to keep track of lifetimes yourself.
Err codemadness.org 70 i 8394
Err codemadness.org 70 i 8396
You have to wrap every function that returns a plain "char *"...
Err codemadness.org 70 i 8397
Err codemadness.org 70 i 8399
... and every function that takes a plain "char *" as an argument, Err codemadness.org 70 i 8400 without a length parameter, because...
Err codemadness.org 70 i 8401
Err codemadness.org 70 i 8403
You CANNOT take slice.ptr and pass it to a function that just Err codemadness.org 70 i 8404 expects a plain "char *", because your slice does not include a Err codemadness.org 70 i 8405 nul terminator (the '\0 byte at the end of a C string). This is Err codemadness.org 70 i 8406 what kills the whole plan.
Err codemadness.org 70 i 8407

Err codemadness.org 70 i 8409

Even if you had a helper library that implements C string slices Err codemadness.org 70 i 8410 like that, you would have a mismatch every time you needed to call a C Err codemadness.org 70 i 8411 function that expects a conventional C string in the form of a Err codemadness.org 70 i 8412 "char *". You need to put a nul terminator somewhere, and if you Err codemadness.org 70 i 8413 only have a slice, you need to allocate memory, copy the slice into Err codemadness.org 70 i 8414 it, and slap a 0 byte at the end. Then you can pass that to a Err codemadness.org 70 i 8415 function that expects a normal C string.

Err codemadness.org 70 i 8416

There is hacky C code that needs to pass a substring to another Err codemadness.org 70 i 8417 function, so it overwrites the byte after the substring with a 0, Err codemadness.org 70 i 8418 passes the substring, and overwrites the byte back. This is Err codemadness.org 70 i 8419 horrible, and doesn't work with strings that live in read-only Err codemadness.org 70 i 8420 memory. But that's the best that C lets you do.

Err codemadness.org 70 i 8421

I'm very happy with string slices in Rust, which work exactly like the Err codemadness.org 70 i 8422 StringSlice above, but &str is actually at the language level and Err codemadness.org 70 i 8423 everything knows how to handle it.

Err codemadness.org 70 i 8424

The glib-rs crate has conversion traits to go from Rust strings or Err codemadness.org 70 i 8425 slices into C, and vice-versa. We alredy saw some of those in the Err codemadness.org 70 i 8426 blog post about conversions in Glib-rs.

Err codemadness.org 70 i 8427

Sizes of things

Err codemadness.org 70 i 8428

Rust uses usize to specify the size of things; it's an unsigned Err codemadness.org 70 i 8429 integer; 32 bits on 32-bit machines, and 64 bits on 64-bit machines; Err codemadness.org 70 i 8430 it's like C's size_t.

Err codemadness.org 70 i 8431

In the Glib/C world, we have an assortment of types to represent the Err codemadness.org 70 i 8432 sizes of things:

Err codemadness.org 70 i 8433

Err codemadness.org 70 i 8435
gsize, the same as size_t. This is an unsigned integer; it's okay.
Err codemadness.org 70 i 8436
Err codemadness.org 70 i 8438
gssize, a signed integer of the same size as gsize. This is Err codemadness.org 70 i 8439 okay if used to represent a negative offset, and really funky in Err codemadness.org 70 i 8440 the Glib functions like Err codemadness.org 70 i 8441 g_string_new_len (const char *str, gssize len), where len == -1 Err codemadness.org 70 i 8442 means "call strlen(str) for me because I'm too lazy to compute the Err codemadness.org 70 i 8443 length myself".
Err codemadness.org 70 i 8444
Err codemadness.org 70 i 8446
int - broken, as in libxml2, but we can't change the API. On Err codemadness.org 70 i 8447 64-bit machines, an int to specify a length means you can't pass Err codemadness.org 70 i 8448 objects bigger than 2 GB.
Err codemadness.org 70 i 8449
Err codemadness.org 70 i 8451
long - marginally better than int, since it has a better chance Err codemadness.org 70 i 8452 of actually being the same size as size_t, but still funky. Err codemadness.org 70 i 8453 Probably okay for negative offsets; problematic for sizes which Err codemadness.org 70 i 8454 should really be unsigned.
Err codemadness.org 70 i 8455
Err codemadness.org 70 i 8457
etc.
Err codemadness.org 70 i 8458

Err codemadness.org 70 i 8460

I'm not sure how old size_t is in the C standard library, but it Err codemadness.org 70 i 8461 can't have been there since the beginning of time — otherwise Err codemadness.org 70 i 8462 people wouldn't have been using int to specify the sizes of things.

Code Hospitality

2017-11-17T14:21:04-06:00

Recently on the Greater than Code podcast there was an episode Err codemadness.org 70 i 8468 called "Code Hospitality", by Err codemadness.org 70 i 8469 Nadia Odunayo.

Err codemadness.org 70 i 8470

Nadia talks about thinking of how to make people comfortable in your Err codemadness.org 70 i 8471 code and in your team/organization/etc., and does it in terms of Err codemadness.org 70 i 8472 thinking about host/guest relationships. Have you ever stayed in an Err codemadness.org 70 i 8473 AirBnB where the host carefully prepares some "welcome instructions" Err codemadness.org 70 i 8474 for you, or puts little notes in their apartment to orient/guide you, Err codemadness.org 70 i 8475 or gives you basic guidance around their city's transportation system? Err codemadness.org 70 i 8476 We can think in similar ways of how to make people comfortable with Err codemadness.org 70 i 8477 code bases.

Err codemadness.org 70 i 8478

This of course hit me on so many levels, because in the past I've Err codemadness.org 70 i 8479 written about analogies between software and urbanism/architecture. Err codemadness.org 70 i 8480 Software that has the Quality Without A Name Err codemadness.org 70 i 8481 talks about Christopher Alexander's architecture/urbanism patterns in Err codemadness.org 70 i 8482 the context of software, based on Richard Gabriel's ideas, Err codemadness.org 70 i 8483 and Nikos Salingaros's formalization of the design Err codemadness.org 70 i 8484 process. Legacy Systems as Old Cities talks about Err codemadness.org 70 i 8485 how GNOME evolved parts of its user-visible software, and makes an Err codemadness.org 70 i 8486 analogy with cities that evolve over time instead of being torn down Err codemadness.org 70 i 8487 and rebuilt, based on urbanism ideas by Jane Jacobs, and Err codemadness.org 70 i 8488 architecture/construction ideas by Stewart Brand.

Err codemadness.org 70 i 8489

I definitely intend to do some thinking on Nadia's ideas for Code Err codemadness.org 70 i 8490 Hospitality and try to connect them with this.

Err codemadness.org 70 i 8491

In the meantime, I've just rewritten the README in Err codemadness.org 70 i 8492 gnome-class to make it suitable as an Err codemadness.org 70 i 8493 introduction to hacking there.

Rust+GNOME Hackfest in Berlin, 2017

2017-11-16T18:33:58-06:00

Last weekend I was in Berlin for the second Rust+GNOME Err codemadness.org 70 i 8499 Hackfest, kindly hosted at the Kinvolk office. Err codemadness.org 70 i 8500 This is in a great location, half a block away from the Kottbusser Err codemadness.org 70 i 8501 Tor station, right at the entrance of the trendy Kreuzberg Err codemadness.org 70 i 8502 neighborhood — full of interesting people, incredible graffitti, and Err codemadness.org 70 i 8503 good, diverse food.

Err codemadness.org 70 i 8504

Err codemadness.org 70 i 8505

My goals for the hackfest

Err codemadness.org 70 i 8506

Over the past weeks I had been converting gnome-class Err codemadness.org 70 i 8507 from the old lalrpop-based parser into the new Procedural Err codemadness.org 70 i 8508 Macros framework for Rust, or proc-macro2 for short. To do this the Err codemadness.org 70 i 8509 parser for the gnome-class mini-language needs to be rewritten from Err codemadness.org 70 i 8510 being specified in a lalrpop grammar, to using Rust's syn Err codemadness.org 70 i 8511 crate.

Err codemadness.org 70 i 8512

Syn is a parser for Rust source code, written as a set of nom Err codemadness.org 70 i 8513 combinator parser macros. For gnome-class we want to extend the Rust Err codemadness.org 70 i 8514 language with a few conveniences to be able to specify GObject Err codemadness.org 70 i 8515 classes/subclasses, methods, signals, properties, interfaces, and all Err codemadness.org 70 i 8516 the goodies that GObject Introspection would expect.

Err codemadness.org 70 i 8517

During the hackfest, Alex Crichton, from the Rust core team, Err codemadness.org 70 i 8518 kindly took over my baby steps in compiler writing and made everything Err codemadness.org 70 i 8519 much more functional. It was invaluable to have him there to reason Err codemadness.org 70 i 8520 about macro hygiene (we are generating an unhygienic macro!), bugs Err codemadness.org 70 i 8521 in the quoting system, and general Rust-iness of the whole thing.

Err codemadness.org 70 i 8522

I was also able to talk to Sebastian Dröge about his work in Err codemadness.org 70 i 8523 writing GObjects in Rust by hand, for GStreamer, and what sort of Err codemadness.org 70 i 8524 things gnome-class could make easier. Sebastian knows GObject very Err codemadness.org 70 i 8525 well, and has been doing awesome work in making it easy to derive Err codemadness.org 70 i 8526 GObjects by hand in Rust, without lots of boilerplate — something with Err codemadness.org 70 i 8527 which gnome-class can certainly help.

Err codemadness.org 70 i 8528

I was also looking forward to talking again with Guillaume Err codemadness.org 70 i 8529 Gomez, one of the maintainers of gtk-rs, and who Err codemadness.org 70 i 8530 does so much work in the Rust ecosystem that I can't believe he has Err codemadness.org 70 i 8531 time for it all.

Err codemadness.org 70 i 8532

Err codemadness.org 70 i 8533

Extend the Rust language for GObject? Like Vala?

Err codemadness.org 70 i 8534

Yeah, pretty much.

Err codemadness.org 70 i 8535

Except that instead of a wholly new language, we use Rust as-is, and Err codemadness.org 70 i 8536 we just add syntactic constructs that make it easy to write GObjects Err codemadness.org 70 i 8537 without boilerplate. For example, this works right now:

Err codemadness.org 70 i 8538

#![feature(proc_macro)]	Err	codemadness.org	70
i  8539 	Err	codemadness.org	70
i  8540 extern crate gobject_gen;	Err	codemadness.org	70
i  8541 	Err	codemadness.org	70
i  8542 #[macro_use]	Err	codemadness.org	70
i  8543 extern crate glib;	Err	codemadness.org	70
i  8544 use gobject_gen::gobject_gen;	Err	codemadness.org	70
i  8545 	Err	codemadness.org	70
i  8546 gobject_gen! {	Err	codemadness.org	70
i  8547     // Derives from GObject	Err	codemadness.org	70
i  8548     class One {	Err	codemadness.org	70
i  8549     }	Err	codemadness.org	70
i  8550 	Err	codemadness.org	70
i  8551     impl One {	Err	codemadness.org	70
i  8552         // non-virtual method	Err	codemadness.org	70
i  8553         pub fn one(&self) -> u32 {	Err	codemadness.org	70
i  8554             1	Err	codemadness.org	70
i  8555         }	Err	codemadness.org	70
i  8556 	Err	codemadness.org	70
i  8557         virtual fn get(&self) -> u32 {	Err	codemadness.org	70
i  8558             1	Err	codemadness.org	70
i  8559         }	Err	codemadness.org	70
i  8560     }	Err	codemadness.org	70
i  8561 	Err	codemadness.org	70
i  8562     // Inherits from our other class	Err	codemadness.org	70
i  8563     class Two: One {	Err	codemadness.org	70
i  8564     }	Err	codemadness.org	70
i  8565 	Err	codemadness.org	70
i  8566     impl One for Two {	Err	codemadness.org	70
i  8567         // overrides the virtual method	Err	codemadness.org	70
i  8568         // maybe we should use "override" instead of "virtual" here?	Err	codemadness.org	70
i  8569         virtual fn get(&self) -> u32 {	Err	codemadness.org	70
i  8570             2	Err	codemadness.org	70
i  8571         }	Err	codemadness.org	70
i  8572     }	Err	codemadness.org	70
i  8573 }	Err	codemadness.org	70
i  8574 	Err	codemadness.org	70
i  8575 #[test]	Err	codemadness.org	70
i  8576 fn test() {	Err	codemadness.org	70
i  8577     let one = One::new();	Err	codemadness.org	70
i  8578     let two = Two::new();	Err	codemadness.org	70
i  8579 	Err	codemadness.org	70
i  8580     assert!(one.one() == 1);	Err	codemadness.org	70
i  8581     assert!(one.get() == 1);	Err	codemadness.org	70
i  8582     assert!(two.one() == 1);	Err	codemadness.org	70
i  8583     assert!(two.get() == 2);	Err	codemadness.org	70
i  8584 }	Err	codemadness.org	70
i  8585

Err codemadness.org 70 i 8586 Err codemadness.org 70 i 8587

This generates a little boatload of generated code, Err codemadness.org 70 i 8588 including a good number of unsafe calls to GObject functions Err codemadness.org 70 i 8589 like g_type_register_static_simple(). It also creates all the Err codemadness.org 70 i 8590 traits and paraphernalia that Glib-rs would create for the Rust Err codemadness.org 70 i 8591 binding of a normal GObject written in C.

Err codemadness.org 70 i 8592

The idea is that from the outside world, your generated GObject Err codemadness.org 70 i 8593 classes are indistinguishable from GObjects implemented in C.

Err codemadness.org 70 i 8594

The idea is to write GObject libraries in a better language than C, Err codemadness.org 70 i 8595 which can then be consumed from language bindings.

Err codemadness.org 70 i 8596

Current status of gnome-class

Err codemadness.org 70 i 8597

Up to about two weeks before the hackfest, the syntax for this Err codemadness.org 70 i 8598 mini-language was totally ad-hoc and limited. After a very productive Err codemadness.org 70 i 8599 discussion on the mailing list, we came up with a better Err codemadness.org 70 i 8600 syntax that definitely looks more Rust-like. It is also easier to Err codemadness.org 70 i 8601 implement, since the Rust parser in syn can be mostly reused as-is, or Err codemadness.org 70 i 8602 pruned down for the parts where we only support GObject-like methods, Err codemadness.org 70 i 8603 and not all the Rust bells and whistles (generics, lifetimes, trait Err codemadness.org 70 i 8604 bounds).

Err codemadness.org 70 i 8605

Gnome-class supports deriving classes directly from the basic GObject, Err codemadness.org 70 i 8606 or from other GObject subclasses in the style of glib-rs.

Err codemadness.org 70 i 8607

You can define virtual and non-virtual methods. You can override Err codemadness.org 70 i 8608 virtual methods from your superclasses.

Err codemadness.org 70 i 8609

Not all argument types are supported. In the end we should support Err codemadness.org 70 i 8610 argument types which are convertible from Rust to C types. We need to Err codemadness.org 70 i 8611 finish figuring out the annotations for ownership transfer of Err codemadness.org 70 i 8612 references.

Err codemadness.org 70 i 8613

We don't support GObject signals yet; I think that's my next task.

Err codemadness.org 70 i 8614

We don't support GObject properties yet.

Err codemadness.org 70 i 8615

We don't support defining new GType interfaces yet, but it is planned. Err codemadness.org 70 i 8616 It should be easy to support implementing existing interfaces, as it Err codemadness.org 70 i 8617 is pretty much the same as implementing a subclass.

Err codemadness.org 70 i 8618

The best way to see what works right now is probably to look at the Err codemadness.org 70 i 8619 examples, which also work as tests.

Err codemadness.org 70 i 8620

Digression on macro hygiene

Err codemadness.org 70 i 8621

Rust macros are hygienic, unlike C macros which work just through Err codemadness.org 70 i 8622 textual substitution. That is, names declared inside Rust macros will Err codemadness.org 70 i 8623 not clash with names in the calling code.

Err codemadness.org 70 i 8624

One peculiar thing about gnome-class is that the user gives us a few Err codemadness.org 70 i 8625 names, like a class name Foo and some things inside it, say, a Err codemadness.org 70 i 8626 method name bar, and a signal baz and a property qux. Err codemadness.org 70 i 8627 From there we want to generate a bunch of boilerplate for GObject Err codemadness.org 70 i 8628 registration and implementaiton. Some of the generated names in that Err codemadness.org 70 i 8629 boilerplate would be

Err codemadness.org 70 i 8630

Foo              // base name	Err	codemadness.org	70
i  8631 FooClass         // generated name for the class struct	Err	codemadness.org	70
i  8632 Foo::bar()       // A method	Err	codemadness.org	70
i  8633 Foo::emit_baz()  // Generated from the signal name	Err	codemadness.org	70
i  8634 Foo::set_qux()   // Generated property setter	Err	codemadness.org	70
i  8635 foo_bar()        // Generated C function for a method call	Err	codemadness.org	70
i  8636 foo_get_type()   // Generated C function that all GObjects have	Err	codemadness.org	70
i  8637

Err codemadness.org 70 i 8638 Err codemadness.org 70 i 8639

However, if we want to actually generate those names inside our Err codemadness.org 70 i 8640 gnome-class macro and make them visible to the caller, we need to do Err codemadness.org 70 i 8641 so unhygienically. Alex started started a very interesting discussion Err codemadness.org 70 i 8642 on macro hygiene, so expect some news in the Rust world Err codemadness.org 70 i 8643 soon.

Err codemadness.org 70 i 8644

TL;DR: there is a difference between a code generator, which Err codemadness.org 70 i 8645 gnome-class mostly intends to be, and a macro system which is just Err codemadness.org 70 i 8646 an aid in typing repetitive code.

Err codemadness.org 70 i 8647

Err codemadness.org 70 i 8648

People for whom to to be thankful

Err codemadness.org 70 i 8649

During the hackfest, Nirbheek has been porting librsvg Err codemadness.org 70 i 8650 from Autotools to the Meson build system, and dealing with Rust Err codemadness.org 70 i 8651 peculiarities along the way. This is exactly what I needed! Thanks, Err codemadness.org 70 i 8652 Nirbheek!

Err codemadness.org 70 i 8653

Sebastian answered many of my questions about GObject Err codemadness.org 70 i 8654 internals and how to use them from the Rust side.

Err codemadness.org 70 i 8655

Zeeshan took us to a bunch of good restaurants. Korean, Err codemadness.org 70 i 8656 ramen, Greek, excellent pizza... My stomach is definitely thankful.

Err codemadness.org 70 i 8657

Berlin

Err codemadness.org 70 i 8658

I love Berlin. It is a cosmopolitan, progressive, LGBTQ-friendly Err codemadness.org 70 i 8659 city, with lots of things to do, vast distances to be traveled, with Err codemadness.org 70 i 8660 good public transport and bike lanes, diverse food to be eaten along Err codemadness.org 70 i 8661 the way...

Err codemadness.org 70 i 8662

But damnit, it's also cold at this time of the year. I don't think Err codemadness.org 70 i 8663 the weather was ever above 10°C while we were there, and mostly in a Err codemadness.org 70 i 8664 constant state of not-quite-rain. This is much different from the Berlin Err codemadness.org 70 i 8665 in the summer that I knew!

Err codemadness.org 70 i 8666

Err codemadness.org 70 i 8667

This is my third time visiting Berlin. The first one was during the Err codemadness.org 70 i 8668 Desktop Summit in 2011, and the second one was when my family and I Err codemadness.org 70 i 8669 visited the city two years ago. It is a city that I would definitely Err codemadness.org 70 i 8670 like to know better.

Err codemadness.org 70 i 8671

Thanks to the GNOME Foundation...

Err codemadness.org 70 i 8672

... for sponsoring my travel and accomodation during the hackfest.

Err codemadness.org 70 i 8673

Compilation notifications in Emacs

2017-11-07T10:47:52-06:00

Here is a little Emacs Lisp snippet that I've started using. It makes Err codemadness.org 70 i 8679 Emacs pop up a desktop-wide notification when a compilation finishes, Err codemadness.org 70 i 8680 i.e. after "M-x compile" is done. Let's see if that keeps me from Err codemadness.org 70 i 8681 wasting time in the web when I launch a compilation.

Err codemadness.org 70 i 8682

(setq compilation-finish-functions	Err	codemadness.org	70
i  8683       (append compilation-finish-functions	Err	codemadness.org	70
i  8684           '(fmq-compilation-finish)))	Err	codemadness.org	70
i  8685 	Err	codemadness.org	70
i  8686 (defun fmq-compilation-finish (buffer status)	Err	codemadness.org	70
i  8687   (call-process "notify-send" nil nil nil	Err	codemadness.org	70
i  8688         "-t" "0"	Err	codemadness.org	70
i  8689         "-i" "emacs"	Err	codemadness.org	70
i  8690         "Compilation finished in Emacs"	Err	codemadness.org	70
i  8691         status))	Err	codemadness.org	70
i  8692

How glib-rs works, part 3: Boxed types

2017-09-08T19:28:11-05:00

(First part of the series, with index to all the articles)

Err codemadness.org 70 i 8697

Now let's get on and see how glib-rs handles boxed types.

Err codemadness.org 70 i 8698

Boxed types?

Err codemadness.org 70 i 8699

Let's say you are given a sealed cardboard box with something, but you Err codemadness.org 70 i 8700 can't know what's inside. You can just pass it on to someone else, or Err codemadness.org 70 i 8701 burn it. And since computers are magic duplication machines, you may Err codemadness.org 70 i 8702 want to copy the box and its contents... and maybe some day you will Err codemadness.org 70 i 8703 get around to opening it.

Err codemadness.org 70 i 8704

That's a boxed type. You get a pointer to something, who knows Err codemadness.org 70 i 8705 what's inside. You can just pass it on to someone else, burn it — I Err codemadness.org 70 i 8706 mean, free it — or since computers are magic, copy the pointer and Err codemadness.org 70 i 8707 whatever it points to.

Err codemadness.org 70 i 8708

That's exactly the API for boxed types.

Err codemadness.org 70 i 8709

typedef gpointer (*GBoxedCopyFunc) (gpointer boxed);	Err	codemadness.org	70
i  8710 typedef void (*GBoxedFreeFunc) (gpointer boxed);	Err	codemadness.org	70
i  8711 	Err	codemadness.org	70
i  8712 GType g_boxed_type_register_static (const gchar   *name,	Err	codemadness.org	70
i  8713                                     GBoxedCopyFunc boxed_copy,	Err	codemadness.org	70
i  8714                                     GBoxedFreeFunc boxed_free);	Err	codemadness.org	70
i  8715

Err codemadness.org 70 i 8716 Err codemadness.org 70 i 8717

Simple copying, simple freeing

Err codemadness.org 70 i 8718

Imagine you have a color...

Err codemadness.org 70 i 8719

typedef struct {	Err	codemadness.org	70
i  8720     guchar r;	Err	codemadness.org	70
i  8721     guchar g;	Err	codemadness.org	70
i  8722     guchar b;	Err	codemadness.org	70
i  8723 } Color;	Err	codemadness.org	70
i  8724

Err codemadness.org 70 i 8725 Err codemadness.org 70 i 8726

If you had a pointer to a Color, how would you copy it? Easy:

Err codemadness.org 70 i 8727

Color *copy_color (Color *a)	Err	codemadness.org	70
i  8728 {	Err	codemadness.org	70
i  8729     Color *b = g_new (Color, 1);	Err	codemadness.org	70
i  8730     *b = *a;	Err	codemadness.org	70
i  8731     return b;	Err	codemadness.org	70
i  8732 }	Err	codemadness.org	70
i  8733

Err codemadness.org 70 i 8734 Err codemadness.org 70 i 8735

That is, allocate a new Color, and essentially memcpy() the Err codemadness.org 70 i 8736 contents.

Err codemadness.org 70 i 8737

And to free it? A simple g_free() works — there are no internal Err codemadness.org 70 i 8738 things that need to be freed individually.

Err codemadness.org 70 i 8739

Complex copying, complex freeing

Err codemadness.org 70 i 8740

And if we had a color with a name?

Err codemadness.org 70 i 8741

typedef struct {	Err	codemadness.org	70
i  8742     guchar r;	Err	codemadness.org	70
i  8743     guchar g;	Err	codemadness.org	70
i  8744     guchar b;	Err	codemadness.org	70
i  8745     char *name;	Err	codemadness.org	70
i  8746 } ColorWithName;	Err	codemadness.org	70
i  8747

Err codemadness.org 70 i 8748 Err codemadness.org 70 i 8749

We can't just *a = *b here, as we actually need to copy the string Err codemadness.org 70 i 8750 name. Okay:

Err codemadness.org 70 i 8751

ColorWithName *copy_color_with_name (ColorWithName *a)	Err	codemadness.org	70
i  8752 {	Err	codemadness.org	70
i  8753     ColorWithName *b = g_new (ColorWithName, 1);	Err	codemadness.org	70
i  8754     b->r = a->r;	Err	codemadness.org	70
i  8755     b->g = a->g;	Err	codemadness.org	70
i  8756     b->b = a->b;	Err	codemadness.org	70
i  8757     b->name = g_strdup (a->name);	Err	codemadness.org	70
i  8758     return b;	Err	codemadness.org	70
i  8759 }	Err	codemadness.org	70
i  8760

Err codemadness.org 70 i 8761 Err codemadness.org 70 i 8762

The corresponding free_color_with_name() would g_free(b->name) and then Err codemadness.org 70 i 8763 g_free(b), of course.

Err codemadness.org 70 i 8764

Glib-rs and boxed types

Err codemadness.org 70 i 8765

Let's look at this by parts. First, a BoxedMemoryManager Err codemadness.org 70 i 8766 trait to define the basic API to manage the Err codemadness.org 70 i 8767 memory of boxed types. This is what defines the copy and free Err codemadness.org 70 i 8768 functions, like above.

Err codemadness.org 70 i 8769

pub trait BoxedMemoryManager<T>: 'static {	Err	codemadness.org	70
i  8770     unsafe fn copy(ptr: *const T) -> *mut T;	Err	codemadness.org	70
i  8771     unsafe fn free(ptr: *mut T);	Err	codemadness.org	70
i  8772 }	Err	codemadness.org	70
i  8773

Err codemadness.org 70 i 8774 Err codemadness.org 70 i 8775

Second, the actual representation of a Boxed type:

Err codemadness.org 70 i 8776

pub struct Boxed<T: 'static, MM: BoxedMemoryManager<T>> {	Err	codemadness.org	70
i  8777     inner: AnyBox<T>,	Err	codemadness.org	70
i  8778     _dummy: PhantomData<MM>,	Err	codemadness.org	70
i  8779 }	Err	codemadness.org	70
i  8780

Err codemadness.org 70 i 8781 Err codemadness.org 70 i 8782

This struct is generic over T, the actual type that we will be Err codemadness.org 70 i 8783 wrapping, and MM, something which must implement the Err codemadness.org 70 i 8784 BoxedMemoryManager trait.

Err codemadness.org 70 i 8785

Inside, it stores inner, an AnyBox, which we will see shortly. Err codemadness.org 70 i 8786 The _dummy: PhantomData<MM> is a Rust-ism to indicate that although this Err codemadness.org 70 i 8787 struct doesn't actually store a memory manager, it acts as if it does Err codemadness.org 70 i 8788 — it does not concern us here.

Err codemadness.org 70 i 8789

The actual representation of boxed data

Err codemadness.org 70 i 8790

Let's look at that AnyBox that is stored inside a Boxed:

Err codemadness.org 70 i 8791

enum AnyBox<T> {	Err	codemadness.org	70
i  8792     Native(Box<T>),	Err	codemadness.org	70
i  8793     ForeignOwned(*mut T),	Err	codemadness.org	70
i  8794     ForeignBorrowed(*mut T),	Err	codemadness.org	70
i  8795 }	Err	codemadness.org	70
i  8796

Err codemadness.org 70 i 8797 Err codemadness.org 70 i 8798

We have three cases:

Err codemadness.org 70 i 8799

Err codemadness.org 70 i 8801
Native(Box<T>) - this boxed value T comes from Rust itself, so we Err codemadness.org 70 i 8802 know everything about it!
Err codemadness.org 70 i 8803
Err codemadness.org 70 i 8805
ForeignOwned(*mut T) - this boxed value T came from the outside, but Err codemadness.org 70 i 8806 we own it now. We will have to free it when we are done with it.
Err codemadness.org 70 i 8807
Err codemadness.org 70 i 8809
ForeignBorrowed(*mut T) - this boxed value T came from the Err codemadness.org 70 i 8810 outside, but we are just borrowing it temporarily: we don't want to Err codemadness.org 70 i 8811 free it when we are done with it.
Err codemadness.org 70 i 8812

Err codemadness.org 70 i 8814

For example, if we look at the implementation of the Drop Err codemadness.org 70 i 8815 trait for the Boxed struct, we will indeed see that it Err codemadness.org 70 i 8816 calls the BoxedMemoryManager::free() only if we have a Err codemadness.org 70 i 8817 ForeignOwned value:

Err codemadness.org 70 i 8818

impl<T: 'static, MM: BoxedMemoryManager<T>> Drop for Boxed<T, MM> {	Err	codemadness.org	70
i  8819     fn drop(&mut self) {	Err	codemadness.org	70
i  8820         unsafe {	Err	codemadness.org	70
i  8821             if let AnyBox::ForeignOwned(ptr) = self.inner {	Err	codemadness.org	70
i  8822                 MM::free(ptr);	Err	codemadness.org	70
i  8823             }	Err	codemadness.org	70
i  8824         }	Err	codemadness.org	70
i  8825     }	Err	codemadness.org	70
i  8826 }	Err	codemadness.org	70
i  8827

Err codemadness.org 70 i 8828 Err codemadness.org 70 i 8829

If we had a Native(Box<T>) value, it means it came from Rust itself, Err codemadness.org 70 i 8830 and Rust knows how to Drop its own Box<T> (i.e. a chunk of memory Err codemadness.org 70 i 8831 allocated in the heap).

Err codemadness.org 70 i 8832

But for external resources, we must tell Rust how to manage them. Err codemadness.org 70 i 8833 Again: in the case where the Rust side owns the reference to the Err codemadness.org 70 i 8834 external boxed data, we have a ForeignOwned and Drop it by Err codemadness.org 70 i 8835 free()ing it; in the case where the Rust side is just borrowing the Err codemadness.org 70 i 8836 data temporarily, we have a ForeignBorrowed and don't touch it when Err codemadness.org 70 i 8837 we are done.

Err codemadness.org 70 i 8838

Copying

Err codemadness.org 70 i 8839

When do we have to copy a boxed value? For example, when we transfer Err codemadness.org 70 i 8840 from Rust to Glib with full transfer of ownership, i.e. the Err codemadness.org 70 i 8841 to_glib_full() pattern that we saw Err codemadness.org 70 i 8842 before. This is how that trait method is Err codemadness.org 70 i 8843 implemented for Boxed:

Err codemadness.org 70 i 8844

impl<'a, T: 'static, MM: BoxedMemoryManager<T>> ToGlibPtr<'a, *const T> for Boxed<T, MM> {	Err	codemadness.org	70
i  8845     fn to_glib_full(&self) -> *const T {	Err	codemadness.org	70
i  8846         use self::AnyBox::*;	Err	codemadness.org	70
i  8847         let ptr = match self.inner {	Err	codemadness.org	70
i  8848             Native(ref b) => &**b as *const T,	Err	codemadness.org	70
i  8849             ForeignOwned(p) | ForeignBorrowed(p) => p as *const T,	Err	codemadness.org	70
i  8850         };	Err	codemadness.org	70
i  8851         unsafe { MM::copy(ptr) }	Err	codemadness.org	70
i  8852     }	Err	codemadness.org	70
i  8853 }	Err	codemadness.org	70
i  8854

Err codemadness.org 70 i 8855 Err codemadness.org 70 i 8856

See the MM:copy(ptr) in the last line? That's where the copy Err codemadness.org 70 i 8857 happens. The lines above just get the appropriate pointer to the data Err codemadness.org 70 i 8858 data from the AnyBox and cast it.

Err codemadness.org 70 i 8859

There is extra boilerplate in boxed.rs which you can look at; it's Err codemadness.org 70 i 8860 mostly a bunch of trait implementations to copy the boxed data at the Err codemadness.org 70 i 8861 appropriate times (e.g. the FromGlibPtrNone trait), also an Err codemadness.org 70 i 8862 implementation of the Deref trait to get to the contents of a Boxed Err codemadness.org 70 i 8863 / AnyBox easily, etc. The trait implementations are there just to Err codemadness.org 70 i 8864 make it as convenient as possible to handle Boxed types.

Err codemadness.org 70 i 8865

Who implements BoxedMemoryManager?

Err codemadness.org 70 i 8866

Up to now, we have seen things like the implementation of Drop for Err codemadness.org 70 i 8867 Boxed, which uses BoxedMemoryManager::free(), and the Err codemadness.org 70 i 8868 implementation of ToGlibPtr which uses ::copy().

Err codemadness.org 70 i 8869

But those are just the trait's "abstract" methods, so to speak. What Err codemadness.org 70 i 8870 actually implements them?

Err codemadness.org 70 i 8871

Glib-rs has a general-purpose macro to wrap Glib types. It can wrap Err codemadness.org 70 i 8872 boxed types, shared pointer types, and GObjects. For now we will just Err codemadness.org 70 i 8873 look at boxed types.

Err codemadness.org 70 i 8874

Glib-rs comes with a macro, glib_wrapper!(), that can be used in Err codemadness.org 70 i 8875 different ways. You can use it to automatically write the boilerplate Err codemadness.org 70 i 8876 for a boxed type like this:

Err codemadness.org 70 i 8877

glib_wrapper! {	Err	codemadness.org	70
i  8878     pub struct Color(Boxed<ffi::Color>);	Err	codemadness.org	70
i  8879 	Err	codemadness.org	70
i  8880     match fn {	Err	codemadness.org	70
i  8881         copy => |ptr| ffi::color_copy(mut_override(ptr)),	Err	codemadness.org	70
i  8882         free => |ptr| ffi::color_free(ptr),	Err	codemadness.org	70
i  8883         get_type => || ffi::color_get_type(),	Err	codemadness.org	70
i  8884     }	Err	codemadness.org	70
i  8885 }	Err	codemadness.org	70
i  8886

Err codemadness.org 70 i 8887 Err codemadness.org 70 i 8888

This expands to an internal Err codemadness.org 70 i 8889 glib_boxed_wrapper!() macro that does a few Err codemadness.org 70 i 8890 things. We will only look at particularly interesting bits.

Err codemadness.org 70 i 8891

First, the macro creates a newtype around a tuple with 1) the actual Err codemadness.org 70 i 8892 data type you want to box, and 2) a memory manager. In the example Err codemadness.org 70 i 8893 above, the newtype would be called Color, and it would wrap an Err codemadness.org 70 i 8894 ffi:Color (say, a C struct).

Err codemadness.org 70 i 8895

        pub struct $name(Boxed<$ffi_name, MemoryManager>);	Err	codemadness.org	70
i  8896

Err codemadness.org 70 i 8897 Err codemadness.org 70 i 8898

Aha! And that MemoryManager? The macro defines it as a zero-sized Err codemadness.org 70 i 8899 type:

Err codemadness.org 70 i 8900

        pub struct MemoryManager;	Err	codemadness.org	70
i  8901

Err codemadness.org 70 i 8902 Err codemadness.org 70 i 8903

Then it implements the BoxedMemoryManager trait for that Err codemadness.org 70 i 8904 MemoryManager struct:

Err codemadness.org 70 i 8905

        impl BoxedMemoryManager<$ffi_name> for MemoryManager {	Err	codemadness.org	70
i  8906             #[inline]	Err	codemadness.org	70
i  8907             unsafe fn copy($copy_arg: *const $ffi_name) -> *mut $ffi_name {	Err	codemadness.org	70
i  8908                 $copy_expr	Err	codemadness.org	70
i  8909             }	Err	codemadness.org	70
i  8910 	Err	codemadness.org	70
i  8911             #[inline]	Err	codemadness.org	70
i  8912             unsafe fn free($free_arg: *mut $ffi_name) {	Err	codemadness.org	70
i  8913                 $free_expr	Err	codemadness.org	70
i  8914             }	Err	codemadness.org	70
i  8915         }	Err	codemadness.org	70
i  8916

Err codemadness.org 70 i 8917 Err codemadness.org 70 i 8918

There! This is where the copy/free methods are implemented, based Err codemadness.org 70 i 8919 on the bits of code with which you invoked the macro. In the call to Err codemadness.org 70 i 8920 glib_wrapper!() we had this:

Err codemadness.org 70 i 8921

        copy => |ptr| ffi::color_copy(mut_override(ptr)),	Err	codemadness.org	70
i  8922         free => |ptr| ffi::color_free(ptr),	Err	codemadness.org	70
i  8923

Err codemadness.org 70 i 8924 Err codemadness.org 70 i 8925

In the impl aboe, the $copy_expr will expand to Err codemadness.org 70 i 8926 ffi::color_copy(mut_override(ptr)) and $free_expr will expand to Err codemadness.org 70 i 8927 ffi::color_free(ptr), which defines our implementation of a memory Err codemadness.org 70 i 8928 manager for our Color boxed type.

Err codemadness.org 70 i 8929

Zero-sized what?

Err codemadness.org 70 i 8930

Within the macro's definition, let's look again at the definitions of Err codemadness.org 70 i 8931 our boxed type and the memory manager object that actually implements Err codemadness.org 70 i 8932 the BoxedMemoryManager trait. Here is what the macro would expand Err codemadness.org 70 i 8933 to with our Color example:

Err codemadness.org 70 i 8934

        pub struct Color(Boxed<ffi::Color, MemoryManager>);	Err	codemadness.org	70
i  8935 	Err	codemadness.org	70
i  8936         pub struct MemoryManager;	Err	codemadness.org	70
i  8937 	Err	codemadness.org	70
i  8938         impl BoxedMemoryManager<ffi::Color> for MemoryManager {	Err	codemadness.org	70
i  8939             unsafe fn copy(...) -> *mut ffi::Color { ... }	Err	codemadness.org	70
i  8940             unsafe fn free(...) { ... }	Err	codemadness.org	70
i  8941         }	Err	codemadness.org	70
i  8942

Err codemadness.org 70 i 8943 Err codemadness.org 70 i 8944

Here, MemoryManager is a zero-sized type. This means it doesn't Err codemadness.org 70 i 8945 take up any space in the Color tuple! When a Color is allocated Err codemadness.org 70 i 8946 in the heap, it is really as if it contained an ffi::Color (the Err codemadness.org 70 i 8947 C struct we are wrapping) and nothing else.

Err codemadness.org 70 i 8948

All the knowledge about how to copy/free ffi::Color lives only in Err codemadness.org 70 i 8949 the compiler thanks to the trait implementation. When the compiler Err codemadness.org 70 i 8950 expands all the macros and monomorphizes all the generic functions, Err codemadness.org 70 i 8951 the calls to ffi::color_copy() and ffi::color_free() will be Err codemadness.org 70 i 8952 inlined at the appropriate spots. There is no need to have Err codemadness.org 70 i 8953 auxiliary structures taking up space in the heap, just to store Err codemadness.org 70 i 8954 function pointers to the copy/free functions, or anything like that.

Err codemadness.org 70 i 8955

Next up

Err codemadness.org 70 i 8956

You may have seen that our example call to glib_wrapper!() also Err codemadness.org 70 i 8957 passed in a ffi::color_get_type() function. We haven't talked about Err codemadness.org 70 i 8958 how glib-rs wraps Glib's GType, GValue, and all of that. We are Err codemadness.org 70 i 8959 getting closer and closer to being able to wrap GObject.

Err codemadness.org 70 i 8960

Stay tuned!

Initial posts about librsvg's C to Rust conversion

2017-09-07T15:50:46-05:00

The initial articles about librsvg's conversion to Rust are in my old Err codemadness.org 70 i 8966 blog, so they may be a bit hard to find from this new blog. Err codemadness.org 70 i 8967 Here is a list of those posts, just so they are easier to find:

Err codemadness.org 70 i 8968

Librsvg gets Rusty
Porting a few C functions to Rust
Bézier curves, markers, and SVG's concept of directionality
Refactoring C to make Rustification easier
Exposing Rust objects to C code
Debugging Rust code inside a C library
Reproducible font rendering for librsvg's tests
Algebraic data types in Rust, and basic parsing
How librsvg exports reference-counted objects from Rust to C
Griping about parsers and shitty specifications
Porting librsvg's tree of nodes to Rust
gboolean is not Rust bool

Err codemadness.org 70 i 8982

Within this new blog, you can look for articles with the librsvg tag.

The Magic of GObject Introspection

2017-09-07T11:46:03-05:00

Before continuing with the glib-rs architecture, let's take Err codemadness.org 70 i 8988 a detour and look at GObject Introspection. Although it can Err codemadness.org 70 i 8989 seem like an obscure part of the GNOME platform, it is an absolutely Err codemadness.org 70 i 8990 vital part of it: it is what lets people write GNOME applications in Err codemadness.org 70 i 8991 any language.

Err codemadness.org 70 i 8992

Let's start with a bit of history.

Err codemadness.org 70 i 8993

Brief history of language bindings in GNOME

Err codemadness.org 70 i 8994

When we started GNOME in 1997, we didn't want to write all of it in Err codemadness.org 70 i 8995 C. We had some inspiration from elsewhere.

Err codemadness.org 70 i 8996

Prehistory: GIMP and the Procedural Database

Err codemadness.org 70 i 8997

There was already good precedent for software written in a combination of Err codemadness.org 70 i 8998 programming languages. Emacs, the flagship text editor of the Err codemadness.org 70 i 8999 GNU project, was written with a relatively small core in C, and the Err codemadness.org 70 i 9000 majority of the program in Emacs Lisp.

Err codemadness.org 70 i 9001

In similar fashion, we were very influenced by the design of the GIMP, Err codemadness.org 70 i 9002 which was very innovative at that time. The GIMP has a large core Err codemadness.org 70 i 9003 written in C. However, it supports plug-ins or scripts written in a Err codemadness.org 70 i 9004 variety of languages. Initially the only scripting language available Err codemadness.org 70 i 9005 for the GIMP was Scheme.

Err codemadness.org 70 i 9006

The GIMP's plug-ins and scripts run as separate processes, so Err codemadness.org 70 i 9007 they don't have immediate access to the data of the image being Err codemadness.org 70 i 9008 edited, or to the core functions of the program like "paint with a Err codemadness.org 70 i 9009 brush at this location". To let plug-ins and scripts access these Err codemadness.org 70 i 9010 data and these functions, the GIMP has what it calls a Err codemadness.org 70 i 9011 Procedural Database (PDB). This is a Err codemadness.org 70 i 9012 list of functions that the core program or plug-ins wish to export. Err codemadness.org 70 i 9013 For example, there are functions like gimp-scale-image and Err codemadness.org 70 i 9014 gimp-move-layer. Once these functions are registered in the Err codemadness.org 70 i 9015 PDB, any part of the program or plug-ins can call them. Scripts are Err codemadness.org 70 i 9016 often written to automate common tasks — for example, when one wants Err codemadness.org 70 i 9017 to adjust the contrast of photos and scale them in bulk. Scripts can Err codemadness.org 70 i 9018 call functions in the PDB easily, irrespective of the programming Err codemadness.org 70 i 9019 language they are written in.

Err codemadness.org 70 i 9020

We wanted to write GNOME's core libraries in C, and write a similar Err codemadness.org 70 i 9021 Procedural Database to allow those libraries to be called from any Err codemadness.org 70 i 9022 programming language. Eventually it turned out that a PDB was not Err codemadness.org 70 i 9023 necessary, and there were better ways to go about enabling different Err codemadness.org 70 i 9024 programming languages.

Err codemadness.org 70 i 9025

Enabling sane memory management

Err codemadness.org 70 i 9026

GTK+ started out with a very simple scheme for memory management: a Err codemadness.org 70 i 9027 container owned its child widgets, and so on recursively. When you Err codemadness.org 70 i 9028 freed a container, it would be responsible for freeing its children.

Err codemadness.org 70 i 9029

However, consider what happens when a widget needs to hold a reference Err codemadness.org 70 i 9030 to another widget that is not one of its children. For example, a Err codemadness.org 70 i 9031 GtkLabel with an underlined mnemonic ("_N_ame:") needs to have a Err codemadness.org 70 i 9032 reference to the GtkEntry that should be focused when you press Err codemadness.org 70 i 9033 Alt-N. In the very earliest versions of GTK+, how to do this was Err codemadness.org 70 i 9034 undefined: C programmers were already used to having shared pointers Err codemadness.org 70 i 9035 everywhere, and they were used to being responsible for managing their Err codemadness.org 70 i 9036 memory.

Err codemadness.org 70 i 9037

Of course, this was prone to bugs. If you have something like

Err codemadness.org 70 i 9038

typedef struct {	Err	codemadness.org	70
i  9039     GtkWidget parent;	Err	codemadness.org	70
i  9040 	Err	codemadness.org	70
i  9041     char *label_string;	Err	codemadness.org	70
i  9042     GtkWidget *widget_to_focus;	Err	codemadness.org	70
i  9043 } GtkLabel;	Err	codemadness.org	70
i  9044

Err codemadness.org 70 i 9045 Err codemadness.org 70 i 9046

then if you are writing the destructor, you may simply want to

Err codemadness.org 70 i 9047

static void	Err	codemadness.org	70
i  9048 gtk_label_free (GtkLabel *label)	Err	codemadness.org	70
i  9049 {	Err	codemadness.org	70
i  9050     g_free (label_string);	Err	codemadness.org	70
i  9051     gtk_widget_free (widget_to_focus);          /* oops, we don't own this */	Err	codemadness.org	70
i  9052 	Err	codemadness.org	70
i  9053     free_parent_instance (&label->parent);	Err	codemadness.org	70
i  9054 }	Err	codemadness.org	70
i  9055

Err codemadness.org 70 i 9056 Err codemadness.org 70 i 9057

Say you have a GtkBox with the label and its associated GtkEntry. Err codemadness.org 70 i 9058 Then, freeing the GtkBox would recursively free the label with that Err codemadness.org 70 i 9059 gtk_label_free(), and then the entry with its own function. But by Err codemadness.org 70 i 9060 the time the entry gets freed, the line Err codemadness.org 70 i 9061 gtk_widget_free (widget_to_focus) has already freed the entry, and Err codemadness.org 70 i 9062 we get a double-free bug!

Err codemadness.org 70 i 9063

Madness!

Err codemadness.org 70 i 9064

That is, we had no idea what we were doing. Or rather, our Err codemadness.org 70 i 9065 understanding of widgets had not evolved to the point of acknowledging Err codemadness.org 70 i 9066 that a widget tree is not a simply tree, but rather a Err codemadness.org 70 i 9067 directed graph of container-child relationships, plus Err codemadness.org 70 i 9068 random-widget-to-random-widget relationships. And of course, other Err codemadness.org 70 i 9069 parts of the program which are not even widget implementations may Err codemadness.org 70 i 9070 need to keep references to widgets and free them or not as Err codemadness.org 70 i 9071 appropriate.

Err codemadness.org 70 i 9072

I think Marius Vollmer was the first person to start formalizing Err codemadness.org 70 i 9073 this. He came from the world of GNU Guile, a Scheme interpreter, and Err codemadness.org 70 i 9074 so he already knew how garbage collection and seas of shared Err codemadness.org 70 i 9075 references ought to work.

Err codemadness.org 70 i 9076

Marius implemented reference-counting for GTK+ — that's where Err codemadness.org 70 i 9077 gtk_object_ref() and gtk_object_unref() come from; they eventually Err codemadness.org 70 i 9078 got moved to the base GObject class, so we now have g_object_ref() Err codemadness.org 70 i 9079 and g_object_unref() and a host of functions to have weak Err codemadness.org 70 i 9080 references, notification of destruction, and all the things required Err codemadness.org 70 i 9081 to keep garbage collectors happy.

Err codemadness.org 70 i 9082

The first language bindings

Err codemadness.org 70 i 9083

The very first language bindings were written by hand. The GTK+ API Err codemadness.org 70 i 9084 was small, and it seemed feasible to take

Err codemadness.org 70 i 9085

void gtk_widget_show (GtkWidget *widget);	Err	codemadness.org	70
i  9086 void gtk_widget_hide (GtkWidget *widget);	Err	codemadness.org	70
i  9087 	Err	codemadness.org	70
i  9088 void gtk_container_add (GtkContainer *container, GtkWidget *child);	Err	codemadness.org	70
i  9089 void gtk_container_remove (GtkContainer *container, GtkWidget *child);	Err	codemadness.org	70
i  9090

Err codemadness.org 70 i 9091 Err codemadness.org 70 i 9092

and just wrap those functions in various languages, by hand, on an Err codemadness.org 70 i 9093 as-needed basis.

Err codemadness.org 70 i 9094

Of course, there is a lot of duplication when doing things that way. Err codemadness.org 70 i 9095 As the C API grows, one needs to do more and more manual work to Err codemadness.org 70 i 9096 keep up with it.

Err codemadness.org 70 i 9097

Also, C structs with public fields are problematic. If we had

Err codemadness.org 70 i 9098

typedef struct {	Err	codemadness.org	70
i  9099     guchar r;	Err	codemadness.org	70
i  9100     guchar g;	Err	codemadness.org	70
i  9101     guchar b;	Err	codemadness.org	70
i  9102 } GdkColor;	Err	codemadness.org	70
i  9103

Err codemadness.org 70 i 9104 Err codemadness.org 70 i 9105

and we expect program code to fill in a GdkColor by hand and Err codemadness.org 70 i 9106 pass it to a drawing function like

Err codemadness.org 70 i 9107

void gdk_set_foreground_color (GdkDrawingContext *gc, GdkColor *color);	Err	codemadness.org	70
i  9108

Err codemadness.org 70 i 9109 Err codemadness.org 70 i 9110

then it is no problem to do that in C:

Err codemadness.org 70 i 9111

GdkColor magenta = { 255, 0, 255 };	Err	codemadness.org	70
i  9112 	Err	codemadness.org	70
i  9113 gdk_set_foreground_color (gc, &magenta);	Err	codemadness.org	70
i  9114

Err codemadness.org 70 i 9115 Err codemadness.org 70 i 9116

But to do that in a high level language? You don't have access to C Err codemadness.org 70 i 9117 struct fields! And back then, libffi wasn't generally available.

Err codemadness.org 70 i 9118

Authors of language bindings had to write some glue code, in C, by Err codemadness.org 70 i 9119 hand, to let people access a C struct and then pass it on to GTK+. Err codemadness.org 70 i 9120 For example, for Python, they would need to write something like

Err codemadness.org 70 i 9121

PyObject *	Err	codemadness.org	70
i  9122 make_wrapped_gdk_color (PyObject *args, PyObject *kwargs)	Err	codemadness.org	70
i  9123 {	Err	codemadness.org	70
i  9124     GdkColor *g_color;	Err	codemadness.org	70
i  9125     PyObject *py_color;	Err	codemadness.org	70
i  9126 	Err	codemadness.org	70
i  9127     g_color = g_new (GdkColor, 1);	Err	codemadness.org	70
i  9128     /* ... fill in g_color->r, g, b from the Python args */	Err	codemadness.org	70
i  9129 	Err	codemadness.org	70
i  9130     py_color = wrap_g_color (g_color);	Err	codemadness.org	70
i  9131     return py_color;	Err	codemadness.org	70
i  9132 }	Err	codemadness.org	70
i  9133

Err codemadness.org 70 i 9134 Err codemadness.org 70 i 9135

Writing that by hand is an incredible amount of drudgery.

Err codemadness.org 70 i 9136

What language bindings needed was a description of the API in a Err codemadness.org 70 i 9137 machine-readable format, so that the glue code could be written by a Err codemadness.org 70 i 9138 code generator.

Err codemadness.org 70 i 9139

The first API descriptions

Err codemadness.org 70 i 9140

I don't remember if it was the GNU Guile people, or the PyGTK people, Err codemadness.org 70 i 9141 who started to write descriptions of the GNOME API by hand. For ease Err codemadness.org 70 i 9142 of parsing, it was done in a Scheme-like dialect. A description may Err codemadness.org 70 i 9143 look like

Err codemadness.org 70 i 9144

(class GtkWidget	Err	codemadness.org	70
i  9145        ;;; void gtk_widget_show (GtkWidget *widget);	Err	codemadness.org	70
i  9146        (method show	Err	codemadness.org	70
i  9147                (args nil)	Err	codemadness.org	70
i  9148                (retval nil))	Err	codemadness.org	70
i  9149 	Err	codemadness.org	70
i  9150        ;;; void gtk_widget_hide (GtkWidget *widget);	Err	codemadness.org	70
i  9151        (method hide	Err	codemadness.org	70
i  9152                (args nil)	Err	codemadness.org	70
i  9153                (retval nil)))	Err	codemadness.org	70
i  9154 	Err	codemadness.org	70
i  9155 (class GtkContainer	Err	codemadness.org	70
i  9156        ;;; void gtk_container_add (GtkContainer *container, GtkWidget *child);	Err	codemadness.org	70
i  9157        (method add	Err	codemadness.org	70
i  9158                (args GtkWidget)	Err	codemadness.org	70
i  9159                (retval nil)))	Err	codemadness.org	70
i  9160 	Err	codemadness.org	70
i  9161 (struct GdkColor	Err	codemadness.org	70
i  9162         (field r (type 'guchar))	Err	codemadness.org	70
i  9163         (field g (type 'guchar))	Err	codemadness.org	70
i  9164         (field b (type 'guchar))) 	Err	codemadness.org	70
i  9165

Err codemadness.org 70 i 9166 Err codemadness.org 70 i 9167

Again, writing those descriptions by hand (and keeping up with the C Err codemadness.org 70 i 9168 API) was a lot of work, but the glue code to implement the binding Err codemadness.org 70 i 9169 could be done mostly automatically. The generated code may need Err codemadness.org 70 i 9170 subsequent tweaks by hand to deal with details that the Scheme-like Err codemadness.org 70 i 9171 descriptions didn't contemplate, but it was better than writing Err codemadness.org 70 i 9172 everything by hand.

Err codemadness.org 70 i 9173

Glib gets a real type system

Err codemadness.org 70 i 9174

Tim Janik took over the parts of Glib that implement Err codemadness.org 70 i 9175 objects/signals/types, and added a lot of things to create a good type Err codemadness.org 70 i 9176 system for C. This is where things like GType, GValue, GParamSpec, and Err codemadness.org 70 i 9177 fundamental types come from.

Err codemadness.org 70 i 9178

For example, a GType is an identifier for a type, and a GValue is a Err codemadness.org 70 i 9179 type plus, well, a value of that type. You can ask a GValue, "are Err codemadness.org 70 i 9180 you an int? are you a GObject?".

Err codemadness.org 70 i 9181

You can register new types: for example, there would be code in Gdk Err codemadness.org 70 i 9182 that registers a new GType for GdkColor, so you can ask a value, Err codemadness.org 70 i 9183 "are you a color?".

Err codemadness.org 70 i 9184

Registering a type involves telling the GObject system things like how Err codemadness.org 70 i 9185 to copy values of that type, and how to free them. For GdkColor Err codemadness.org 70 i 9186 this may be just g_new() / g_free(); for reference-counted objects Err codemadness.org 70 i 9187 it may be g_object_ref() / g_object_unref().

Err codemadness.org 70 i 9188

Objects can be queried about some of their properties

Err codemadness.org 70 i 9189

A widget can tell you when you press a mouse button mouse on it: it Err codemadness.org 70 i 9190 will emit the button-press-event signal. When GtkWidget's Err codemadness.org 70 i 9191 implementation registers this signal, it calls something like

Err codemadness.org 70 i 9192

    g_signal_new ("button-press-event",	Err	codemadness.org	70
i  9193         gtk_widget_get_type(), /* type of object for which this signal is being created */	Err	codemadness.org	70
i  9194         ...	Err	codemadness.org	70
i  9195         G_TYPE_BOOLEAN,  /* type of return value */	Err	codemadness.org	70
i  9196         1,               /* number of arguments */	Err	codemadness.org	70
i  9197         GDK_TYPE_EVENT); /* type of first and only argument */	Err	codemadness.org	70
i  9198

Err codemadness.org 70 i 9199 Err codemadness.org 70 i 9200

This tells GObject that GtkWidget will have a signal called Err codemadness.org 70 i 9201 button-press-event, with a return type of G_TYPE_BOOLEAN, and with Err codemadness.org 70 i 9202 a single argument of type GDK_TYPE_EVENT. This lets GObject do the Err codemadness.org 70 i 9203 appropriate marshalling of arguments when the signal is emitted.

Err codemadness.org 70 i 9204

But also! You can query the signal for its argument types! You can Err codemadness.org 70 i 9205 run g_signal_query(), which will then tell you all the details of Err codemadness.org 70 i 9206 the signal: its name, return type, argument types, etc. A language Err codemadness.org 70 i 9207 binding could run g_signal_query() and generate a description of the Err codemadness.org 70 i 9208 signal automatically to the Scheme-like description language. And Err codemadness.org 70 i 9209 then generate the binding from that.

Err codemadness.org 70 i 9210

Not all of an object's properties can be queried

Err codemadness.org 70 i 9211

Unfortunately, although GObject signals and properties can be Err codemadness.org 70 i 9212 queried, methods can't be. C doesn't have classes with methods, and GObject does Err codemadness.org 70 i 9213 not really have any provisions to implement them.

Err codemadness.org 70 i 9214

Conventionally, for a static method one would just do

Err codemadness.org 70 i 9215

void	Err	codemadness.org	70
i  9216 gtk_widget_set_flags (GtkWidget *widget, GtkWidgetFlags flags)	Err	codemadness.org	70
i  9217 {	Err	codemadness.org	70
i  9218     /* modify a struct field within "widget" or whatever */	Err	codemadness.org	70
i  9219     /* repaint or something */	Err	codemadness.org	70
i  9220 }	Err	codemadness.org	70
i  9221

Err codemadness.org 70 i 9222 Err codemadness.org 70 i 9223

And for a virtual method one would put a function pointer in the class Err codemadness.org 70 i 9224 structure, and provide a convenient way to call it:

Err codemadness.org 70 i 9225

typedef struct {	Err	codemadness.org	70
i  9226     GtkObjectClass parent_class;	Err	codemadness.org	70
i  9227 	Err	codemadness.org	70
i  9228     void (* draw) (GtkWidget *widget, cairo_t *cr);	Err	codemadness.org	70
i  9229 } GtkWidgetClass;	Err	codemadness.org	70
i  9230 	Err	codemadness.org	70
i  9231 void	Err	codemadness.org	70
i  9232 gtk_widget_draw (GtkWidget *widget, cairo_t *cr)	Err	codemadness.org	70
i  9233 {	Err	codemadness.org	70
i  9234     GtkWidgetClass *klass = find_widget_class (widget);	Err	codemadness.org	70
i  9235 	Err	codemadness.org	70
i  9236     (* klass->draw) (widget, cr);	Err	codemadness.org	70
i  9237 }	Err	codemadness.org	70
i  9238

Err codemadness.org 70 i 9239 Err codemadness.org 70 i 9240

And GObject has no idea about this method — there is no way to query Err codemadness.org 70 i 9241 it; it just exists in C-space.

Err codemadness.org 70 i 9242

Now, historically, GTK+'s header files have been written in a very Err codemadness.org 70 i 9243 consistent style. It is quite possible to write a tool that will take Err codemadness.org 70 i 9244 a header file like

Err codemadness.org 70 i 9245

/* gtkwidget.h */	Err	codemadness.org	70
i  9246 typedef struct {	Err	codemadness.org	70
i  9247     GtkObject parent_class;	Err	codemadness.org	70
i  9248 	Err	codemadness.org	70
i  9249     void (* draw) (GtkWidget *widget, cairo_t *cr);	Err	codemadness.org	70
i  9250 } GtkWidgetClass;	Err	codemadness.org	70
i  9251 	Err	codemadness.org	70
i  9252 void gtk_widget_set_flags (GtkWidget *widget, GtkWidgetFlags flags);	Err	codemadness.org	70
i  9253 void gtk_widget_draw (GtkWidget *widget, cairo_t *cr);	Err	codemadness.org	70
i  9254

Err codemadness.org 70 i 9255 Err codemadness.org 70 i 9256

and parse it, even if it is with a simple parser that does not Err codemadness.org 70 i 9257 completely understand the C language, and have heuristics like

Err codemadness.org 70 i 9258

Err codemadness.org 70 i 9260
Is there a class_name_foo() function prototype with no Err codemadness.org 70 i 9261 corresponding foo field in the Class structure? It's probably a Err codemadness.org 70 i 9262 static method.
Err codemadness.org 70 i 9263
Err codemadness.org 70 i 9265
Is there a class_name_bar() function with a bar field in the Err codemadness.org 70 i 9266 Class structure? It's probably a virtual method.
Err codemadness.org 70 i 9267
Err codemadness.org 70 i 9269
Etc.
Err codemadness.org 70 i 9270

Err codemadness.org 70 i 9272

And in fact, that's what we had. C header files would get parsed Err codemadness.org 70 i 9273 with those heuristics, and the Scheme-like description files would get Err codemadness.org 70 i 9274 generated.

Err codemadness.org 70 i 9275

Scheme-like descriptions get reused, kind of

Err codemadness.org 70 i 9276

Language binding authors started reusing the Scheme-like Err codemadness.org 70 i 9277 descriptions. Sometimes they would cannibalize the descriptions from Err codemadness.org 70 i 9278 PyGTK, or Guile (again, I don't remember where the canonical version Err codemadness.org 70 i 9279 was maintained) and use them as they were.

Err codemadness.org 70 i 9280

Other times they would copy the files, modify them by hand some more, Err codemadness.org 70 i 9281 and then use them to generate their language binding.

Err codemadness.org 70 i 9282

C being hostile

Err codemadness.org 70 i 9283

From just reading/parsing a C function prototype, you cannot know Err codemadness.org 70 i 9284 certain things. If one function argument is of type Foo *, does it mean:

Err codemadness.org 70 i 9285

Err codemadness.org 70 i 9287
the function gets a pointer to something which it should not modify Err codemadness.org 70 i 9288 ("in" parameter)
Err codemadness.org 70 i 9289
Err codemadness.org 70 i 9291
the function gets a pointer to uninitialized data which it will set Err codemadness.org 70 i 9292 ("out" parameter)
Err codemadness.org 70 i 9293
Err codemadness.org 70 i 9295
the function gets a pointer to initialized data which it will use Err codemadness.org 70 i 9296 and modify ("inout" parameter)
Err codemadness.org 70 i 9297
Err codemadness.org 70 i 9299
the function will copy that pointer and hold a reference to the Err codemadness.org 70 i 9300 pointed data, and not free it when it's done
Err codemadness.org 70 i 9301
Err codemadness.org 70 i 9303
the function will take over the ownership of the pointed data, and Err codemadness.org 70 i 9304 free it when it's done
Err codemadness.org 70 i 9305
Err codemadness.org 70 i 9307
etc.
Err codemadness.org 70 i 9308

Err codemadness.org 70 i 9310

Sometimes people would include these annotations in the Scheme-like Err codemadness.org 70 i 9311 description language. But wouldn't it be better if those annotations Err codemadness.org 70 i 9312 came from the C code itself?

Err codemadness.org 70 i 9313

GObject Introspection appears

Err codemadness.org 70 i 9314

For GNOME 3, we wanted a unified solution for language bindings:

Err codemadness.org 70 i 9315

Err codemadness.org 70 i 9317
Have a single way to extract the machine-readable descriptions of Err codemadness.org 70 i 9318 the C API.
Err codemadness.org 70 i 9319
Err codemadness.org 70 i 9321
Have every language binding be automatically generated from those Err codemadness.org 70 i 9322 descriptions.
Err codemadness.org 70 i 9323
Err codemadness.org 70 i 9325
In the descriptions, have all the information necessary to Err codemadness.org 70 i 9326 generate a correct language binding...
Err codemadness.org 70 i 9327
Err codemadness.org 70 i 9329
... including documentation.
Err codemadness.org 70 i 9330

Err codemadness.org 70 i 9332

We had to do a lot of work to accomplish this. For example:

Err codemadness.org 70 i 9333

Err codemadness.org 70 i 9335
Remove C-isms from the public API. Varargs functions, those that Err codemadness.org 70 i 9336 have foo (int x, ...), can't be easily described and called from Err codemadness.org 70 i 9337 other languages. Instead, have something like Err codemadness.org 70 i 9338 foov (int x, int num_args, GValue *args_array) that can be easily Err codemadness.org 70 i 9339 consumed by other languages.
Err codemadness.org 70 i 9340
Err codemadness.org 70 i 9342
Add annotations throughout the code so that the ad-hoc C parser Err codemadness.org 70 i 9343 can know about in/out/inout arguments, and whether pointer arguments Err codemadness.org 70 i 9344 are borrowed references or a full transfership of ownership.
Err codemadness.org 70 i 9345
Err codemadness.org 70 i 9347
Take the in-line documentation comments and store them as part of Err codemadness.org 70 i 9348 the machine-readable description of the API.
Err codemadness.org 70 i 9349
Err codemadness.org 70 i 9351
When compiling a library, automatically do all the things like Err codemadness.org 70 i 9352 g_signal_query() and spit out machine-readable descriptions of Err codemadness.org 70 i 9353 those parts of the API.
Err codemadness.org 70 i 9354

Err codemadness.org 70 i 9356

So, GObject Introspection is all of those things.

Err codemadness.org 70 i 9357

Annotations

Err codemadness.org 70 i 9358

If you have looked at the C code for a GNOME library, you may have Err codemadness.org 70 i 9359 seen something like this:

Err codemadness.org 70 i 9360

/**	Err	codemadness.org	70
i  9361  * gtk_widget_get_parent:	Err	codemadness.org	70
i  9362  * @widget: a #GtkWidget	Err	codemadness.org	70
i  9363  *	Err	codemadness.org	70
i  9364  * Returns the parent container of @widget.	Err	codemadness.org	70
i  9365  *	Err	codemadness.org	70
i  9366  * Returns: (transfer none) (nullable): the parent container of @widget, or %NULL	Err	codemadness.org	70
i  9367  **/	Err	codemadness.org	70
i  9368 GtkWidget *	Err	codemadness.org	70
i  9369 gtk_widget_get_parent (GtkWidget *widget)	Err	codemadness.org	70
i  9370 {	Err	codemadness.org	70
i  9371     ...	Err	codemadness.org	70
i  9372 }	Err	codemadness.org	70
i  9373

Err codemadness.org 70 i 9374 Err codemadness.org 70 i 9375

See that "(transfer none) (nullable)" in the documentation comments? Err codemadness.org 70 i 9376 The (transfer none) means that the return value is a pointer whose Err codemadness.org 70 i 9377 ownership does not get transferred to the caller, i.e. the widget Err codemadness.org 70 i 9378 retains ownership. Finally, the (nullable) indicates that the Err codemadness.org 70 i 9379 function can return NULL, when the widget has no parent.

Err codemadness.org 70 i 9380

A language binding will then use this information as follows:

Err codemadness.org 70 i 9381

Err codemadness.org 70 i 9383
It will not unref() the parent widget when it is done with it.
Err codemadness.org 70 i 9384
Err codemadness.org 70 i 9386
It will deal with a NULL pointer in a special way, instead of Err codemadness.org 70 i 9387 assuming that references are not null.
Err codemadness.org 70 i 9388

Err codemadness.org 70 i 9390

Every now and then someone discovers a public function which is Err codemadness.org 70 i 9391 lacking an annotation of that sort — for GNOME's purposes this is a Err codemadness.org 70 i 9392 bug; fortunately, it is easy to add that annotation to the C sources Err codemadness.org 70 i 9393 and regenerate the machine-readable descriptions.

Err codemadness.org 70 i 9394

Machine-readable descriptions, or repository files

Err codemadness.org 70 i 9395

So, what do those machine-readable descriptions actually look like? Err codemadness.org 70 i 9396 They moved away from a Scheme-like language and got turned into XML, Err codemadness.org 70 i 9397 because early XXIst century.

Err codemadness.org 70 i 9398

The machine-readable descriptions are called GObject Introspection Err codemadness.org 70 i 9399 Repository files, or GIR for short.

Err codemadness.org 70 i 9400

Let's look at some parts of Gtk-3.0.gir, which your distro may put in Err codemadness.org 70 i 9401 /usr/share/gir-1.0/Gtk-3.0.gir.

Err codemadness.org 70 i 9402

<repository version="1.2" ...>	Err	codemadness.org	70
i  9403 	Err	codemadness.org	70
i  9404   <namespace name="Gtk"	Err	codemadness.org	70
i  9405              version="3.0"	Err	codemadness.org	70
i  9406              shared-library="libgtk-3.so.0,libgdk-3.so.0"	Err	codemadness.org	70
i  9407              c:identifier-prefixes="Gtk"	Err	codemadness.org	70
i  9408              c:symbol-prefixes="gtk">	Err	codemadness.org	70
i  9409

Err codemadness.org 70 i 9410 Err codemadness.org 70 i 9411

For the toplevel "Gtk" namespace, this is what the .so library is Err codemadness.org 70 i 9412 called. All identifiers have "Gtk" or "gtk" prefixes.

Err codemadness.org 70 i 9413

A class with methods and a signal

Err codemadness.org 70 i 9414

Let's look at the description for GtkEntry...

Err codemadness.org 70 i 9415

    <class name="Entry"	Err	codemadness.org	70
i  9416            c:symbol-prefix="entry"	Err	codemadness.org	70
i  9417            c:type="GtkEntry"	Err	codemadness.org	70
i  9418            parent="Widget"	Err	codemadness.org	70
i  9419            glib:type-name="GtkEntry"	Err	codemadness.org	70
i  9420            glib:get-type="gtk_entry_get_type"	Err	codemadness.org	70
i  9421            glib:type-struct="EntryClass">	Err	codemadness.org	70
i  9422 	Err	codemadness.org	70
i  9423       <doc xml:space="preserve">The #GtkEntry widget is a single line text entry	Err	codemadness.org	70
i  9424 widget. A fairly large set of key bindings are supported	Err	codemadness.org	70
i  9425 by default. If the entered text is longer than the allocation	Err	codemadness.org	70
i  9426 ...	Err	codemadness.org	70
i  9427        </doc>	Err	codemadness.org	70
i  9428

Err codemadness.org 70 i 9429 Err codemadness.org 70 i 9430

This is the start of the description for GtkEntry. We already know Err codemadness.org 70 i 9431 that everything is prefixed with "Gtk", so the name is just given as Err codemadness.org 70 i 9432 "Entry". Its parent class is Widget and the function which Err codemadness.org 70 i 9433 registers it against the GObject type system is gtk_entry_get_type.

Err codemadness.org 70 i 9434

Also, there are the toplevel documentation comments for the Entry Err codemadness.org 70 i 9435 class.

Err codemadness.org 70 i 9436

Onwards!

Err codemadness.org 70 i 9437

      <implements name="Atk.ImplementorIface"/>	Err	codemadness.org	70
i  9438       <implements name="Buildable"/>	Err	codemadness.org	70
i  9439       <implements name="CellEditable"/>	Err	codemadness.org	70
i  9440       <implements name="Editable"/>	Err	codemadness.org	70
i  9441

Err codemadness.org 70 i 9442 Err codemadness.org 70 i 9443

GObject classes can implement various interfaces; this is the list Err codemadness.org 70 i 9444 that GtkEntry supports.

Err codemadness.org 70 i 9445

Next, let's look at a single method:

Err codemadness.org 70 i 9446

      <method name="get_text" c:identifier="gtk_entry_get_text">	Err	codemadness.org	70
i  9447         <doc xml:space="preserve">Retrieves the contents of the entry widget. ... </doc>	Err	codemadness.org	70
i  9448 	Err	codemadness.org	70
i  9449         <return-value transfer-ownership="none">	Err	codemadness.org	70
i  9450           <type name="utf8" c:type="const gchar*"/>	Err	codemadness.org	70
i  9451         </return-value>	Err	codemadness.org	70
i  9452 	Err	codemadness.org	70
i  9453         <parameters>	Err	codemadness.org	70
i  9454           <instance-parameter name="entry" transfer-ownership="none">	Err	codemadness.org	70
i  9455             <type name="Entry" c:type="GtkEntry*"/>	Err	codemadness.org	70
i  9456           </instance-parameter>	Err	codemadness.org	70
i  9457         </parameters>	Err	codemadness.org	70
i  9458       </method>	Err	codemadness.org	70
i  9459

Err codemadness.org 70 i 9460 Err codemadness.org 70 i 9461

The method get_text and its corresponding C symbol. Its return Err codemadness.org 70 i 9462 value is an UTF-8 encoded string, and ownership of the memory for that Err codemadness.org 70 i 9463 string is not transferred to the caller.

Err codemadness.org 70 i 9464

The method takes a single parameter which is the entry instance itself.

Err codemadness.org 70 i 9465

Now, let's look at a signal:

Err codemadness.org 70 i 9466

      <glib:signal name="activate" when="last" action="1">	Err	codemadness.org	70
i  9467         <doc xml:space="preserve">The ::activate signal is emitted when the user hits	Err	codemadness.org	70
i  9468 the Enter key. ...</doc>	Err	codemadness.org	70
i  9469 	Err	codemadness.org	70
i  9470         <return-value transfer-ownership="none">	Err	codemadness.org	70
i  9471           <type name="none" c:type="void"/>	Err	codemadness.org	70
i  9472         </return-value>	Err	codemadness.org	70
i  9473       </glib:signal>	Err	codemadness.org	70
i  9474 	Err	codemadness.org	70
i  9475     </class>	Err	codemadness.org	70
i  9476

Err codemadness.org 70 i 9477 Err codemadness.org 70 i 9478

The "activate" signal takes no arguments, and has a return value of Err codemadness.org 70 i 9479 type void, i.e. no return value.

Err codemadness.org 70 i 9480

A struct with public fields

Err codemadness.org 70 i 9481

The following comes from Gdk-3.0.gir; it's the description for Err codemadness.org 70 i 9482 GdkRectangle.

Err codemadness.org 70 i 9483

    <record name="Rectangle"	Err	codemadness.org	70
i  9484             c:type="GdkRectangle"	Err	codemadness.org	70
i  9485             glib:type-name="GdkRectangle"	Err	codemadness.org	70
i  9486             glib:get-type="gdk_rectangle_get_type"	Err	codemadness.org	70
i  9487             c:symbol-prefix="rectangle">	Err	codemadness.org	70
i  9488 	Err	codemadness.org	70
i  9489       <field name="x" writable="1">	Err	codemadness.org	70
i  9490         <type name="gint" c:type="int"/>	Err	codemadness.org	70
i  9491       </field>	Err	codemadness.org	70
i  9492       <field name="y" writable="1">	Err	codemadness.org	70
i  9493         <type name="gint" c:type="int"/>	Err	codemadness.org	70
i  9494       </field>	Err	codemadness.org	70
i  9495       <field name="width" writable="1">	Err	codemadness.org	70
i  9496         <type name="gint" c:type="int"/>	Err	codemadness.org	70
i  9497       </field>	Err	codemadness.org	70
i  9498       <field name="height" writable="1">	Err	codemadness.org	70
i  9499         <type name="gint" c:type="int"/>	Err	codemadness.org	70
i  9500       </field>	Err	codemadness.org	70
i  9501 	Err	codemadness.org	70
i  9502     </record>	Err	codemadness.org	70
i  9503

Err codemadness.org 70 i 9504 Err codemadness.org 70 i 9505

So that's the x/y/width/height fields in the struct, in the same Err codemadness.org 70 i 9506 order as they are defined in the C code.

Err codemadness.org 70 i 9507

And so on. The idea is for the whole API exported by a GObject Err codemadness.org 70 i 9508 library to be describable by that format. If something can't be Err codemadness.org 70 i 9509 described, it's a bug in the library, or a bug in the format.

Err codemadness.org 70 i 9510

Making language bindings start up quickly: typelib files

Err codemadness.org 70 i 9511

As we saw, the GIR files are the XML descriptions of GObject APIs. Err codemadness.org 70 i 9512 Dynamic languages like Python would prefer to generate the language Err codemadness.org 70 i 9513 binding on the fly, as needed, instead of pre-generating a huge Err codemadness.org 70 i 9514 binding.

Err codemadness.org 70 i 9515

However, GTK+ is a big API: Gtk-3.0.gir is 7 MB of XML. Parsing Err codemadness.org 70 i 9516 all of that just to be able to generate gtk_widget_show() on the fly Err codemadness.org 70 i 9517 would be too slow. Also, there are GTK+'s dependencies: Atk, Gdk, Err codemadness.org 70 i 9518 Cairo, etc. You don't want to parse everything just to start up!

Err codemadness.org 70 i 9519

So, we have an extra step that compiles the GIR files down to binary Err codemadness.org 70 i 9520 .typelib files. For example, Err codemadness.org 70 i 9521 /usr/lib64/girepository-1.0/Gtk-3.0.typelib is about 600 KB on my Err codemadness.org 70 i 9522 machine. Those files get mmap()ed for fast access, and can be Err codemadness.org 70 i 9523 shared between processes.

Err codemadness.org 70 i 9524

How dynamic language bindings use typelib files

Err codemadness.org 70 i 9525

GObject Introspection comes with a library that language binding Err codemadness.org 70 i 9526 implementors can use to consume those .typelib files. The Err codemadness.org 70 i 9527 libgirepository library has functions like "list all the classes Err codemadness.org 70 i 9528 available in this namespace", or "call this function with these Err codemadness.org 70 i 9529 values for arguments, and give me back the return value here".

Err codemadness.org 70 i 9530

Internally, libgirepository uses libffi to actually call the C Err codemadness.org 70 i 9531 functions in the dynamically-linked libraries.

Err codemadness.org 70 i 9532

So, when you write foo.py and do

Err codemadness.org 70 i 9533

import gi	Err	codemadness.org	70
i  9534 gi.require_version('Gtk', '3.0')	Err	codemadness.org	70
i  9535 from gi.repository import Gtk	Err	codemadness.org	70
i  9536 win = Gtk.Window()	Err	codemadness.org	70
i  9537

Err codemadness.org 70 i 9538 Err codemadness.org 70 i 9539

what happens is that pygobject calls libgirepository to mmap() Err codemadness.org 70 i 9540 the .typelib, and sees that the constructor for Gtk.Window is a C Err codemadness.org 70 i 9541 function called gtk_window_new(). After seeing how that function Err codemadness.org 70 i 9542 wants to be called, it calls the function using libffi, wraps the Err codemadness.org 70 i 9543 result with a PyObject, and that's what you get on the Python side.

Err codemadness.org 70 i 9544

Static languages

Err codemadness.org 70 i 9545

A static language like Rust prefers to have the whole language binding Err codemadness.org 70 i 9546 pre-generated. This is what the various crates in gtk-rs Err codemadness.org 70 i 9547 do.

Err codemadness.org 70 i 9548

The gir crate takes a .gir file (i.e. the XML descriptions) Err codemadness.org 70 i 9549 and does two things:

Err codemadness.org 70 i 9550

Err codemadness.org 70 i 9552
Reconstructs the C function prototypes and C struct declarations, Err codemadness.org 70 i 9553 but in a way Rust can understand them. This gets output to the sys Err codemadness.org 70 i 9554 crate.
Err codemadness.org 70 i 9555
Err codemadness.org 70 i 9557
Creates idiomatic Rust code for the language binding. This gets Err codemadness.org 70 i 9558 output to the various crates; for example, the gtk one.
Err codemadness.org 70 i 9559

Err codemadness.org 70 i 9561

When reconstructing the C structs and prototypes, we get stuff like

Err codemadness.org 70 i 9562

#[repr(C)]	Err	codemadness.org	70
i  9563 pub struct GtkWidget {	Err	codemadness.org	70
i  9564     pub parent_instance: gobject::GInitiallyUnowned,	Err	codemadness.org	70
i  9565     pub priv_: *mut GtkWidgetPrivate,	Err	codemadness.org	70
i  9566 }	Err	codemadness.org	70
i  9567 	Err	codemadness.org	70
i  9568 extern "C" {	Err	codemadness.org	70
i  9569     pub fn gtk_entry_new() -> *mut GtkWidget;	Err	codemadness.org	70
i  9570 }	Err	codemadness.org	70
i  9571

Err codemadness.org 70 i 9572 Err codemadness.org 70 i 9573

And the idiomatic bindings? Stay tuned!

Librsvg's build infrastructure: Autotools and Rust

2017-09-01T18:15:29-05:00

Today I released librsvg 2.41.1, and it's a big release! Err codemadness.org 70 i 9577 Apart from all the Rust goodness, and the large number of bug fixes, I Err codemadness.org 70 i 9578 am very happy with the way the build system works these days. I've Err codemadness.org 70 i 9579 found it invaluable to have good examples of Autotools incantations to Err codemadness.org 70 i 9580 copy&paste, so hopefully this will be useful to someone else.

Err codemadness.org 70 i 9581

There are some subtleties that a "good" autotools setup demands, and Err codemadness.org 70 i 9582 so far I think librsvg is doing well:

Err codemadness.org 70 i 9583

Err codemadness.org 70 i 9585
The configure script checks for cargo and rustc.
Err codemadness.org 70 i 9586
Err codemadness.org 70 i 9588
"make distcheck" works. This means that the build can be Err codemadness.org 70 i 9589 performed with builddir != srcdir, and also that make check runs Err codemadness.org 70 i 9590 the available tests and they all pass.
Err codemadness.org 70 i 9591
Err codemadness.org 70 i 9593
The rsvg_internals library is built with Rust, and our Err codemadness.org 70 i 9594 Makefile.am calls cargo build with the correct options. It is Err codemadness.org 70 i 9595 able to handle debug and release builds.
Err codemadness.org 70 i 9596
Err codemadness.org 70 i 9598
"make clean" cleans up the Rust build directories as well.
Err codemadness.org 70 i 9599
Err codemadness.org 70 i 9601
If you change a .rs file and type make, only the necessary stuff Err codemadness.org 70 i 9602 gets rebuilt.
Err codemadness.org 70 i 9603
Err codemadness.org 70 i 9605
Etcetera. I think librsvg feels like a normal autotool'ed library. Err codemadness.org 70 i 9606 Let's see how this is done.
Err codemadness.org 70 i 9607

Err codemadness.org 70 i 9609

Librsvg's basic autotools setup

Err codemadness.org 70 i 9610

Librsvg started out with a fairly traditional autotools setup with a Err codemadness.org 70 i 9611 configure.ac and Makefile.am. For Err codemadness.org 70 i 9612 historical reasons the .[ch] source files live in the toplevel Err codemadness.org 70 i 9613 librsvg/ directory, not in a src subdirectory or something like Err codemadness.org 70 i 9614 that.

Err codemadness.org 70 i 9615

librsvg	Err	codemadness.org	70
i  9616 ├ configure.ac	Err	codemadness.org	70
i  9617 ├ Makefile.am	Err	codemadness.org	70
i  9618 ├ *.[ch]	Err	codemadness.org	70
i  9619 ├ src/	Err	codemadness.org	70
i  9620 ├ doc/	Err	codemadness.org	70
i  9621 ├ tests/	Err	codemadness.org	70
i  9622 └ win32/	Err	codemadness.org	70
i  9623

Err codemadness.org 70 i 9624 Err codemadness.org 70 i 9625

Adding Rust to the build

Err codemadness.org 70 i 9626

The Rust source code lives in librsvg/rust; that's Err codemadness.org 70 i 9627 where Cargo.toml lives, and of course there is the conventional Err codemadness.org 70 i 9628 src subdirectory with the *.rs files.

Err codemadness.org 70 i 9629

librsvg	Err	codemadness.org	70
i  9630 ├ configure.ac	Err	codemadness.org	70
i  9631 ├ Makefile.am	Err	codemadness.org	70
i  9632 ├ *.[ch]	Err	codemadness.org	70
i  9633 ├ src/	Err	codemadness.org	70
i  9634 ├ rust/         <--- this is new!	Err	codemadness.org	70
i  9635 │ ├ Cargo.toml	Err	codemadness.org	70
i  9636 │ └ src/	Err	codemadness.org	70
i  9637 ├ doc/	Err	codemadness.org	70
i  9638 ├ tests/	Err	codemadness.org	70
i  9639 └ win32/	Err	codemadness.org	70
i  9640

Err codemadness.org 70 i 9641 Err codemadness.org 70 i 9642

Detecting the presence of `cargo` and `rustc` in `configure.ac`

Err codemadness.org 70 i 9643

This goes in configure.ac:

Err codemadness.org 70 i 9644

AC_CHECK_PROG(CARGO, [cargo], [yes], [no])	Err	codemadness.org	70
i  9645 AS_IF(test x$CARGO = xno,	Err	codemadness.org	70
i  9646     AC_MSG_ERROR([cargo is required.  Please install the Rust toolchain from https://www.rust-lang.org/])	Err	codemadness.org	70
i  9647 )	Err	codemadness.org	70
i  9648 AC_CHECK_PROG(RUSTC, [rustc], [yes], [no])	Err	codemadness.org	70
i  9649 AS_IF(test x$RUSTC = xno,	Err	codemadness.org	70
i  9650     AC_MSG_ERROR([rustc is required.  Please install the Rust toolchain from https://www.rust-lang.org/])	Err	codemadness.org	70
i  9651 )	Err	codemadness.org	70
i  9652

Err codemadness.org 70 i 9653 Err codemadness.org 70 i 9654

These two try to execute cargo and rustc, respectively, and abort Err codemadness.org 70 i 9655 with an error message if they are not present.

Err codemadness.org 70 i 9656

Supporting debug or release mode for the Rust build

Err codemadness.org 70 i 9657

One can call cargo like "cargo build --release" to turn on expensive Err codemadness.org 70 i 9658 optimizations, or normally like just "cargo build" to build with Err codemadness.org 70 i 9659 debug information. That is, the latter is the default: if you don't Err codemadness.org 70 i 9660 pass any options, cargo does a debug build.

Err codemadness.org 70 i 9661

Autotools and C compilers normally work a bit differently; one must Err codemadness.org 70 i 9662 call the configure script like "CFLAGS='-g -O0' ./configure" for a Err codemadness.org 70 i 9663 debug build, or "CFLAGS='-O2 -fomit-frame-pointer' ./configure" for Err codemadness.org 70 i 9664 a release build.

Err codemadness.org 70 i 9665

Linux distros already have all the infrastructure to pass the Err codemadness.org 70 i 9666 appropriate CFLAGS to configure. We need to be able to pass the Err codemadness.org 70 i 9667 appropriate flag to Cargo. My main requirement for this was:

Err codemadness.org 70 i 9668

Distros shouldn't have to substantially change their RPM specfiles Err codemadness.org 70 i 9670 (or whatever) to accomodate the Rust build.
I assume that distros will want to make release builds by default.
I as a developer am comfortable with passing extra options to make Err codemadness.org 70 i 9673 debug builds on my machine.

Err codemadness.org 70 i 9675

The scheme in librsvg lets you run "configure --enable-debug" to Err codemadness.org 70 i 9676 make it call a plain cargo build, or a plain "configure" to make Err codemadness.org 70 i 9677 it use cargo build --release instead. The CFLAGS are passed as Err codemadness.org 70 i 9678 usual through an environment variable. This way, distros don't have Err codemadness.org 70 i 9679 to change their packaging to keep on making release builds as usual.

Err codemadness.org 70 i 9680

This goes in configure.ac:

Err codemadness.org 70 i 9681

dnl Specify --enable-debug to make a development release.  By default,	Err	codemadness.org	70
i  9682 dnl we build in public release mode.	Err	codemadness.org	70
i  9683 	Err	codemadness.org	70
i  9684 AC_ARG_ENABLE(debug,	Err	codemadness.org	70
i  9685               AC_HELP_STRING([--enable-debug],	Err	codemadness.org	70
i  9686                              [Build Rust code with debugging information [default=no]]),	Err	codemadness.org	70
i  9687               [debug_release=$enableval],	Err	codemadness.org	70
i  9688               [debug_release=no])	Err	codemadness.org	70
i  9689 	Err	codemadness.org	70
i  9690 AC_MSG_CHECKING(whether to build Rust code with debugging information)	Err	codemadness.org	70
i  9691 if test "x$debug_release" = "xyes" ; then	Err	codemadness.org	70
i  9692     AC_MSG_RESULT(yes)	Err	codemadness.org	70
i  9693     RUST_TARGET_SUBDIR=debug	Err	codemadness.org	70
i  9694 else	Err	codemadness.org	70
i  9695     AC_MSG_RESULT(no)	Err	codemadness.org	70
i  9696     RUST_TARGET_SUBDIR=release	Err	codemadness.org	70
i  9697 fi	Err	codemadness.org	70
i  9698 AM_CONDITIONAL([DEBUG_RELEASE], [test "x$debug_release" = "xyes"])	Err	codemadness.org	70
i  9699 	Err	codemadness.org	70
i  9700 AC_SUBST([RUST_TARGET_SUBDIR])	Err	codemadness.org	70
i  9701

Err codemadness.org 70 i 9702 Err codemadness.org 70 i 9703

This defines an Automake conditional called DEBUG_RELEASE, which we Err codemadness.org 70 i 9704 will use in Makefile.am later.

Err codemadness.org 70 i 9705

It also causes @RUST_TARGET_SUBDIR@ to be substituted in Makefile.am Err codemadness.org 70 i 9706 with either debug or release; we will see what these are about.

Err codemadness.org 70 i 9707

Adding Rust source files

Err codemadness.org 70 i 9708

The librsvg/rust/src directory has all the *.rs files, and cargo Err codemadness.org 70 i 9709 tracks their dependencies and whether they need to be rebuilt if one changes. Err codemadness.org 70 i 9710 However, since that directory is not tracked by make, it won't Err codemadness.org 70 i 9711 rebuild things if a Rust source file changes! So, we need to tell our Err codemadness.org 70 i 9712 Makefile.am about those files:

Err codemadness.org 70 i 9713

RUST_SOURCES =                   \	Err	codemadness.org	70
i  9714         rust/build.rs            \	Err	codemadness.org	70
i  9715         rust/Cargo.toml          \	Err	codemadness.org	70
i  9716         rust/src/aspect_ratio.rs \	Err	codemadness.org	70
i  9717         rust/src/bbox.rs         \	Err	codemadness.org	70
i  9718         rust/src/cnode.rs        \	Err	codemadness.org	70
i  9719         rust/src/color.rs        \	Err	codemadness.org	70
i  9720         ...	Err	codemadness.org	70
i  9721 	Err	codemadness.org	70
i  9722 RUST_EXTRA =                     \	Err	codemadness.org	70
i  9723         rust/Cargo.lock	Err	codemadness.org	70
i  9724 	Err	codemadness.org	70
i  9725 EXTRA_DIST += $(RUST_SOURCES) $(RUST_EXTRA)	Err	codemadness.org	70
i  9726

Err codemadness.org 70 i 9727 Err codemadness.org 70 i 9728

It's a bit unfortunate that the change tracking is duplicated in the Err codemadness.org 70 i 9729 Makefile, but we are already used to listing all the C source files Err codemadness.org 70 i 9730 in there, anyway.

Err codemadness.org 70 i 9731

Most notably, the rust subdirectory is not listed in the SUBDIRS Err codemadness.org 70 i 9732 in Makefile.am, since there is no rust/Makefile at all!

Err codemadness.org 70 i 9733

Cargo release or debug build?

Err codemadness.org 70 i 9734

if DEBUG_RELEASE	Err	codemadness.org	70
i  9735 CARGO_RELEASE_ARGS=	Err	codemadness.org	70
i  9736 else	Err	codemadness.org	70
i  9737 CARGO_RELEASE_ARGS=--release	Err	codemadness.org	70
i  9738 endif	Err	codemadness.org	70
i  9739

Err codemadness.org 70 i 9740 Err codemadness.org 70 i 9741

We will call cargo build with that argument later.

Err codemadness.org 70 i 9742

Verbose or quiet build?

Err codemadness.org 70 i 9743

Librsvg uses AM_SILENT_RULES([yes]) in configure.ac. This lets Err codemadness.org 70 i 9744 you just run "make" for a quiet build, or "make V=1" to get the Err codemadness.org 70 i 9745 full command lines passed to the compiler. Cargo supports something Err codemadness.org 70 i 9746 similar, so let's add it to Makefile.am:

Err codemadness.org 70 i 9747

CARGO_VERBOSE = $(cargo_verbose_$(V))	Err	codemadness.org	70
i  9748 cargo_verbose_ = $(cargo_verbose_$(AM_DEFAULT_VERBOSITY))	Err	codemadness.org	70
i  9749 cargo_verbose_0 =	Err	codemadness.org	70
i  9750 cargo_verbose_1 = --verbose	Err	codemadness.org	70
i  9751

Err codemadness.org 70 i 9752 Err codemadness.org 70 i 9753

This expands the V variable to empty, 0, or 1. The result of Err codemadness.org 70 i 9754 expanding that gives us the final command-line argument in the Err codemadness.org 70 i 9755 CARGO_VERBOSE variable.

Err codemadness.org 70 i 9756

What's the filename of the library we are building?

Err codemadness.org 70 i 9757

RUST_LIB=@abs_top_builddir@/rust/target/@RUST_TARGET_SUBDIR@/librsvg_internals.a	Err	codemadness.org	70
i  9758

Err codemadness.org 70 i 9759 Err codemadness.org 70 i 9760

Remember our @RUST_TARGET_SUBDIR@ from configure.ac? If you call Err codemadness.org 70 i 9761 plain "cargo build", it will put the binaries in Err codemadness.org 70 i 9762 rust/target/debug. But if you call "cargo build --release", it Err codemadness.org 70 i 9763 will put the binaries in rust/target/release.

Err codemadness.org 70 i 9764

With the bit above, the RUST_LIB variable now has the correct path Err codemadness.org 70 i 9765 for the built library. The @abs_top_builddir@ makes it work when Err codemadness.org 70 i 9766 the build directory is not the same as the source directory.

Err codemadness.org 70 i 9767

Okay, so how do we call `cargo`?

Err codemadness.org 70 i 9768

@abs_top_builddir@/rust/target/@RUST_TARGET_SUBDIR@/librsvg_internals.a: $(RUST_SOURCES)	Err	codemadness.org	70
i  9769     cd $(top_srcdir)/rust && \	Err	codemadness.org	70
i  9770     CARGO_TARGET_DIR=@abs_top_builddir@/rust/target cargo build $(CARGO_VERBOSE) $(CARGO_RELEASE_ARGS)	Err	codemadness.org	70
i  9771

Err codemadness.org 70 i 9772 Err codemadness.org 70 i 9773

We make the funky library filename depend on $(RUST_SOURCES). Err codemadness.org 70 i 9774 That's what will cause make to rebuild the Rust library if one of Err codemadness.org 70 i 9775 the Rust source files changes.

Err codemadness.org 70 i 9776

We override the CARGO_TARGET_DIR with Automake's preference, and Err codemadness.org 70 i 9777 call cargo build with the correct arguments.

Err codemadness.org 70 i 9778

Linking into the main C library

Err codemadness.org 70 i 9779

librsvg_@RSVG_API_MAJOR_VERSION@_la_LIBADD = \	Err	codemadness.org	70
i  9780         $(LIBRSVG_LIBS)                      \	Err	codemadness.org	70
i  9781         $(LIBM)                              \	Err	codemadness.org	70
i  9782         $(RUST_LIB)	Err	codemadness.org	70
i  9783

Err codemadness.org 70 i 9784 Err codemadness.org 70 i 9785

This expands our $(RUST_LIB) from above into our linker line, along Err codemadness.org 70 i 9786 with librsvg's other dependencies.

Err codemadness.org 70 i 9787

`make check`

Err codemadness.org 70 i 9788

This is our hook so that make check will cause cargo test to run:

Err codemadness.org 70 i 9789

check-local:	Err	codemadness.org	70
i  9790         cd $(srcdir)/rust && \	Err	codemadness.org	70
i  9791         CARGO_TARGET_DIR=@abs_top_builddir@/rust/target cargo test	Err	codemadness.org	70
i  9792

Err codemadness.org 70 i 9793 Err codemadness.org 70 i 9794

`make clean`

Err codemadness.org 70 i 9795

Same thing for make clean and cargo clean:

Err codemadness.org 70 i 9796

clean-local:	Err	codemadness.org	70
i  9797         cd $(top_srcdir)/rust && \	Err	codemadness.org	70
i  9798         CARGO_TARGET_DIR=@abs_top_builddir@/rust/target cargo clean	Err	codemadness.org	70
i  9799

Err codemadness.org 70 i 9800 Err codemadness.org 70 i 9801

Vendoring dependencies

Err codemadness.org 70 i 9802

Linux distros probably want Rust packages to come bundled with their Err codemadness.org 70 i 9803 dependencies, so that they can replace them later with newer/patched Err codemadness.org 70 i 9804 versions.

Err codemadness.org 70 i 9805

Here is a hook so that make dist will cause cargo vendor to be Err codemadness.org 70 i 9806 run before making the tarball. That command will creates a Err codemadness.org 70 i 9807 rust/vendor directory with a copy of all the Rust crates that Err codemadness.org 70 i 9808 librsvg depends on.

Err codemadness.org 70 i 9809

RUST_EXTRA += rust/cargo-vendor-config	Err	codemadness.org	70
i  9810 	Err	codemadness.org	70
i  9811 dist-hook:	Err	codemadness.org	70
i  9812     (cd $(distdir)/rust && \	Err	codemadness.org	70
i  9813     cargo vendor -q && \	Err	codemadness.org	70
i  9814     mkdir .cargo && \	Err	codemadness.org	70
i  9815     cp cargo-vendor-config .cargo/config)	Err	codemadness.org	70
i  9816

Err codemadness.org 70 i 9817 Err codemadness.org 70 i 9818

The tarball needs to have a rust/.cargo/config to know where to find Err codemadness.org 70 i 9819 the vendored sources (i.e. the embedded dependencies), but we don't Err codemadness.org 70 i 9820 want that in our development source tree. Instead, we generate it Err codemadness.org 70 i 9821 from a rust/cargo-vendor-config file in our Err codemadness.org 70 i 9822 source tree:

Err codemadness.org 70 i 9823

# This is used after `cargo vendor` is run from `make dist`.	Err	codemadness.org	70
i  9824 #	Err	codemadness.org	70
i  9825 # In the distributed tarball, this file should end up in	Err	codemadness.org	70
i  9826 # rust/.cargo/config	Err	codemadness.org	70
i  9827 	Err	codemadness.org	70
i  9828 [source.crates-io]	Err	codemadness.org	70
i  9829 registry = 'https://github.com/rust-lang/crates.io-index'	Err	codemadness.org	70
i  9830 replace-with = 'vendored-sources'	Err	codemadness.org	70
i  9831 	Err	codemadness.org	70
i  9832 [source.vendored-sources]	Err	codemadness.org	70
i  9833 directory = './vendor'	Err	codemadness.org	70
i  9834

Err codemadness.org 70 i 9835 Err codemadness.org 70 i 9836

One last thing

Err codemadness.org 70 i 9837

If you put this in your Cargo.toml, release binaries will be a lot Err codemadness.org 70 i 9838 smaller. This turns on link-time optimizations (LTO), which removes Err codemadness.org 70 i 9839 unused functions from the binary.

Err codemadness.org 70 i 9840

[profile.release]	Err	codemadness.org	70
i  9841 lto = true	Err	codemadness.org	70
i  9842

Err codemadness.org 70 i 9843 Err codemadness.org 70 i 9844

Summary and thanks

Err codemadness.org 70 i 9845

I think the above is some good boilerplate that you can put in your Err codemadness.org 70 i 9846 configure.ac / Makefile.am to integrate a Rust sub-library into Err codemadness.org 70 i 9847 your C code. It handles make-y things like make clean and make Err codemadness.org 70 i 9848 check; debug and release builds; verbose and quiet builds; Err codemadness.org 70 i 9849 builddir != srcdir; all the goodies.

Err codemadness.org 70 i 9850

I think the only thing I'm missing is to check for the cargo-vendor Err codemadness.org 70 i 9851 binary. I'm not sure how to only check for that if I'm the one making Err codemadness.org 70 i 9852 tarballs... maybe an --enable-maintainer-mode flag?

Err codemadness.org 70 i 9853

This would definitely not have been possible without prior work. Err codemadness.org 70 i 9854 Thanks to everyone who figured out Autotools before me, so I could Err codemadness.org 70 i 9855 cut&paste your goodies:

Err codemadness.org 70 i 9856

Update 2017/Nov/11: Fixed the initialization of RUST_EXTRA; thanks Err codemadness.org 70 i 9857 to Tobias Mueller for catching this.

Err codemadness.org 70 i 9858

Hubert Figuière's "Rust and Automake"
Luke Nukem's "Autotools and Rust"
OStree's incantation for cargo vendor
Havoc's "Cargo.lock should be in version control"

How Glib-rs works, part 2: Transferring lists and arrays

2017-08-28T20:26:47-05:00

(First part of the series, with index to all the articles)

Err codemadness.org 70 i 9868

In the first part, we saw how glib-rs provides Err codemadness.org 70 i 9869 the FromGlib and ToGlib traits to let Rust Err codemadness.org 70 i 9870 code convert from/to Glib's simple types, like to convert from a Glib Err codemadness.org 70 i 9871 gboolean to a Rust bool and vice-versa. We also saw the special Err codemadness.org 70 i 9872 needs of strings; since they are passed by reference and are not Err codemadness.org 70 i 9873 copied as simple values, we can use Err codemadness.org 70 i 9874 FromGlibPtrNone and Err codemadness.org 70 i 9875 FromGlibPtrFull depending on what kind of Err codemadness.org 70 i 9876 ownership transfer we want, none for "just make it look like we are Err codemadness.org 70 i 9877 using a borrowed reference", or full for "I'll take over the data and Err codemadness.org 70 i 9878 free it when I'm done". Going the other way around, we can use Err codemadness.org 70 i 9879 ToGlibPtr and its methods to pass things from Rust to Err codemadness.org 70 i 9880 Glib.

Err codemadness.org 70 i 9881

In this part, we'll see the tools that glib-rs provides to do Err codemadness.org 70 i 9882 conversions of more complex data types. We'll look at two cases:

Err codemadness.org 70 i 9883

Err codemadness.org 70 i 9885
Passing null-terminated arrays of strings Err codemadness.org 70 i 9886 from Glib to Rust
Err codemadness.org 70 i 9887
Err codemadness.org 70 i 9889
Passing GLists to Rust
Err codemadness.org 70 i 9890

Err codemadness.org 70 i 9892

And one final case just in passing:

Err codemadness.org 70 i 9893

Passing containers from Rust to Glib

Err codemadness.org 70 i 9896

Passing arrays from Glib to Rust

Err codemadness.org 70 i 9897

We'll look at the case for transferring null-terminated arrays of Err codemadness.org 70 i 9898 strings, since it's an interesting one. There are other traits to Err codemadness.org 70 i 9899 convert from Glib arrays whose length is known, not implied with a Err codemadness.org 70 i 9900 NULL element, but for now we'll only look at arrays of strings.

Err codemadness.org 70 i 9901

Null-terminated arrays of strings

Err codemadness.org 70 i 9902

Look at this function for GtkAboutDialog:

Err codemadness.org 70 i 9903

/**	Err	codemadness.org	70
i  9904  * gtk_about_dialog_add_credit_section:	Err	codemadness.org	70
i  9905  * @about: A #GtkAboutDialog	Err	codemadness.org	70
i  9906  * @section_name: The name of the section	Err	codemadness.org	70
i  9907  * @people: (array zero-terminated=1): The people who belong to that section	Err	codemadness.org	70
i  9908  * ...	Err	codemadness.org	70
i  9909  */	Err	codemadness.org	70
i  9910 void	Err	codemadness.org	70
i  9911 gtk_about_dialog_add_credit_section (GtkAboutDialog  *about,	Err	codemadness.org	70
i  9912                                      const gchar     *section_name,	Err	codemadness.org	70
i  9913                                      const gchar    **people)	Err	codemadness.org	70
i  9914

Err codemadness.org 70 i 9915 Err codemadness.org 70 i 9916

You would use this like

Err codemadness.org 70 i 9917

const gchar *translators[] = {	Err	codemadness.org	70
i  9918     "Alice <alice@example.com>",	Err	codemadness.org	70
i  9919     "Bob <bob@example.com>",	Err	codemadness.org	70
i  9920     "Clara <clara@example.com>",	Err	codemadness.org	70
i  9921     NULL	Err	codemadness.org	70
i  9922 };	Err	codemadness.org	70
i  9923 	Err	codemadness.org	70
i  9924 gtk_about_dialog_add_credit_section (my_about_dialog, _("Translators"), translators);	Err	codemadness.org	70
i  9925

Err codemadness.org 70 i 9926 Err codemadness.org 70 i 9927

The function expects an array of gchar *, where the last element is Err codemadness.org 70 i 9928 a NULL. Instead of passing an explicit length for the array, it's Err codemadness.org 70 i 9929 done implicitly by requiring a NULL pointer after the last element. Err codemadness.org 70 i 9930 The gtk-doc annotation says (array zero-terminated=1). When we Err codemadness.org 70 i 9931 generate information for the GObject-Introspection Repository (GIR), Err codemadness.org 70 i 9932 this is what comes out:

Err codemadness.org 70 i 9933

 1	Err	codemadness.org	70
i  9934  2	Err	codemadness.org	70
i  9935  3	Err	codemadness.org	70
i  9936  4	Err	codemadness.org	70
i  9937  5	Err	codemadness.org	70
i  9938  6	Err	codemadness.org	70
i  9939  7	Err	codemadness.org	70
i  9940  8	Err	codemadness.org	70
i  9941  9	Err	codemadness.org	70
i  9942 10

<method name="add_credit_section"	Err	codemadness.org	70
i  9943         c:identifier="gtk_about_dialog_add_credit_section"	Err	codemadness.org	70
i  9944         version="3.4">	Err	codemadness.org	70
i  9945   ..	Err	codemadness.org	70
i  9946     <parameter name="people" transfer-ownership="none">	Err	codemadness.org	70
i  9947       <doc xml:space="preserve">The people who belong to that section</doc>	Err	codemadness.org	70
i  9948       <array c:type="gchar**">	Err	codemadness.org	70
i  9949         <type name="utf8" c:type="gchar*"/>	Err	codemadness.org	70
i  9950       </array>	Err	codemadness.org	70
i  9951     </parameter>	Err	codemadness.org	70
i  9952

Err codemadness.org 70 i 9953

Err codemadness.org 70 i 9954

You can see the transfer-ownership="none" in line 5. This means Err codemadness.org 70 i 9955 that the function will not take ownership of the passed array; it will Err codemadness.org 70 i 9956 make its own copy instead. By convention, GIR assumes that arrays of Err codemadness.org 70 i 9957 strings are NULL-terminated, so there is no special annotation for Err codemadness.org 70 i 9958 that here. If we were implementing this function in Rust, how would we Err codemadness.org 70 i 9959 read that C array of UTF-8 strings and turn it into a Rust Err codemadness.org 70 i 9960 Vec<String> or something? Easy:

Err codemadness.org 70 i 9961

let c_char_array: *mut *mut c_char = ...; // comes from Glib	Err	codemadness.org	70
i  9962 let rust_translators = FromGlibPtrContainer::from_glib_none(c_char_array);	Err	codemadness.org	70
i  9963 // rust_translators is a Vec<String>	Err	codemadness.org	70
i  9964

Err codemadness.org 70 i 9965 Err codemadness.org 70 i 9966

Let's look at how this bad boy is implemented.

Err codemadness.org 70 i 9967

First stage: `impl FromGlibPtrContainer for Vec<T>`

Err codemadness.org 70 i 9968

We want to go from a "*mut *mut c_char" (in C parlance, a "gchar **") Err codemadness.org 70 i 9969 to a Vec<String>. Indeed, there is an implementation of the Err codemadness.org 70 i 9970 FromGlibPtrContainer trait for Vecs Err codemadness.org 70 i 9971 here. These are the first few lines:

Err codemadness.org 70 i 9972

impl <P: Ptr, PP: Ptr, T: FromGlibPtrArrayContainerAsVec<P, PP>> FromGlibPtrContainer<P, PP> for Vec<T> {	Err	codemadness.org	70
i  9973     unsafe fn from_glib_none(ptr: PP) -> Vec<T> {	Err	codemadness.org	70
i  9974         FromGlibPtrArrayContainerAsVec::from_glib_none_as_vec(ptr)	Err	codemadness.org	70
i  9975     }	Err	codemadness.org	70
i  9976

Err codemadness.org 70 i 9977 Err codemadness.org 70 i 9978

So... that from_glib_none() will return a Vec<T>, which is what we Err codemadness.org 70 i 9979 want. Let's look at the first few lines of FromGlibPtrArrayContainerAsVec:

Err codemadness.org 70 i 9980

1	Err	codemadness.org	70
i  9981 2	Err	codemadness.org	70
i  9982 3	Err	codemadness.org	70
i  9983 4

    impl FromGlibPtrArrayContainerAsVec<$ffi_name, *mut $ffi_name> for $name {	Err	codemadness.org	70
i  9984         unsafe fn from_glib_none_as_vec(ptr: *mut $ffi_name) -> Vec<Self> {	Err	codemadness.org	70
i  9985             FromGlibContainerAsVec::from_glib_none_num_as_vec(ptr, c_ptr_array_len(ptr))	Err	codemadness.org	70
i  9986         }	Err	codemadness.org	70
i  9987

Err codemadness.org 70 i 9988

Err codemadness.org 70 i 9989

Aha! This is inside a macro, thus the $ffi_name garbage. Err codemadness.org 70 i 9990 It's done like that so the same trait can be implemented for const and Err codemadness.org 70 i 9991 mut pointers to c_char.

Err codemadness.org 70 i 9992

See the call to c_ptr_array_len() in line 3? That's what figures Err codemadness.org 70 i 9993 out where the NULL pointer is at the end of the array: it figures out Err codemadness.org 70 i 9994 the array's length.

Err codemadness.org 70 i 9995

Second stage: `impl FromGlibContainerAsVec::from_glib_none_num_as_vec()`

Err codemadness.org 70 i 9996

Now that the length of the array is known, the implementation calls Err codemadness.org 70 i 9997 FromGlibContainerAsVec::from_glib_none_num_as_vec()

Err codemadness.org 70 i 9998

 1	Err	codemadness.org	70
i  9999  2	Err	codemadness.org	70
i 10000  3	Err	codemadness.org	70
i 10001  4	Err	codemadness.org	70
i 10002  5	Err	codemadness.org	70
i 10003  6	Err	codemadness.org	70
i 10004  7	Err	codemadness.org	70
i 10005  8	Err	codemadness.org	70
i 10006  9	Err	codemadness.org	70
i 10007 10	Err	codemadness.org	70
i 10008 11	Err	codemadness.org	70
i 10009 12

    impl FromGlibContainerAsVec<$ffi_name, *const $ffi_name> for $name {	Err	codemadness.org	70
i 10010         unsafe fn from_glib_none_num_as_vec(ptr: *const $ffi_name, num: usize) -> Vec<Self> {	Err	codemadness.org	70
i 10011             if num == 0 || ptr.is_null() {	Err	codemadness.org	70
i 10012                 return Vec::new();	Err	codemadness.org	70
i 10013             }	Err	codemadness.org	70
i 10014 	Err	codemadness.org	70
i 10015             let mut res = Vec::with_capacity(num);	Err	codemadness.org	70
i 10016             for i in 0..num {	Err	codemadness.org	70
i 10017                 res.push(from_glib_none(ptr::read(ptr.offset(i as isize)) as $ffi_name));	Err	codemadness.org	70
i 10018             }	Err	codemadness.org	70
i 10019             res	Err	codemadness.org	70
i 10020         }	Err	codemadness.org	70
i 10021

Err codemadness.org 70 i 10022

Err codemadness.org 70 i 10023

Lines 3/4: If the number of elements is zero, or the array is NULL, Err codemadness.org 70 i 10024 return an empty Vec.

Err codemadness.org 70 i 10025

Line 7: Allocate a Vec of suitable size.

Err codemadness.org 70 i 10026

Lines 8/9: For each of the pointers in the C array, call Err codemadness.org 70 i 10027 from_glib_none() to convert it from a *const c_char to a String, Err codemadness.org 70 i 10028 like we saw in the first part.

Err codemadness.org 70 i 10029

Done! We started with a *mut *mut c_char or a *const *const Err codemadness.org 70 i 10030 c_char and ended up with a Vec<String>, which is what we wanted.

Err codemadness.org 70 i 10031

Passing `GList`s to Rust

Err codemadness.org 70 i 10032

Some functions don't give you an array; they give you a GList or Err codemadness.org 70 i 10033 GSList. There is an implementation of Err codemadness.org 70 i 10034 FromGlibPtrArrayContainerAsVec that understands Err codemadness.org 70 i 10035 GList:

Err codemadness.org 70 i 10036

impl<T> FromGlibPtrArrayContainerAsVec<<T as GlibPtrDefault>::GlibType, *mut glib_ffi::GList> for T	Err	codemadness.org	70
i 10037 where T: GlibPtrDefault + FromGlibPtrNone<<T as GlibPtrDefault>::GlibType> + FromGlibPtrFull<<T as GlibPtrDefault>::GlibType> {	Err	codemadness.org	70
i 10038 	Err	codemadness.org	70
i 10039     unsafe fn from_glib_none_as_vec(ptr: *mut glib_ffi::GList) -> Vec<T> {	Err	codemadness.org	70
i 10040         let num = glib_ffi::g_list_length(ptr) as usize;	Err	codemadness.org	70
i 10041         FromGlibContainer::from_glib_none_num(ptr, num)	Err	codemadness.org	70
i 10042     }	Err	codemadness.org	70
i 10043

Err codemadness.org 70 i 10044 Err codemadness.org 70 i 10045

The impl declaration is pretty horrible, so just look at the Err codemadness.org 70 i 10046 method: from_glib_none_as_vec() takes in a GList, then calls Err codemadness.org 70 i 10047 g_list_length() on it, and finally calls Err codemadness.org 70 i 10048 FromGlibContainer::from_glib_none_num() with the length it computed.

Err codemadness.org 70 i 10049

I have a Glib container and its length

Err codemadness.org 70 i 10050

In turn, that from_glib_none_num() goes here:

Err codemadness.org 70 i 10051

impl <P, PP: Ptr, T: FromGlibContainerAsVec<P, PP>> FromGlibContainer<P, PP> for Vec<T> {	Err	codemadness.org	70
i 10052     unsafe fn from_glib_none_num(ptr: PP, num: usize) -> Vec<T> {	Err	codemadness.org	70
i 10053         FromGlibContainerAsVec::from_glib_none_num_as_vec(ptr, num)	Err	codemadness.org	70
i 10054     }	Err	codemadness.org	70
i 10055

Err codemadness.org 70 i 10056 Err codemadness.org 70 i 10057

Okay, getting closer to the actual implementation.

Err codemadness.org 70 i 10058

Give me a vector already

Err codemadness.org 70 i 10059

Finally, we get to the function that walks the GList:

Err codemadness.org 70 i 10060

 1	Err	codemadness.org	70
i 10061  2	Err	codemadness.org	70
i 10062  3	Err	codemadness.org	70
i 10063  4	Err	codemadness.org	70
i 10064  5	Err	codemadness.org	70
i 10065  6	Err	codemadness.org	70
i 10066  7	Err	codemadness.org	70
i 10067  8	Err	codemadness.org	70
i 10068  9	Err	codemadness.org	70
i 10069 10	Err	codemadness.org	70
i 10070 11	Err	codemadness.org	70
i 10071 12	Err	codemadness.org	70
i 10072 13	Err	codemadness.org	70
i 10073 14	Err	codemadness.org	70
i 10074 15	Err	codemadness.org	70
i 10075 16	Err	codemadness.org	70
i 10076 17

impl<T> FromGlibContainerAsVec<<T as GlibPtrDefault>::GlibType, *mut glib_ffi::GList> for T	Err	codemadness.org	70
i 10077 where T: GlibPtrDefault + FromGlibPtrNone<<T as GlibPtrDefault>::GlibType> + FromGlibPtrFull<<T as GlibPtrDefault>::GlibType> {	Err	codemadness.org	70
i 10078 	Err	codemadness.org	70
i 10079     unsafe fn from_glib_none_num_as_vec(mut ptr: *mut glib_ffi::GList, num: usize) -> Vec<T> {	Err	codemadness.org	70
i 10080         if num == 0 || ptr.is_null() {	Err	codemadness.org	70
i 10081             return Vec::new()	Err	codemadness.org	70
i 10082         }	Err	codemadness.org	70
i 10083         let mut res = Vec::with_capacity(num);	Err	codemadness.org	70
i 10084         for _ in 0..num {	Err	codemadness.org	70
i 10085             let item_ptr: <T as GlibPtrDefault>::GlibType = Ptr::from((*ptr).data);	Err	codemadness.org	70
i 10086             if !item_ptr.is_null() {	Err	codemadness.org	70
i 10087                 res.push(from_glib_none(item_ptr));	Err	codemadness.org	70
i 10088             }	Err	codemadness.org	70
i 10089             ptr = (*ptr).next;	Err	codemadness.org	70
i 10090         }	Err	codemadness.org	70
i 10091         res	Err	codemadness.org	70
i 10092     }	Err	codemadness.org	70
i 10093

Err codemadness.org 70 i 10094

Err codemadness.org 70 i 10095

Again, ignore the horrible impl declaration and just look at Err codemadness.org 70 i 10096 from_glib_none_num_as_vec().

Err codemadness.org 70 i 10097

Line 4: that function takes in a ptr to a GList, and a num with Err codemadness.org 70 i 10098 the list's length, which we already computed above.

Err codemadness.org 70 i 10099

Line 5: Return an empty vector if we have an empty list.

Err codemadness.org 70 i 10100

Line 8: Allocate a vector of suitable capacity.

Err codemadness.org 70 i 10101

Line 9: For each element, convert it with from_glib_none() and push Err codemadness.org 70 i 10102 it to the array.

Err codemadness.org 70 i 10103

Line 14: Walk to the next element in the list.

Err codemadness.org 70 i 10104

Passing containers from Rust to Glib

Err codemadness.org 70 i 10105

This post is getting a bit long, so I'll just mention this briefly. Err codemadness.org 70 i 10106 There is a trait ToGlibContainerFromSlice that takes a Rust slice, Err codemadness.org 70 i 10107 and can convert it to various Glib types.

Err codemadness.org 70 i 10108

Err codemadness.org 70 i 10110
To GSlist and GList. These have Err codemadness.org 70 i 10111 methods like to_glib_none_from_slice() and Err codemadness.org 70 i 10112 to_glib_full_from_slice()
Err codemadness.org 70 i 10113
Err codemadness.org 70 i 10115
To an array of fundamental types. Here, you can choose Err codemadness.org 70 i 10116 between to_glib_none_from_slice(), which gives you a Stash like Err codemadness.org 70 i 10117 we saw the last time. Or, you can use Err codemadness.org 70 i 10118 to_glib_full_from_slice(), which gives you back a g_malloc()ed Err codemadness.org 70 i 10119 array with copied items. Finally, to_glib_container_from_slice() Err codemadness.org 70 i 10120 gives you back a g_malloc()ed array of pointers to values rather Err codemadness.org 70 i 10121 than plain values themselves. Which function you choose depends on Err codemadness.org 70 i 10122 which C API you want to call.
Err codemadness.org 70 i 10123

Err codemadness.org 70 i 10125

I hope this post gives you enough practice to be able to "follow the Err codemadness.org 70 i 10126 traits" for each of those if you want to look at the implementations.

Err codemadness.org 70 i 10127

Next up

Err codemadness.org 70 i 10128

Passing boxed types, like public structs.

Err codemadness.org 70 i 10129

Passing reference-counted types.

Err codemadness.org 70 i 10130

How glib-rs wraps GObjects.

How Glib-rs works, part 1: Type conversions

2017-08-25T16:07:47-05:00

How Glib-rs works, part 1: Type conversions
How Glib-rs works, part 2: Transferring lists and arrays
How Glib-rs works, part 3: Boxed types

Err codemadness.org 70 i 10142

During the GNOME+Rust hackfest in Mexico City, Niko Matsakis Err codemadness.org 70 i 10143 started the implementation of gnome-class, a procedural macro Err codemadness.org 70 i 10144 that will let people implement new GObject classes in Rust and export Err codemadness.org 70 i 10145 them to the world. Currently, if you want to write a new GObject Err codemadness.org 70 i 10146 (e.g. a new widget) and put it in a library so that it can be used Err codemadness.org 70 i 10147 from language bindings via GObject-Introspection, you have to do it in Err codemadness.org 70 i 10148 C. It would be nice to be able to do this in a safe language like Err codemadness.org 70 i 10149 Rust.

Err codemadness.org 70 i 10150

How would it be done by hand?

Err codemadness.org 70 i 10151

In a C implementation of a new GObject subclass, one calls things like Err codemadness.org 70 i 10152 g_type_register_static() and g_signal_new() by hand, while being Err codemadness.org 70 i 10153 careful to specify the correct GType for each value, and being Err codemadness.org 70 i 10154 super-careful about everything, as C demands.

Err codemadness.org 70 i 10155

In Rust, one can in fact do exactly the same thing. You can call the Err codemadness.org 70 i 10156 same, low-level GObject and GType functions. You can use Err codemadness.org 70 i 10157 #[repr(C)]] for the instance and class structs that GObject will Err codemadness.org 70 i 10158 allocate for you, and which you then fill in.

Err codemadness.org 70 i 10159

You can see an example of this in gst-plugins-rs. This is where it implements a Sink Err codemadness.org 70 i 10160 GObject, in Rust, by calling Glib functions by Err codemadness.org 70 i 10161 hand: struct declarations, class_init() function, Err codemadness.org 70 i 10162 registration of type and interfaces.

Err codemadness.org 70 i 10163

How would it be done by a machine?

Err codemadness.org 70 i 10164

That's what Niko's gnome-class is about. During the hackfest it Err codemadness.org 70 i 10165 got to the point of being able to generate the code to create a new Err codemadness.org 70 i 10166 GObject subclass, register it, and export functions for methods. The Err codemadness.org 70 i 10167 syntax is not finalized yet, but it looks something like this:

Err codemadness.org 70 i 10168

gobject_gen! {	Err	codemadness.org	70
i 10169     class Counter {	Err	codemadness.org	70
i 10170         struct CounterPrivate {	Err	codemadness.org	70
i 10171             val: Cell<u32>	Err	codemadness.org	70
i 10172         }	Err	codemadness.org	70
i 10173 	Err	codemadness.org	70
i 10174         signal value_changed(&self);	Err	codemadness.org	70
i 10175 	Err	codemadness.org	70
i 10176         fn set_value(&self, v: u32) {	Err	codemadness.org	70
i 10177             let private = self.private();	Err	codemadness.org	70
i 10178             private.val.set(v);	Err	codemadness.org	70
i 10179             // private.emit_value_changed();	Err	codemadness.org	70
i 10180         }	Err	codemadness.org	70
i 10181 	Err	codemadness.org	70
i 10182         fn get_value(&self) -> u32 {	Err	codemadness.org	70
i 10183             let private = self.private();	Err	codemadness.org	70
i 10184             private.val.get()	Err	codemadness.org	70
i 10185         }	Err	codemadness.org	70
i 10186     }	Err	codemadness.org	70
i 10187 }	Err	codemadness.org	70
i 10188

Err codemadness.org 70 i 10189 Err codemadness.org 70 i 10190

I started adding support for declaring GObject signals — mainly being Err codemadness.org 70 i 10191 able to parse them from what goes inside gobject_gen!() — and then Err codemadness.org 70 i 10192 being able to call g_signal_newv() at the appropriate time during Err codemadness.org 70 i 10193 the class_init() implementation.

Err codemadness.org 70 i 10194

Types in signals

Err codemadness.org 70 i 10195

Creating a signal for a GObject class is basically like specifying a Err codemadness.org 70 i 10196 function prototype: the object will invoke a callback function with Err codemadness.org 70 i 10197 certain arguments and return value when the signal is emitted. For Err codemadness.org 70 i 10198 example, this is how GtkButton registers its button-press-event Err codemadness.org 70 i 10199 signal:

Err codemadness.org 70 i 10200

  button_press_event_id =	Err	codemadness.org	70
i 10201     g_signal_new (I_("button-press-event"),	Err	codemadness.org	70
i 10202                   ...	Err	codemadness.org	70
i 10203                   G_TYPE_BOOLEAN,    /* type of return value */	Err	codemadness.org	70
i 10204                   1,                 /* how many arguments? */	Err	codemadness.org	70
i 10205                   GDK_TYPE_EVENT);   /* type of first and only argument */	Err	codemadness.org	70
i 10206

Err codemadness.org 70 i 10207 Err codemadness.org 70 i 10208

g_signal_new() creates the signal and returns a signal id, an Err codemadness.org 70 i 10209 integer. Later, when the object wants to emit the signal, it uses Err codemadness.org 70 i 10210 that signal id like this:

Err codemadness.org 70 i 10211

GtkEventButton event = ...;	Err	codemadness.org	70
i 10212 gboolean return_val;	Err	codemadness.org	70
i 10213 	Err	codemadness.org	70
i 10214 g_signal_emit (widget, button_press_event_id, 0, event, &return_val);	Err	codemadness.org	70
i 10215

Err codemadness.org 70 i 10216 Err codemadness.org 70 i 10217

In the nice gobject_gen!() macro, if I am going to have a signal Err codemadness.org 70 i 10218 declaration like

Err codemadness.org 70 i 10219

signal button_press_event(&self, event: &ButtonPressEvent) -> bool;	Err	codemadness.org	70
i 10220

Err codemadness.org 70 i 10221 Err codemadness.org 70 i 10222

then I will need to be able to translate the type names for Err codemadness.org 70 i 10223 ButtonPressEvent and bool into something that g_signal_newv() will Err codemadness.org 70 i 10224 understand: I need the GType values for those. Fundamental Err codemadness.org 70 i 10225 types like gboolean get constants like G_TYPE_BOOLEAN. Types Err codemadness.org 70 i 10226 that are defined at runtime, like GDK_TYPE_EVENT, get GType values Err codemadness.org 70 i 10227 generated at runtime, too, when one registers the type with Err codemadness.org 70 i 10228 g_type_register_*().

Err codemadness.org 70 i 10229 Err codemadness.org 70 i 10230 Err codemadness.org 70 i 10231 Err codemadness.org 70 i 10232 Err codemadness.org 70 i 10233 Err codemadness.org 70 i 10234 Err codemadness.org 70 i 10235 Err codemadness.org 70 i 10236 Err codemadness.org 70 i 10237 Err codemadness.org 70 i 10238 Err codemadness.org 70 i 10239 Err codemadness.org 70 i 10240 Err codemadness.org 70 i 10241 Err codemadness.org 70 i 10242 Err codemadness.org 70 i 10243 Err codemadness.org 70 i 10244 Err codemadness.org 70 i 10245 Err codemadness.org 70 i 10246 Err codemadness.org 70 i 10247 Err codemadness.org 70 i 10248 Err codemadness.org 70 i 10249 Err codemadness.org 70 i 10250 Err codemadness.org 70 i 10251 Err codemadness.org 70 i 10252 Err codemadness.org 70 i 10253 Err codemadness.org 70 i 10254

Rust type	GType
i32	G_TYPE_INT
u32	G_TYPE_UINT
bool	G_TYPE_BOOLEAN
etc.	etc.

Err codemadness.org 70 i 10255

Glib types in Rust

Err codemadness.org 70 i 10256

How does glib-rs, the Rust binding to Glib and GObject, handle Err codemadness.org 70 i 10257 types?

Err codemadness.org 70 i 10258

Going from Glib to Rust

Err codemadness.org 70 i 10259

First we need a way to convert Glib's types to Rust, and vice-versa. Err codemadness.org 70 i 10260 There is a trait to convert simple Glib types into Rust types:

Err codemadness.org 70 i 10261

pub trait FromGlib<T>: Sized {	Err	codemadness.org	70
i 10262     fn from_glib(val: T) -> Self;	Err	codemadness.org	70
i 10263 }	Err	codemadness.org	70
i 10264

Err codemadness.org 70 i 10265 Err codemadness.org 70 i 10266

This means, if I have a T which is a Glib type, this trait will give Err codemadness.org 70 i 10267 you a from_glib() function which will convert it to a Rust type Err codemadness.org 70 i 10268 which is Sized, i.e. a type whose size is known at compilation time.

Err codemadness.org 70 i 10269

For example, this is how it is implemented for booleans:

Err codemadness.org 70 i 10270

impl FromGlib<glib_ffi::gboolean> for bool {	Err	codemadness.org	70
i 10271     #[inline]	Err	codemadness.org	70
i 10272     fn from_glib(val: glib_ffi::gboolean) -> bool {	Err	codemadness.org	70
i 10273         !(val == glib_ffi::GFALSE)	Err	codemadness.org	70
i 10274     }	Err	codemadness.org	70
i 10275 }	Err	codemadness.org	70
i 10276

Err codemadness.org 70 i 10277 Err codemadness.org 70 i 10278

and you use it like this:

Err codemadness.org 70 i 10279

let my_gboolean: glib_ffi::gboolean = g_some_function_that_returns_gboolean ();	Err	codemadness.org	70
i 10280 	Err	codemadness.org	70
i 10281 let my_rust_bool: bool = from_glib (my_gboolean);	Err	codemadness.org	70
i 10282

Err codemadness.org 70 i 10283 Err codemadness.org 70 i 10284

Booleans in glib and Rust have different sizes, and also Err codemadness.org 70 i 10285 different values. Glib's booleans use the C convention: 0 is false Err codemadness.org 70 i 10286 and anything else is true, while in Rust booleans are strictly false Err codemadness.org 70 i 10287 or true, and the size is undefined (with the current Rust ABI, it's Err codemadness.org 70 i 10288 one byte).

Err codemadness.org 70 i 10289

Going from Rust to Glib

Err codemadness.org 70 i 10290

And to go the other way around, from a Rust bool to a gboolean? Err codemadness.org 70 i 10291 There is this trait:

Err codemadness.org 70 i 10292

pub trait ToGlib {	Err	codemadness.org	70
i 10293     type GlibType;	Err	codemadness.org	70
i 10294 	Err	codemadness.org	70
i 10295     fn to_glib(&self) -> Self::GlibType;	Err	codemadness.org	70
i 10296 }	Err	codemadness.org	70
i 10297

Err codemadness.org 70 i 10298 Err codemadness.org 70 i 10299

This means, if you have a Rust type that maps to a corresponding Err codemadness.org 70 i 10300 GlibType, this will give you a to_glib() function to do the Err codemadness.org 70 i 10301 conversion.

Err codemadness.org 70 i 10302

This is the implementation for booleans:

Err codemadness.org 70 i 10303

impl ToGlib for bool {	Err	codemadness.org	70
i 10304     type GlibType = glib_ffi::gboolean;	Err	codemadness.org	70
i 10305 	Err	codemadness.org	70
i 10306     #[inline]	Err	codemadness.org	70
i 10307     fn to_glib(&self) -> glib_ffi::gboolean {	Err	codemadness.org	70
i 10308         if *self { glib_ffi::GTRUE } else { glib_ffi::GFALSE }	Err	codemadness.org	70
i 10309     }	Err	codemadness.org	70
i 10310 }	Err	codemadness.org	70
i 10311

Err codemadness.org 70 i 10312 Err codemadness.org 70 i 10313

And it is used like this:

Err codemadness.org 70 i 10314

let my_rust_bool: bool = true;	Err	codemadness.org	70
i 10315 	Err	codemadness.org	70
i 10316 g_some_function_that_takes_gboolean (my_rust_bool.to_glib ());	Err	codemadness.org	70
i 10317

Err codemadness.org 70 i 10318 Err codemadness.org 70 i 10319

(If you are thinking "a function call to marshal a boolean" — note how Err codemadness.org 70 i 10320 the functions are inlined, and the optimizer basically compiles them Err codemadness.org 70 i 10321 down to nothing.)

Err codemadness.org 70 i 10322

Pointer types - from Glib to Rust

Err codemadness.org 70 i 10323

That's all very nice for simple types like booleans and ints. Err codemadness.org 70 i 10324 Pointers to other objects are slightly more complicated.

Err codemadness.org 70 i 10325

GObject-Introspection allows one to specify how pointer arguments to Err codemadness.org 70 i 10326 functions are handled by using a transfer specifier.

Err codemadness.org 70 i 10327

`(transfer none)`

Err codemadness.org 70 i 10328

For example, if you call gtk_window_set_title(window, "Hello"), you Err codemadness.org 70 i 10329 would expect the function to make its own copy of the "Hello" Err codemadness.org 70 i 10330 string. In Rust terms, you would be passing it a simple borrowed Err codemadness.org 70 i 10331 reference. GObject-Introspection (we'll abbreviate it as GI) calls Err codemadness.org 70 i 10332 this GI_TRANSFER_NOTHING, and it's specified by using Err codemadness.org 70 i 10333 (transfer none) in the documentation strings for function arguments Err codemadness.org 70 i 10334 or return values.

Err codemadness.org 70 i 10335

The corresponding trait to bring in pointers from Glib to Rust, Err codemadness.org 70 i 10336 without taking ownership, is this. It's unsafe because it will be Err codemadness.org 70 i 10337 used to de-reference pointers that come from the wild west:

Err codemadness.org 70 i 10338

pub trait FromGlibPtrNone<P: Ptr>: Sized {	Err	codemadness.org	70
i 10339     unsafe fn from_glib_none(ptr: P) -> Self;	Err	codemadness.org	70
i 10340 }	Err	codemadness.org	70
i 10341

Err codemadness.org 70 i 10342 Err codemadness.org 70 i 10343

And you use it via this generic function:

Err codemadness.org 70 i 10344

#[inline]	Err	codemadness.org	70
i 10345 pub unsafe fn from_glib_none<P: Ptr, T: FromGlibPtrNone<P>>(ptr: P) -> T {	Err	codemadness.org	70
i 10346     FromGlibPtrNone::from_glib_none(ptr)	Err	codemadness.org	70
i 10347 }	Err	codemadness.org	70
i 10348

Err codemadness.org 70 i 10349 Err codemadness.org 70 i 10350

Let's look at how this works. Here is the FromGlibPtrNone trait Err codemadness.org 70 i 10351 implemented for strings.

Err codemadness.org 70 i 10352

1	Err	codemadness.org	70
i 10353 2	Err	codemadness.org	70
i 10354 3	Err	codemadness.org	70
i 10355 4	Err	codemadness.org	70
i 10356 5	Err	codemadness.org	70
i 10357 6	Err	codemadness.org	70
i 10358 7

impl FromGlibPtrNone<*const c_char> for String {	Err	codemadness.org	70
i 10359     #[inline]	Err	codemadness.org	70
i 10360     unsafe fn from_glib_none(ptr: *const c_char) -> Self {	Err	codemadness.org	70
i 10361         assert!(!ptr.is_null());	Err	codemadness.org	70
i 10362         String::from_utf8_lossy(CStr::from_ptr(ptr).to_bytes()).into_owned()	Err	codemadness.org	70
i 10363     }	Err	codemadness.org	70
i 10364 }	Err	codemadness.org	70
i 10365

Err codemadness.org 70 i 10366

Err codemadness.org 70 i 10367

Line 1: given a pointer to a c_char, the conversion to String...

Err codemadness.org 70 i 10368

Line 4: check for NULL pointers

Err codemadness.org 70 i 10369

Line 5: Use the CStr to wrap the C Err codemadness.org 70 i 10370 ptr, like we looked at last time, validate it as UTF-8 and Err codemadness.org 70 i 10371 copy the string for us.

Err codemadness.org 70 i 10372

Unfortunately, there's a copy involved in the last step. It may be Err codemadness.org 70 i 10373 possible to use Cow<&str> there instead to avoid a copy if Err codemadness.org 70 i 10374 the char* from Glib is indeed valid UTF-8.

Err codemadness.org 70 i 10375

`(transfer full)`

Err codemadness.org 70 i 10376

And how about transferring ownership of the pointed-to value? There Err codemadness.org 70 i 10377 is this trait:

Err codemadness.org 70 i 10378

pub trait FromGlibPtrFull<P: Ptr>: Sized {	Err	codemadness.org	70
i 10379     unsafe fn from_glib_full(ptr: P) -> Self;	Err	codemadness.org	70
i 10380 }	Err	codemadness.org	70
i 10381

Err codemadness.org 70 i 10382 Err codemadness.org 70 i 10383

And the implementation for strings is as follows. In Glib's scheme of Err codemadness.org 70 i 10384 things, "transferring ownership of a string" means that the recipient Err codemadness.org 70 i 10385 of the string must eventually g_free() it.

Err codemadness.org 70 i 10386

1	Err	codemadness.org	70
i 10387 2	Err	codemadness.org	70
i 10388 3	Err	codemadness.org	70
i 10389 4	Err	codemadness.org	70
i 10390 5	Err	codemadness.org	70
i 10391 6	Err	codemadness.org	70
i 10392 7	Err	codemadness.org	70
i 10393 8

impl FromGlibPtrFull<*const c_char> for String {	Err	codemadness.org	70
i 10394     #[inline]	Err	codemadness.org	70
i 10395     unsafe fn from_glib_full(ptr: *const c_char) -> Self {	Err	codemadness.org	70
i 10396         let res = from_glib_none(ptr);	Err	codemadness.org	70
i 10397         glib_ffi::g_free(ptr as *mut _);	Err	codemadness.org	70
i 10398         res	Err	codemadness.org	70
i 10399     }	Err	codemadness.org	70
i 10400 }	Err	codemadness.org	70
i 10401

Err codemadness.org 70 i 10402

Err codemadness.org 70 i 10403

Line 1: given a pointer to a c_char, the conversion to String...

Err codemadness.org 70 i 10404

Line 4: Do the conversion with from_glib_none() with the trait we Err codemadness.org 70 i 10405 saw before, put it in res.

Err codemadness.org 70 i 10406

Line 5: Call g_free() on the original C string.

Err codemadness.org 70 i 10407

Line 6: Return the res, a Rust string which we own.

Err codemadness.org 70 i 10408

Pointer types - from Rust to Glib

Err codemadness.org 70 i 10409

Consider the case where you want to pass a String from Rust to a Glib function Err codemadness.org 70 i 10410 that takes a *const c_char — in C parlance, a char *, without the Err codemadness.org 70 i 10411 Glib function acquiring ownership of the string. For example, assume Err codemadness.org 70 i 10412 that the C version of gtk_window_set_title() is in the gtk_ffi Err codemadness.org 70 i 10413 module. You may want to call it like this:

Err codemadness.org 70 i 10414

fn rust_binding_to_window_set_title(window: &Gtk::Window, title: &String) {	Err	codemadness.org	70
i 10415     gtk_ffi::gtk_window_set_title(..., make_c_string_from_rust_string(title));	Err	codemadness.org	70
i 10416 }	Err	codemadness.org	70
i 10417

Err codemadness.org 70 i 10418 Err codemadness.org 70 i 10419

Now, what would that make_c_string_from_rust_string() look like?

Err codemadness.org 70 i 10420

Err codemadness.org 70 i 10422
We have: a Rust String — UTF-8, known length, no nul terminator
Err codemadness.org 70 i 10423
Err codemadness.org 70 i 10425
We want: a *const char — nul-terminated UTF-8
Err codemadness.org 70 i 10426

Err codemadness.org 70 i 10428

So, let's write this:

Err codemadness.org 70 i 10429

1	Err	codemadness.org	70
i 10430 2	Err	codemadness.org	70
i 10431 3	Err	codemadness.org	70
i 10432 4	Err	codemadness.org	70
i 10433 5

fn make_c_string_from_rust_string(s: &String) -> *const c_char {	Err	codemadness.org	70
i 10434     let cstr = CString::new(&s[..]).unwrap();	Err	codemadness.org	70
i 10435     let ptr = cstr.into_raw() as *const c_char;	Err	codemadness.org	70
i 10436     ptr	Err	codemadness.org	70
i 10437 }	Err	codemadness.org	70
i 10438

Err codemadness.org 70 i 10439

Err codemadness.org 70 i 10440

Line 1: Take in a &String; return a *const c_char.

Err codemadness.org 70 i 10441

Line 2: Build a CString like we way a few days ago: this Err codemadness.org 70 i 10442 allocates a byte buffer with space for a nul terminator, and copies Err codemadness.org 70 i 10443 the string's bytes. We unwrap() for this simple example, because Err codemadness.org 70 i 10444 CString::new() will return an error if the String contained nul Err codemadness.org 70 i 10445 characters in the middle of the string, which C doesn't understand.

Err codemadness.org 70 i 10446

Line 3: Call into_raw() to get a pointer to the byte buffer, and Err codemadness.org 70 i 10447 cast it to a *const c_char. We'll need to free this value later.

Err codemadness.org 70 i 10448

But this kind of sucks, because we the have to use this function, pass Err codemadness.org 70 i 10449 the pointer to a C function, and then reconstitute the CString so it Err codemadness.org 70 i 10450 can free the byte buffer:

Err codemadness.org 70 i 10451

let buf = make_c_string_from_rust_string(my_string);	Err	codemadness.org	70
i 10452 unsafe { c_function_that_takes_a_string(buf); }	Err	codemadness.org	70
i 10453 let _ = CString::from_raw(buf as *mut c_char);	Err	codemadness.org	70
i 10454

Err codemadness.org 70 i 10455 Err codemadness.org 70 i 10456

The solution that Glib-rs provides for this is very Rusty, and rather Err codemadness.org 70 i 10457 elegant.

Err codemadness.org 70 i 10458

Stashes

Err codemadness.org 70 i 10459

We want:

Err codemadness.org 70 i 10460

A temporary place to put a piece of data
A pointer to that buffer
Automatic memory management for both of those

Err codemadness.org 70 i 10465

Glib-rs defines a Stash for this:

Err codemadness.org 70 i 10466

1	Err	codemadness.org	70
i 10467 2	Err	codemadness.org	70
i 10468 3	Err	codemadness.org	70
i 10469 4	Err	codemadness.org	70
i 10470 5	Err	codemadness.org	70
i 10471 6

pub struct Stash<'a,                                 // we have a lifetime	Err	codemadness.org	70
i 10472                  P: Copy,                            // the pointer must be copy-able	Err	codemadness.org	70
i 10473                  T: ?Sized + ToGlibPtr<'a, P>> (     // Type for the temporary place	Err	codemadness.org	70
i 10474     pub P,                                           // We store a pointer...	Err	codemadness.org	70
i 10475     pub <T as ToGlibPtr<'a, P>>::Storage             // ... to a piece of data with that lifetime ...	Err	codemadness.org	70
i 10476 );	Err	codemadness.org	70
i 10477

Err codemadness.org 70 i 10478

Err codemadness.org 70 i 10479

... and the piece of data must be of of the associated type Err codemadness.org 70 i 10480 ToGlibPtr::Storage, which we will see shortly.

Err codemadness.org 70 i 10481

This struct Stash goes along with the ToGlibPtr trait:

Err codemadness.org 70 i 10482

pub trait ToGlibPtr<'a, P: Copy> {	Err	codemadness.org	70
i 10483     type Storage;	Err	codemadness.org	70
i 10484 	Err	codemadness.org	70
i 10485     fn to_glib_none(&'a self) -> Stash<'a, P, Self>;  // returns a Stash whose temporary storage	Err	codemadness.org	70
i 10486                                                       // has the lifetime of our original data	Err	codemadness.org	70
i 10487 }	Err	codemadness.org	70
i 10488

Err codemadness.org 70 i 10489 Err codemadness.org 70 i 10490

Let's unpack this by looking at the implementation of the "transfer a Err codemadness.org 70 i 10491 String to a C function while keeping ownership":

Err codemadness.org 70 i 10492

1	Err	codemadness.org	70
i 10493 2	Err	codemadness.org	70
i 10494 3	Err	codemadness.org	70
i 10495 4	Err	codemadness.org	70
i 10496 5	Err	codemadness.org	70
i 10497 6	Err	codemadness.org	70
i 10498 7	Err	codemadness.org	70
i 10499 8	Err	codemadness.org	70
i 10500 9

impl <'a> ToGlibPtr<'a, *const c_char> for String {	Err	codemadness.org	70
i 10501     type Storage = CString;	Err	codemadness.org	70
i 10502 	Err	codemadness.org	70
i 10503     #[inline]	Err	codemadness.org	70
i 10504     fn to_glib_none(&self) -> Stash<'a, *const c_char, String> {	Err	codemadness.org	70
i 10505         let tmp = CString::new(&self[..]).unwrap();	Err	codemadness.org	70
i 10506         Stash(tmp.as_ptr(), tmp)	Err	codemadness.org	70
i 10507     }	Err	codemadness.org	70
i 10508 }	Err	codemadness.org	70
i 10509

Err codemadness.org 70 i 10510

Err codemadness.org 70 i 10511

Line 1: We implement ToGlibPtr<'a *const c_char> for String, Err codemadness.org 70 i 10512 declaring the lifetime 'a for the Stash.

Err codemadness.org 70 i 10513

Line 2: Our temporary storage is a CString.

Err codemadness.org 70 i 10514

Line 6: Make a CString like before.

Err codemadness.org 70 i 10515

Line 7: Create the Stash with a pointer to the CString's contents, Err codemadness.org 70 i 10516 and the CString itself.

Err codemadness.org 70 i 10517

`(transfer none)`

Err codemadness.org 70 i 10518

Now, we can use ".0" to extract the first field from our Stash, Err codemadness.org 70 i 10519 which is precisely the pointer we want to a byte buffer:

Err codemadness.org 70 i 10520

let my_string = ...;	Err	codemadness.org	70
i 10521 unsafe { c_function_which_takes_a_string(my_string.to_glib_none().0); }	Err	codemadness.org	70
i 10522

Err codemadness.org 70 i 10523 Err codemadness.org 70 i 10524

Now Rust knows that the temporary buffer inside the Stash has the lifetime of Err codemadness.org 70 i 10525 my_string, and it will free it automatically when the string goes Err codemadness.org 70 i 10526 out of scope. If we can accept the .to_glib_none().0 incantation Err codemadness.org 70 i 10527 for "lending" pointers to C, this works perfectly.

Err codemadness.org 70 i 10528

`(transfer full)`

Err codemadness.org 70 i 10529

And for transferring ownership to the C function? The ToGlibPtr Err codemadness.org 70 i 10530 trait has another method:

Err codemadness.org 70 i 10531

pub trait ToGlibPtr<'a, P: Copy> {	Err	codemadness.org	70
i 10532     ...	Err	codemadness.org	70
i 10533 	Err	codemadness.org	70
i 10534     fn to_glib_full(&self) -> P;	Err	codemadness.org	70
i 10535 }	Err	codemadness.org	70
i 10536

Err codemadness.org 70 i 10537 Err codemadness.org 70 i 10538

And here is the implementation for strings:

Err codemadness.org 70 i 10539

impl <'a> ToGlibPtr<'a, *const c_char> for String {	Err	codemadness.org	70
i 10540     fn to_glib_full(&self) -> *const c_char {	Err	codemadness.org	70
i 10541         unsafe {	Err	codemadness.org	70
i 10542             glib_ffi::g_strndup(self.as_ptr() as *const c_char, 	Err	codemadness.org	70
i 10543                                 self.len() as size_t)	Err	codemadness.org	70
i 10544                 as *const c_char	Err	codemadness.org	70
i 10545         }	Err	codemadness.org	70
i 10546     }	Err	codemadness.org	70
i 10547

Err codemadness.org 70 i 10548 Err codemadness.org 70 i 10549

We basically g_strndup() the Rust string's contents from its byte Err codemadness.org 70 i 10550 buffer and its len(), and we can then pass this on to C. That Err codemadness.org 70 i 10551 code will be responsible for g_free()ing the C-side string.

Err codemadness.org 70 i 10552

Next up

Err codemadness.org 70 i 10553

Transferring lists and arrays. Stay tuned!

Correctness in Rust: building strings

2017-08-16T20:26:39-05:00

Rust tries to follow the "make illegal states unrepresentable" mantra Err codemadness.org 70 i 10559 in several ways. In this post I'll show several things related to the Err codemadness.org 70 i 10560 process of building strings, from bytes in memory, or from a file, or Err codemadness.org 70 i 10561 from char * things passed from C.

Err codemadness.org 70 i 10562

Strings in Rust

Err codemadness.org 70 i 10563

The easiest way to build a string is to do it directly at compile Err codemadness.org 70 i 10564 time:

Err codemadness.org 70 i 10565

let my_string = "Hello, world!";	Err	codemadness.org	70
i 10566

Err codemadness.org 70 i 10567 Err codemadness.org 70 i 10568

In Rust, strings are UTF-8. Here, the compiler checks our string Err codemadness.org 70 i 10569 literal is valid UTF-8. If we try to be sneaky and insert an Err codemadness.org 70 i 10570 invalid character...

Err codemadness.org 70 i 10571

let my_string = "Hello \xf0";	Err	codemadness.org	70
i 10572

Err codemadness.org 70 i 10573 Err codemadness.org 70 i 10574

We get a compiler error:

Err codemadness.org 70 i 10575

error: this form of character escape may only be used with characters in the range [\x00-\x7f]	Err	codemadness.org	70
i 10576  --> foo.rs:2:30	Err	codemadness.org	70
i 10577   |	Err	codemadness.org	70
i 10578 2 |     let my_string = "Hello \xf0";	Err	codemadness.org	70
i 10579   |                              ^^	Err	codemadness.org	70
i 10580

Err codemadness.org 70 i 10581 Err codemadness.org 70 i 10582

Rust strings know their length, unlike C strings. They can contain Err codemadness.org 70 i 10583 a nul character in the middle, because they don't need a nul Err codemadness.org 70 i 10584 terminator at the end.

Err codemadness.org 70 i 10585

let my_string = "Hello \x00 zero";	Err	codemadness.org	70
i 10586 println!("{}", my_string);	Err	codemadness.org	70
i 10587

Err codemadness.org 70 i 10588 Err codemadness.org 70 i 10589

The output is what you expect:

Err codemadness.org 70 i 10590

$ ./foo | hexdump -C	Err	codemadness.org	70
i 10591 00000000  48 65 6c 6c 6f 20 00 20  7a 65 72 6f 0a           |Hello . zero.|	Err	codemadness.org	70
i 10592 0000000d                    ^ note the nul char here	Err	codemadness.org	70
i 10593 $	Err	codemadness.org	70
i 10594

Err codemadness.org 70 i 10595 Err codemadness.org 70 i 10596

So, to summarize, in Rust:

Err codemadness.org 70 i 10597

Strings are encoded in UTF-8
Strings know their length
Strings can have nul chars in the middle

Err codemadness.org 70 i 10602

This is a bit different from C:

Err codemadness.org 70 i 10603

Strings don't exist!

Err codemadness.org 70 i 10606

Okay, just kidding. In C:

Err codemadness.org 70 i 10607

A lot of software has standardized on UTF-8.
Strings don't know their length - a char * is a raw pointer to the Err codemadness.org 70 i 10610 beginning of the string.
Strings conventionally have a nul terminator, that is, a zero byte Err codemadness.org 70 i 10612 that marks the end of the string. Therefore, you can't have nul Err codemadness.org 70 i 10613 characters in the middle of strings.

Err codemadness.org 70 i 10615

Building a string from bytes

Err codemadness.org 70 i 10616

Let's say you have an array of bytes and want to make a string from Err codemadness.org 70 i 10617 them. Rust won't let you just cast the array, like C would. First Err codemadness.org 70 i 10618 you need to do UTF-8 validation. For example:

Err codemadness.org 70 i 10619

 1	Err	codemadness.org	70
i 10620  2	Err	codemadness.org	70
i 10621  3	Err	codemadness.org	70
i 10622  4	Err	codemadness.org	70
i 10623  5	Err	codemadness.org	70
i 10624  6	Err	codemadness.org	70
i 10625  7	Err	codemadness.org	70
i 10626  8	Err	codemadness.org	70
i 10627  9	Err	codemadness.org	70
i 10628 10	Err	codemadness.org	70
i 10629 11	Err	codemadness.org	70
i 10630 12

fn convert_and_print(bytes: Vec<u8>) {	Err	codemadness.org	70
i 10631     let result = String::from_utf8(bytes);	Err	codemadness.org	70
i 10632     match result {	Err	codemadness.org	70
i 10633         Ok(string) => println!("{}", string),	Err	codemadness.org	70
i 10634         Err(e) => println!("{:?}", e)	Err	codemadness.org	70
i 10635     }	Err	codemadness.org	70
i 10636 }	Err	codemadness.org	70
i 10637 	Err	codemadness.org	70
i 10638 fn main() {	Err	codemadness.org	70
i 10639     convert_and_print(vec![0x48, 0x65, 0x6c, 0x6c, 0x6f]);	Err	codemadness.org	70
i 10640     convert_and_print(vec![0x48, 0x65, 0xf0, 0x6c, 0x6c, 0x6f]);	Err	codemadness.org	70
i 10641 }	Err	codemadness.org	70
i 10642

Err codemadness.org 70 i 10643

Err codemadness.org 70 i 10644

In lines 10 and 11, we call convert_and_print() with different Err codemadness.org 70 i 10645 arrays of bytes; the first one is valid UTF-8, and the second one Err codemadness.org 70 i 10646 isn't.

Err codemadness.org 70 i 10647

Line 2 calls String::from_utf8(), which returns a Result, Err codemadness.org 70 i 10648 i.e. something with a success value or an error. In lines 3-5 we Err codemadness.org 70 i 10649 unpack this Result. If it's Ok, we print the converted string, Err codemadness.org 70 i 10650 which has been validated for UTF-8. Otherwise, we print the debug Err codemadness.org 70 i 10651 representation of the error.

Err codemadness.org 70 i 10652

The program prints the following:

Err codemadness.org 70 i 10653

$ ~/foo	Err	codemadness.org	70
i 10654 Hello	Err	codemadness.org	70
i 10655 FromUtf8Error { bytes: [72, 101, 240, 108, 108, 111], error: Utf8Error { valid_up_to: 2, error_len: Some(1) } }	Err	codemadness.org	70
i 10656

Err codemadness.org 70 i 10657 Err codemadness.org 70 i 10658

Here, in the error case, the Utf8Error tells us that the bytes Err codemadness.org 70 i 10659 are UTF-8 and are valid_up_to index 2; that is the first problematic Err codemadness.org 70 i 10660 index. We also get some extra information which lets the program know Err codemadness.org 70 i 10661 if the problematic sequence was incomplete and truncated at the end of Err codemadness.org 70 i 10662 the byte array, or if it's complete and in the middle.

Err codemadness.org 70 i 10663

And for a "just make this printable, pls" API? We can Err codemadness.org 70 i 10664 use String::from_utf8_lossy(), which replaces invalid UTF-8 Err codemadness.org 70 i 10665 sequences with U+FFFD REPLACEMENT CHARACTER:

Err codemadness.org 70 i 10666

fn convert_and_print(bytes: Vec<u8>) {	Err	codemadness.org	70
i 10667     let string = String::from_utf8_lossy(&bytes);	Err	codemadness.org	70
i 10668     println!("{}", string);	Err	codemadness.org	70
i 10669 }	Err	codemadness.org	70
i 10670 	Err	codemadness.org	70
i 10671 fn main() {	Err	codemadness.org	70
i 10672     convert_and_print(vec![0x48, 0x65, 0x6c, 0x6c, 0x6f]);	Err	codemadness.org	70
i 10673     convert_and_print(vec![0x48, 0x65, 0xf0, 0x6c, 0x6c, 0x6f]);	Err	codemadness.org	70
i 10674 }	Err	codemadness.org	70
i 10675

Err codemadness.org 70 i 10676 Err codemadness.org 70 i 10677

This prints the following:

Err codemadness.org 70 i 10678

$ ~/foo	Err	codemadness.org	70
i 10679 Hello	Err	codemadness.org	70
i 10680 He�llo	Err	codemadness.org	70
i 10681

Err codemadness.org 70 i 10682 Err codemadness.org 70 i 10683

Reading from files into strings

Err codemadness.org 70 i 10684

Now, let's assume you want to read chunks of a file and put them into Err codemadness.org 70 i 10685 strings. Let's go from the low-level parts up to the high level "read Err codemadness.org 70 i 10686 a line" API.

Err codemadness.org 70 i 10687

Single bytes and single UTF-8 characters

Err codemadness.org 70 i 10688

When you open a File, you get an object that implements the Err codemadness.org 70 i 10689 Read trait. In addition to the usual "read me some bytes" method, Err codemadness.org 70 i 10690 it can also give you back an iterator over bytes, or an iterator Err codemadness.org 70 i 10691 over UTF-8 characters.

Err codemadness.org 70 i 10692

The Read.bytes() method gives you back a Bytes iterator, Err codemadness.org 70 i 10693 whose next() method returns Result<u8, io::Error>. When you ask Err codemadness.org 70 i 10694 the iterator for its next item, that Result means you'll get a byte Err codemadness.org 70 i 10695 out of it successfully, or an I/O error.

Err codemadness.org 70 i 10696

In contrast, the Read.chars() method gives you back Err codemadness.org 70 i 10697 a Chars iterator, and its next() method returns Err codemadness.org 70 i 10698 Result<char, CharsError>, not io::Error. This Err codemadness.org 70 i 10699 extended CharsError has a NotUtf8 case, which you get back Err codemadness.org 70 i 10700 when next() tries to read the next UTF-8 sequence from the file and Err codemadness.org 70 i 10701 the file has invalid data. CharsError also has a case for normal Err codemadness.org 70 i 10702 I/O errors.

Err codemadness.org 70 i 10703

Reading lines

Err codemadness.org 70 i 10704

While you could build a UTF-8 string one character at a time, there Err codemadness.org 70 i 10705 are more efficient ways to do it.

Err codemadness.org 70 i 10706

You can create a BufReader, a buffered reader, out of anything Err codemadness.org 70 i 10707 that implements the Read trait. BufReader has a Err codemadness.org 70 i 10708 convenient read_line() method, to which you pass a mutable Err codemadness.org 70 i 10709 String and it returns a Result<usize, io::Error> with either the Err codemadness.org 70 i 10710 number of bytes read, or an error.

Err codemadness.org 70 i 10711

That method is declared in the BufRead trait, which BufReader Err codemadness.org 70 i 10712 implements. Why the separation? Because other concrete structs also Err codemadness.org 70 i 10713 implement BufRead, such as Cursor — a nice wrapper that lets Err codemadness.org 70 i 10714 you use a vector of bytes like an I/O Read or Write Err codemadness.org 70 i 10715 implementation, similar to GMemoryInputStream.

Err codemadness.org 70 i 10716

If you prefer an iterator rather than the read_line() function, Err codemadness.org 70 i 10717 BufRead also gives you a lines() method, which gives you back Err codemadness.org 70 i 10718 a Lines iterator.

Err codemadness.org 70 i 10719

In both cases — the read_line() method or the Lines iterator, the Err codemadness.org 70 i 10720 error that you can get back can be of ErrorKind::InvalidData, Err codemadness.org 70 i 10721 which indicates that there was an invalid UTF-8 sequence in the line Err codemadness.org 70 i 10722 to be read. It can also be a normal I/O error, of course.

Err codemadness.org 70 i 10723

Summary so far

Err codemadness.org 70 i 10724

There is no way to build a String, or a &str slice, from invalid Err codemadness.org 70 i 10725 UTF-8 data. All the methods that let you turn bytes into string-like Err codemadness.org 70 i 10726 things perform validation, and return a Result to let you know if Err codemadness.org 70 i 10727 your bytes validated correctly.

Err codemadness.org 70 i 10728

The exceptions are in the unsafe methods, Err codemadness.org 70 i 10729 like String::from_utf8_unchecked(). You should really only use Err codemadness.org 70 i 10730 them if you are absolutely sure that your bytes were validated as Err codemadness.org 70 i 10731 UTF-8 beforehand.

Err codemadness.org 70 i 10732

There is no way to bring in data from a file (or anything file-like, Err codemadness.org 70 i 10733 that implements the Read trait) and turn it into a String Err codemadness.org 70 i 10734 without going through functions that do UTF-8 validation. There is Err codemadness.org 70 i 10735 not an unsafe "read a line" API without validation — you would have to Err codemadness.org 70 i 10736 build one yourself, but the I/O hit is probably going to be slower than Err codemadness.org 70 i 10737 validating data in memory, anyway, so you may as well validate.

Err codemadness.org 70 i 10738

C strings and Rust

Err codemadness.org 70 i 10739

For unfortunate historical reasons, C flings around char * to mean Err codemadness.org 70 i 10740 different things. In the context of Glib, it can mean

Err codemadness.org 70 i 10741

A valid, nul-terminated UTF-8 sequence of bytes (a "normal string")
A nul-terminated file path, which has no meaningful encoding
A nul-terminated sequence of bytes, not validated as UTF-8.

Err codemadness.org 70 i 10746

What a particular char * means depends on which API you got it from.

Err codemadness.org 70 i 10747

Bringing a string from C to Rust

Err codemadness.org 70 i 10748

From Rust's viewpoint, getting a raw char * from C (a "*const Err codemadness.org 70 i 10749 c_char" in Rust parlance) means that it gets a pointer to a buffer of Err codemadness.org 70 i 10750 unknown length.

Err codemadness.org 70 i 10751

Now, that may not be entirely accurate:

Err codemadness.org 70 i 10752

You may indeed only have a pointer to a buffer of unknown length
You may have a pointer to a buffer, and also know its length Err codemadness.org 70 i 10755 (i.e. the offset at which the nul terminator is)

Err codemadness.org 70 i 10757

The Rust standard library provides a CStr object, which means, Err codemadness.org 70 i 10758 "I have a pointer to an array of bytes, and I know its length, and I Err codemadness.org 70 i 10759 know the last byte is a nul".

Err codemadness.org 70 i 10760

CStr provides an unsafe from_ptr() constructor which takes a Err codemadness.org 70 i 10761 raw pointer, and walks the memory to which it points until it finds a Err codemadness.org 70 i 10762 nul byte. You must give it a valid pointer, and you had better Err codemadness.org 70 i 10763 guarantee that there is a nul terminator, or CStr will walk until Err codemadness.org 70 i 10764 the end of your process' address space looking for one.

Err codemadness.org 70 i 10765

Alternatively, if you know the length of your byte array, and you know Err codemadness.org 70 i 10766 that it has a nul byte at the end, you can Err codemadness.org 70 i 10767 call CStr::from_bytes_with_nul(). You pass it a &[u8] slice; Err codemadness.org 70 i 10768 the function will check that a) the last byte in that slice is indeed Err codemadness.org 70 i 10769 a nul, and b) there are no nul bytes in the middle.

Err codemadness.org 70 i 10770

The unsafe version of this last function Err codemadness.org 70 i 10771 is unsafe CStr::from_bytes_with_nul_unchecked(): it also takes Err codemadness.org 70 i 10772 an &[u8] slice, but you must guarantee that the last byte is a nul Err codemadness.org 70 i 10773 and that there are no nul bytes in the middle.

Err codemadness.org 70 i 10774

I really like that the Rust documentation tells you when functions Err codemadness.org 70 i 10775 are not "instantaneous" and must instead walks arrays, like to do Err codemadness.org 70 i 10776 validation or to look for the nul terminator above.

Err codemadness.org 70 i 10777

Turning a CStr into a string-like

Err codemadness.org 70 i 10778

Now, the above indicates that a CStr is a nul-terminated array of Err codemadness.org 70 i 10779 bytes. We have no idea what the bytes inside look like; we just know Err codemadness.org 70 i 10780 that they don't contain any other nul bytes.

Err codemadness.org 70 i 10781

There is a CStr::to_str() method, which returns a Err codemadness.org 70 i 10782 Result<&str, Utf8Error>. It performs UTF-8 validation on the array Err codemadness.org 70 i 10783 of bytes. If the array is valid, the function just returns a slice of Err codemadness.org 70 i 10784 the validated bytes minus the nul terminator (i.e. just what you Err codemadness.org 70 i 10785 expect for a Rust string slice). Otherwise, it returns an Utf8Error Err codemadness.org 70 i 10786 with the details like we discussed before.

Err codemadness.org 70 i 10787

There is also CStr::to_string_lossy() which does the Err codemadness.org 70 i 10788 replacement of invalid UTF-8 sequences like we discussed before.

Err codemadness.org 70 i 10789

Conclusion

Err codemadness.org 70 i 10790

Strings in Rust are UTF-8 encoded, they know their length, and they Err codemadness.org 70 i 10791 can have nul bytes in the middle.

Err codemadness.org 70 i 10792

To build a string from raw bytes, you must go through functions that Err codemadness.org 70 i 10793 do UTF-8 validation and tell you if it failed. There are unsafe Err codemadness.org 70 i 10794 functions that let you skip validation, but then of course you are on Err codemadness.org 70 i 10795 your own.

Err codemadness.org 70 i 10796

The low-level functions which read data from files operate on bytes. Err codemadness.org 70 i 10797 On top of those, there are convenience functions to read validated Err codemadness.org 70 i 10798 UTF-8 characters, lines, etc. All of these tell you when there was Err codemadness.org 70 i 10799 invalid UTF-8 or an I/O error.

Err codemadness.org 70 i 10800

Rust lets you wrap a raw char * that you got from C into something Err codemadness.org 70 i 10801 that can later be validated and turned into a string. Anything that Err codemadness.org 70 i 10802 manipulates a raw pointer is unsafe; this includes the "wrap me this Err codemadness.org 70 i 10803 pointer into a C string abstraction" API, and the "build me an array Err codemadness.org 70 i 10804 of bytes from this raw pointer" API. Later, you can validate those Err codemadness.org 70 i 10805 as UTF-8 and build real Rust strings — or know if the validation Err codemadness.org 70 i 10806 failed.

Err codemadness.org 70 i 10807

Rust builds these little "corridors" through the API so that illegal Err codemadness.org 70 i 10808 states are unrepresentable.

GUADEC 2017 presentation

2017-08-09T20:20:49-05:00

During GUADEC this year I gave a presentation Err codemadness.org 70 i 10816 called Err codemadness.org 70 i 10817 Replacing C library code with Rust: what I learned with librsvg. Err codemadness.org 70 i 10818 This is the PDF file; be sure to scroll past the full-page Err codemadness.org 70 i 10819 presentation pages until you reach the speaker's notes, especially for Err codemadness.org 70 i 10820 the code sections!

Err codemadness.org 70 i 10821

Err codemadness.org 70 i 10822

You can also get the ODP file for the presentation. This is Err codemadness.org 70 i 10823 released under a CC-BY-SA license.

Err codemadness.org 70 i 10824

For the presentation, my daughter Luciana made some drawings of Err codemadness.org 70 i 10825 Ferris, the Rust mascot, also released under the same license:

Err codemadness.org 70 i 10826

Err codemadness.org 70 i 10827 Err codemadness.org 70 i 10828 Err codemadness.org 70 i 10829

Surviving a rust-cssparser API break

2017-08-01T06:02:30-05:00

Yesterday I looked into updating librsvg's Rust dependencies. There have Err codemadness.org 70 i 10835 been some API breaks (!!!) in the unstable libraries that it uses Err codemadness.org 70 i 10836 since the last time I locked them. This post is about an interesting Err codemadness.org 70 i 10837 case of API breakage.

Err codemadness.org 70 i 10838

rust-cssparser is the crate that Servo uses for parsing Err codemadness.org 70 i 10839 CSS. Well, more like tokenizing CSS: you give it a string, it Err codemadness.org 70 i 10840 gives you back tokens, and you are supposed to compose CSS selector Err codemadness.org 70 i 10841 information or other CSS values from the tokens.

Err codemadness.org 70 i 10842

Librsvg uses rust-cssparser now for most of the micro-languages in Err codemadness.org 70 i 10843 SVG's attribute values, instead of its old, fragile C parsers. I hope Err codemadness.org 70 i 10844 to be able to use it in conjunction with Servo's rust-selectors Err codemadness.org 70 i 10845 crate to fully parse CSS data and replace libcroco.

Err codemadness.org 70 i 10846

A few months ago, rust-cssparser's API looked more or less like the Err codemadness.org 70 i 10847 following. This is the old representation of a Token:

Err codemadness.org 70 i 10848

pub enum Token<'a> {	Err	codemadness.org	70
i 10849     // an identifier	Err	codemadness.org	70
i 10850     Ident(Cow<'a, str>),	Err	codemadness.org	70
i 10851 	Err	codemadness.org	70
i 10852     // a plain number	Err	codemadness.org	70
i 10853     Number(NumericValue),	Err	codemadness.org	70
i 10854 	Err	codemadness.org	70
i 10855     // a percentage value normalized to [0.0, 1.0]	Err	codemadness.org	70
i 10856     Percentage(PercentageValue),	Err	codemadness.org	70
i 10857 	Err	codemadness.org	70
i 10858     WhiteSpace(&'a str),	Err	codemadness.org	70
i 10859     Comma,	Err	codemadness.org	70
i 10860 	Err	codemadness.org	70
i 10861     ...	Err	codemadness.org	70
i 10862 }	Err	codemadness.org	70
i 10863

Err codemadness.org 70 i 10864 Err codemadness.org 70 i 10865

That is, a Token can be an Identifier with a string name, or a Err codemadness.org 70 i 10866 Number, a Percentage, whitespace, a comma, and many others.

Err codemadness.org 70 i 10867

On top of that is the old API for a Parser, which you construct with Err codemadness.org 70 i 10868 a string and then it gives you back tokens:

Err codemadness.org 70 i 10869

impl<'i> Parser<'i> {	Err	codemadness.org	70
i 10870     pub fn new(input: &'i str) -> Parser<'i, 'i> {	Err	codemadness.org	70
i 10871 	Err	codemadness.org	70
i 10872     pub fn next(&mut self) -> Result<Token<'i>, ()> { ... }	Err	codemadness.org	70
i 10873 	Err	codemadness.org	70
i 10874     ...	Err	codemadness.org	70
i 10875 }	Err	codemadness.org	70
i 10876

Err codemadness.org 70 i 10877 Err codemadness.org 70 i 10878

This means the following. You create the parser out of a string slice Err codemadness.org 70 i 10879 with new(). You can then extract a Result with a Token Err codemadness.org 70 i 10880 sucessfully, or with an empty error value. The parser uses a lifetime Err codemadness.org 70 i 10881 'i on the string from which it is constructed: the Tokens that Err codemadness.org 70 i 10882 return identifiers, for example, could return sub-string slices that Err codemadness.org 70 i 10883 come from the original string, and the parser has to be marked with a Err codemadness.org 70 i 10884 lifetime so that it does not outlive its underlying string.

Err codemadness.org 70 i 10885

A few commits later, rust-cssparser got changed to return detailed Err codemadness.org 70 i 10886 error values, so that instead of () you get a a BasicParseError Err codemadness.org 70 i 10887 with sub-cases like UnexpectedToken or EndOfInput.

Err codemadness.org 70 i 10888

After the changes to the error values for results, I didn't pay much Err codemadness.org 70 i 10889 attention to rust-cssparser for while. Yesterday I wanted to update Err codemadness.org 70 i 10890 librsvg to use the newest rust-cssparser, and had some interesting Err codemadness.org 70 i 10891 problems.

Err codemadness.org 70 i 10892

First, Parser::new() was changed from taking just a &str slice to Err codemadness.org 70 i 10893 taking a ParserInput struct. This is an implementation detail which Err codemadness.org 70 i 10894 lets the parser cache the last token it saw. Not a big deal:

Err codemadness.org 70 i 10895

// instead of constructing a parser like	Err	codemadness.org	70
i 10896 let mut parser = Parser::new (my_string);	Err	codemadness.org	70
i 10897 	Err	codemadness.org	70
i 10898 // you now construct it like	Err	codemadness.org	70
i 10899 let mut input = ParserInput::new (my_string);	Err	codemadness.org	70
i 10900 let mut parser = Parser::new (&mut input);	Err	codemadness.org	70
i 10901

Err codemadness.org 70 i 10902 Err codemadness.org 70 i 10903

I am not completely sure why this is exposed to the public API, since Err codemadness.org 70 i 10904 Rust won't allow you to have two mutable references to a Err codemadness.org 70 i 10905 ParserInput, and the only consumer of a (mutable) ParserInput is Err codemadness.org 70 i 10906 the Parser, anyway.

Err codemadness.org 70 i 10907

However, the parser.next() function changed:

Err codemadness.org 70 i 10908

// old version	Err	codemadness.org	70
i 10909 pub fn next(&mut self) -> Result<Token<'i>, ()> { ... }	Err	codemadness.org	70
i 10910 	Err	codemadness.org	70
i 10911 // new version	Err	codemadness.org	70
i 10912 pub fn next(&mut self) -> Result<&Token<'i>, BasicParseError<'i>> {... }	Err	codemadness.org	70
i 10913 // note this bad boy here -------^	Err	codemadness.org	70
i 10914

Err codemadness.org 70 i 10915 Err codemadness.org 70 i 10916

The successful Result from next() is now a reference to a Err codemadness.org 70 i 10917 Token, not a plain Token value which you now own. The parser is Err codemadness.org 70 i 10918 giving you a borrowed reference to its internally-cached token.

Err codemadness.org 70 i 10919

My parsing functions for the old API looked similar to the Err codemadness.org 70 i 10920 following. This is a function that parses a string into an angle; it Err codemadness.org 70 i 10921 can look like "45deg" or "1.5rad", for example.

Err codemadness.org 70 i 10922

 1	Err	codemadness.org	70
i 10923  2	Err	codemadness.org	70
i 10924  3	Err	codemadness.org	70
i 10925  4	Err	codemadness.org	70
i 10926  5	Err	codemadness.org	70
i 10927  6	Err	codemadness.org	70
i 10928  7	Err	codemadness.org	70
i 10929  8	Err	codemadness.org	70
i 10930  9	Err	codemadness.org	70
i 10931 10	Err	codemadness.org	70
i 10932 11	Err	codemadness.org	70
i 10933 12	Err	codemadness.org	70
i 10934 13	Err	codemadness.org	70
i 10935 14	Err	codemadness.org	70
i 10936 15	Err	codemadness.org	70
i 10937 16	Err	codemadness.org	70
i 10938 17	Err	codemadness.org	70
i 10939 18	Err	codemadness.org	70
i 10940 19	Err	codemadness.org	70
i 10941 20	Err	codemadness.org	70
i 10942 21	Err	codemadness.org	70
i 10943 22	Err	codemadness.org	70
i 10944 23	Err	codemadness.org	70
i 10945 24	Err	codemadness.org	70
i 10946 25	Err	codemadness.org	70
i 10947 26

pub fn parse_angle_degrees (s: &str) -> Result <f64, ParseError> {	Err	codemadness.org	70
i 10948     let mut parser = Parser::new (s);	Err	codemadness.org	70
i 10949 	Err	codemadness.org	70
i 10950     let token = parser.next ()	Err	codemadness.org	70
i 10951         .map_err (|_| ParseError::new ("expected angle"))?;	Err	codemadness.org	70
i 10952 	Err	codemadness.org	70
i 10953     match token {	Err	codemadness.org	70
i 10954         Token::Number (NumericValue { value, .. }) => Ok (value as f64),	Err	codemadness.org	70
i 10955 	Err	codemadness.org	70
i 10956         Token::Dimension (NumericValue { value, .. }, unit) => {	Err	codemadness.org	70
i 10957             let value = value as f64;	Err	codemadness.org	70
i 10958 	Err	codemadness.org	70
i 10959             match unit.as_ref () {	Err	codemadness.org	70
i 10960                 "deg"  => Ok (value),	Err	codemadness.org	70
i 10961                 "grad" => Ok (value * 360.0 / 400.0),	Err	codemadness.org	70
i 10962                 "rad"  => Ok (value * 180.0 / PI),	Err	codemadness.org	70
i 10963                 _      => Err (ParseError::new ("expected angle"))	Err	codemadness.org	70
i 10964             }	Err	codemadness.org	70
i 10965         },	Err	codemadness.org	70
i 10966 	Err	codemadness.org	70
i 10967         _ => Err (ParseError::new ("expected angle"))	Err	codemadness.org	70
i 10968     }.and_then (|r|	Err	codemadness.org	70
i 10969                 parser.expect_exhausted ()	Err	codemadness.org	70
i 10970                 .map (|_| r)	Err	codemadness.org	70
i 10971                 .map_err (|_| ParseError::new ("expected angle")))	Err	codemadness.org	70
i 10972 }	Err	codemadness.org	70
i 10973

Err codemadness.org 70 i 10974

Err codemadness.org 70 i 10975

This is a bit ugly, but it was the first version that passed the Err codemadness.org 70 i 10976 tests. Lines 4 and 5 mean, "get the first token or return an error". Err codemadness.org 70 i 10977 Line 17 means, "anything except deg, grad, or rad for the units Err codemadness.org 70 i 10978 causes the match expression to generate an error". Finally, I was Err codemadness.org 70 i 10979 feeling very proud of using and_then() in line 22, with Err codemadness.org 70 i 10980 parser.expect_exhausted(), to ensure that the parser would not find Err codemadness.org 70 i 10981 any more tokens after the angle/units.

Err codemadness.org 70 i 10982

However, in the new version of rust-cssparser, Parser.next() gives Err codemadness.org 70 i 10983 back a Result with a &Token success value — a reference to a Err codemadness.org 70 i 10984 token —, while the old version returned a plain Token. No problem, Err codemadness.org 70 i 10985 I thought, I'm just going to de-reference the value in the match and Err codemadness.org 70 i 10986 be done with it:

Err codemadness.org 70 i 10987

    let token = parser.next ()	Err	codemadness.org	70
i 10988         .map_err (|_| ParseError::new ("expected angle"))?;	Err	codemadness.org	70
i 10989 	Err	codemadness.org	70
i 10990     match *token {	Err	codemadness.org	70
i 10991     //    ^ dereference here...	Err	codemadness.org	70
i 10992         Token::Number { value, .. } => value as f64,	Err	codemadness.org	70
i 10993 	Err	codemadness.org	70
i 10994         Token::Dimension { value, ref unit, .. } => {	Err	codemadness.org	70
i 10995     //                            ^ avoid moving the unit value	Err	codemadness.org	70
i 10996

Err codemadness.org 70 i 10997 Err codemadness.org 70 i 10998

The compiler complained elsewhere. The whole function now looked like Err codemadness.org 70 i 10999 this:

Err codemadness.org 70 i 11000

 1	Err	codemadness.org	70
i 11001  2	Err	codemadness.org	70
i 11002  3	Err	codemadness.org	70
i 11003  4	Err	codemadness.org	70
i 11004  5	Err	codemadness.org	70
i 11005  6	Err	codemadness.org	70
i 11006  7	Err	codemadness.org	70
i 11007  8	Err	codemadness.org	70
i 11008  9	Err	codemadness.org	70
i 11009 10	Err	codemadness.org	70
i 11010 11	Err	codemadness.org	70
i 11011 12	Err	codemadness.org	70
i 11012 13

pub fn parse_angle_degrees (s: &str) -> Result <f64, ParseError> {	Err	codemadness.org	70
i 11013     let mut parser = Parser::new (s);	Err	codemadness.org	70
i 11014 	Err	codemadness.org	70
i 11015     let token = parser.next ()	Err	codemadness.org	70
i 11016         .map_err (|_| ParseError::new ("expected angle"))?;	Err	codemadness.org	70
i 11017 	Err	codemadness.org	70
i 11018     match token {	Err	codemadness.org	70
i 11019         // ...	Err	codemadness.org	70
i 11020     }.and_then (|r|	Err	codemadness.org	70
i 11021                 parser.expect_exhausted ()	Err	codemadness.org	70
i 11022                 .map (|_| r)	Err	codemadness.org	70
i 11023                 .map_err (|_| ParseError::new ("expected angle")))	Err	codemadness.org	70
i 11024 }	Err	codemadness.org	70
i 11025

Err codemadness.org 70 i 11026

Err codemadness.org 70 i 11027

But in line 4, token is now a reference to something that lives Err codemadness.org 70 i 11028 inside parser, and parser is therefore borrowed mutably. The Err codemadness.org 70 i 11029 compiler didn't like that line 10 (the call to Err codemadness.org 70 i 11030 parser.expect_exhausted()) was trying to borrow parser mutably Err codemadness.org 70 i 11031 again.

Err codemadness.org 70 i 11032

I played a bit with creating a temporary scope around the assignment Err codemadness.org 70 i 11033 to token so that it would only borrow parser mutably inside that Err codemadness.org 70 i 11034 scope. Things ended up like this, without the call to and_then() Err codemadness.org 70 i 11035 after the match:

Err codemadness.org 70 i 11036

 1	Err	codemadness.org	70
i 11037  2	Err	codemadness.org	70
i 11038  3	Err	codemadness.org	70
i 11039  4	Err	codemadness.org	70
i 11040  5	Err	codemadness.org	70
i 11041  6	Err	codemadness.org	70
i 11042  7	Err	codemadness.org	70
i 11043  8	Err	codemadness.org	70
i 11044  9	Err	codemadness.org	70
i 11045 10	Err	codemadness.org	70
i 11046 11	Err	codemadness.org	70
i 11047 12	Err	codemadness.org	70
i 11048 13	Err	codemadness.org	70
i 11049 14	Err	codemadness.org	70
i 11050 15	Err	codemadness.org	70
i 11051 16	Err	codemadness.org	70
i 11052 17	Err	codemadness.org	70
i 11053 18	Err	codemadness.org	70
i 11054 19	Err	codemadness.org	70
i 11055 20	Err	codemadness.org	70
i 11056 21	Err	codemadness.org	70
i 11057 22	Err	codemadness.org	70
i 11058 23	Err	codemadness.org	70
i 11059 24	Err	codemadness.org	70
i 11060 25	Err	codemadness.org	70
i 11061 26	Err	codemadness.org	70
i 11062 27	Err	codemadness.org	70
i 11063 28	Err	codemadness.org	70
i 11064 29	Err	codemadness.org	70
i 11065 30

pub fn angle_degrees (s: &str) -> Result <f64, ParseError> {	Err	codemadness.org	70
i 11066     let mut input = ParserInput::new (s);	Err	codemadness.org	70
i 11067     let mut parser = Parser::new (&mut input);	Err	codemadness.org	70
i 11068 	Err	codemadness.org	70
i 11069     let angle = {	Err	codemadness.org	70
i 11070         let token = parser.next ()	Err	codemadness.org	70
i 11071             .map_err (|_| ParseError::new ("expected angle"))?;	Err	codemadness.org	70
i 11072 	Err	codemadness.org	70
i 11073         match *token {	Err	codemadness.org	70
i 11074             Token::Number { value, .. } => value as f64,	Err	codemadness.org	70
i 11075 	Err	codemadness.org	70
i 11076             Token::Dimension { value, ref unit, .. } => {	Err	codemadness.org	70
i 11077                 let value = value as f64;	Err	codemadness.org	70
i 11078 	Err	codemadness.org	70
i 11079                 match unit.as_ref () {	Err	codemadness.org	70
i 11080                     "deg"  => value,	Err	codemadness.org	70
i 11081                     "grad" => value * 360.0 / 400.0,	Err	codemadness.org	70
i 11082                     "rad"  => value * 180.0 / PI,	Err	codemadness.org	70
i 11083                     _      => return Err (ParseError::new ("expected 'deg' | 'grad' | 'rad'"))	Err	codemadness.org	70
i 11084                 }	Err	codemadness.org	70
i 11085             },	Err	codemadness.org	70
i 11086 	Err	codemadness.org	70
i 11087             _ => return Err (ParseError::new ("expected angle"))	Err	codemadness.org	70
i 11088         }	Err	codemadness.org	70
i 11089     };	Err	codemadness.org	70
i 11090 	Err	codemadness.org	70
i 11091     parser.expect_exhausted ().map_err (|_| ParseError::new ("expected angle"))?;	Err	codemadness.org	70
i 11092 	Err	codemadness.org	70
i 11093     Ok (angle)	Err	codemadness.org	70
i 11094 }	Err	codemadness.org	70
i 11095

Err codemadness.org 70 i 11096

Err codemadness.org 70 i 11097

Lines 5 through 25 are basically

Err codemadness.org 70 i 11098

    let angle = {	Err	codemadness.org	70
i 11099         // parse out the angle; return if error	Err	codemadness.org	70
i 11100     };	Err	codemadness.org	70
i 11101

Err codemadness.org 70 i 11102 Err codemadness.org 70 i 11103

And after that is done, I test for parser.expect_exhausted(). Err codemadness.org 70 i 11104 There is no chaining of results with helper functions; instead it's Err codemadness.org 70 i 11105 just going through each token linearly.

Err codemadness.org 70 i 11106

The API break was annoying to deal with, but fortunately the calling Err codemadness.org 70 i 11107 code ended up cleaner, and I didn't have to change anything in the Err codemadness.org 70 i 11108 tests. I hope rust-cssparser can stabilize its API for consumers that Err codemadness.org 70 i 11109 are not Servo.

Legacy Systems as Old Cities

2017-06-28T21:26:00-05:00

I just realized that I only tweeted about this a couple of months ago, Err codemadness.org 70 i 11114 but never blogged about it. Shame on me!

Err codemadness.org 70 i 11115

I wrote an article, Legacy Systems as Old Cities, for The Err codemadness.org 70 i 11116 Recompiler magazine. Is GNOME, now at 20 years old, legacy software? Err codemadness.org 70 i 11117 Is it different from mainframe software because "everyone" can change Err codemadness.org 70 i 11118 it? Does long-lived software have the same patterns of change as Err codemadness.org 70 i 11119 cities and physical artifacts? Can we learn from the building trades Err codemadness.org 70 i 11120 and urbanism for maintaining software in the long term? Could we Err codemadness.org 70 i 11121 turn legacy software into a good legacy?

Err codemadness.org 70 i 11122

You can read the article here.

Err codemadness.org 70 i 11123

Also, let me take this opportunity to recommend The Recompiler Err codemadness.org 70 i 11124 magazine. It is the most enjoyable technical publication I read. Err codemadness.org 70 i 11125 Their podcast is also excellent!

Err codemadness.org 70 i 11126

Update 2017/06/10 - Spanish version of the article, Los Sistemas Heredados como Ciudades Viejas

Setting Alt-Tab behavior in gnome-shell

2017-06-22T10:25:02-05:00

After updating my distro a few months ago, I somehow lost my tweaks to Err codemadness.org 70 i 11131 the Alt-Tab behavior in gnome-shell.

Err codemadness.org 70 i 11132

The default is to have Alt-Tab switch you between applications in the Err codemadness.org 70 i 11133 current workspace. One can use Alt-backtick (or whatever key you Err codemadness.org 70 i 11134 have above Tab) to switch between windows in the current application.

Err codemadness.org 70 i 11135

I prefer a Windows-like setup, where Alt-Tab switches between Err codemadness.org 70 i 11136 windows in the current workspace, regardless of the application to Err codemadness.org 70 i 11137 which they belong.

Err codemadness.org 70 i 11138

Many moons ago there was a gnome-shell extension to change this Err codemadness.org 70 i 11139 behavior, but these days (GNOME 3.24) it can be done without Err codemadness.org 70 i 11140 extensions. It is a bit convoluted.

Err codemadness.org 70 i 11141

With the GUI

Err codemadness.org 70 i 11142

If you are using X instead of Wayland, this works:

Err codemadness.org 70 i 11143

Err codemadness.org 70 i 11145
Unset the Switch applications command. To do this, run Err codemadness.org 70 i 11146 gnome-control-center, go to Keyboard, and find the Switch Err codemadness.org 70 i 11147 applications command. Click on it, and hit Backspace in the Err codemadness.org 70 i 11148 dialog that prompts you for the keyboard shortcut. Click on the Err codemadness.org 70 i 11149 Set button.
Err codemadness.org 70 i 11150
Err codemadness.org 70 i 11152
Set the Switch windows command. While still in the Err codemadness.org 70 i 11153 Keyboard settings, find the Switch windows command. Click on Err codemadness.org 70 i 11154 it, and hit Alt-Tab. Click Set.
Err codemadness.org 70 i 11155

Err codemadness.org 70 i 11157

That should be all you need, unless you are in Wayland. In that case, Err codemadness.org 70 i 11158 you need to do it on the command line.

Err codemadness.org 70 i 11159

With the command line, or in Wayland

Err codemadness.org 70 i 11160

The kind people on #gnome-hackers tell me that as of GNOME Err codemadness.org 70 i 11161 3.24, changing Alt-Tab doesn't work on Wayland as in (2) above, Err codemadness.org 70 i 11162 because the compositor captures the Alt-Tab key when you type it Err codemadness.org 70 i 11163 inside the dialog that prompts you for a keyboard shortcut. In that Err codemadness.org 70 i 11164 case, you have to change the configuration keys directly instead of Err codemadness.org 70 i 11165 using the GUI:

Err codemadness.org 70 i 11166

gsettings set org.gnome.desktop.wm.keybindings switch-applications "[]"	Err	codemadness.org	70
i 11167 gsettings set org.gnome.desktop.wm.keybindings switch-applications-backward "[]"	Err	codemadness.org	70
i 11168 gsettings set org.gnome.desktop.wm.keybindings switch-windows "['<Alt>Tab', '<Super>Tab']"	Err	codemadness.org	70
i 11169 gsettings set org.gnome.desktop.wm.keybindings switch-windows-backward  "['<Alt><Shift>Tab', '<Super><Shift>Tab']"	Err	codemadness.org	70
i 11170

Err codemadness.org 70 i 11171 Err codemadness.org 70 i 11172

Of course the above also works in X, too.

Err codemadness.org 70 i 11173

Changing windows across all workspaces

Err codemadness.org 70 i 11174

If you'd like to switch between windows in all workspaces, rather than Err codemadness.org 70 i 11175 in the current workspace, find the org.gnome.shell.window-switcher Err codemadness.org 70 i 11176 current-workspace-only GSettings key and change it. You can do this Err codemadness.org 70 i 11177 in dconf-editor, or on the command line with

Err codemadness.org 70 i 11178

gsettings set org.gnome.shell.window-switcher current-workspace-only true	Err	codemadness.org	70
i 11179

Exploring Rust's standard library: system calls and errors

2017-06-12T10:55:26-05:00

In this post I'll show you the code path that Rust takes inside its Err codemadness.org 70 i 11184 standard library when you open a file. I wanted to learn how Rust Err codemadness.org 70 i 11185 handles system calls and errno, and all the little subtleties of the Err codemadness.org 70 i 11186 POSIX API. This is what I learned!

Err codemadness.org 70 i 11187

The C side of things

Err codemadness.org 70 i 11188

When you open a file, or create a socket, or do anything else that Err codemadness.org 70 i 11189 returns an object that can be accessed like a file, you get a file Err codemadness.org 70 i 11190 descriptor in the form of an int.

Err codemadness.org 70 i 11191

/* All of these return a int with a file descriptor, or	Err	codemadness.org	70
i 11192  * -1 in case of error.	Err	codemadness.org	70
i 11193  */	Err	codemadness.org	70
i 11194 int open(const char *pathname, int flags, ...);	Err	codemadness.org	70
i 11195 int socket(int domain, int type, int protocol);	Err	codemadness.org	70
i 11196

Err codemadness.org 70 i 11197 Err codemadness.org 70 i 11198

You get a nonnegative integer in case of success, or -1 in case of an Err codemadness.org 70 i 11199 error. If there's an error, you look at errno, which gives you an Err codemadness.org 70 i 11200 integer error code.

Err codemadness.org 70 i 11201

int fd;	Err	codemadness.org	70
i 11202 	Err	codemadness.org	70
i 11203 retry_open:	Err	codemadness.org	70
i 11204 fd = open ("/foo/bar/baz.txt", 0);	Err	codemadness.org	70
i 11205 if (fd == -1) {	Err	codemadness.org	70
i 11206     if (errno == ENOENT) {	Err	codemadness.org	70
i 11207         /* File doesn't exist */	Err	codemadness.org	70
i 11208     } else if (errno == ...) [	Err	codemadness.org	70
i 11209         ...	Err	codemadness.org	70
i 11210     } else if (errno == EINTR) {	Err	codemadness.org	70
i 11211         goto retry_open; /* interrupted system call; let's retry */	Err	codemadness.org	70
i 11212     }	Err	codemadness.org	70
i 11213 }	Err	codemadness.org	70
i 11214

Err codemadness.org 70 i 11215 Err codemadness.org 70 i 11216

Many system calls can return EINTR, which means "interrupted system Err codemadness.org 70 i 11217 call", which means that something interrupted the kernel while it Err codemadness.org 70 i 11218 was doing your system call and it returned control to userspace, with Err codemadness.org 70 i 11219 the syscall unfinished. For example, your process may have received a Err codemadness.org 70 i 11220 Unix signal (e.g. you send it SIGSTOP by pressing Ctrl-Z on a Err codemadness.org 70 i 11221 terminal, or you resized the terminal and your process got a Err codemadness.org 70 i 11222 SIGWINCH). Most of the time EINTR means simply that you must Err codemadness.org 70 i 11223 retry the operation: if you Control-Z a program to suspend it, and Err codemadness.org 70 i 11224 then fg to continue it again; and if the program was in the middle Err codemadness.org 70 i 11225 of open()ing a file, you would expect it to continue at that exact Err codemadness.org 70 i 11226 point and to actually open the file. Software that doesn't check for Err codemadness.org 70 i 11227 EINTR can fail in very subtle ways!

Err codemadness.org 70 i 11228

Once you have an open file descriptor, you can read from it:

Err codemadness.org 70 i 11229

ssize_t	Err	codemadness.org	70
i 11230 read_five_bytes (int fd, void *buf)	Err	codemadness.org	70
i 11231 {	Err	codemadness.org	70
i 11232     ssize_t result;	Err	codemadness.org	70
i 11233 	Err	codemadness.org	70
i 11234     retry:	Err	codemadness.org	70
i 11235     result = read (fd, buf, 5);	Err	codemadness.org	70
i 11236     if (result == -1) {	Err	codemadness.org	70
i 11237         if (errno == EINTR) {	Err	codemadness.org	70
i 11238             goto retry;	Err	codemadness.org	70
i 11239         } else {	Err	codemadness.org	70
i 11240             return -1; /* the caller should cherk errno */	Err	codemadness.org	70
i 11241         }	Err	codemadness.org	70
i 11242     } else {	Err	codemadness.org	70
i 11243         return result; /* success */	Err	codemadness.org	70
i 11244     }	Err	codemadness.org	70
i 11245 }	Err	codemadness.org	70
i 11246

Err codemadness.org 70 i 11247 Err codemadness.org 70 i 11248

... and one has to remember that if read() returns 0, it means we Err codemadness.org 70 i 11249 were at the end-of-file; if it returns less than the number of bytes Err codemadness.org 70 i 11250 requested it means we were close to the end of file; if this is a Err codemadness.org 70 i 11251 nonblocking socket and it returns EWOULDBLOCK or EAGAIN then one Err codemadness.org 70 i 11252 must decide to retry the operation or actually wait and try again Err codemadness.org 70 i 11253 later.

Err codemadness.org 70 i 11254

There is a lot of buggy software written in C that tries to use the Err codemadness.org 70 i 11255 POSIX API directly, and gets these subtleties wrong. Most programs Err codemadness.org 70 i 11256 written in high-level languages use the I/O facilities provided by Err codemadness.org 70 i 11257 their language, which hopefully make things easier.

Err codemadness.org 70 i 11258

I/O in Rust

Err codemadness.org 70 i 11259

Rust makes error handling convenient and safe. If you decide to Err codemadness.org 70 i 11260 ignore an error, the code looks like it is ignoring the error Err codemadness.org 70 i 11261 (e.g. you can grep for unwrap() and find lazy code). The Err codemadness.org 70 i 11262 code actually looks better if it doesn't ignore the error and Err codemadness.org 70 i 11263 properly propagates it upstream (e.g. you can use the ? shortcut to Err codemadness.org 70 i 11264 propagate errors to the calling function).

Err codemadness.org 70 i 11265

I keep recommending this article on error models to people; it Err codemadness.org 70 i 11266 discusses POSIX-like error codes vs. exceptions vs. more modern Err codemadness.org 70 i 11267 approaches like Haskell's and Rust's - definitely worth studying over Err codemadness.org 70 i 11268 a few of days (also, see Miguel's valiant effort to move C# I/O away Err codemadness.org 70 i 11269 from exceptions for I/O errors).

Err codemadness.org 70 i 11270

So, what happens when one opens a file in Rust, from the toplevel API Err codemadness.org 70 i 11271 down to the system calls? Let's go down the rabbit hole.

Err codemadness.org 70 i 11272

You can open a file like this:

Err codemadness.org 70 i 11273

use std::fs::File;	Err	codemadness.org	70
i 11274 	Err	codemadness.org	70
i 11275 fn main () {	Err	codemadness.org	70
i 11276     let f = File::open ("foo.txt");	Err	codemadness.org	70
i 11277     ...	Err	codemadness.org	70
i 11278 }	Err	codemadness.org	70
i 11279

Err codemadness.org 70 i 11280 Err codemadness.org 70 i 11281

This does not give you a raw file descriptor; it gives you an Err codemadness.org 70 i 11282 io::Result<fs::File, io::Error>, which you must pick apart to see if Err codemadness.org 70 i 11283 you actually got back a File that you can operate on, or an error.

Err codemadness.org 70 i 11284

Let's look at the implementation of File::open() and File::create().

Err codemadness.org 70 i 11285

impl File {	Err	codemadness.org	70
i 11286     pub fn open<P: AsRef<Path>>(path: P) -> io::Result<File> {	Err	codemadness.org	70
i 11287         OpenOptions::new().read(true).open(path.as_ref())	Err	codemadness.org	70
i 11288     }	Err	codemadness.org	70
i 11289 	Err	codemadness.org	70
i 11290     pub fn create<P: AsRef<Path>>(path: P) -> io::Result<File> {	Err	codemadness.org	70
i 11291         OpenOptions::new().write(true).create(true).truncate(true).open(path.as_ref())	Err	codemadness.org	70
i 11292     }	Err	codemadness.org	70
i 11293     ...	Err	codemadness.org	70
i 11294 }	Err	codemadness.org	70
i 11295

Err codemadness.org 70 i 11296 Err codemadness.org 70 i 11297

Here, OpenOptions is an auxiliary struct that implements a "builder" Err codemadness.org 70 i 11298 pattern. Instead of passing bitflags for the various Err codemadness.org 70 i 11299 O_CREATE/O_APPEND/etc. flags from the open(2) system call, one Err codemadness.org 70 i 11300 builds a struct with the desired options, and finally calls .open() Err codemadness.org 70 i 11301 on it.

Err codemadness.org 70 i 11302

So, let's look at the implementation of OpenOptions.open():

Err codemadness.org 70 i 11303

    pub fn open<P: AsRef<Path>>(&self, path: P) -> io::Result<File> {	Err	codemadness.org	70
i 11304         self._open(path.as_ref())	Err	codemadness.org	70
i 11305     }	Err	codemadness.org	70
i 11306 	Err	codemadness.org	70
i 11307     fn _open(&self, path: &Path) -> io::Result<File> {	Err	codemadness.org	70
i 11308         let inner = fs_imp::File::open(path, &self.0)?;	Err	codemadness.org	70
i 11309         Ok(File { inner: inner })	Err	codemadness.org	70
i 11310     }	Err	codemadness.org	70
i 11311

Err codemadness.org 70 i 11312 Err codemadness.org 70 i 11313

See that fs_imp::File::open()? That's what we want: it's the Err codemadness.org 70 i 11314 platform-specific wrapper for opening files. Let's look Err codemadness.org 70 i 11315 at its implementation for Unix:

Err codemadness.org 70 i 11316

    pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {	Err	codemadness.org	70
i 11317         let path = cstr(path)?;	Err	codemadness.org	70
i 11318         File::open_c(&path, opts)	Err	codemadness.org	70
i 11319     }	Err	codemadness.org	70
i 11320

Err codemadness.org 70 i 11321 Err codemadness.org 70 i 11322

The first line, let path = cstr(path)? tries to convert a Path Err codemadness.org 70 i 11323 into a nul-terminated C string. The second line calls the following:

Err codemadness.org 70 i 11324

    pub fn open_c(path: &CStr, opts: &OpenOptions) -> io::Result<File> {	Err	codemadness.org	70
i 11325         let flags = libc::O_CLOEXEC |	Err	codemadness.org	70
i 11326                     opts.get_access_mode()? |	Err	codemadness.org	70
i 11327                     opts.get_creation_mode()? |	Err	codemadness.org	70
i 11328                     (opts.custom_flags as c_int & !libc::O_ACCMODE);	Err	codemadness.org	70
i 11329         let fd = cvt_r(|| unsafe {	Err	codemadness.org	70
i 11330             open64(path.as_ptr(), flags, opts.mode as c_int)	Err	codemadness.org	70
i 11331         })?;	Err	codemadness.org	70
i 11332         let fd = FileDesc::new(fd);	Err	codemadness.org	70
i 11333 	Err	codemadness.org	70
i 11334         ...	Err	codemadness.org	70
i 11335 	Err	codemadness.org	70
i 11336         Ok(File(fd))	Err	codemadness.org	70
i 11337     }	Err	codemadness.org	70
i 11338

Err codemadness.org 70 i 11339 Err codemadness.org 70 i 11340

Here, let flags = ... converts the OpenOptions we had in the Err codemadness.org 70 i 11341 beginning to an int with bit flags.

Err codemadness.org 70 i 11342

Then, it does let fd = cvt_r (LAMBDA), and that lambda function Err codemadness.org 70 i 11343 calls the actual open64() from libc (a Rust wrapper for the system's Err codemadness.org 70 i 11344 libc): it returns a file descriptor, or -1 on error. Why is this Err codemadness.org 70 i 11345 done in a lambda? Let's look at cvt_r():

Err codemadness.org 70 i 11346

pub fn cvt_r<T, F>(mut f: F) -> io::Result<T>	Err	codemadness.org	70
i 11347     where T: IsMinusOne,	Err	codemadness.org	70
i 11348           F: FnMut() -> T	Err	codemadness.org	70
i 11349 {	Err	codemadness.org	70
i 11350     loop {	Err	codemadness.org	70
i 11351         match cvt(f()) {	Err	codemadness.org	70
i 11352             Err(ref e) if e.kind() == ErrorKind::Interrupted => {}	Err	codemadness.org	70
i 11353             other => return other,	Err	codemadness.org	70
i 11354         }	Err	codemadness.org	70
i 11355     }	Err	codemadness.org	70
i 11356 }	Err	codemadness.org	70
i 11357

Err codemadness.org 70 i 11358 Err codemadness.org 70 i 11359

Okay! Here f is the lambda that calls open64(); cvt_r() calls Err codemadness.org 70 i 11360 it in a loop and translates the POSIX-like result into something Err codemadness.org 70 i 11361 friendly to Rust. This loop is where it handles EINTR, which gets Err codemadness.org 70 i 11362 translated into ErrorKind::Interrupted. I suppose cvt_r() stands Err codemadness.org 70 i 11363 for convert_retry()? Let's look at Err codemadness.org 70 i 11364 the implementation of cvt(), which fetches the error code:

Err codemadness.org 70 i 11365

pub fn cvt<T: IsMinusOne>(t: T) -> io::Result<T> {	Err	codemadness.org	70
i 11366     if t.is_minus_one() {	Err	codemadness.org	70
i 11367         Err(io::Error::last_os_error())	Err	codemadness.org	70
i 11368     } else {	Err	codemadness.org	70
i 11369         Ok(t)	Err	codemadness.org	70
i 11370     }	Err	codemadness.org	70
i 11371 }	Err	codemadness.org	70
i 11372

Err codemadness.org 70 i 11373 Err codemadness.org 70 i 11374

(The IsMinusOne shenanigans are just a Rust-ism to help convert Err codemadness.org 70 i 11375 multiple integer types without a lot of as casts.)

Err codemadness.org 70 i 11376

The above means, if the POSIX-like result was -1, return an Err() from Err codemadness.org 70 i 11377 the last error returned by the operating system. That should surely Err codemadness.org 70 i 11378 be errno internally, correct? Let's look at Err codemadness.org 70 i 11379 the implementation for io::Error::last_os_error():

Err codemadness.org 70 i 11380

    pub fn last_os_error() -> Error {	Err	codemadness.org	70
i 11381         Error::from_raw_os_error(sys::os::errno() as i32)	Err	codemadness.org	70
i 11382     }	Err	codemadness.org	70
i 11383

Err codemadness.org 70 i 11384 Err codemadness.org 70 i 11385

We don't need to look at Error::from_raw_os_error(); it's just a Err codemadness.org 70 i 11386 conversion function from an errno value into a Rust enum value. Err codemadness.org 70 i 11387 However, let's look at sys::os::errno():

Err codemadness.org 70 i 11388

pub fn errno() -> i32 {	Err	codemadness.org	70
i 11389     unsafe {	Err	codemadness.org	70
i 11390         (*errno_location()) as i32	Err	codemadness.org	70
i 11391     }	Err	codemadness.org	70
i 11392 }	Err	codemadness.org	70
i 11393

Err codemadness.org 70 i 11394 Err codemadness.org 70 i 11395

Here, errno_location() is an extern function defined in GNU libc Err codemadness.org 70 i 11396 (or whatever C library your Unix uses). It returns a pointer to the Err codemadness.org 70 i 11397 actual int which is the errno thread-local variable. Since non-C Err codemadness.org 70 i 11398 code can't use libc's global variables directly, there needs to be a Err codemadness.org 70 i 11399 way to get their addresses via function calls - that's what Err codemadness.org 70 i 11400 errno_location() is for.

Err codemadness.org 70 i 11401

And on Windows?

Err codemadness.org 70 i 11402

Remember the internal File.open()? This is what it looks Err codemadness.org 70 i 11403 like on Windows:

Err codemadness.org 70 i 11404

    pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {	Err	codemadness.org	70
i 11405         let path = to_u16s(path)?;	Err	codemadness.org	70
i 11406         let handle = unsafe {	Err	codemadness.org	70
i 11407             c::CreateFileW(path.as_ptr(),	Err	codemadness.org	70
i 11408                            opts.get_access_mode()?,	Err	codemadness.org	70
i 11409                            opts.share_mode,	Err	codemadness.org	70
i 11410                            opts.security_attributes as *mut _,	Err	codemadness.org	70
i 11411                            opts.get_creation_mode()?,	Err	codemadness.org	70
i 11412                            opts.get_flags_and_attributes(),	Err	codemadness.org	70
i 11413                            ptr::null_mut())	Err	codemadness.org	70
i 11414         };	Err	codemadness.org	70
i 11415         if handle == c::INVALID_HANDLE_VALUE {	Err	codemadness.org	70
i 11416             Err(Error::last_os_error())	Err	codemadness.org	70
i 11417         } else {	Err	codemadness.org	70
i 11418             Ok(File { handle: Handle::new(handle) })	Err	codemadness.org	70
i 11419         }	Err	codemadness.org	70
i 11420     }	Err	codemadness.org	70
i 11421

Err codemadness.org 70 i 11422 Err codemadness.org 70 i 11423

CreateFileW() is the Windows API function to open files. The Err codemadness.org 70 i 11424 conversion of error codes inside Error::last_os_error() happens Err codemadness.org 70 i 11425 analogously - it calls GetLastError() from the Windows API and Err codemadness.org 70 i 11426 converts it.

Err codemadness.org 70 i 11427

Can we not call C libraries?

Err codemadness.org 70 i 11428

The Rust/Unix code above depends on the system's libc for open() and Err codemadness.org 70 i 11429 errno, which are entirely C constructs. Libc is what actually does Err codemadness.org 70 i 11430 the system calls. There are efforts to make the Rust standard library Err codemadness.org 70 i 11431 not use libc and use syscalls directly.

Err codemadness.org 70 i 11432

As an example, you can look at Err codemadness.org 70 i 11433 the Rust standard library for Redox. Redox is a new operating Err codemadness.org 70 i 11434 system kernel entirely written in Rust. Fun times!

Err codemadness.org 70 i 11435

Update: If you want to see what a C-less libstd would look Err codemadness.org 70 i 11436 like, take a look at steed, an effort to reimplement Rust's libstd Err codemadness.org 70 i 11437 without C dependencies.

Err codemadness.org 70 i 11438

Conclusion

Err codemadness.org 70 i 11439

Rust is very meticulous about error handling, but it succeeds in Err codemadness.org 70 i 11440 making it pleasant to read. I/O functions give you back an Err codemadness.org 70 i 11441 io::Result<>, which you piece apart to see if it succeeded or got an Err codemadness.org 70 i 11442 error.

Err codemadness.org 70 i 11443

Internally, and for each platform it supports, the Rust standard Err codemadness.org 70 i 11444 library translates errno from libc into an io::ErrorKind Rust Err codemadness.org 70 i 11445 enum. The standard library also automatically handles Unix-isms like Err codemadness.org 70 i 11446 retrying operations on EINTR.

Err codemadness.org 70 i 11447

I've been enjoying reading the Rust standard library code: it Err codemadness.org 70 i 11448 has taught me many Rust-isms, and it's nice to see how the Err codemadness.org 70 i 11449 hairy/historical libc constructs are translated into clean Rust Err codemadness.org 70 i 11450 idioms. I hope this little trip down the rabbit hole for the Err codemadness.org 70 i 11451 open(2) system call lets you look in other interesting places, too.

Federico's Blog

GNOME themes, an incomplete status report, and how you can help

Frame of reference

Libadwaita and the rest of the ecosystem

CSS parity across implementations

Accent colors

Custom widgets

Recoloring styles

Recoloring icons

How you can help

Bzip2's experimental repository is changing maintainership

Librsvg, Rust, and non-mainstream architectures

Memories of discontinued architectures

Unsupported architectures at the LLVM level

Non-tier-1 platforms and "$distro does not build Rust there yet"

Some anecdotes from Suse

Endianness bugs

Who gets paid to fix this?

Do not use librsvg 2.40.x

Librsvg's test suite is now in Rust

Changes to continuous integration

Test suite in Rust

What's next: cargo-c and publish to crates.io

What remains to be ported to Rust?

Librsvg is accepting interns for Outreachy's December 2020 round

"Rust does not have a stable ABI"

How do distributions use shared libraries?

Bug fixes and security fixes

Bundled libraries

Rust does not have a stable ABI

Data layout for C versus Rust

Calling conventions

Summary of ABI so far

Rust bundles its entire standard library with Rust-built .so's

Stable APIs/ABIs and distros

But we only have a single stable ABI anyway

Actually, in GNOME we have gone further than that

Single copies of libraries with a C ABI

Monomorphization and template bloat

Should all libraries export APIs with generics and exotic types?

Library sizes

A metaphor which I haven't fully explored

Looking for candidates for the 2020 GNOME Foundation elections

Bringing my Emacs from the past

Everything in ~/.emacs/init.el and with use-package

~/.emacs.d/custom.el for M-x customize stuff

Which-key to get hints when typing command prefixes

Don't pollute the modeline with common modes

Magit to use git in a civilized fashion

Move between windows with Shift-arrows

Pretty colors

Nyan cat instead of scrollbars

Move buffers to adjacent windows

Change buffer names for files with the same name

Helm to auto-complete in grand style

Ripgrep to search in grand style

Rust mode and Language Server

Performatively not get distracted

Stuff from custom.el

Results

Reducing memory consumption in librsvg, part 4: compact representation for Bézier paths

What is in a PathCommand?

A more compact representation

PackedCommand

Packing a PathCommand into a PackedCommand

Creating the compact Path

An iterator for Path

Results

References

Reducing memory consumption in librsvg, part 3: slack space in Bézier paths

First step: convert to Box<[T]>

Second step: make the allocator do less work

Making the code prettier

Next steps

References

Reducing memory consumption in librsvg, part 2: SpecifiedValues

What is SpecifiedValues?

Not all properties are specified

Another way to represent the set of properties

Initialization

Everything in `~/.emacs/init.el` and with `use-package`

`~/.emacs.d/custom.el` for `M-x customize` stuff

Making programming errors a `g_critical`

Removing the `dir` field

Removing the `Data` associated type