I’ve recently wanted to debug gnome-shell, that was crashing at seemingly
random times for me (especially after hibernating), but attaching a debugger
to gnome-shell isn’t as easy as it appears. I’ve noticed that it tended
to dump a
core.$pid file in my
$HOME, which is a coredump file one
can load into a debugger to print the backtrace among other things to
after the fact. It annoyed me a little that it cluttered my
I figured it’d be nice having a manager for these coredumps so one can
see what programs crashed, why and how and to easily rotate them. I’ve remembered
using systemd-coredumpd at some point and kind of liked that, but since I’m
using Alpine Linux that wasn’t an option, so I wrote corecollector.
If corecollector is packaged in your distribution installing corecollector should be as simple as installing it via your package manager. From then on corecollector will collect coredumps for you (see further down for the usage).
If corecollector hasn’t been packaged yet, you can install it manually:
Simply download a release from https://github.com/Cogitri/corecollector/releases ,
unpack the tarball and
cd into the resulting dir. Then run:
ninja --buildtype=release build ninja -C build install
You’ll need a functional D compiler for this to work. You’ll also have to
corecollector user and group, as which
corehelper, which is invoked
by the kernel when a program crashes and a core is dumped, will save
the coredumps to avoid running as root for longer than required. You can add
users to the
corecollector group to allow them to access coredumps without
root, but be mindful that coredumps are snapshots of the memory of a program
and can contain highly sensitive data.
Using (the CLI)
The main binary to interact with is
corectl, here’s it’s
--help which explains
Usage: corectl <subcommand> [OPTION...] List and interact with coredumps Subcommands: backtrace [ID] - Print the backtrace of the coredump identified by ID. debug [ID] - Open the coredump identified by ID in a debugger. dump [ID] [FILE] - Dump the coredump identified by ID to file denoted by FILE. Defaults to 'stdout'. info [ID] - Get more detailed information for the coredump identified by ID. list - List all available coredumps. Help Options: -h, --help - Show help options. Application Options: -v, --version - Print program version. -d, --debug [0-3] - Specify the debug level.
If you want to know what programs have crashed you can simply run
$ corectl list ID SIGNAL UID GID PID TIMESTAMP EXE 1 6 1000 1000 4837 2020-Feb-18 18:32:05 /usr/bin/Xwayland 2 5 105 109 30052 2020-Feb-18 18:32:06 /usr/bin/gnome-shell 3 11 1000 1000 5296 2020-Feb-18 21:17:49 /usr/bin/gnome-software 4 31 1000 1000 6830 2020-Feb-18 21:19:21 /usr/libexec/tracker-extract 5 11 1000 1000 7770 2020-Feb-18 21:24:41 /usr/local/bin/gnome-software
You can then run
corectl debug $ID to attach a debugger to one of the coredumps:
$ corectl debug 3 GNU gdb (GDB) 9.1 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-alpine-linux-musl". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/bin/gnome-software... [New LWP 28750] [New LWP 28749] [New LWP 28755] [New LWP 28753] [New LWP 28747] [New LWP 28748] [New LWP 28757] [New LWP 28751] [New LWP 28752] [New LWP 28759] [New LWP 28756] [New LWP 28754] Core was generated by `gnome-software'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720 720 cairo-spans-compositor.c: No such file or directory. [Current thread is 1 (LWP 28750)] (gdb) bt #0 0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720 ... .. .
You can also print the backtrace directly via
corectl backtrace $ID which tends
to be the most frequently used feature:
$corectl backtrace 3 #0 0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720 ... .. .
Corecollector also has a few configuration options, see
# What compression to use for coredumps. # Available choices: none zlib compression = zlib # The maximum size of coredumps. # Unset or set to 0 to set no size limit. # Measured in KByte, must be whole numbers. maxSize = 0 # The path to save coredumps to. targetPath = /var/lib/corecollector # Where to log to. Pass /dev/null to only log to syslog logPath = /var/log/corecollector.log # Enable debugging enableDebug = false # The maximum size of the coredumpdir. # Unset or set to 0 to set no size limit. # Measured in KByte, must be whole numbers. maxDirSize = 0
It uses zlib for compression by default, since coredumps tend to be very easy to compress. As such a coredump might be some 400KiB big when uncompressed but be as small as 5 KiB when compressed. Corecollector also offers options to limit the maximum size of coredumps, rotate the coredump directory once it reaches a certain size or to log to a certain path (additionally to syslog).
Corecollector is made to make debugging programs easy and to make users aware
of crashing programs, since it can be easy to miss them in the logs or
I hope it proves useful to you :)
Corecollector was my first project in DLang and it was a very pleasant experience. Since DLang has a pretty small community (and as such doesn’t have a too broad library landscape) it can seem a little frightening to get started with it, but I’ve found it to be an incredibly productive language: I don’t feel held back because it’s somewhat “low-level-ish”, but due to it’s GC (which can easily be controlled or even turned off completely for performance critical areas) it’s easy to use without introducing security vulnerabilities due to use-after-frees or double-frees and without having to worry about lifetimes or similar. It’s also easy to bind to C libraries in D (one only needs the function declaration and link against the right library and it’s as if you’re calling it from C), so the library landscape doesn’t look so bad after all :)