Post History

Note that if you want to debug the currently installed version of NetworkManager, you don't have to compile NetworkManager yourself. Instead, make GDB download symbol files via [`debuginfod`](https://www.mankier.com/8/debuginfod). Section "Debug NetworkManager with GDB" of this answer describes how.

# Clone and configure

Clone NetworkManager's repository:

    git clone https://gitlab.freedesktop.org/NetworkManager/NetworkManager.git --depth 1

Before configuring and compiling, make sure you have the required packages installed. I was missing [`gtk-doc`](https://archlinux.org/packages/extra/any/gtk-doc/), [`gobject-introspection`](https://archlinux.org/packages/extra/x86_64/gobject-introspection/), and [`ppp`](https://archlinux.org/packages/core/x86_64/ppp/) which caused errors:

>./autogen.sh: line 25: gtkdocize: command not found

>configure: error: introspection enabled but can't be used

>configure: error: "couldn't find pppd.h. pppd development headers are required."

You'll probably need [`base-devel`](https://archlinux.org/packages/core/any/base-devel/) installed as well:

    sudo pacman -S base-devel gtk-doc gobject-introspection ppp

Although NetworkManager will run with root privileges, it needs to be installed in a path entirely owned by root. If you put your self-compiled NetworkManager in a directory like `/home/me/nm_install_dir`, even if you `chown root:root /home/me/install_dir/`, it will start but won't assign IP addresses and print errors like these:

>Cannot create parents for '/home/me/nm_install_dir/var/lib/NetworkManager': Read-only file system

>secret-key: failure to persist secret key in "/home/me/nm_install_dir/var/lib/NetworkManager/secret_key" (failed to create file /home/me/nm_install_dir/var/lib/NetworkManager/secret_key.KG6J61: Read-only file system) (use non-persistent key)

>config: device-state: write #3 (/home/me/nm_install_dir/var/run/NetworkManager/devices/3) failed: Failed to create file “/home/me/nm_install_dir/var/run/NetworkManager/devices/3.C2ZI61”: Read-only file system

Thus, I used a directory in the root directory for putting NetworkManager's binary, configuration, and runtime files:

      ./autogen.sh --prefix=/nm_install_dir

Then, compile and install NetworkManager with:

      make -j 8 && sudo make install

If you encounter issues while configuring or compiling, see NetworkManager's [CONTRIBUTING.md](https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/CONTRIBUTING.md#user-content-building-from-source) which also links to required packages.

## Increase logging

It can be useful to have NetworkManager produce more detailed log messages; Create a file `/nm_install_dir/etc/NetworkManager/NetworkManager.conf` containing this:

      [logging]
      domains=ALL:TRACE

# Run self-compiled NetworkManager

Stop your currently running instance of NetworkManager with `systemctl stop NetworkManager` and (optionally) remove current IP addresses from your interfaces with `ip addr flush up`.

On Arch Linux, systemd starts NetworkManager as defined in the unit file `/usr/lib/systemd/system/NetworkManager.service`. In that file, I changed the line

    ExecStart=/usr/bin/NetworkManager --no-daemon

to

    ExecStart=/nm_install_dir/sbin/NetworkManager --no-daemon

and ran `systemctl daemon-reload && systemctl start NetworkManager`.

There are other ways to do that, as described [here](https://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services).

Check the logs with `journalctl --since=-5m -u NetworkManager` to see if there are any warnings or errors.

# Debug NetworkManager with GDB

If you want to debug NetworkManager without it being started by systemd, run this as root:

    # Stop the systemd-started NetworkManager
    systemctl stop NetworkManager
    DEBUGINFOD_URLS="https://debuginfod.archlinux.org/" gdb /nm_install_dir/sbin/NetworkManager

The environment variable `DEBUGINFOD_URLS` enables GDB to download debugging symbols on demand using [`debuginfod`](https://www.mankier.com/8/debuginfod). We set it here explicitly since per default, user root doesn't have it.

## Start NetworkManager with systemd and debug with GDB

We'll attach GDB to the NetworkManager process once systemd has started it. Prerequisite for this is GDB having the [capability](https://man7.org/linux/man-pages/man7/capabilities.7.html) to attach to a running process, discussed [here](https://stackoverflow.com/questions/19215177/how-to-solve-ptrace-operation-not-permitted-when-trying-to-attach-gdb-to-a-pro):

    sudo setcap CAP_SYS_PTRACE=+eip /usr/bin/gdb

Make sure NetworkManager is not running with `systemctl stop NetworkManager`.

Save the following script, adapted from [here](https://stackoverflow.com/questions/4382348/is-there-any-way-to-tell-gdb-to-wait-for-a-process-to-start-and-attach-to-it#11147567), as `gdbwait`:

    #!/bin/sh
    process_name="$1"
    breakpoint_location="$2"
    echo "Waiting for $process_name to start"
    pid=$(pgrep -o "$process_name")
    while [ "$pid" = "" ]; do
      pid=$(pgrep -o "$process_name")
    done

    # Per default, user root doesn't have the environment variable DEBUGINFOD_URLS → Set it to download symbols as needed
    DEBUGINFOD_URLS="https://debuginfod.archlinux.org/" gdb -ex "break $breakpoint_location" -ex continue -p "$pid"

To avoid GDB's prompt "Enable debuginfod for this session?", you can `echo "set debuginfod enabled on" >> /root/.gdbinit`.

Then, run `./gdbwait NetworkManager yourDebugLocation` as root and `systemctl start NetworkManager` afterwards. GDB needs root privileges, otherwise GDB won't be able to debug the process and print errors:

>warning: "target:/nm_install_dir/sbin/NetworkManager": could not open as an executable file: Operation not permitted.

>warning: `target:/nm_install_dir/sbin/NetworkManager': can't open to read symbols: Operation not permitted.

>warning: opening /proc/self/mem file failed: Permission denied (13)

Mind that `gdbwait` won't be able to break at early functions like `main` since detecting the process and attaching to it takes longer than for main to have started.