Attaching to something already running

Everything so far started the program under GDB. What if the process is already running? Well, we would need to attach to it…

Something to attach to

First we need a process to attach to.

#include <stdio.h>
#include <unistd.h>

int counter = 0;

int main(void) {
    printf("pid %d\n", getpid());
    while (1) {
        printf("counter = %d\n", counter);
        counter++;
        sleep(1);
    }
    return 0;
}

Build it and leave it running in another terminal:

$ gcc -g -O0 -no-pie -o counter counter.c
$ ./counter
pid 12
counter = 0
counter = 1
counter = 2
counter = 3
counter = 4

Attach

gdb -p <pid> attaches to a running process. If you do not know the pid, pgrep counter will tell you.

If that comes back with a permission error rather than a prompt, that is expected on most distributions. Jump to when it says permission denied, fix it, and come back here.

$ gdb -q -p 12
(gdb) bt
#0  0x0000770df18c4b7a in __GI___clock_nanosleep (...) at clock_nanosleep.c:78
#1  0x0000770df18d1b27 in __GI___nanosleep (...) at nanosleep.c:25
#2  0x0000770df18e6d93 in __sleep (seconds=0) at sleep.c:55
#3  0x00000000004011b2 in main () at counter.c:10

The process stops when you attach and waits for you to let it go.

The backtrace shows exactly where it was. Frame #3 is your main, at line 10. Frames #2 to #0 are inside libc, descending into the nanosleep syscall, because you caught it mid sleep(1). Attaching to a process at a random moment usually lands you somewhere like this, in library code you did not write, and the frame you care about is further up the stack.

Read it, then change it

(gdb) print counter
$1 = 5
(gdb) set var counter = 1000
(gdb) print counter
$2 = 1000

You have just changed a variable in a process you did not start. Rather than detaching, leave GDB attached and set a rule: whenever the counter reaches 1010, put it back to 1000. That is a conditional breakpoint with a list of commands hanging off it.

(gdb) break counter.c:10 if counter == 1010
Breakpoint 1 at 0x4011b5: file counter.c, line 10.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>set var counter = 1000
>continue
>end
(gdb) continue
Continuing.

Four things are doing the work there:

if counter == 1010 makes it a conditional breakpoint. It is checked every time through the loop, and the process only stops when it is true.
commands attaches a script to that breakpoint. Everything up to end runs each time it fires.
silent suppresses the “Breakpoint 1, main () at counter.c:10” banner. Without it your terminal fills with stop messages.
continue at the end of the list means the process starts again by itself. You are not in the loop at all.

Now watch the other terminal:

counter = 1007
counter = 1008
counter = 1009
counter = 1010
counter = 1001
counter = 1002

The program is stuck in a range it was never written to have. It counts to 1010, GDB catches it, rewrites the variable, and lets it go, over and over, without you touching the keyboard.

Look at the value it comes back on. You set 1000, and the next line printed is 1001. Line 10 is counter++, so the breakpoint fires before the increment. GDB writes 1000, the program then runs the increment it was about to run, and prints 1001 on the next pass. A breakpoint on a line stops before that line has done anything.

That is worth stopping on for a second. You have changed the behaviour of a running program, with no source change, no rebuild, and no restart, and the program has no idea. This is the same mechanism that makes a debugger useful on a service you cannot take down, and the same one that makes an attached debugger something you would rather an attacker did not have on your box.

detach, not quit

When you are done, detach releases the process and leaves it running:

(gdb) detach
[Inferior 1 (process 12) detached]
(gdb) quit

The breakpoint goes with it. The counter climbs past 1010 and keeps going, because nothing you did lives in the process once GDB lets go of it.

Quitting while still attached is the other outcome, and GDB will ask first:

(gdb) quit
A debugging session is active.

	Inferior 1 [process 12] will be killed.

Quit anyway? (y or n)

On your own laptop that prompt is a nuisance. On a production box it is the difference between a debugging session and an outage. Get into the habit of typing detach before quit, so the muscle memory is right when it matters.

Two more things that will bite you. Attaching stops the process for as long as you sit at the prompt, so anything with a timeout on the other end will give up while you are reading a backtrace. And a process can only have one tracer, so if something is already attached you get an error instead of a session.

When it says permission denied

This is the one that stops people:

$ gdb -q -p 262516
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Inappropriate ioctl for device.

The Yama LSM is what stopped you:

$ cat /proc/sys/kernel/yama/ptrace_scope
1

1 means a process may only be traced by one of its own ancestors. Starting a program under GDB works, because GDB is its parent. Attaching to something you did not start does not, because you are a sibling. It exists so that malware which lands as your user cannot walk around reading the memory of every other process you are running, your browser and ssh-agent included.

Set it to 0 while you need it:

$ sudo sysctl -w kernel.yama.ptrace_scope=0

That is a system-wide loosening of a hardening control, on a machine you own, for as long as you need it. sysctl -w does not survive a reboot, which is the right default. Put it back with 1 when you are done.

The other way through is sudo gdb -p <pid>, since CAP_SYS_PTRACE bypasses Yama entirely. That is also why attaching works without any of this inside a container started with --cap-add=SYS_PTRACE.

Next: turning the commands you keep retyping into commands of your own.