puck tools — test your egress policy against real C2 traffic →
Offensive ContextOffensive ContextContact
#linux#gdb

How debugging actually works

What is happening here? A debugger is not watching or simulating your program. A debugger is a second process that has been granted the right to reach into the first one and rewrite it while it runs.

One syscall does all of it

On Linux the whole relationship is managed through ptrace(2). GDB calls ptrace(PTRACE_ATTACH, ...), or the child calls PTRACE_TRACEME before exec. From that point the kernel routes the target’s signals through GDB.

Reading memory is another call with PTRACE_PEEKDATA, writing it is PTRACE_POKEDATA, reading registers is PTRACE_GETREGS, and resuming is PTRACE_CONT.

A breakpoint is vandalism

When you set a breakpoint, GDB overwrites the first byte of the instruction at that address with 0xcc. That is the x86 encoding of int3, the one-byte software interrupt. GDB keeps the original byte in its own memory.

When execution reaches that address the CPU hits int3, raises SIGTRAP, the kernel stops the process and notifies the tracer. GDB then puts the original byte back, rewinds the instruction pointer by one, and tells you it stopped at a breakpoint. To continue, it single-steps that one restored instruction, re-plants the 0xcc, and lets go.

Let’s do it by hand to see it in action.

Start by looking at the real bytes of check. The /r flag prints the raw machine code next to each instruction:

$ gdb -q ./hello
Reading symbols from ./hello...
(gdb) disassemble /r check
Dump of assembler code for function check:
   0x0000000000401176 <+0>:	f3 0f 1e fa        	endbr64
   0x000000000040117a <+4>:	55                 	push   rbp
   0x000000000040117b <+5>:	48 89 e5           	mov    rbp,rsp
   0x000000000040117e <+8>:	48 83 ec 10        	sub    rsp,0x10
   0x0000000000401182 <+12>:	48 89 7d f8        	mov    QWORD PTR [rbp-0x8],rdi
   0x0000000000401186 <+16>:	48 8b 45 f8        	mov    rax,QWORD PTR [rbp-0x8]

The instruction at 0x401186 begins with 0x48. Remember that byte.

Now get the program running so there is memory to write to, and look at that address directly:

(gdb) break main
Breakpoint 1 at 0x4011a5: file hello.c, line 9.
(gdb) run Evan
Breakpoint 1, main (argc=2, argv=0x7fffffffdd78) at hello.c:9
9	    if (argc < 2) { puts("usage: hello NAME"); return 2; }
(gdb) x/1bx 0x401186
0x401186 <check+16>:	0x48

0x48, exactly as the compiler emitted it. Now write the trap byte yourself:

(gdb) set var *(unsigned char *)0x401186 = 0xcc
(gdb) x/1bx 0x401186
0x401186 <check+16>:	0xcc

You have just manually created a breakpoint. Let it run:

(gdb) continue
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x0000000000401187 in check (name=0x7fffffffe0f7 "Evan") at hello.c:5
5	    return strcmp(name, "CONTEXT") == 0;

And you get one stopped process.

Look at the address it stopped on. You wrote the 0xcc at 0x401186 and execution halted at 0x401187, one byte later. The CPU had already consumed the one-byte instruction before it trapped. This is why GDB rewinds the instruction pointer by one before showing you anything, and why it has to put the original byte back before continuing. You have now done every part of that by hand except the tidying up.

GDB hides its own

Do the same inspection on a breakpoint GDB set, and you will see something else entirely:

$ gdb -q ./hello
(gdb) break check
Breakpoint 1 at 0x401186: file hello.c, line 5.
(gdb) run Evan
Breakpoint 1, check (name=0x7fffffffe0f7 "Evan") at hello.c:5
5	    return strcmp(name, "CONTEXT") == 0;
(gdb) x/1bx 0x401186
0x401186 <check+16>:	0x48
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000000401186 in check at hello.c:5
	breakpoint already hit 1 time

This is a little confusing, the breakpoint is real, it has been hit. The reads still show 0x48. GDB filters its own breakpoints out of the memory reads you make, so x and disassemble show you the program you wrote rather than the program it is running.

Hardware breakpoints do not touch memory

The breakpoint command creates a software breakpoint. Sometimes you cannot overwrite the instruction. The page may be read-only in a way you cannot change, or the code may checksum itself. x86 has four debug registers, DR0 to DR3, which hold addresses the CPU traps on without any modification to memory at all.

(gdb) hbreak check
(gdb) watch counter

hbreak is a hardware breakpoint. watch is the same machinery pointed at data: the CPU traps when a location is written. That is how you catch the one line in a large program that is corrupting a variable. You get four. Spend them well, because this is a CPU limit and not a GDB one.

What this buys you

Every strange thing GDB does downstream follows from the above:

  • Breakpoints in shared libraries do not work until the library loads, because there is no memory to vandalise yet. That is what “make breakpoint pending” is asking you.
  • Self-modifying or packed code loses your breakpoints, because it overwrote the page you planted them in.
  • Optimised builds stop in nonsensical places, because the address that claims to be line 12 is genuinely shared by lines 9 through 14.
  • A process can only have one tracer. If something is already attached, you get Operation not permitted. That is also why some hardened boxes set /proc/sys/kernel/yama/ptrace_scope to 1 and break gdb -p entirely.

Next: doing all of this to a process that is already running.

← GDB from zero to hero