Anti-Debugging

You open GDB, set a breakpoint at main, type run... and suddenly the binary prints "Segmentation fault" or "Debugger detected!" and immediately exits. What happened? You just hit an Anti-Debugging Check! Anti-debugging includes techniques used by binary authors (and malware developers) to detect if a program is being executed under a debugger (such as GDB, pwndbg, or x64dbg) and stop you from analyzing it dynamically. In this section, we'll cover how anti-debugging checks work under the hood on Linux and four ways to bypass them.

Detection Mechanisms: ptrace and rdtsc

The most common anti-debugging technique on Linux relies on the ptrace system call. ptrace is the core Linux system call that debuggers (like GDB) use to attach to, inspect, and control running processes. Linux enforces a strict security rule: A process can only be traced by ONE debugger at a time. A binary can take advantage of this rule by calling ptrace(PTRACE_TRACEME) on itself when it starts up: if no debugger is attached, ptrace succeeds and returns 0; if GDB is already attached, ptrace fails and returns -1.

In C, this check is written as if (ptrace(PTRACE_TRACEME, 0, 1, 0) < 0) exit(1);. However, to prevent reverse engineers from hooking the ptrace C library function, authors often invoke ptrace directly via raw x86-64 system calls (syscall 101):

long ret;
__asm__ volatile (
    "mov $101, %%rax\n\t"    // Syscall 101 = __NR_ptrace on x86_64
    "xor %%rdi, %%rdi\n\t"   // PTRACE_TRACEME (0)
    "syscall\n\t"
    : "=a"(ret)
);

if (ret < 0) {
    puts("Segmentation fault"); // Fake error message to trick you!
    exit(1);
}

When you see a program throw a fake "Segmentation fault" immediately after starting under GDB, check if it executed a ptrace syscall!

Another common check uses Timing Checks (rdtsc). When you step through instructions line-by-line in GDB using ni or si, human reaction time introduces a massive delay between instructions: millions of CPU clock cycles! The rdtsc (Read Time-Stamp Counter) instruction reads the total number of CPU clock cycles executed since the machine booted into the EDX:EAX registers. By measuring the cycle count difference between two points in code, a binary can detect single-stepping:

uint32_t lo1, hi1, lo2, hi2;
uint64_t t1, t2;

__asm__ volatile ("rdtsc" : "=a"(lo1), "=d"(hi1)); // 1. Read cycle counter before block

// Protected code block...

__asm__ volatile ("rdtsc" : "=a"(lo2), "=d"(hi2)); // 2. Read cycle counter after block

t1 = ((uint64_t)hi1 << 32) | lo1;
t2 = ((uint64_t)hi2 << 32) | lo2;

if ((t2 - t1) > 0x100000) { // If time delta is huge, someone is single-stepping!
    puts("Timing anomaly detected! Exiting...");
    exit(1);
}

How to Bypass Anti-Debugging

Now for the fun part: four practical ways to bypass these anti-debugging checks!

1. Binary Patching (Cleanest and Most Reliable): Edit the binary executable file directly in IDA or Ghidra. Locate the syscall or ptrace call, find the conditional jump right after it (e.g. js failure_label), and change the jump instruction to 0x90 bytes (NOP - No Operation). Export the patched executable file, and you can debug it in GDB without triggering anti-debug checks!

2. Dynamic Register Overriding in GDB: If you don't want to modify the binary file, override the syscall return value dynamically inside GDB. Set a breakpoint right after the syscall instruction (break *check_debugger+25), run the program, and when GDB hits the breakpoint, force RAX to return 0 (success) with set $rax = 0 before continuing.

3. LD_PRELOAD Hooking (for Libc Calls): If the binary calls standard libc ptrace(), intercept the function call using Linux's LD_PRELOAD environment variable. Create a file fake_ptrace.c returning 0 (long ptrace(...) { return 0; }), compile it into a shared library (gcc -shared -fPIC fake_ptrace.c -o fake_ptrace.so), and load it with GDB (LD_PRELOAD=./fake_ptrace.so gdb ./binary).

4. angr SimProcedure Hooking: If you are using angr to solve the challenge, hook the ptrace symbol with an angr SimProcedure (proj.hook_symbol("ptrace", HookPtrace())) and set add_options={angr.options.BYPASS_UNSUPPORTED_SYSCALL} on the entry state to handle rdtsc and raw syscalls automatically.