Direct Syscalls: Bypassing API Hooking

Background

Modern EDRs hook Win32 APIs at the NTDLL layer - they overwrite the first bytes of functions like NtAllocateVirtualMemory with a JMP into their own monitoring code. Direct syscalls skip NTDLL entirely, placing the CPU-level syscall instruction inline in your own code. There's nothing to hook.

This is a lab overview of the technique and its detection surface. All syscall numbers are illustrative; they vary by Windows build.

How NTDLL hooking works

A hooked NtAllocateVirtualMemory stub looks like this after EDR injects:

NtAllocateVirtualMemory:
  jmp <edr_monitor_function>   ; EDR overwrote the real prologue
  ; original bytes relocated into EDR's trampoline

The real prologue - mov r10, rcx / mov eax, <SSN> / syscall - is gone. Every call to VirtualAlloc passes through the EDR's lens.

The direct syscall pattern

To bypass the hook, emit the syscall stub yourself:

NtAllocVirtMem_stub:
  mov r10, rcx
  mov eax, 0x18       ; SSN for NtAllocateVirtualMemory on this build
  syscall
  ret

Declare it as a C function pointer and call it directly. NTDLL is never touched.

Resolving SSNs at runtime

Parse the real syscall numbers from the on-disk NTDLL (which EDR doesn't hook):

// Map NTDLL from disk, not the loaded (hooked) copy
HANDLE f = CreateFileW(L"C:\\Windows\\System32\\ntdll.dll",
                       GENERIC_READ, FILE_SHARE_READ, NULL,
                       OPEN_EXISTING, 0, NULL);
// Map, walk the EAT for Nt* exports, read the SSN from the
// unhoooked mov eax instruction at offset +4

This gives you SSNs that match the running kernel even when the in-memory NTDLL is fully patched by the EDR.

Indirect syscalls

A refinement: instead of emitting your own syscall instruction, jump to the syscall instruction already present in NTDLL's stub - but only after setting up your own registers. This makes the kernel-mode call stack look like it came from NTDLL, which is what some EDRs verify:

NtAllocVirtMem_indirect:
  mov r10, rcx
  mov eax, 0x18
  jmp [ntdll_syscall_gadget]  ; land on NTDLL's own syscall instruction

The syscall instruction, and thus the kernel call stack return address, now points inside NTDLL's legitimate stub range.

Detection surface

Direct/indirect syscalls are harder to catch than userland API hooking, but defenders have options:

• Kernel callbacks (PsSetCreateThreadNotifyRoutine, ObRegisterCallbacks) run regardless of how the syscall was reached.
• ETW-TI (Threat Intelligence) kernel telemetry fires on many sensitive operations without any userland hook.
• Call stack analysis: a syscall instruction with a return address outside any known NTDLL range is a strong anomaly signal. Indirect syscalls address this - but the set-up instructions before the jmp can still be spotted.
• NtCreateSection + NtMapViewOfSection on ntdll.dll from disk is a common enumeration pattern with a clear signature.

Takeaway

Direct syscalls are a fundamental technique for understanding how EDR products work and where their blind spots are. The arms race between stub-emission and kernel-level telemetry is ongoing - ETW-TI and kernel call stack validation have raised the bar significantly since the original technique was published.

References