OS Structures, Security & Reverse Engineering

how OS components fit together, how software interfaces with hardware, and how attackers or analysts manipulate those interactions

I/O (Input/Output)

data transfer between CPU/memory and external devices (disk, network, etc)

Device Driver

kernel module managing communication between hardware and OS

Interrupt Handler/ISR

function executed in response to an interrupt

DMA (Direct Memory Access)

allows hardware to transfer data directly to memory without CPU intervention

Polling

continuously checking device status (inefficient compared to interrupts)

System Clock

generates periodic interrupts used for timing and scheduling

Bootloader

initializes hardware and loads the OS kernel at startup

Monolithic Kernel

where all services (I/O, FS, drivers) run in kernel space (e.g., Linux)

Microkernel

where most services are in user space; more moduler (e.g., Minix)

System Call Interface (SCI)

boundary between user space and kernel space

Shell

user command interpreter (bash, PowerShell, etc)

File System

structure for storing and organizing data on disk

Inode (Unix)

data structure storing file metadata (permissions, owner, size, etc)

Process Control Block (PCB)

data structure storing process state (registers, stack pointer, memory info, etc)

PID (Process ID)

unique identifier for each process

Fork()

system call creating a new process by duplicating the current one (UNIX)

Exec()

system call that replaces current process image with a new program

Wait() / Sleep() / Exit()

common process management calls

Signals

software interrupts used for process communication (SIGINT, SIGKILL)

Pipes/Sockets

inter-process communication (IPC) mechanisms

User/Kernel Privilege Levels

CPU modes that restrict access to hardware (ring levels on x86: Ring 3 - user, Ring 0 = kernel)

Access Control

mechanisms that determine which users/processes can access which resources

Privilege Escalation

exploit allowing unauthorized elevation to higher privileges

System Call Hooking

intercepting system calls to alter or monitor behavior (often used in malware)

Sandbox

isolated environment restricting program access

Breakpoint

pauses program execution at a specific instruction

Disassembly

translating machine code into assembly code for analysis

Decompilation

translating binary code into approximate high-level code

Symbol Table

maps names of functions/variables to memory addresses (useful for debugging)

Import Table (PE Files)

lists external functions the program calls (e.g., Windows API Functions)

Obfuscation

making code intentionally hard to understand (used by malware)

Packers

compress/encrypt executables and unpack themselves at runtime