Memory Safety & Buffer Overflow — Study Notes

1. What Is a Buffer?

  • A buffer is a region of memory used for temporary storage of data.

  • Commonly used for:

    • Strings

    • Input data

    • Network packets

    • File contents

  • Buffers have fixed sizes — exceeding these limits is dangerous.

2. What Is a Buffer Overflow?

  • Occurs when more data is written into a buffer than it can hold.

  • Extra data spills into adjacent memory and overwrites other variables, pointers, or control data.

  • Often results in:

    • Program crashes

    • Corrupted variables

    • Hijacked control flow (exploits)

3. Why Overflows Are Common in C

  • C has no built-in bounds checking for arrays or pointers.

  • Many standard C functions (strcpy, gets, etc.) assume inputs are safe.

  • Manual memory management leads to frequent developer mistakes.

  • Direct access to system memory makes errors more dangerous.

4. What Happens During an Overflow

  • Adjacent memory gets overwritten:

    • Local variables

    • Saved registers

    • Return addresses (on the stack)

    • Heap metadata (on the heap)

  • This behavior allows attackers to:

    • Change program behavior

    • Redirect program execution

    • Trigger crashes or arbitrary code execution

5. Stack Overflow vs Heap Overflow

Stack Overflow

  • Overwrites local variables, saved frame pointers, and the return address.

  • Overwriting the return address can yield arbitrary code execution.

  • Typical exploit technique in classic buffer overflow attacks.

Heap Overflow

  • Corrupts dynamic memory structures:

    • malloc metadata

    • object fields

  • Often used to manipulate program logic, not directly hijack the CPU.

  • Requires more complex exploitation.

6. Unicode and Multi-Byte Handling Issues

  • Some languages expand characters during encoding.

  • If a buffer holds fewer bytes than the expanded form, this may overflow.

  • Multi-byte → single-byte mismatch leads to incorrect size assumptions.

7. What Variables Can Be Affected?

  • Any variable adjacent in memory can be corrupted.

  • Example:

    int A;
char buf[32];
int B;

  • Overflowing buf likely overwrites B, not A (because A is usually stored before buf in memory).

8. Security Consequences of Buffer Overflows

  • Crashing the application

  • Overwriting data used later (logic corruption)

  • Leaking sensitive memory contents

  • Redirecting execution flow

  • Remote code execution (RCE)

  • Privilege escalation vulnerabilities

9. Mitigation Techniques

Stack Canary

  • Inserts secret value next to return address.

  • If overflow overwrites it → program aborts before exploitation.

ASLR (Address Space Layout Randomization)

  • Randomizes memory locations of:

    • Stack

    • Heap

    • Libraries

    • Executables

  • Makes guessing addresses for attacks extremely difficult.

DEP / NX (Non-Executable Memory)

  • Marks memory regions like stack/heap as non-executable.

  • Prevents injected shellcode from running.

Safe APIs

  • Use strncpy, snprintf, bounds-checked loops, etc.

10. Investigating a Potential Buffer Overflow

Safe investigation steps:

  • Examine source for unsafe functions.

  • Test with inputs of increasing length.

  • Use a debugger to inspect:

    • Register values

    • Stack frames

    • Crash location

  • Look for repeated crashes in the same code path.

  • Do NOT attempt malicious exploitation in real systems.

11. Impact of Compiler Optimizations on Security

  • Removing frame pointers makes stack tracing harder for analysis.

  • Optimization (-O2) may inline functions or reorder instructions.

  • This makes:

    • Control flow harder to understand

    • Stack layout less predictable

  • -O0 builds are easier for attackers and analysts because:

    • Stack layouts are stable

    • Function boundaries are visible

12. NOP Sleds in Exploitation

  • A NOP sled is a sequence of “no operation” instructions leading to shellcode.

  • Helps when the attacker cannot guess the exact jump address.

  • CPU “slides” through NOPs until it reaches the payload.

  • Increases tolerance for inaccurate address targeting.

13. Why Different Hosts May Behave Differently

Same malicious input may:

  • Crash on one system

  • Run fine (or be exploitable) on another

Due to differences in:

  • ASLR configuration

  • Stack canaries

  • DEP/NX settings

  • Compiler version

  • Optimization flags

  • System architecture (32-bit vs 64-bit)

Forensic evidence to examine:

  • Crash logs

  • Stack trace locations

  • Memory maps

  • Binary build metadata

  • Core dumps