Comprehensive Guide to NTFS Architecture and Forensics

Introduction to NTFS and Design Philosophy

History and Prevalence: * NTFS (New Technology File System) has been the default Windows file system since $1993$ . * It remains the dominant format for Windows desktops and servers ( $Windows\,10$ , $Windows\,11$ , $Server\,2019$ , $Server\,2022$ ). * While FAT persists in removable media and embedded devices, NTFS is the standard for hard drives and SSDs on Windows machines.
The "Everything is a File" Principle: * Every byte on an NTFS volume belongs to a file, including administrative metadata and the structures describing file locations. * This uniformity differs significantly from FAT, which uses fixed regions ( $Boot\,Sector$ , $FAT\,Region$ , $Root\,Directory$ ). * Administrative metadata stored as files includes: * The Master File Table (MFT). * Volume bitmap. * Transaction log. * Bad cluster list.
Volume Layout: * Any sector can hold any kind of data; there are no fixed regions except for the boot sector in sector $0$ . * The boot sector contains a pointer to the starting cluster of the MFT. * Forensic implication: Locating data requires walking the MFT, which may be fragmented anywhere on the disk.
Documentation and Specifications: * Microsoft has never published a complete on-disk specification for NTFS. * Knowledge is derived from reverse engineering and empirical research by experts like Brian Carrier, Mark Russinovich, and the Linux NTFS project. * Joachim Metz's LibSemp (also referred to as LibSense) documentation is the closest canonical reference. * NTFS is not static; it mutates with major Windows releases (updates to attribute types, journaling formats, and kernel behaviors).

The Master File Table (MFT)

Structural Overview: * The MFT is an array of fixed-size entries, typically $1024\,bytes$ ( $1\,KB$ ) each. * Every file or directory has at least one entry in the MFT. * The MFT itself is a file (Entry $0$ ). To read it, the system gets the starting cluster from the boot sector, then reads Entry $0$ to find the full run list of clusters belonging to the table.
MFT Entry Header (42 bytes): * Signature: A 4-byte ASCII value. FILE indicates a healthy entry; BAAD indicates the kernel detected corruption. * Flags: Includes the "in-use" bit (indicates an active file) and the "directory" bit. * Sequence Number: A $16\text{-bit}$ counter incremented each time an entry is freed. It helps detect stale references. * Base Entry Reference: Used when a file's attributes overflow into additional MFT records. * First Attribute Offset: Byte offset to the start of the attributes (typically follows the header).
Address Structure (File Reference Address): * A $64\text{-bit}$ value composed of: 1. File Number ( $48\text{-bits}$ ): The sequential index in the MFT array. 2. Sequence Number ( $16\text{-bits}$ ): Located in the upper bits to track reuse. * Forensic Utility: By comparing the sequence number in a recovered reference to the current entry, analysts can determine if a deleted file's metadata has been overwritten (reused) or remains valid.

Reserved System Files (Entries 0–15)

Entry 0 ( $MFT$ ): The table itself.
Entry 1 ( $MFT\,mer$ ): A backup of the first few (typically four) MFT entries to prevent total volume loss if the start of the MFT is damaged.
Entry 2 ( $log\,file$ ): The transaction journal for crash recovery; contains preimage copies of metadata changes.
Entry 3: Volume label and identifier.
Entry 4 ( $attrdef$ ): Defines the attribute types supported by the volume.
Entry 5: The Root Directory (represented as . in some tools).
Entry 6 ( $bitmap$ ): Tracks cluster allocation across the volume.
Entry 7 ( $Boot$ ): Contains the boot sector and boot code.
Entry 8 ( $Bad\,Clus$ ): Lists clusters marked as physically defective.
Entry 9 ( $Secure$ ): Consolidated security descriptors and ACLs (added in $Windows\,2000$ for deduplication).
Entries 10–15: Reserved for future use.
User Files: Typically begin at Entry $24$ on freshly formatted volumes.

Attribute Mechanics and Overflow

Everything is an Attribute: * NTFS reads and writes attributes, not "file content" directly. File content is merely the content of a $DATA attribute. * Attributes are packed end-to-end after the header. They are variable length and have no padding.
Standard Attribute Header: * Type Identifier: A numeric value (e.g., $16$ , $32$ , $48$ , $128$ ). * Total Size: Includes header and content; used to find the next attribute. * Resident vs. Non-Resident Flag: Determines if content is in the MFT entry or in external clusters. * Attribute Identifier: A unique ID used to distinguish multiple attributes of the same type (e.g., multiple data streams).
Attribute Overflow: * When file attributes (e.g., extensive fragmentation, long names, many streams) exceed the $1024\text{-byte}$ MFT entry, NTFS uses Attribute Lists (Type $32$ ). * The Base Entry contains the $ATTRIBUTE_LIST which maps attributes to additional (non-base) MFT entries. * Maximum attribute count per file: $65,536$ (limited by the $16\text{-bit}$ ID).
Resident vs. Non-Resident Attributes: * Resident: Content fits inside the MFT record (cutoff is roughly $700\,bytes$ ). Metadata like $STANDARD_INFORMATION and $FILE_NAME are always resident. * Non-Resident: Content is stored in regular clusters. The MFT entry holds a Run List (mapping Virtual Cluster Numbers/VCN to Logical Cluster Numbers/LCN). * Run List Encoding: Compact pairs of (Starting Cluster, Length). Knowing LCN and VCN is essential if tools fail to parse corrupted metadata.

Specific Attribute Types

Type 16 ( $STANDARD\,INFORMATION$ ): * Contains timestamps (Modified, Accessed, Created - MAC), flags (hidden, system), owner SID, and security ID. * User-space tools/malware can modify these via Win32 APIs (e.g., SetFileTime).
Type 48 ( $FILE\,NAME$ ): * Contains the filename (Unicode), parent directory reference, file size, and a second set of MAC timestamps. * These timestamps are set by the kernel and are not accessible via standard APIs. * Timestomping Indicator: If timestamps in Type $16$ and Type $48$ mismatch, it suggests intentional manipulation of the user-modifiable timestamps.
Type 128 ( $DATA$ ): * Contains the file's primary content. * Alternate Data Streams (ADS): NTFS allows multiple $DATA attributes per file if they are named. These are often used for metadata (e.g., Zone.Identifier for internet downloads) but can be used by malware to hide executables, as standard tools (Explorer, dir, ls) only show the default unnamed stream.
Directory Index Types: * Type $144$ ($INDEX_ROOT): Always resident; the root of the B-tree. * Type $160$ ($INDEX_ALLOCATION): Non-resident; additional B-tree nodes. * Type $176$ ($BITMAP): Tracks use of index records in allocation space.
Security and Encryption: * Type $256$ ($LOGGED_UTILITY_STREAM): Often contains EFS (Encrypting File System) key material, identified by the name EFS.

Directory B-Tree Structures and Rebalancing

B-Tree Mechanics: * NTFS stores directories as balanced, multi-way search trees (B-trees) for efficiency with large directories. * Entries are kept in sorted order. * Nodes consist of a root (in $INDEX_ROOT) and children (in $INDEX_ALLOCATION). * Each index record is typically $4096\,bytes$ .
The Problem for Forensic Recovery: * Unlike FAT (which just marks an entry with $0xE5$ ), NTFS B-tree operations (insert/delete) often require rebalancing. * Insert/Splits: Adding a file can cause a node to split, overwriting the slack space of old nodes and destroying deleted file remnants. * Delete/Rebalance: Deleting a file can cause entries to move between nodes to maintain balance. A moved entry in unallocated space may look like a deleted file (a false positive). * Forensic recovery of filenames is "probabilistic, not deterministic" due to these constant rewrites.

Special Storage Modes: Sparse, Compressed, and Encrypted

Sparse Files: * Reduces disk usage by not writing regions containing only zeros to physical clusters. * The run list contains "sparse runs": length is defined, but there is no LCN address. The OS returns zeros for those regions.
Compression: * Applied only to non-resident data attributes. * Data is divided into Compression Units (CU), typically $16\,clusters$ ( $64\,KB$ ). * Each unit is stored as either zeroed (sparse run), compressed (data + sparse padding), or uncompressed. * Forensic tools must be "unit-aware" and merge fragmented physical runs before decompressing, or they will produce garbage data.
EFS (Encrypting File System): * Encrypts only the $DATA attribute content; metadata (filenames, sizes, timestamps) remains in the clear. * Uses a symmetric File Encryption Key (FEK) (AES-256 on modern systems; DESX or 3DES on legacy). * The FEK is itself encrypted using each authorized user's public key, stored in Data Decryption Fields (DDF) within the $LOGGED_UTILITY_STREAM. * Data Recovery Field (DRF): Keys for recovery agents (domain admins). * EFS Recovery Chain: Password $\rightarrow$ Private Key $\rightarrow$ DDF $\rightarrow$ FEK $\rightarrow$ Plaintext. * Forensic Artifact - EFS0.TMP: A temporary plaintext copy created during the encryption process. If not zeroed out or if the system crashes, plaintext may survive in unallocated space.

Forensic Tooling and Techniques

Deletion Behavior: In NTFS, MFT entries are not erased; the "in-use" flag is changed to $0$ . Contents remain until reused.
Sleuth Kit Operations: * istat: Displays MFT entry headers, attribute types, names, identifiers, flags, and run lists. * icat: Extracts attribute content. Syntax: icat <image> <entry>[-<type>-<id>]. * Example to extract an ADS: icat image 34-128-5 ( $128$ is $DATA, $5$ is the identifier for the stream). * fls: Lists directory contents by walking the B-tree index.
Standard Recovery Workflow: 1. Walk the live B-tree. 2. Scan for "dead" or unallocated index records/MFT entries. 3. Check Slack space within live index records for recently removed entries.