Smart NTFS and Block-Level Backups

A common limitation in backup operations is the metadata scan required by traditional file-based backup software. These solutions typically traverse the directory tree and use stat() operations to check modification timestamps. On file servers with a large number of files, this scan can take significant time before data transfer begins.

To improve incremental backup efficiency, StateWarden bypasses user-space filesystem traversal in favor of block-level tracking.

1. Limitations of File-Based Backups

When traditional software performs a file-based backup, it operates within user-space:

1. It queries the OS for the directory and file list.
2. It queries the OS for file metadata (e.g., size, modification date).
3. It opens the file and reads it sequentially.

If a large file (e.g., a 50GB virtual machine disk image) has a minor modification, a file-based backup solution often reads and hashes the entire file to process the changes.

2. Block-Level Backup Architecture

StateWarden processes storage volumes at the block level. By integrating with low-level host kernel APIs, the system identifies and transfers only modified sectors.

Windows: Smart NTFS and the USN Journal

On Windows environments, StateWarden uses the following process:

1. USN Journal Parsing: The Agent monitors the NTFS Update Sequence Number (USN) Journal, which records file modifications. The Agent parses this journal to identify modified files without scanning the directory tree.
2. Physical LCN Mapping: The Agent uses FSCTL_GET_RETRIEVAL_POINTERS to map the modified files directly to their physical Logical Cluster Numbers (LCNs) on the disk.
3. Raw Sector Reads: The Agent performs raw sector reads from \\.\PhysicalDrive0 rather than standard user-space file reads. This improves throughput and bypasses traditional file-locking conflicts.

Linux: Kernel-Level Tracking and Logical Volume Management

On Linux environments, StateWarden utilizes the Device Mapper framework:

1. Write Interception: StateWarden integrates with epoch-based tracking targets in the Linux kernel (dm-era). These structures monitor and record write operations at the block device level.
2. Filesystem Agnostic: Because tracking occurs at the sector level on the volume (e.g., /dev/md0p1), the Changed Block Tracking (CBT) operates independently of the underlying filesystem (e.g., ext4, xfs, btrfs).

3. Operational Benefits

Operating below the filesystem abstraction layer provides several benefits:

• Efficient Incremental Scans: The metadata scan is minimal. The Agent identifies, hashes, and transmits only the modified data chunks.
• Reduced IOPS: By utilizing raw sector reads, StateWarden reduces the read/write IOPS load on the production server during a backup window.
• Block-Level Deduplication: Mathematical chunking of the raw disk stream supports efficient cross-device deduplication at the Data Plane (Driad). Duplicate blocks across different servers (e.g., common OS updates) are stored only once.