Simulating a hard disk at the block level to hear it defragment

Nothing recorded, from the file system to the speaker.

Here is Windows 98 starting on a 4 GB, 5,400 rpm drive, as the model hears it: power-on, the motor spinning up, the heads leaving the landing zone, the kernel read in one long run, then the drivers, scattered all over the platter.

Windows 98 boot · 1999 · 20 s

Five stages produce that sound, and none of them plays a sample:

1. A volume, generated from a history Years of use replayed through the allocator of the disk’s file system: which files exist, and exactly which clusters each one holds.
2. A planner A boot, an installation, a day of use or a defragmentation decides which files to read or write, in which order, and how long the computer computes between two requests.
3. Block requests, one at a time A date, a logical block address, a number of sectors, read or write — what an operating system actually sends a disk.
4. The drive’s mechanics Buffer and bus, the block address turned into cylinder, head and sector on a zoned platter, the seek, the rotational latency, the transfer.
5. Sound, map and haptics The timeline of seeks, head switches, track steps and transfers drives the synthesiser, the cluster map and the Taptic Engine, on one clock.

The app ships twenty-four such disks — four users in each of six eras from 1993 to 2012: a developer, a family, a gamer and a secretary, with a power user standing in for the family in 1993 — and lets you build more. The rest of this page follows the chain from the top.

Fragmentation is not a setting. It is a residue.

The obvious way to build a fragmented disk is to ask for one: so many files, so many pieces each. It sounds wrong, because real fragmentation has a texture — the system files in one piece at the start, the compiler’s objects chopped into the holes the previous build left, a swap file nobody moves. That texture comes from what happened to the disk, in what order.

So nobody asks for a fragmentation. A disk is written as a story — an installation, builds, downloads, documents saved over and over, updates — and the story is replayed, write by write and delete by delete, through the allocator of the file system it lives on. The story is written before any allocation and knows nothing of the format, which makes a clean experiment possible: the same developer history, replayed on three allocators over the same 512 MB.

None of those numbers is a parameter. MS-DOS FAT serves the first free cluster from the start of the volume at every write: holes are plugged at once, the front of the disk becomes a dense Swiss cheese and recent files are chopped up. FAT32 resumes at the last cluster it allocated: writing is clean while the cursor advances, then it wraps round and finds the holes left months before — fragmentation arrives in waves, and one file ends in 98 pieces. NTFS looks for a hole that fits.

One detail matters more than the distribution of sizes: deletes are interleaved with writes. A build that created all its objects and then deleted them all would reopen one large contiguous space and fragment nothing. In reality the compiler writes while the linker reads, the precompiled header is rebuilt in the middle, and whatever is written at that moment falls into the holes that are still open.

Directories exist and grow like files, away from their first cluster. Files whose size is not known in advance — a compiler’s output, a browser cache, a document being saved — grow as they are written rather than getting their space at once. On NTFS that one distinction produces the long tail of files in two to sixteen pieces.

What you hear is the algorithm.

A defragmenter is a planner like any other: it reads the volume’s map, decides what to move where, and emits reads and writes. The drive does not care why. So two tools on the same volume sound different for the same reason they take different times — they do not ask the disk for the same things.

The volume below is the 1993 developer’s: a 210 MB FAT16 drive at 3,600 rpm, 72 % full after two years of Borland C++ builds, with 342 of its 4,027 files in pieces. On the left, the defragmenter of the time, MS-DOS 6 DEFRAG, which the model runs on the same engine as the Windows 95 defragmenter: they did the same thing in the same order. On the right, a strategy written for this project.

DEFRAG packs every file against the start of the volume, in the order of the directory tree. Whatever stands where the next file should go is evacuated to the end of the volume, to be brought back when the sweep gets there. On this disk it moves 3,997 files, makes 3,090 evacuations and moves 229 MB — more than the volume holds — in 41,703 seeks. Twenty of its twenty-five minutes go into the temporary files and caches: thousands of one-cluster fragments. That is the hammering.

MS-DOS 6 DEFRAG · 10 min into the pass · 15 s

Frontier compaction starts from one rule: the order on the disk is the order to keep. A frontier sweeps the volume from its start, and everything below it is in place. A file in one piece that begins at the frontier stays where it is. A small hole is filled by a file that had to move anyway; otherwise the file after the hole slides down, in chunks the size of the hole, each written into the space the previous one just left. Two free clusters are enough to pack a whole volume, and the hole grows as it climbs. The tables are written once per batch, and no write in a batch lands on a cluster the batch has just freed — so no write ever lands on data still in use, and a power cut mid-pass leaves a consistent volume.

Frontier compaction · 3 min 20 into the pass · 15 s

It moves 1,484 files and 120 MB, and takes 5 min 31 s. Across the twelve FAT volumes of the gallery the difference in time is small and fragile — 4 h 32 in all against 4 h 51, and the smallest change to the model can swap them — but it moves 1.4 times less data and on the 1999 volumes leaves at most two holes in the free space, where Windows 95 leaves up to 59. It is also the only one of the three FAT tools that tidies a volume 99 % full: where DEFRAG finds nowhere to evacuate and gives the volume back as it was.

On large NTFS volumes the gap is not a matter of degree. The Windows 95 strategy on the 320 GB family disk of 2007 would pack hundreds of gigabytes through a 256 KB buffer: 4.8 million requests and 5 h 47 of simulated pass. The Windows XP defragmenter on the same volume issues 1.2 million and takes 29 min 21 s.

The app has seven tools in all: the two of the era (DEFRAG and the XP defragmenter, renamed for Vista and 7), JkDefrag 3.36 and UltraDefrag 7.1.1 modelled on their source code, and three written here. The README has a table per tool and per volume.

Between the files and the disk, an operating system.

A boot is not a list of files read in turn. For everything dated Windows XP, the model reproduces what the system did between the file system and the drive, down to its constants:

• The prefetcher reads the boot back in the order it was first used, in large batches issued at once — MFT records first, then data, then executable images.
• The IDE driver serves what is waiting in increasing block order from its current position, so a batch is heard as one sweep of the arm.
• The cache’s lazy writer wakes once a second, three seconds after the first page is dirtied, and writes an eighth of the dirty pages, stream by stream, the log first.
• The NTFS allocator takes the smallest free run that fits, from a cache of free runs rebuilt at mount; freed clusters stay masked until the next checkpoint.
• The defragmenter lays out the boot files first, pulls the MFT back together, repairs broken files into the smallest hole that holds them, empties a region when none does, then packs, in 64 KB blocks, each a transaction.

Windows XP boot · 2003 · 7,200 rpm, fluid bearing · 15 s

Windows 95 and 98, Vista and 7 do not get that treatment. They keep a model built from their documented behaviour, and the README says, point by point, where it is an assumption.

Deduced from real datasheets, and from the year.

A capacity alone does not describe a disk: the same gigabyte is a whole drive in 1996 and a corner of a platter, read five times faster, in 2003. So the model interpolates in time between eight drives that were really sold — from a Conner CFA170A of 1993 with a 64 KB buffer to a Seagate Barracuda 7200.14 of 2012 with 64 MB — taking from their manuals the number of tracks per surface, which sets the stroke of the arm and so the acoustics of a seek, and the capacity of a surface, which sets the number of platters.

• The seek follows the two-regime law of Ruemmler and Wilkes (IEEE Computer, 1994) — a square root of the distance for short seeks, a straight line beyond — split into speed-up, coast, slow-down and settle. A seek before a write settles longer, as the datasheets publish.
• Rotation and transfer are simulated sector by sector on a zoned platter, with track steps and head switches at the end of a cylinder.
• The buffer reads ahead after a read; the write cache acknowledges a write at once and lays it down later, in elevator order — so the disk works while the computer computes.
• Power-on has three stages: the motor spins up, the heads unstick from the platter, the drive seeks track 0. Drives of 2012 park on a ramp instead, and neither unstick nor land.
• Drives from before 1997 recalibrate two minutes after they are ready, then every four minutes: a second of crackle, the signature of that generation — and the reason “AV” drives without recalibration were sold for video editing.

Resonators that do not change pitch, excited by a moving arm.

The first idea is to map seek distance to a sample and pitch it. It sounds like a toy. An actuator is a mechanical structure with its own modes; a longer seek does not transpose them, it excites them differently. So the head is a bank of resonators at fixed frequencies — modes near 4.5 kHz (the arm’s sway) and 5.5 kHz, and five more — driven by a current profile drawn from the four phases of each seek. A short seek has no coasting phase, so the shape of the sound changes with the distance, not only its loudness. Its level follows the manual: the seek sound power the manufacturer publishes, less the idle level.

Then the rule that made it sound like a disk rather than a typewriter. Two seeks close together never fire two separate sounds: they are merged into one continuous excitation passed once through the resonators. The idea comes from MAME’s floppy-drive emulation, where retriggering the step sample sounded “much too loud”. Head switches and track steps less than 30 ms apart are treated the same way — which is why a long sequential read is heard as a pitch rather than a string of ticks, as in the kernel read of the first sound on this page.

The same boot · head only

The same boot · spindle only

The spindle is filtered noise shaped by three figures from the datasheet: speed, number of platters, and bearing. Ball bearings, used until about 2000, whistle and beat once per revolution; fluid bearings leave only the breath of air around the platters.

Spindle · 1996 · 5,400 rpm, ball bearing

Spindle · 2003 · 7,200 rpm, fluid bearing

The Taptic Engine receives the same cues. A single seek is played as its four phases; a dense train becomes one continuous event, because at 150 seeks a second a burst of transients is mush to the hand.

Live on the phone, bit-exact on the desk.

In the app nothing is computed in advance. The planner runs on its own thread and emits its requests one by one; each goes straight through the drive simulation into a buffer, and the thread sleeps once it is about eight seconds ahead of what you hear. A pass has no known length before it ends, there is no rewind, and a block on the map changes colour exactly when its write is heard.

The same code builds as a command-line renderer. Its output is streamed too — a pass of several hours renders without holding more than a few seconds in memory — and it keeps the order of the additions of the live mix, so the WAV is identical to the bit. That property carries the whole project:

• A refactor that should change nothing is checked by hashing 58 renders before and after.
• A change to a planner is measured on a few hundred pass reports — requests, seeks, files moved, pieces and holes left — in a handful of minutes, without rendering a note.
• Every figure the README quotes from a report is regenerated by a script, which says which one no longer holds.
• The App Store previews are filmed on a simulator, which records no sound; the sound is rendered offline for the same seconds of the same pass, and lands within a frame.

Every sound and film on this page comes out of that renderer. None was recorded from a device.

What is measured, and what is tuned by ear.

Geometry, seek times, buffers, spindle speeds and bearings come from manufacturers’ manuals. The timbre does not: no manual publishes the spectrum of its drive, so the frequencies of the spindle bands, the depth of the bearing’s beat and the way the head’s modes are dosed with distance are set by listening. The hum is the weakest link — computed from the spindle speed, for lack of a recording.

Some behaviours are choices too, and are labelled as such in the code: where DEFRAG evacuates to — the end of the volume, a choice calibrated on the pass durations of the period; the order in which a recalibrating drive visits its reference tracks; and, for NT 4, Vista and 7, four constants that bound the NTFS allocator’s search for holes. On the 2007 family disk, one of them alone moves the share of fragmented files from 22 % to 35 %.

If you know a drive or a defragmenter of that era better than its manual does, the issues are open.

The app

Winchester: Defrag Sounds, free, iPhone and iPad

Source code

github.com/glandais/winchester

The mechanism, stage by stage

How it works

Questions and corrections

github.com/glandais/winchester/issues