Packfiles: how git ships its database

In part one we opened .git and found a database of loose objects: one zlib-compressed file per blob, tree and commit, each named by the SHA-1 of its own bytes. In part two we met the 41 byte pointers into that database. And if you hovered carefully through part one's file explorer, one entry gave the game away early: objects/pack was annotated as a topic for part three of this series. This is part three. Time to pay that debt.

The same tiny git from part one is still running behind every figure here, with the same seven objects and the same hashes. On top of it I wrote a pack encoder and a delta codec in TypeScript, so the pack you'll dissect below is built live in your browser. It's not a mockup of a packfile, it is a packfile: I fed the exact bytes to real git and it unpacked all seven objects back, byte for byte. As always, don't take my word for anything; by the end you can check everything yourself.

01 · A database of confetti

Loose objects are a great format for a database at rest. Writing is atomic (an object is created once and never modified), deduplication is free, and corruption is local to one file. But let the repository age and the flaws show up:

Every version of every file is a separate full snapshot. Change one line in a 10 MB file and commit: that's a second 10 MB blob, compressed alone, sharing nothing with the first even though 99.9% of the bytes are identical.
Thousands of tiny files are expensive. Our toy repo's seven loose objects hold just 541 bytes of zlib between them, but on a typical filesystem each file occupies a whole 4 KiB block, plus a directory entry, plus an inode.
Shipping them is worse. A clone of a repository with a million objects can't reasonably open a million tiny downloads. It wants one stream.

The Pro Git book describes the fix:

"The packfile is a single file containing the contents of all the objects that were removed from your filesystem. The index is a file that contains offsets into that packfile so you can quickly seek to a specific object."

Git packs loose objects into packfiles when you have too many of them, when you push, or when you run git gc yourself, which delegates the actual work to git repack and git pack-objects. Run git gc in our repo from part one and .git/objects transforms: the seven xx/yyyy… files disappear, and objects/pack/ gains a pack-7e9b502f….pack with all seven objects inside, plus a pack-7e9b502f….idx sidecar (recent versions of git also drop a small .rev reverse index next to them). The whole pack is 478 bytes. One file, smaller than the seven files it replaced, and we'll see exactly why.

02 · One file to ship them all

Here is that idea at byte level. The figure below is a real pack built in your browser from part one's seven objects, following the pack format spec. Every byte belongs to a colored region, and the ASCII gutter on the right shows how little survives compression: PACK, a few lucky fragments, and noise:

The container is almost insultingly simple:

A 12 byte header. The four ASCII letters PACK, a 4 byte version number (git accepts versions 2 and 3 but only ever writes 2), and a 4 byte object count. All big-endian, which caps a single pack at 4 billion objects. The spec itself jokes about that observation.
One record per object, back to back. Each record is a small header (type + size, next section) followed by the object's body as a raw zlib stream. The blob 10\0 prefix that every loose object carries is not in the pack: the record header already encodes type and size, so git re-derives the prefix when it needs to hash the object.
A 20 byte trailer: the SHA-1 of everything above it. Corrupt a single bit and the checksum stops matching. Look at the file name git chose for the pack, pack-7e9b502f….pack, and then hover the trailer in the figure: the pack is literally named after its own checksum. Content addressing all the way down, again.

My encoder doesn't compress objects against each other, so the browser-built pack above stores seven full bodies and comes out slightly larger than git's 478 bytes. Real git does one extra trick inside the same container, and that trick is section 04.

03 · Reading a record

Each record starts with a header that packs a type and a size into as few bytes as possible. It's a variable-length integer with a twist: the first byte gives one bit to "there's more", three bits to the object type, and its last four bits to the size. Every continuation byte gives one bit to "there's more" and seven bits to the size. Size chunks accumulate little-endian: later bytes are more significant. These are real headers from our pack; hover the bit groups:

The three type bits map to six valid values:

Value	Type	What it is
1	`OBJ_COMMIT`	a commit, body stored whole
2	`OBJ_TREE`	a tree, body stored whole
3	`OBJ_BLOB`	a blob, body stored whole
4	`OBJ_TAG`	an annotated tag, body stored whole
6	`OBJ_OFS_DELTA`	a delta against a base found by offset, in this same pack
7	`OBJ_REF_DELTA`	a delta against a base named by its 20 byte object ID

Type 0 is invalid and type 5 is reserved for future expansion. The first four are the object types you already know from part one wearing a compact coat. The last two are new, and they only exist inside packs: an object stored not as its bytes, but as instructions to rebuild it from another object.

One subtlety about that size: decompression never needs it. The zlib stream that follows each header is self-terminating, so a reader always knows where a record ends without trusting the header. Git records the size anyway and verifies it when unpacking: an integrity check, not a navigation aid.

04 · Deltas, or why the pack is small

Remember the observation from part one: git stores full snapshots, never diffs. That's still true at the object model level. But inside a pack, git is free to represent an object however it wants, as long as the original bytes come back out. So when two objects look alike, git stores one of them whole (the base) and the other as a delta: a recipe that rebuilds it from the base.

The two delta types differ only in how they point at the base. OBJ_REF_DELTA embeds the base's full 20 byte object ID right after the record header. OBJ_OFS_DELTA embeds a relative offset instead, meaning "the base starts this many bytes before me in this same file", which is cheaper (a few bytes instead of 20) and is what git prefers on disk. That offset is its own little varint with opposite conventions (big-endian this time, plus a +1 bias per continuation byte); flip the header figure above to its ofs-delta example and it decodes the real one for you. In both cases the delta payload that follows is the same format, and it's fully specified in gitformat-pack: two varint sizes (base size, target size), then a sequence of exactly two kinds of instructions:

Copy (first bit 1): take size bytes from the base starting at offset. The remaining seven bits of the opcode say which offset and size bytes follow, so small numbers cost fewer bytes. A quirk straight from the spec: a copy size of zero means 0x10000, so the one-byte instruction 0x80 copies 64 KiB from offset 0.
Insert (first bit 0): the opcode's low seven bits are a count, and that many literal bytes follow, spliced into the target as-is. That's where genuinely new content travels. Opcode 0x00 is reserved.

This isn't hypothetical for our repo. Run git verify-pack -v on the pack git gc built and it confesses what it did with our two commits:

e268ce4f68b11ce8305d2b478e1c66681f6e3df5 commit 258 161 12
76309f46dfb50ee333bb88409baa488be4f622a2 commit  67  78 173 1 e268ce4f…
83baae61804e65cc73a7201a7252750c76066a30 blob    10  19 251

The second commit is stored whole. The first commit, 209 bytes of body, is stored as a 67 byte delta with the second commit as its base (that trailing column). Newest whole, oldest as delta: you're most likely to want the newest version fast, so that's the one that costs zero reconstruction work.

Those 67 bytes are sitting in the real pack on my disk, and I extracted them for the figure below. Watch them rebuild first-commit out of second-commit: three instructions, and the SHA-1 at the end lands exactly on 76309f4, the commit ID you saw in part one:

Read what the delta actually says, because it's delightfully unsentimental. The two commits share almost everything (same author, same pinned dates), so one copy instruction lifts a 151 byte run covering the author and committer lines in one go. The parts that differ (the tree line, the message) travel as literal inserts. Git found that copy without understanding commits at all: git pack-objects slides a window (default 10 candidates, configurable with --window) over similarly named and sized objects and keeps whichever pairing produces the smallest delta, with --depth (default 50) capping how long a chain of delta-upon-delta may get before reconstruction cost outweighs the savings.

05 · The delta lab

Now make some deltas yourself. Both boxes below are yours to edit; the delta between them is computed live by the same encoder and decoder used in the figures above, and every recomputed delta is round-trip checked against the target on the spot:

Try the classics. Append a sentence and watch it become one copy plus one insert. Change a single word in the middle and watch the delta bridge it with two copies and a tiny insert between them. Then paste two completely unrelated texts and watch the delta degenerate into pure inserts, larger than the target itself, which is exactly when git shrugs and stores the object whole instead.

06 · The sidecar index

One puzzle left. Objects in a pack sit at arbitrary offsets, and git cat-file still finds any of them instantly. Scanning 478 bytes is nothing, but scanning a 2 GB pack for every lookup would be absurd. That's the job of the .idx file that always travels next to the .pack:

The version 2 index starts with a magic number (\377tOc), a version number, and then a fan-out table: 256 cumulative counts, where entry N says how many object names in the pack start with a byte less than or equal to N. One array read narrows any lookup to a small range of the sorted name table that follows, a binary search does the rest, and a parallel table maps the found name to its byte offset in the pack. Version 2 also stores a CRC32 per object (so compressed bytes can be copied between packs and still be checked) and ends with two SHA-1s: a copy of the pack's checksum, chaining the pair together, and the index's own.

The index contains nothing you couldn't recompute from the pack itself, and that's the point: the .pack is the truth that gets shipped, the .idx is a local acceleration structure, rebuildable any time with git index-pack.

07 · What's next

The mental model, compressed (sorry): loose objects are the write-optimized face of the database and packs are the read-and-ship-optimized face. Same objects, same hashes, two physical layouts. A pack is a 12 byte header, a stream of type-and-size prefixed records, some stored whole and some as copy/insert recipes against a base, and a checksum. The index bolted next to it makes the archive seekable.

And "the ship-optimized face" is not a metaphor: when you git clone, the server builds precisely one of these and streams it at you. The protocol that negotiates which objects go into that pack, and how both sides agree on it byte by byte, is the next post. There we'll also meet the thin pack, a pack whose ref-deltas point at base objects that aren't in the file at all, because the sender knows you already have them.

If you're arriving mid-series: part one builds the object database these packs contain, and part two covers the refs that point into it. The pack parser and delta reconstructor also exist in my Python implementation, py-git, which clones real repositories over HTTP by doing exactly what this post described.