Hashing & Integrity

A cryptographic hash function takes an input of any size and produces a fixed-length digest — SHA-256 always outputs 256 bits, whether you feed it one byte or a terabyte. It is a one-way street: computing the digest from the input is fast, but recovering the input from the digest is infeasible. A hash is a fingerprint for data, not a means of encryption — there is no key and no way to reverse it.

To be trustworthy, a hash function must satisfy three properties:

• Preimage resistance — given a digest, you cannot find any input that produces it.
• Second-preimage resistance — given one input, you cannot find a different input with the same digest.
• Collision resistance — you cannot find any two distinct inputs that hash to the same value.

A vital feature is the avalanche effect: changing a single bit of input flips roughly half the output bits, so similar inputs produce wildly different digests. This is why hashes work for integrity checking, deduplication, digital signatures, and version control.

Not all hashes have aged well. MD5 (128-bit) and SHA-1 (160-bit) are both broken for collision resistance and must not be used for security.

A collision means two different inputs share the same digest — devastating for signatures, because a valid signature on one file would also validate a malicious one. In 2017 Google and CWI Amsterdam produced the first practical SHA-1 collision (SHAttered), two distinct PDFs with the same hash. SHA-2 remains solid. SHA-3, based on the Keccak sponge construction, was standardized by NIST in 2015 not because SHA-2 was failing, but to have a fundamentally different backup design in reserve.

Storing passwords as raw hashes is a classic mistake. If two users share the password hunter2, their hashes are identical, and an attacker who steals the database can use a precomputed lookup table — a rainbow table — to reverse common hashes instantly.

The fix is a salt: a unique, random value generated per user and stored alongside the hash. You hash the salt together with the password, so identical passwords produce completely different digests, and precomputed tables become useless — an attacker must crack each account separately.

A pepper goes one step further: a secret value added to every password before hashing, but stored separately from the database — in application config or, better, a hardware security module. If the database alone leaks, the pepper the attacker lacks makes cracking far harder. Salt defeats precomputation; pepper adds a secret the database dump does not contain.

Here is a counterintuitive truth: for passwords, you specifically do not want a fast hash. General-purpose hashes like SHA-256 are designed to be lightning fast, which means an attacker with GPUs can try billions of guesses per second against a stolen database.

Password hashing needs slow, tunable, resource-hungry functions built for the job:

• bcrypt (1999, based on Blowfish) — includes an adjustable cost factor so you can make it slower as hardware improves.
• scrypt (2009) — deliberately memory-hard, forcing attackers to spend RAM as well as time, which neutralizes cheap parallel GPU cracking.
• Argon2 — winner of the 2015 Password Hashing Competition and the current recommendation; the Argon2id variant tunes memory, time, and parallelism.

PBKDF2 is an older, still-acceptable option that relies on many iterations but is not memory-hard. The universal principle: pick a function whose slowness you control, and turn the cost up until a single guess takes a meaningful fraction of a second.

Two constructions built on hashes appear constantly in the wild.

HMAC (Hash-based Message Authentication Code) combines a hash with a secret key to prove both the integrity and the authenticity of a message. A plain hash tells you data was not corrupted by accident; an HMAC-SHA256 tag tells you it was not altered by an attacker, because forging the tag requires the secret key. HMAC secures API requests, webhooks, session tokens, and message authentication inside TLS. Its keyed design also sidesteps the length-extension weakness that naive "hash the key plus message" schemes suffer.

A Merkle tree (hash tree) hashes data in pairs, then hashes those hashes, repeatedly, until a single root hash summarizes an entire dataset. Its magic is efficient verification: you can prove a single item belongs to the set by revealing only a short path of hashes, not the whole dataset. Merkle trees are foundational to git commits, blockchains like Bitcoin, Certificate Transparency logs, the ZFS filesystem, and peer-to-peer systems like BitTorrent and IPFS.