Compare Files with PowerShell: a faster way

Black thumb drive or "USB stick" as we call them here in The Netherlands.

Sometimes you need to test whether two files are the same. As files get larger, comparisons take longer, so we need to consider performance. In this article, I'll show how to compare two files using a buffered approach in PowerShell.

When talking about performance, it is better to measure multiple times on multiple systems. BenchmarkDotNet's good practices also warn against extrapolating results across environments. I only measured once on one system here, so treat these numbers as a relative indication rather than a general benchmark.

Strategies

There are several ways to compare files:

Extra performance: use LINQ

PowerShell has native comparison operators, but calling LINQ from PowerShell makes the buffer comparison much faster. We'll import the System.Linq namespace with using namespace System.Linq and call SequenceEqual like this: [Enumerable]::SequenceEqual($one, $two). In my single test, this LINQ method was substantially faster than the PowerShell comparison.

FilesAreEqual function

Let's create a function that takes two files, compares them, and returns $true when both files are equal:

using namespace System.Linq

function FilesAreEqual {
    param(
        [System.IO.FileInfo] $first,
        [System.IO.FileInfo] $second,
        [uint32] $bufferSize = 524288)

    function Read-FullBuffer {
        param(
            [System.IO.Stream] $stream,
            [byte[]] $buffer)

        $totalBytesRead = 0

        while ($totalBytesRead -lt $buffer.Length) {
            $bytesRead = $stream.Read(
                $buffer,
                $totalBytesRead,
                $buffer.Length - $totalBytesRead)

            if ($bytesRead -eq 0) { break }
            $totalBytesRead += $bytesRead
        }

        return $totalBytesRead
    }

    if ($first.Length -ne $second.Length) { return $false }
    if ($bufferSize -eq 0) { $bufferSize = 524288 }

    $fs1 = $null
    $fs2 = $null

    try {
        $fs1 = $first.OpenRead()
        $fs2 = $second.OpenRead()

        $one = New-Object byte[] $bufferSize
        $two = New-Object byte[] $bufferSize

        while ($true) {
            $bytesRead1 = Read-FullBuffer $fs1 $one
            $bytesRead2 = Read-FullBuffer $fs2 $two

            if ($bytesRead1 -ne $bytesRead2) { return $false }
            if ($bytesRead1 -eq 0) { return $true }

            if ($bytesRead1 -lt $bufferSize) {
                [Array]::Clear($one, $bytesRead1, $bufferSize - $bytesRead1)
                [Array]::Clear($two, $bytesRead2, $bufferSize - $bytesRead2)
            }

            if (-Not [Enumerable]::SequenceEqual($one, $two)) {
                return $false
            }
        }
    }
    finally {
        if ($null -ne $fs1) { $fs1.Dispose() }
        if ($null -ne $fs2) { $fs2.Dispose() }
    }
}

Buffering works

So what happens when we plug different values into $bufferSize? This is what I got on my machine by comparing the Wars of Liberty binary of 140 MB with itself:

[table id=8 /]

A sufficiently large buffer can make a file comparison much faster, which is why I settled on 524,288 bytes (512 KiB) for this PowerShell function. On my machine, comparing the entire 2.2 GB Wars of Liberty setup file took 42 seconds.

So what about a byte-by-byte comparison? In the same test, that took 32 seconds for the 140 MB file: more than 13 times slower than using a 524,288-byte buffer.

Just need a script?

If you just need a script, copy this text to compare.ps1 and run it from the command line like .\compare.ps1 .\article.html .\article2.html. The result will be printed to the console.

<#
.SYNOPSIS
    Compares two files. Returns True if the files are equal.
.DESCRIPTION
    Compares two files. Returns True if the files are equal; otherwise False.
    Use the bufferSize to optimize for speed. Might depend on your system.
.PARAMETER file1
    The first file.
.PARAMETER file2
    The second file.
.PARAMETER bufferSize
    The size of the buffer will influence the speed of the script.
.OUTPUTS
    True when equal otherwise False.
.LINK
    More info: https://keestalkstech.com/comparing-files-with-powershell/
#>
using namespace System.Linq

param(
    [Parameter(Mandatory = $true)]
    [string]
    $file1,
    [Parameter(Mandatory = $true)]
    [string]
    $file2,
    [uint32]
    $bufferSize = 524288)

function Read-FullBuffer {
    param(
        [System.IO.Stream] $stream,
        [byte[]] $buffer)

    $totalBytesRead = 0

    while ($totalBytesRead -lt $buffer.Length) {
        $bytesRead = $stream.Read(
            $buffer,
            $totalBytesRead,
            $buffer.Length - $totalBytesRead)

        if ($bytesRead -eq 0) { break }
        $totalBytesRead += $bytesRead
    }

    return $totalBytesRead
}

$ErrorActionPreference = "Stop"
$PSDefaultParameterValues['*:ErrorAction'] = 'Stop'

$first = Get-Item $file1
$second = Get-Item $file2

# Files with different sizes cannot be equal.
if ($first.Length -ne $second.Length) { return $false }
if ($bufferSize -eq 0) { $bufferSize = 524288 }

$fs1 = $null
$fs2 = $null

try {
    $fs1 = $first.OpenRead()
    $fs2 = $second.OpenRead()

    $one = New-Object byte[] $bufferSize
    $two = New-Object byte[] $bufferSize

    while ($true) {
        $bytesRead1 = Read-FullBuffer $fs1 $one
        $bytesRead2 = Read-FullBuffer $fs2 $two

        if ($bytesRead1 -ne $bytesRead2) { return $false }
        if ($bytesRead1 -eq 0) { return $true }

        if ($bytesRead1 -lt $bufferSize) {
            [Array]::Clear($one, $bytesRead1, $bufferSize - $bytesRead1)
            [Array]::Clear($two, $bytesRead2, $bufferSize - $bytesRead2)
        }

        if (-Not [Enumerable]::SequenceEqual($one, $two)) {
            return $false
        }
    }
}
finally {
    if ($null -ne $fs1) { $fs1.Dispose() }
    if ($null -ne $fs2) { $fs2.Dispose() }
}