Seek Position of a String in a Stream

Yesterday I was working on some C# code that had to read XMP metadata from a file. It is not located at a fixed position, so I had to scan the file. XMP, being plain XML, can be found using simple string matching. After some searching, I found many solutions that read the entire file into memory before performing a regular expression search or string comparison. That’s not going to work for me, because some of my files are over 100 MB. So I wrote a method that searches for a string in a stream without loading the entire file into memory.

Heads-up: This implementation requires a seekable stream because it reads and updates Position. It works with FileStream and MemoryStream, but not directly with many HTTP, network, compression, or encryption streams. In the example, we’ll download the file first and then open it as a FileStream.

Hello from 2026! I’ve rewritten this article for .NET 10.

Encoding, encoding, encoding!

When searching for a string in a byte stream, you need to know which encoding was used to write it. UTF-32 uses four bytes per Unicode scalar value, while ASCII uses one byte for every character it supports. UTF-8 uses between one and four bytes. Let’s make the encoding explicit when converting the search text to bytes:

using System.Text;

public static long Seek(Stream stream, string text, Encoding encoding)
{
    var search = encoding.GetBytes(text);
    return Seek(stream, search);
}

Performance: use a buffer

We could read the stream one byte at a time, but that would require far more calls to the stream. Instead, we’ll use a 64 KB buffer as a practical default. It keeps memory usage bounded while significantly reducing the number of reads. If the search sequence is larger than half the buffer, we’ll increase the buffer to twice its size so there is always enough room for overlap.

There is one complication: a match can start at the end of one buffer and continue in the next. When a buffer contains no complete match, we preserve its final search.Length - 1 bytes and append the next read. Those bytes are enough to detect every possible match that crosses the buffer boundary.

Seek a string in a stream

Let's look at the code:

public static long Seek(Stream stream, byte[] search)
{
    ArgumentNullException.ThrowIfNull(stream);
    ArgumentNullException.ThrowIfNull(search);

    if (search.Length == 0)
    {
        throw new ArgumentException("Search cannot be empty.", nameof(search));
    }

    var bufferSize = Math.Max(64 * 1024, checked(search.Length * 2));
    var buffer = new byte[bufferSize];
    var buffered = 0;
    var position = stream.Position;

    while (true)
    {
        var read = stream.Read(buffer, buffered, buffer.Length - buffered);

        if (read == 0)
        {
            return -1;
        }

        buffered += read;

        var index = buffer.AsSpan(0, buffered).IndexOf(search);

        if (index >= 0)
        {
            var matchPosition = position + index;
            stream.Position = matchPosition + search.Length;
            return matchPosition;
        }

        var keep = Math.Min(search.Length - 1, buffered);
        var consumed = buffered - keep;

        buffer.AsSpan(consumed, keep).CopyTo(buffer);

        position += consumed;
        buffered = keep;
    }
}

ReadOnlySpan<byte>.IndexOf performs the byte search without allocating another buffer. When a match is found, the method returns the position of its first byte and moves the stream to the first byte after the match. This makes consecutive searches safe, even when the stream reads ahead.

XMP POC

We started out with the problem of reading XMP metadata from huge files. I cannot share those files, so let’s use a smaller sample to prove the concept. We’ll download it to a temporary file instead of buffering the HTTP response in memory, open it as a FileStream, and extract the XMP metadata between the two markers:

var url = "https://keestalkstech.com/wp-content/uploads/2020/06/photo-with-xmp.jpg?1";
var tempFile = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());

using var client = new HttpClient();
await using var downloadStream = await client.GetStreamAsync(url);

await using var stream = new FileStream(
    tempFile,
    FileMode.CreateNew,
    FileAccess.ReadWrite,
    FileShare.None,
    64 * 1024,
    FileOptions.Asynchronous | FileOptions.DeleteOnClose);

await downloadStream.CopyToAsync(stream);
stream.Position = 0;

var encoding = Encoding.UTF8;
var start = Seek(stream, "<x:xmpmeta", encoding);

if (start < 0)
{
    throw new InvalidDataException("The XMP start marker was not found.");
}

var end = Seek(stream, "<?xpacket", encoding);

if (end < 0)
{
    throw new InvalidDataException("The XMP end marker was not found.");
}

stream.Position = start;

var length = checked((int)(end - start));
var buffer = new byte[length];

stream.ReadExactly(buffer);
var xmp = encoding.GetString(buffer);

The extracted XMP looks like this:

<x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='Image::ExifTool 10.40'>
<rdf:RDF xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'>

 <rdf:Description rdf:about=''
  xmlns:dc='http://purl.org/dc/elements/1.1/'>
  <dc:creator>
   <rdf:Seq>
    <rdf:li>Chris Reid</rdf:li>
   </rdf:Seq>
  </dc:creator>
  <dc:rights>
   <rdf:Alt>
    <rdf:li xml:lang='x-default'>Unsplash, free to use</rdf:li>
   </rdf:Alt>
  </dc:rights>
  <dc:title>
   <rdf:Alt>
    <rdf:li xml:lang='x-default'>Python Code</rdf:li>
   </rdf:Alt>
  </dc:title>
 </rdf:Description>
</rdf:RDF>
</x:xmpmeta>

Final thoughts

Searching a stream sounds simple until encoding, buffer boundaries, short reads, and positioning get involved. Span<byte>.IndexOf keeps the byte search concise, while preserving the overlap between reads ensures that we don't miss a match crossing a buffer boundary.

This implementation deliberately targets seekable streams. For an HTTP, network, compression, or encryption stream, copy the data to seekable storage first or use a forward-only algorithm. And remember: a stream contains bytes, so always use the encoding that was used to write the text.

Changelog

2026-07-18 Rewrote the article for .NET 10. Added explicit encoding, buffered cross-boundary matching with short-read handling, seekable-stream guidance, and a file-backed XMP example using ReadExactly.
2022-09-18 Swapped out the WebClient for an HttpClient to be more compatible. Code is now written in .NET 6.
2020-06-07 Changed the article to reflect the latest insights and .NET Core 3.1.
2010-11-20 Initial article.