I use C# quite frequently to scrape information from sites. The standard C# WebClient is not always sufficient. I like to:

  • store/retrieve cookies – needed for session support;
  • prevent redirects;
  • retrieve the HttpStatusCode of a request.

I’ve ended up overloading .Net’s WebClient class and overriding the GetWebRequest method to set the cookie container and the redirect setting. To improve performance I’ve also enabled compression.

Hope it helps.

Updates made on 2015-05-21

The code is now part of the KeesTalksTech Utility Pack on GitHub. The class can be viewed here. I’ve updated the following:

  • Updated code to C# 6 with var’s and dot operators (?.)
  • Added a Setup action property to modify the request before it’s done.
  • Added XML documentation.