cURL is a computer software project providing a library (libcurl) and command-line tool (curl) for transferring data using various protocols. It was first released in 1997. The name stands for “Client URL”. The original author and lead developer is the Swedish developer Daniel Stenberg.
Investigate problems due to User-Agent using Bash
Last week we had some problems with the Google Ads bot. It was not able to crawl a bunch of URLs while the browser had no problem getting through. The only difference was the User-Agent. This send us on a debugging journey through Cloudflare, gateways and micro-sites. To assist us, we’ve created a small bash script to visit an URL and show some debug info.
How to scrape a photo from a Twitter Photo URL using PHP
After explaining how to scrape a photo from a Twitter Photo URL using C# (ASP.Net) I decided to port the script PHP so I could use it on WordPress. I ended up using CURL and a regular expression to do the trick.