Last week I was working on our new cockpit application, which is essentially a list of links to parts of our Wehkamp platform. The old application was not being maintained, as the React-stack is not something that’s in the skill-set of most engineers. We kept the new cockpit simple: plain old HTML. Of course we wanted to support a nice dark-theme as well. This article shows how simple it is to implement dark mode.

Last week we wanted to parse some XML data with Spark. We have a column with unstructured XML documents and we need to extract text fields from it. This article shows how you can extract data in Spark using a UDF and ElementTree, the Python XML parser.

This week we’ve been working on processing the access logs from Cloudflare with Databricks (Spark). We now have a job that generates a huge CSV file (+1GB) and sends it on towards by FTP for further processing with an external tool. Creating a DataFrame with the right data was easy. Now, let’s explore how to do a CSV export, secrets management and an FTP transfer!

I operate from the Netherlands and that makes my time zone Central European Summer Time (CEST). The data I handle is usually stored in UTC time. Whenever I need to crunch some data with Spark I struggle to do the right date conversion, especially around summer or winter time (do I need to add 1 or 2 hours?). In this blog, I’ll show how to handle these time zones properly in PySpark.

At Wehkamp, we use Google Analytics extensively to measure how many users are active. It is hard to correlate active users with your other statistics that “live” in Prometheus. In this blog I’ll show how you can use the Google Real Time API to produce statistics for active users, page views and screen views.

It is pretty easy to write your AWS ALB access logs to S3, but if you want to do something with them, you might want to add them to AWS Athena, so you could query them using plain old SQL. Let’s investigate how we can see which upstreams / targets are misbehaving.

Sometimes you just need to record the original message that was submitted to your API. My colleague Onno Pierik and I encountered such an event. We needed to record the original submitted SOAP message (body) and submit it (under certain conditions) to another service. To be honest: most of the scenarios I’ve seen so far end up with memory problems, so use with caution!

To make a setup more resilient we could allow for actions to be retried when they fail. We should not “hammer” our underlaying systems, so it is wise to wait a bit before retrying (exponential backoff). Let’s see how something like this could be done in Python. Note: this only works if actions are idempotent and you can afford to wait.

I’ve installed and configured AWS Vault in Windows. In this blog I’ll show how to setup MFA, automate token rotation, share AWS Vault with Windows Subsystem for Linux (WSL) and how to do an ECR login on Docker.

There are a myriad of tutorials on Redis in almost every programming language. Many will cover how to make a to-do list, so why write another one? Well, I want to write a tutorial that is language agnostic and only uses Redis commands and Lua scripts (the build-in scripting language of Redis).