Today I had some data coming into our event driven landscape, so I needed to know when my data was processed. As it constituted the processing of 400.000+ records (and I had more things to do), I needed a small script that kept an eye on my JSON API endpoint to see if the values are changed.
Read the article Poll JSON endpoint until value changes with bash & curl
I’ve tried to use the ngrok npm package in my application, but as the documentation says: “The ngrok and all tunnels will be killed when node process is done.” I need the process to “survive” my application. Let’s see what we can do about that…
Read the article Incorporate free ngrok in your Node.js application for local development
I love attribute validation! They can be used for all kinds of things. In ASP.NET Core we use them to validate models that come into our controllers. In one of our projects we kept running into the same thing: we need to validate a value against an array of pre-defined values. So we wrote some base validation attributes.
Read the article “Is One Of” and “Is Not One Of” validation attributes
I imagine your first thought is: why? Well, at Wehkamp we do a lot of cross platform development, but sometimes we end up with shell scripts that do stuff with Docker and Python. Usually that’s not a problem for Mac, but for Windows it’s a different thing. I have a MacBook Pro, but I’m a .NET developer, that’s why I prefer Windows, so I run Bootcamp. This article will show how to do Python development in the Windows Subsystem for Linux (WSL) using Visual Studio Code and Docker.
Read the article Visual Studio Code + Docker + Python using WSL in Windows on Boot Camp
This week we’ve been looking at joining two huge tables in Spark into a single table. It turns out that it is not a straightforward exercise to join data based on an array of IDs. In this blog I’ll show one way of doing this.
Read the article Spark: replace array with IDs with values; or: how to join objects?
Today I helped my neighbour to set up her new Honeywell T6 thermostat on a 2.4GHz Wi-Fi. It did not go well… Turned out that the setup does not work on Samsung devices.
Read the article Fix: stuck in Wi-Fi setup Honeywell T6 installation on Samsung
My Synology disk crashed and so did my Docker set up. Basically, the CI/CD pipeline for my programs no longer existed. The wonderful thing of an awful crash like this, is that I could rethink my setup. The result is what I would call “a poor man’s CI/CD”. It’s just Git, Docker, Docker Compose and Cron. It is easy to set up and it might be all you need.
Read the article Docker on Synology: from git to running container; the easy way
For some blogs I need to capture terminal screens. The recording of these types of screens have different requirement then normal application or website recordings. The bottom of the video is the most important part, the letters need to be crisp and readable for the end user.
Read the article Add black borders to terminal screen recordings
Our data strategy specifies that we should store data on S3 for further processing. Raw S3 data is not the best way of dealing with data on Spark, though. In this blog I’ll show how you can use Spark Structured Streaming to write JSON records of a Kafka topic into a Delta table.
Read the article Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming
There is a lot of code that needs to make a selection based on a maximum value. One example are Kafka reads: we only want the latest offset for each key, because that’s the latest record. What is the fastest way of doing this?
Read the article Easy Spark optimization for max record: aggregate instead of join?