This week we had to exfil some data out of a bucket with 5M+ of keys. After doing some calculations and testing with a Bash script that used AWS cli, we decided to go a more performant route and use s3p. They claim to be 5-50 times faster than AWS cli 😊.
Read the article Using the S3P API to copy 1.3M of 5M of AWS S3 keys
Let’s face it, using positional arguments like $1 and $2 for your arguments is not very descriptive and not very flexible. There is a better way to supply arguments: with this simple trick you’ll get named arguments in your-script, which is way better 🤓.
Read the article Named Arguments in a Bash Script
At Wehkamp, we use Google Analytics extensively to measure how many users are active. It is hard to correlate active users with your other statistics that “live” in Prometheus. In this blog I’ll show how you can use the Google Real Time API to produce statistics for active users, page views and screen views.
Read the article .NET Core & the Google Analytics Real Time Data API
At Wehkamp we use Redis a lot. It is fast, available and implemented as a managed AWS service called ElastiCache. Sometimes we need to extract data from Redis, and usually I use the redis-cli to interact from the command-line. But what if you need to get the values of 400k+ keys? What would you do? Is there an effective way to query multiple key/values from Redis?
Read the article Reading 400k+ key/values from Redis fast
This week I needed to query an ElastiCache instance on AWS – which is Amazons version of Redis. I could not find a decent free client to query this remote dictionary, so I ended up using redis-cli on Ubuntu. Turns out: Redis is a wonderful and powerful system to work with.
Read the article Connect to AWS ElastiCache (Redis) with redis-cli
I have no idea how I came to this point, but the yellow colors in my terminal (both cmd and PowerShell) are not bright yellow anymore. So I want to reset my colors back to the old values! Turns out that getting them back is not as straightforward as I had hoped…
Read the article Resetting the terminal colors on Windows
Today I had some data coming into our event driven landscape, so I needed to know when my data was processed. As it constituted the processing of 400.000+ records (and I had more things to do), I needed a small script that kept an eye on my JSON API endpoint to see if the values are changed.
Read the article Poll JSON endpoint until value changes with bash & curl
I imagine your first thought is: why? Well, at Wehkamp we do a lot of cross platform development, but sometimes we end up with shell scripts that do stuff with Docker and Python. Usually that’s not a problem for Mac, but for Windows it’s a different thing. I have a MacBook Pro, but I’m a .NET developer, that’s why I prefer Windows, so I run Bootcamp. This article will show how to do Python development in the Windows Subsystem for Linux (WSL) using Visual Studio Code and Docker.
Read the article Visual Studio Code + Docker + Python using WSL in Windows on Boot Camp
My Synology disk crashed and so did my Docker set up. Basically, the CI/CD pipeline for my programs no longer existed. The wonderful thing of an awful crash like this, is that I could rethink my setup. The result is what I would call “a poor man’s CI/CD”. It’s just Git, Docker, Docker Compose and Cron. It is easy to set up and it might be all you need.
Read the article Docker on Synology: from git to running container; the easy way
Last week I was working on a Databricks script that needed to produce a Slack message as its final outcome. I lifted some code that used a Slack client that was PIP-installed. Unfortunately, I could not use the package on my cluster. Fortunately, the Slack API is so simple, that you don’t really need a package to post a simple message to a channel. In this blog I’ll show you the simplest way of producing awesome messages in Slack.
Read the article Simple Python code to send messages to a Slack channel (without packages)