Technology

Caching resized images on S3 with Databricks

When you are training a machine learning image classification model, you often need to resize the images your dataset into smaller ones. When you retrain your model on new data, you resize the images once more. In this blog I’ll share how S3 can be used to cache the resized images.

Sorting an array of a complex data type in Spark

Today we’ll be looking at sorting and reducing an array of a complex data type. I’m using Databricks to do Spark, but I’m sure the code is compatible. I’ll be using Spark SQL to show the steps. I’ve tried to keep the data as simple as possible. The example should apply to scenarios that are more complex.

Adding True/False and list value widgets to your Databricks notebook

As an engineer, I love to parametrise my applications. That’s why I love the widget-feature of Databricks notebooks, which allows me to do this with a nice UI. In this blog I’ll explore how to build a True/False widget and a list widget. I also show how to validate the values of required fields.

Trigger Lambda for large S3 Bucket with SQS

At Wehkamp we use AWS Lambda to classify images on S3. The Lambda is triggered when a new image is uploaded to the S3 bucket. Currently we have over 6.400.000 images in the bucket. Now we would like to run the Lambda for all images of the bucket. In this blog I’ll show how we did this with a Python 3.6 script.

AWS Lambda Size: PIL+TF+Keras+Numpy?

At Wehkamp we’ve been using machine learning for a while now. We’re training models in Databricks (Spark) and Keras. This produces a Keras file that we use to make the actual predictions. Training is one thing, but getting them to production is quite another!

The main problem we’ve faced was that it was too big to actually fit into a lambda. This blogs shows how we’ve dealt with that problem.

Installing Ubuntu to Windows Server 2019 (WSL)

These scripts will help to install Ubuntu to a Windows Server 2019 installation. It will download Ubuntu and activate the Linux Subsystem on Windows (WSL).

Building a high performing last viewed list using Redis

We live in a day and age in which we can choose a data-store that matches the characteristics of our apps and (micro) services. Lately we’ve been looking into Redis as a high performing store for last viewed items. In this blog I’ll look show how to create a POC with the redis-cli and then implement it using .NET Core. We’ll be using the sorted set structure.

Finetuning screen brightness with PowerShell

When I work in a low-light environment I like to have fine-grained control over the brightness of my monitor. When I change the brightness using the special function keys on my keyboard, it changes in steps of 10%! That’s a lot. PowerShell to the rescue!

PowerShell snippet: check if software is installed

Some deployment scripts need to check if certain required software is installed on a Windows Machine. You could check if the file is present at a certain location, but there is a better way to check if software is installed: the uninstall database in the Windows Registry. PowerShell makes it really easy to query the registry!

Swap alt and cmd in Windows 10 (MacBook Pro)

As a .Net Developer, I need my MacBook Pro to run Windows. I did so using Boot Camp (which was okay, but my mouse wouldn’t scroll) In Windows I use alt+tab a lot. But the cmd and alt keys are not in the right place (from a Windows keyboard perspective). I use the Windows key a lot too (like Windows+r for run). So let’s swap those keys!