Caching resized images on S3 with Databricks
When you are training a machine learning image classification model, you often need to resize the images your dataset into smaller ones. When you retrain your model on new data, you resize the images once more. In this blog I’ll share how S3 can be used to cache the resized images.
Sorting an array of a complex data type in Spark
Today we’ll be looking at sorting and reducing an array of a complex data type. I’m using Databricks to do Spark, but I’m sure the code is compatible. I’ll be using Spark SQL to show the steps. I’ve tried to keep the data as simple as possible. The example should apply to scenarios that are more complex.
Adding True/False and list value widgets to your Databricks notebook
As an engineer, I love to parametrise my applications. That’s why I love the widget-feature of Databricks notebooks, which allows me to do this with a nice UI. In this blog I’ll explore how to build a True/False widget and a list widget. I also show how to validate the values of required fields.
Trigger Lambda for large S3 Bucket with SQS
At Wehkamp we use AWS Lambda to classify images on S3. The Lambda is triggered when a new image is uploaded to the S3 bucket. Currently we have over 6.400.000 images in the bucket. Now we would like to run the Lambda for all images of the bucket. In this blog I’ll show how we did this with a Python 3.6 script.
AWS Lambda Size: PIL+TF+Keras+Numpy?
At Wehkamp we’ve been using machine learning for a while now. We’re training models in Databricks (Spark) and Keras. This produces a Keras file that we use to make the actual predictions. Training is one thing, but getting them to production is quite another!
The main problem we’ve faced was that it was too big to actually fit into a lambda. This blogs shows how we’ve dealt with that problem.
Building a high performing last viewed list using Redis
We live in a day and age in which we can choose a data-store that matches the characteristics of our apps and (micro) services. Lately we’ve been looking into Redis as a high performing store for last viewed items. In this blog I’ll look show how to create a POC with the redis-cli and then implement it using .NET Core. We’ll be using the sorted set structure.
Finetuning screen brightness with PowerShell
When I work in a low-light environment I like to have fine-grained control over the brightness of my monitor. When I change the brightness using the special function keys on my keyboard, it changes in steps of 10%! That’s a lot. PowerShell to the rescue!
Setup Jenkins (Blue Ocean) using Docker on Synology NAS
Using Docker on a Synology NAS has become really easy. It’s a bit harder to install Jenkins in a container and let it use the Docker installation it’s part of. But we are living in exciting times: bringing a fully functioning CI/CD-pipeline to your home automation has never been easier!