Python

Python

Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python’s design philosophy emphasizes code readability with its notable use of significant white-space. Python is dynamically typed and garbage-collected. It supports multiple programming paradigms, including procedural, object-oriented, and functional programming.

Visual Studio Code + Docker + Python using WSL in Windows on Boot Camp
Visual Studio Code + Docker + Python using WSL in Windows on Boot Camp
Automation, Boot Camp, Docker, Python, WSL

Visual Studio Code + Docker + Python using WSL in Windows on Boot Camp

I imagine your first thought is: why? Well, at Wehkamp we do a lot of cross platform development, but sometimes we end up with shell scripts that do stuff with Docker and Python. Usually that’s not a problem for Mac, but for Windows it’s a different thing. I have a MacBook Pro, but I’m a .NET developer, that’s why I prefer Windows, so I run Bootcamp. This article will show how to do Python development in the Windows Subsystem for Linux (WSL) using Visual Studio Code and Docker.

Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming
Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming
Amazon S3, Databricks / Spark, Kafka, PySpark

Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming

Our data strategy specifies that we should store data on S3 for further processing. Raw S3 data is not the best way of dealing with data on Spark, though. In this blog I’ll show how you can use Spark Structured Streaming to write JSON records of a Kafka topic into a Delta table.

Kafka, Spark and schema inference
Kafka, Spark and schema inference
Databricks / Spark, Kafka, PySpark

Kafka, Spark and schema inference

At Wehkamp we use Apache Kafka in our event driven service architecture. It handles high loads of messages really well. We use Apache Spark to run analysis. From time to time, I need to read a Kafka topic into my Databricks notebook. In this article, I’ll show what I use to read from a Kafka topic that has no schema attached to it. We’ll also dive into how we can render the JSON schema in a human-readable format.

Simple Python code to send message to Slack channel (without packages)
Simple Python code to send message to Slack channel (without packages)
Chatops, Databricks / Spark, Python, Slack

Simple Python code to send message to Slack channel (without packages)

Last week I was working on a Databricks script that needed to produce a Slack message as its final outcome. I lifted some code that used a Slack client that was PIP-installed. Unfortunately, I could not use the package on my cluster. Fortunately, the Slack API is so simple, that you don’t really need a package to post a simple message to a channel. In this blog I’ll show you the simplest way of producing awesome messages in Slack.

Adding True/False and list value widgets to your Databricks notebook
Adding True/False and list value widgets to your Databricks notebook
Databricks / Spark, Python

Adding True/False and list value widgets to your Databricks notebook

As an engineer, I love to parametrise my applications. That’s why I love the widget-feature of Databricks notebooks, which allows me to do this with a nice UI. In this blog I’ll explore how to build a True/False widget and a list widget. I also show how to validate the values of required fields.