For some blogs I need to capture terminal screens. The recording of these types of screens have different requirement then normal application or website recordings. The bottom of the video is the most important part, the letters need to be crisp and readable for the end user.
Read the article Add black borders to terminal screen recordings
Our data strategy specifies that we should store data on S3 for further processing. Raw S3 data is not the best way of dealing with data on Spark, though. In this blog I’ll show how you can use Spark Structured Streaming to write JSON records of a Kafka topic into a Delta table.
Read the article Streaming a Kafka Topic to a Delta Table on S3 with Spark Structured Streaming
There is a lot of code that needs to make a selection based on a maximum value. One example are Kafka reads: we only want the latest offset for each key, because that’s the latest record. What is the fastest way of doing this?
Read the article Easy Spark optimization for max record: aggregate instead of join?![Thumbnail. Click to read 'Add more color to the Python code of your Databricks notebook [retired]'.](https://keestalkstech.com/wp-content/uploads/2019/11/anna-kolosyuk-D5nh6mCW52c-unsplash.jpg)
Tired of the dull Python syntax highlighting in Databricks? Just copy this code into your Magic CSS editor, change it (to your own style), pin it & enjoy!
Read the article Add more color to the Python code of your Databricks notebook [retired]
At Wehkamp we use Apache Kafka in our event driven service architecture. It handles high loads of messages really well. We use Apache Spark to run analysis. From time to time, I need to read a Kafka topic into my Databricks notebook. In this article, I’ll show what I use to read from a Kafka topic that has no schema attached to it. We’ll also dive into how we can render the JSON schema in a human-readable format.
Read the article Kafka, Spark and schema inference
Last week I was working on a Databricks script that needed to produce a Slack message as its final outcome. I lifted some code that used a Slack client that was PIP-installed. Unfortunately, I could not use the package on my cluster. Fortunately, the Slack API is so simple, that you don’t really need a package to post a simple message to a channel. In this article I’ll show you the simplest way of producing awesome messages in Slack.
Read the article Simple Python code to send messages to a Slack channel (without packages)
When you are training a machine learning image classification model, you often need to resize the images your dataset into smaller ones. When you retrain your model on new data, you resize the images once more. In this blog I’ll share how S3 can be used to cache the resized images.
Read the article Caching resized images on S3 with Databricks
Today we’ll be looking at sorting and reducing an array of a complex data type. I’m using Databricks to do Spark, but I’m sure the code is compatible. I’ll be using Spark SQL to show the steps. I’ve tried to keep the data as simple as possible. The example should apply to scenarios that are more complex.
Read the article Sorting an array of a complex data type in Spark
As an engineer, I love to parametrise my applications. That’s why I love the widget-feature of Databricks notebooks, which allows me to do this with a nice UI. In this blog I’ll explore how to build a True/False widget and a list widget. I also show how to validate the values of required fields.
Read the article Adding True/False and list value widgets to your Databricks notebook
Last week we had some problems with the Google Ads bot. It was not able to crawl a bunch of URLs while the browser had no problem getting through. The only difference was the User-Agent. This send us on a debugging journey through Cloudflare, gateways and micro-sites. To assist us, we’ve created a small bash script to visit an URL and show some debug info.
Read the article Investigate problems due to User-Agent using Bash