Skip to content

Blog

How to speed up I/O-intensive tasks with multithreading and asyncio

Recently I had to perform a batch processing task where a thousands of images were downloaded from S3, the images were processed and then uploaded to a new bucket in S3. As the processing was relatively lightweight, most of the computation time was spent on downloading and uploading images, that is, I/O. Such I/O bound tasks are a great fit for multithreading (CPU-bound tasks better fit multiprocessing, with all its quirks related to serialization). In this post, I'd like to share a small example how to run tasks in a thread pool.

Are you an ideal team player?

The Five Dysfunctions of a Team by Patrick Lencioni is a classic in teamwork literature. Recently I've been reading the author's follow-up book The Ideal Team Player. I've only started reading the fable, but I couldn't help taking a peek at the end of the book where I found an interesting self-evaluation test for anyone interested in improving their team work skills. I thought it's quite cool so I'll share it here!

Getting started with Apache Beam for distributed data processing

MapReduce was revolutionary when it was first published in 2004. It provided a programming model for batch processing datasets with terabytes of data. MapReduce was built on three seemingly simple phases: map, sort, and reduce. It used the general-purpose HDFS (Hadoop Distributed File System) file system for I/O and was therefore capable of processing almost any kind of data.

Functional programming tips to power up your Python code

Hi! In this post, I'd like to present my favorite functional programming techniques (FP) for Python. I'm a big fan of FP as I've found that by following FP principles I can write code that is more readable and easier to debug. Python is not a functional programming language (and it never will be), but I think there are still many things we can learn from languages such as Haskell that are beneficial also in Python.

My three favorite non-programming books for developers

Back when starting my first software development job I thought my success at work would be directly determined by my programming skills. Of course, that's not the case. One can be the most productive and clever programmer in the world, but that alone isn't sufficient for a satisfactory career in software development. You'll need to get along with people, know which problems not to fix, deal with pressure, etc.