Jamie's Blog

Benchmarking Free-threading Performance with Tachyon

2026-02-21T00:00:00+00:00

Despite optimisations we explored in my previous post, there are still many obstructions to achieving free-threading performance.

One specific issue is that: often benign code abstractions can hurt multi-threading performance. This is something we explored a while ago in How free are threads in Python now?:

The following code end up being a bottleneck, as we had to lock the grid object each time we call __getitem__. This turns out to be many times slower than accessing the underlying tuples directly. And the result is true in even the latest version of Python 3.15.0a6t (what a mouthful!)

class Grid:
    tiles: tuple[tuple[bool, ...], ...]
    ...

    def __getitem__(self, key: Pair) -> bool:
        return self.tiles[key[0]][key[1]]

Python is a multi-discipline language but OOP is certainly encouraged, so it's the abstraction I made here is considered 'Pythonic' by normal standards. This creates a disconnect where the best practices would differ based on whether the code is expected to run in a thread or not.

This suggests that Python is missing some optimisations to facilitate abstraction without loss of multithreading performance.

Worst still, there were no real way to debug performance issues like this, profilers didn't really support free-threading, and the process just involved a lot of guess work.

Enter Tachyon

Thankfully, this is all about to change in 3.15 with the introduction of Tachyon a sampling profiler.

The new profiler sports a host of features:

Low overhead
Remotely attachable
Supports many different output formats
Async aware

But most importantly, native support for multiple threads. I ran my original abstracted version of my code with the profiler:

sudo -E uv run -p 3.15.0a6t python -m profiling.sampling run -a --flamegraph -o profile.html -r 20khz d6.py

Noting the following:

sudo is needed on MacOS
Since the profiler is a sampling profiler, we can specify a sampling rate of 20kHz (20,000 times a second)
We're producing a flamegraph in this case, but there're many other options available.
-a is required to profile all threads, only the main thread is profiled by default.

We can see that the flame graph shows big columns of red, the red colour signals a hot path, where the code spends the most amount of time.

In particular 75% of the time is spend on the line that accesses grid via __getitem__:

    if grid[curr] or curr == obstruction:

Generally some hot paths are expected, but this single line taking up such a large proportion of time is unexpected. This gives us a good indication of where the problem might lie.

Verifying the fix

Now we already know the fix, we bypass the __getitem__ call and access the underlying tuples directly:

    if tiles[r][c] or (r, c) == obstruction:

The results are as follows:

The new line, is now blue (cold path) and only takes up 5.32% of execution time. Indicating the bottleneck has now been eliminated.

Closing Thoughts

There's a lot more tachyon has to offer, and it's not only useful for free-threading, for example, I would love to try using it on an asyncio program.

What benchmarking provides is a path for normal developers to take advantage of free-threading and reason about performance. I think with this, we can expose and address more of these performance traps, and perhaps free-threading can become more natural for pure Python developers.

Free-threading is slowly getting there

2026-02-20T00:00:00+00:00

I posted a lot about free threading in the past year:

Over all I had 3 issues with free-threading:

Reference count contention hurts performance on even immutable objects.
Often benign code abstractions can hurt multi-threading performance.
No good profiling support to identify the issues.

Recently I decided to test free-threading performance again, mainly to test the latest 3.15 alpha, but the results surprised me.

Unlike on the pre-released versions, the stable releases of 3.14 actually see massive improvements. We still observe extra reference count contention as we scale up the workers, but it is a lot lower than 3.13.

The latest alpha of 3.15 sees even more improvements as reference count contention appears to be all but eliminated. Bringing the performance on par with an isolated memory space provided by subinterpreters.

It is clear from the graph that improvements are being made incrementally, but what change caused the massive boost of performance we see from 3.13 to 3.14?

This may have come from deferred reference counting. The idea is to delay updating the ref count on an object, therefore avoiding some contention. Unfortunately that's the extent of my current understanding.

What we can conclude is that, for shared access of builtin data-structures, the performance has improved significantly over the past 2 years. Though there are still some sharp edges which I believe gets in the way of broader adoption, something I will address in the next few posts.

In any case I hope that with the hard work of the CPython devs we can get to a place where free-threading feels natural and approachable.

Datetime Arithmetic in Python

2026-02-17T00:00:00+00:00

Working with datetime is hard and error prone, but Python actually provides the right tools with timedelta to work with them.

It's a shame that most developers are not making the best use of it.

Basic Arithmetic

Let's first catch up what the basic operations are:

Advanced Arithmetic

Now, let's look at the more advanced features you should be using:

Closing Thoughts

I think it's evident that many develops are unaware of these techniques, I checked on ChatGPT and other LLMs and I get a variety of different methods. Almost none of which takes advantage of the techniques here.

This also mirrors my own experience and I really hope more developers take advantage of this.

Caching an Asyncio Function the Easy Way

2026-02-16T00:00:00+00:00

If you've used asyncio for some time you've probably noticed a few things that work differently to the synchronous counter parts. I recently had to add some caching to an asyncio function. Let's say something like:

async def get_user(user_id: UUID) -> User:
    ...

Of course naturally we want to use functools.cache, but past experience have taught me that it doesn't work out of the box.

Generally the decorators won't work in both sync and async contexts. This is due to the fact that an async function returns immediately when invoked, and is only actually executed when we await whatever was returned.

But what actually happens here? What's being cached? How does it work?

Invoking function() returns a coroutine object which was cached. The object track the state of execution of the async code, await will advance the state, it can therefore not be awaited more than once.

There are other objects that are awaitable, unlike coroutines, asyncio.Task tracks the result of the coroutine irrespective of the coroutine's state.

As shown, this can be awaited as many times as you want, which is exactly what we need here. We can create a simple wrapper around a coroutine function:

def convert_to_task[**P, R](fn: Callable[P, Coroutine[Any, Any, R]]) -> Callable[P, asyncio.Task[R]]:
    """Wrap a coroutine function to make it a task which is cacheable."""

    @wraps(fn)
    def _fn(*args: P.args, **kwargs: P.kwargs) -> asyncio.Task[R]:
        return asyncio.create_task(fn(*args, **kwargs))

    return _fn

We can now compose convert_to_task with functools.cache to make an async function cacheable:

Closing thoughts

Whilst there are off-the-shelf solutions (e.g. aiocache) for this problem, I find that they are missing the simplicity I crave from the functools version. This is probably a signal that we're missing this utility in the standard library.

My solution composes asyncio primitives in a simple way, it's easy enough that you can plausibly reimplement this each time you need it. In lieu of an official version, I believe this is currently the most 'standard' way to achieve caching.

Finally, though you don't need to understand the difference between a coroutine object and a Task or even a Future object, it helps when you're looking to implement advanced asyncio functionality. I was able to drastically simplify the problem by understanding how a Task works.

Counting down Chinese New Year with Marimo

2026-02-12T00:00:00+00:00

I've been on vacation in China leading up the lunar new year and I'd rather not spend all my time researching some heavy topic. However, I thought I might instead spend a few minutes combining my previous posts on marimo notebooks and the Lunar calendar.

Replacing tox with UV

2026-02-01T00:00:00+00:00

Recently I've been updating some of my libraries to Python 3.10+ after Python 3.9 has finally reached end of life.

The upgrade to Python 3.10 is a relatively simple one, but I thought it be a good idea to run tests on different versions of Python.

Existing Tooling

tox is a common tool for this use case, it allows you to define environments declaratively.

requires = ["tox>=4"]
env_list = ["3.14", "3.13", "3.12", "3.11", "3.10"]

[env_run_base]
description = "run unit tests"
deps = [
    "pytest>=8",
    "pytest-sugar"
]
commands = [["pytest", { replace = "posargs", default = ["tests"], extend = true }]]

Similarly nox let's you do the same but in a more imperative way.

uv to rule them all

The thing is, tox and nox requires learning and using a new set of tools, whilst that's completely fine uv has spoiled us with the convenience of using a single tool.

There's already a lot that can be replaced by uv, for example:

python -m venv
uv venv

pip install ...
uv pip install ...

pyenv install ...
uv python install ...

I read an article from Simon Willison where he already worked out how to test a package with different Python versions with the -p flag.

uv run -p 3.10 pytest

This is pretty much job done here! But we can go a bit further ...

Extras

You can specify any extras and dependency groups via --extra and --group respectively:

uv run -p 3.10 --extra cli pytest

uv run -p 3.10 --group test pytest

Overriding Package Versions

A common use of tox is to test over different package versions, for example, if a package needs to support both pydantic==1.*.* and pydantic==2.*.*. We can also do this in uv with the --with or -w options:

uv run -w 'pydantic==1.*.*' pytest
uv run -w 'pydantic==2.*.*' pytest

This also works for a set of deps in a requirements.txt file:

uv run --with-requirements requirements-production.txt pytest

Simon's article also mentions using --with-editable to test with the local version of the code, which can come in handy:

uv run --with 'pydantic==1.*.*' --with-editable 'mylocal-package' pytest

Using an isolated venv

Simon also mentions using an isolated venv:

uv run --with 'pydantic==1.*.*' --isolated pytest

to avoid your test venv contaminating your development venv or vice versa.

I found this most useful when I needed to run tests in parallel, something like:

parallel "uv run -p {} --isolated pytest" ::: 3.{10..14}

This is a quick proof of concept but you can probably write a more robust bash or python script.

Conclusion

tox still remains useful especially with something like tox-uv. But if you're using uv anyway it may be useful to learn all the options mentioned above.

I've already had a lot of success using uv to quickly debug issues caused by version incompatibilities in pandas, something that just takes a bit more time to setup with tools like tox.

Webasgi - Run python web apps in a browser

2026-01-26T00:00:00+00:00

I recently published a project webasgi a static site allows you run FastAPI and other asgi apps (such as litestar) right in the browser.

As you can imagine, Python web apps are not designed to run statically in the browser, and this project has been particularly challenging to me, in fact it's taken me over a year and a very productive AI coding session to get this done.

Vibe coding

Let's get the AI coding stuff out of the way!

I used Google's new antigravity editor with Claude Opus to do most of the coding. This is mostly a concession for me as I don't have the time or expertise in Javascript to do a lot of the work. I do feel that in some sense I robbed myself the opportunity to learn through the process of coding, however the design of the app is the most interesting and what we'll focus one in this post.

The last thing I'll say is that, this is not the first time I've attempted to use AI tools to help me code up this project, but it's the first time successful. This is probably owing to the newer models and the fact that Pyodide is more popular now and likely more present in the model's knowledge.

In any case, the AI discussion will have to carry on another day ...

Pyodide

I've already used pyodide before. I've used it in several past posts:

My Chinese birthdays, time keeping is hard!: pyscript shell using pyodide.
My SQLAlchemy Cookbook: jupyter-lite notebook powered by pyodide
Trying out marimo notebooks: marimo notebook also powered by pyodide.

Pyodide is an amazing project that brings a Python to the browser using WASM. The entire Python runtime is contained within the browser with no additional server required, therefore it also has the benefit of being able to run user generated Python code safely.

As such, it's already being used to create sandboxes. A prominent example is, pydantic.run which is also the inspiration of my project.

ASGI (Asynchronous Server Gateway Interface)

ASGI provides an abstraction for all Python web applications. Effectively splitting the web server into two components:

The application itself which in its simplest form is an async function that takes a request and returns a response. E.g. fastapi.
The web server that handles the connections and scaling. This comes in the form of libraries with horse sounding names such as uvicorn, hypercorn and granian.

This is quite a clean abstraction, allowing both sides to evolve independently, and giving the user a good amount of flexibility. One of my favourite consequences of ASGI is that we can quite easily compose ASGI applications, as seen here with a FastAPI app that mounts a Litestar app under a subpath.

The server's implementation can also be very flexible, uvicorn and hypercorn are both Python implementation where granian is written in rust.

Naturally a Pyodide implementation is also possible, but instead of connecting to a port in the operating system, we can mock the 'networking layer' in browser javascript.

Browser Emulation

Having worked out the basics of the Pyodide "server" the next step is to display the contents returned by the server. This is necessary as it's part of a standard FastAPI workflow:

write some FastAPI code.
run the code with a server like Uvicorn.
check the server by calling the endpoint in your browser. (often using swagger docs)

Without a clean way to visualise the output, it's always going to feel a little awkward.

Emulating a browser means taking html which may contain linked javascript and css loading everything and rendering it. This all needs to take place without leaking into the parent browser session, that is, we don't want the emulated browser to crash out current tab.

Luckily this is exactly what iframe does and we can pass the html as srcDoc.

The hard part ...

We've figured out how to render the responses however we're only just getting started.

The rendered html page can contain elements that make further requests to the server. These can be relative anchor links:

<a href="/other-endpoint">go to next</a>

Or some non-get requests via javascript or form submission. Without changing anything, these requests will not be able to go to the ASGI server, breaking the "immersion".

If we want everything to work as a browser would, we'd need to load some javascript first that patches all of these requests. This is done by injecting some scripts in the html that register javascript event handlers to intercept these events. The iframe has the ability to them securely send a message to the parent via postMessage

Everything put together

This is roughly what the full architecture looks like, the rest is mind numbing javascript implementation you can check out here.

Why?

So why did we do all this?

Being able to run more things in the browser dramatically lowers the barrier to entry. A functional app can be created iterated and shared in the browser.

My immediate next steps are to improve the user experience using AI or otherwise but also to try and tidy up the AI generated parts of the code.

But my long term goal for this is to:

teach and discuss more advanced patterns in these web app frameworks
encourage adoptions of new frameworks that solve some of my headaches with frameworks like fastapi.

Writing a SQS task framework from scratch

2026-01-18T00:00:00+00:00

Recently I've been working on framework to run LLM tasks using AWS's excellent SQS. And I made the decision to write my own task framework/library as opposed to using a pre-exiting framework. I thought this would be a great opportunity to discuss the considerations and levels of abstractions involved when coding a framework.

Why SQS?

Having come from using traditional message queues like RabbitMQ, I can't help but compare SQS with RabbitMQ. The thing is, whilst they can be interchangeable. I think they are actually built with different use cases in mind.

RabbitMQ is for processing a large volume of messages, the messages are strictly processed in FIFO order. SQS has more flexibility around when a message is processed and how long each task takes. So RabbitMQ is good for handling events, but SQS is better for tasks like LLM calls.

Additionally, SQS uses http to communicate between the server and the client which is easier to monitor and is easier to setup if you already have an AWS cluster.

Why not use ...?

Okay so, the first question I will answer you is why not a pre-existing framework?

When we talk about task framework in Python, there is a general consensus. The most popular choice is Celery, a framework that can handle pretty much any handle workload in any broker (be it SQS or otherwise). If you need a UI and compose tasks in a DAG (Directed Acyclic Graph) then you should use Airflow. If you like new technology and talking about orchestration then use something like temporal. The list really goes on and on.

I haven't used a lot of managed task solutions like temporal or prefect which admittedly is something I'd love to try out.

As for why I don't use something like celery. Celery is a framework that tries to work with everything including all brokers. That is only true to an extent as it was originally designed to work with RabbitMQ. Generally the support is also pretty good for Redis but not so good otherwise. A lot of broker specific features are either not supported or not documented.

Another issue is a lack of asyncio support. For my use case asyncio is great, no need for multiple threads or processes. Celery again tries very hard to be general, it supports the concurrency paradigm most likely to support a user's code without a lot of input from the user. But in my case, I already have an opinion of how I want my code to run, and it just doesn't make sense to use it.

Finally, my general complaint of celery is that it requires a lot of configuration for a production use case. Which can cause quite a bit of a headache.

Creating my abstractions

In my experience writing a good framework is all about how good the abstractions are. Abstractions can come with a cost. Whilst Celery create abstractions that allow task compositions and provides support for many brokers, but it trades broker specific features and increases in configuration complexity.

Therefore a good abstraction is about making useful trade-offs.

Writing the code

Before we start with abstractions, it's a good idea to first write some code to see what the it currently looks like with no abstraction.

Here I ask a LLM to provide an example:

import asyncio
from aiobotocore.session import get_session

# Replace with your actual Queue URL
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue"

async def consume_messages():
    session = get_session()

    # create_client is an async context manager that handles connection cleanup
    async with session.create_client('sqs', region_name='us-east-1') as client:
        print(f"Listening for messages on {QUEUE_URL}...")

        while True:
            try:
                # receive_message calls the SQS API
                # WaitTimeSeconds=20 enables Long Polling (waits for messages to arrive)
                response = await client.receive_message(
                    QueueUrl=QUEUE_URL,
                    MaxNumberOfMessages=10,
                    WaitTimeSeconds=20,
                    VisibilityTimeout=60,
                )

                # Check if 'Messages' key exists in response
                if 'Messages' in response:
                    for msg in response['Messages']:
                        # 1. Process the message
                        print(f"Received body: {msg['Body']}")

                        # 2. Delete the message so it isn't processed again
                        await client.delete_message(
                            QueueUrl=QUEUE_URL,
                            ReceiptHandle=msg['ReceiptHandle']
                        )
                        print("Message deleted.")
                else:
                    print("No messages received in this poll.")

            except Exception as e:
                print(f"Error encountered: {e}")
                await asyncio.sleep(5)  # Backoff on error

Identifying Trade-offs

Next we look at things we want to change and thus the trade-offs.

Let's start with deleting the message, the purpose of deleting the message is to signal that the message is complete and shouldn't be processed again. I personally don't like the naming of this, in the context of message queues this is usually called ack.

Something like:

client.ack(message)

Next, let's look at some of the attributes we can abstract:

VisibilityTimeout: is the amount of time the message can be processed for before going back on the queue, it is also how we can delay the message for when adding a message to the queue. We can make this clearer by calling it keep-alive and delay respectively.
MaxNumberOfMessages: is a useful feature of SQS to fetch a batch of messages at once to increase throughput and reduce the cloud bill. But it may not be a good fit for my use case handling LLM requests as they don't work well in batches
WaitTimeSeconds enables long polling which reduces the number of requests and hence the cloud bill. This is a good idea to keep on. And there's no real need for us to turn it off.

Creating the client

Now that the trade offs are identified, we will start to create the client code, and this is what I've come up with:

@dataclass
class QueueClient:
    client: SQSClient
    queue: str
    poll_interval: int = 20

    async def consume(self, keep_alive: int = 30) -> AsyncIterator[MessageTypeDef]:
        while True:
            messages = await self.client.receive_message(
                QueueUrl=self.queue,
                MaxNumberOfMessages=1,
                WaitTimeSeconds=self.poll_interval,
                VisibilityTimeout=keep_alive,
            )
            match messages:
                case {"Messages": [message]}:
                    yield message
                case _:
                    continue

    async def ack(self, message: MessageTypeDef) -> None:
        await self.client.delete_message(
            QueueUrl=self.queue,
            ReceiptHandle=message["ReceiptHandle"],
        )

In order to pass the messages to the user, I've represented the stream of messages as an AsyncIterator. Iterators are truly one of Python's best features, and here it fits particularly well as are trying to expose the messages to the user without making may assumptions about how to user wants to consume it.

Consuming messages then is as simple as running an async for loop:

async def main():
    async for message in client.consume():
        await handle(message)
        await client.ack(message)

Error Handling

A good task framework should provide utilities to handle errors well, in our scenario we simply terminate if we encounter an error. This could be enough provided that we automatically restart our process, but generally this is not the expectation.

The standard terminology for this is nack the opposite of ack. In SQS terms this is simply change_message_visibility or delete_message depending whether we want to retry.

So inside QueueClient goes:

@dataclass
class QueueClient:
    ...

    async def nack(self, message: MessageTypeDef, retry_timeout: int | None = None) -> None:
        if retry_timeout is None:
            await self.ack(message)
            return

        await self.client.change_message_visibility(
            QueueUrl=self.queue,
            ReceiptHandle=message["ReceiptHandle"],
            VisibilityTimeout=retry_timeout,
        )

Consuming messages with error handling is:

async def main():
    async for message in client.consume():
        try:
            await handle(message)
        except Exception:
            await client.nack(message)
            logger.error(...)
        else:
            await client.ack(message)

This may look a little ugly right now, but we want to avoid abstracting things prematurely, so I've left this for later.

Extending keep alive

One thing I wanted to do over celery is to manage the timeout of the message. Celery is configured with a static value which doesn't work well for a something like LLM queries.

Extending the keep-alive is essentially the same operation as retrying: change_message_visibility in one case we set the timeout so that we can retry after a period of time, in the other case we extend the timeout so that we won't retry too soon.

In the QueueClient goes:

@dataclass
class QueueClient:
    ...

    async def keep_alive(self, duration: int) -> None:
        await self.client.change_message_visibility(
            QueueUrl=self.queue,
            ReceiptHandle=message["ReceiptHandle"],
            VisibilityTimeout=duration,
        )

Abstracting consumption

As it is right now, our QueueClient can be tested and used. We know that it will function quite well because we've been careful to make abstraction decisions that improve usability but minimise the cost.

I hinted early that we could do a bit more abstraction. As we have made our QueueClient simple, the consumer loop is more complex than it needs to be, requiring the user to understand how to manage the lifecycle of a message.

We've left this out because there are many different ways the user may want to manage the message's lifecycle, so enforcing one way may restrict the user's options.

However, we can provide some abstraction with helpful implementation without narrowing the user's options.

To do this we can use a abstract base class or a Protocol to define what we want the LifeCycle to look like:

class LifeCycle(Protocol):
    """Defines the lifecycle of a single message.

    That is what happens 
        - when a message is processed successfully
        - when a message errors
        - during the message processing
    """
    def __call__(self, message: MessageTypeDef) -> AsyncContextManager[MessageTypeDef]:
        ...

We use a AsyncContextManager here because it gives us the most flexibility, wrapping the message handler.

So our basic version of message processing would look something like:

@dataclass
class BasicLifeCycle:
    client: QueueClient

    @asynccontextmanager
    async def __call__(self, message: MessageTypeDef) -> AsyncIterator[MessageTypeDef]:
        try:
            yield message
        except Exception:
            await self.client.nack(message)
            logger.error(...)
        else:
            await self.client.ack(message)

We can improve on this with a retry option:

@dataclass
class RetryLifeCycle:
    client: QueueClient
    retry_interval: int

    @asynccontextmanager
    async def __call__(self, message: MessageTypeDef) -> AsyncIterator[MessageTypeDef]:
        try:
            yield message
        except Exception:
            await self.client.nack(message, self.retry_interval)
            logger.error(...)
        else:
            await self.client.ack(message)

And adding more complexity for convenience, we can also provide a background task to keep the message alive. This allows us to process the message for a functionally infinite amount of time:

@dataclass
class HeartbeatLifeCycle:
    client: QueueClient
    interval: int

    @asynccontextmanager
    async def auto_keep_alive(self):
        async def _keep_alive():
            while True:
                await asyncio.sleep(self.interval * 0.8)
                await self.client.keep_alive(self.interval)

        async with asyncio.TaskGroup() as tg:
            task = tg.create_task(_keep_alive)
            try:
                yield
            finally:
                task.cancel()

    @asynccontextmanager
    async def __call__(self, message: MessageTypeDef) -> AsyncIterator[MessageTypeDef]:
        try:
            async with self.auto_keep_alive():
                yield message
        except Exception:
            await self.client.nack(message)
            logger.error(...)
        else:
            await self.client.ack(message)

Finally putting everything together

Let's compose what a single consumer would look like:

async def main():
    client = QueueClient(...)
    life_cycle = HeartbeatLifeCycle(client, interval=20)
    async for message in client.consume():
        async with life_cycle(message) as message:
            handle(message)  # That's it!

Pretty easy right? And should the user want a different lifecycle or to not use it, they are free to do so without any restrictions enforced by the framework.

The philosophy here is to steer your user to the right direction but ultimately to trust them. A principal I find most frameworks violating.

Scaling it

You might wonder if we can scale the task consumption. After all we must be using asyncio for a reason. We needn't worry about coming up with abstractions ourselves as asyncio already comes with TaskGroup.

async def run(client: QueueClient) -> None:
    life_cycle = HeartbeatLifeCycle(client, interval=20)
    async for message in client.consume():
        async with life_cycle(message) as message:
            handle(message)


async def main(workers: int):
    client = QueueClient(...)
    async with asyncio.TaskGroup() as tg:
        for _ in range(workers):
            tg.create_task(run(client))

When there's a good standard library or well known abstraction, this lowers the cost on the user, as we are not creating anything new.

Closing Words

Creating the perfect abstraction is no small task, a good place to start is something humble and go from there. But even then it doesn't always work out in your first try. Sometimes we need a lot of iterations to get there.

I see some crazy abstractions that grew ad-hoc over a lengthy amount of time, gaining more and more features but not taking care of the abstraction. This is a common issue with even very popular frameworks, it's okay to recognise an abstraction is not working and start over. Because only then can we work towards something better.

If you're interested in using this library/framework, it's published to pypi as simple-async-sqs.

Asyncio is neither fast nor slow

2026-01-12T00:00:00+00:00

Don't listen to random benchmarks..

I recently came across an article benchmarking Python performances in web frameworks, comparing asyncio and sync performance.

The author sets out to measure performance of FastAPI/Django web servers running with postgresql comparing async and non-async workloads. The methodology is pretty reasonable, the following is his result for an endpoint with a single postgres database read:

Server type	workers	RPS	Latency avg	Latency max	Median
Sync Django	1	456	140ms	262ms	153ms
Sync Django	2	669	96ms	262ms	132ms
Sync Django Pooled	1	569	112ms	171ms	117ms
Sync Django Pooled	2	1822	35ms	98ms	50ms
Async Django	1	205	312ms	467ms	331ms
Async Django	2	541	118ms	304ms	196ms
FastAPI	1	236	271ms	372ms	287ms
FastAPI	2	409	156ms	433ms	224ms

As a result the author concludes:

These benchmarks show just how much optimization for sync web services Django and the Python has. Sync Django Pooled outperforms or matches all other configurations. Even FastAPI only performs better when it’s the sole bottleneck.

When I read these result I had some doubts, I noticed that:

1.⁠ ⁠it shows that the best sync django scenario with 2 workers, has more than 4x throughput that the equivalent fastapi async solution. 2.⁠ ⁠2 workers of sync django pooled is 3x the performance of a single worker.

The former doesn't line with the my personal experience. Asyncio does add overhead in some cases, but the scenario laid out here should have postgres as the bottleneck. Overhead caused by asyncio's event loop shouldn't be significant.

The latter also seemed a little odd, as the throughput increase should be proportional to the number of workers.

Reviewing the code

Since the results don't match my own expectation then either my mental model is wrong or it's the benchmark. So let's review the benchmark code.

The code for the fastapi endpoint is as follows:

@app.get("/quote")
async def quote(db: AsyncSession = Depends(get_db)):
    statement = (
        select(Quote).order_by(func.random()).options(selectinload(Quote.author))
    )
    result = await db.execute(statement)
    quote = result.scalar()

    return {
        "quote": quote.quote_text,
        "author": quote.author.name,
    }

Here a single random quote is fetched from the table. SQLAlchemy is used as an orm to facilitate query building. Everything seems in order.

But wait, there's a subtle problem here! By default SQLAlchemy does not batch results, instead it will fetch all results into memory first when ⁠ execute ⁠ is called. So whilst ⁠ .scalar() ⁠only returns the first result, in the background all results are fetched.

Recreating the benchmark

So I recreated the benchmark with my code fix for fastapi.

statement = (
    select(Quote).order_by(func.random()).limit(1).options(selectinload(Quote.author))
)

I tried where possible to keep the same code and parameters to the original.

•⁠ ⁠Hardware: Macbook Pro M3 11 cores 18 GB of RAM •⁠ ⁠Connection Pool Configuration: min 5 - max 15 connections •⁠ ⁠Data: 100 authors with 1000 quotes •⁠ ⁠Benchmark: ⁠ rewrk -d 30s -c 64 --host http://localhost:8000/quote/ ⁠

Results

And the results are as follows:

Server type	workers	RPS	Latency avg	Latency max	Median
Sync Django	1	341.81	186.61ms	418.90ms	223.96ms
Sync Django	2	338.74	188.07ms	467.42ms	227.23ms
Sync Django Pooled	1	440.46	144.92ms	281.12ms	150.94ms
Sync Django Pooled	2	419.01	152.30ms	374.59ms	167.25ms
Async Django	1	323.32	197.28ms	497.76ms	249.85ms
Async Django	2	322.78	197.45ms	453.63ms	249.18ms
FastAPI	1	687.26	92.95ms	326.49ms	110.29ms
FastAPI	2	673.85	94.78ms	332.11ms	121.00ms

As expected the FastAPI implementation does a lot better here compared to before. Performing better than the best django version.

What's weird is that the worker count makes very little difference in the performance, often slowing down rather than speeding up. My guess is that my setup has more rows in the database, therefore we reach a saturation point a lot quicker.

The dilemma of benchmarks

Still you might be a bit disappointed with this benchmark as I was, the original had a much clearer trend between async/sync worker count and pooled database connections. However this is precisely what happens with these "real world" benchmarks. Here we're trying to represent a real use case, not necessarily to prove a point.

On the other hand, there exist many micro-benchmarks out there, where variables are controlled and a smaller more predictable scenario is measured.

There are places for both types of benchmarks, real world benchmarks can be more interesting but the data is messy and requires more critical analysis.

Critical analysis

My results are convenient for my beliefs and one might conclude from it that asyncio is indeed much faster.

But looking at the numbers I wondered why Django is so much slower, where does the slow down actually come from. I expected asyncio to do quite well but I assumed we can make up the performance with more threads or workers, but that does not seem to be the case.

This lead me into a deep dive, trying almost every combination of configuration, implementation and framework.

First I wanted more control over the variables, Django is a very different framework compared to FastAPI and a there are too many possibilities when it comes to the performance discrepancy. Flask is a lighter weight framework compared to Django as such it is a much better sync vs async comparison.

Secondly, the original benchmark makes a point of the benefits of pooled connections. However, async django can also benefit from pooled connections. Sqlalchemy also uses a connection pool by default. So I see no reason to not use connection pools across the board.

The new results

This time around I decided to run different thread and worker configurations through the gauntlet and only keep the best performing configurations.

This was generally 3 workers and 5 threads on my machine, giving me a total of 15 threads. However other configurations with similar total thread counts also performed very similarly.

Server type	workers	threads	RPS	Latency avg	Latency max	Median
FastAPI (Pooled)	1	-	687.26	92.95ms	326.49ms	110.29ms
Flask (Pooled)	3	5	682.73	93.66ms	239.67ms	104.34ms
Django (Pooled)	3	5	411.91	155.17ms	299.19ms	169.44ms
Django Async (Pooled)	1	-	406.90	152.79ms	279.93ms	160.32ms

NOTE: asgi for the async frameworks don't use threads

And this time, the results are more even with the fastapi implementation pulling a little ahead over flask, though over many different runs the results are practically identical.

This is more an expected result, as once again the main bottleneck is the database and not the web framework.

These results show that there is no significant difference between the django implementations. The original discrepancies are easily explained by the the connection pooling.

The only thing that doesn't make sense to me is why django is slower than the flask implementation all else being equal.

Digging even deeper

My immediate thought was skill issues, I don't have a lot of experiences with django and it's possible I made a mistake that lead to performance degradation. Checking the code and the orm generated SQL query, nothing really stood out. Then I confirmed that the database driver we're using is indeed psycopg 3 so no differences there.

Finally, I decided to give a closer look at the connection pool used in django, and that's where it all clicked. django uses psycopg_pool an implementation provided directly by psycopg. Where as sqlalchemy has its own implementation. The differences are actually significant,

sqlalchemy has a lazy pool implementation, which means that connections are created or destroyed only when the pool is being accessed.
⁠On the other hand, psycopg-pool actively maintains the pool and processes tasks using background thread workers.

So one more time, I modified my fastapi and flask implementation to use the same connection pool as Django:

@app.get("/quote/")
async def quote() -> PlainTextResponse:
    async with pool.connection() as conn:
        async with conn.cursor() as cur:

            await cur.execute(
                "SELECT q.id, q.quote_text, a.name FROM quotes_quote q "
                "JOIN quotes_author a ON q.author_id = a.id "
                "ORDER BY RANDOM() LIMIT 1"
            )
            row = await cur.fetchone()
            return PlainTextResponse(f"{row[1]}\n\n--{row[2]}")

And I got the following results:

Server type	workers	threads	RPS	Latency avg	Latency max	Median
FastAPI	1	-	687.26	92.95ms	326.49ms	110.29ms
FastAPI (psycopg_pool)	1	-	526.21	121.49ms	248.73ms	128.85ms
Flask	3	5	682.73	93.66ms	239.67ms	104.34ms
Flask (psycopg_pool)	3	5	532.82	119.95ms	289.91ms	172.75ms
Django	3	5	411.91	155.17ms	299.19ms	169.44ms
Django Async	1	-	406.90	152.79ms	279.93ms	160.32ms

So it's quite clear here that psycopg_pool is the root cause in this particularly scenario.

There is still some differences between django and the other frameworks, but I think that's likely because we're using raw sql queries over django's orm.

Conclusion

If you were hoping for an answer of "just use async" or "don't use async", life is not so simple. The lesson I hope you take is that there are nuances when it comes to this topic.

Time and time again there's been posts on async performances, and we often see misleading benchmarks or analysis.

Let's sum up my own thoughts:

Asyncio's advantage has never been speed in particular, but the cost. We can achieve the same performance with no added threads or processes.
Even then you might consider using django or other sync frameworks if you're more familiar with them or if they provide something you don't get elsewhere, e.g. django's rich middleware plugin system.
Benchmarks cannot be taken at face value, try to reproduce it for your requirements (LLMs are pretty good at this).
Lastly when you get a result from a benchmark that doesn't match expectation, it's prudent that you investigate where the discrepancy comes from.

Future Steps

This whole exercise, as exhausting as it is, barely scratches the itch. There's quite a few unexplored questions:

What is the advantage of psycopg_pool's implementation, are there cases where it performs better?
Are there any configuration or ticks I'm missing to speed up Django?
There's not a lot of analysis on async django as it uses threads to run database queries and middlewares, but I think it's worth looking at which situations where async django wins.
Is there a general rule for thread/process numbers that maximise the performances and what are the trade offs of having more threads.

In the near future I intend to put together a more comprehensive comparison of different concurrency models and explore these questions.

The best I could do right now is to provide the full source code and hopefully encourage people to make their own measurements and share their results. In case there's been any mistakes the feedback is very much welcome there.

I was wrong about Subinterpreters

2026-01-01T00:00:00+00:00

... but that's actually a good thing

I've already written a ton about subinterpreters since a year ago:

Recently I had some time to dig a bit deeper into subinterpreters and check my understanding of them.

Efficient data sharing between interpreters

My first attempt at using it was to solve an Advent of Code. This involved running the same function around 4000 times on the same grid of values.

The function makes a lot of read only access to the grid, which causes high lock contention in a free-threaded implementation. But with interpreters the performance scaled pretty much perfectly. Here's a chart of the performance differences:

In the article I claimed:

There are also other ways to execute code I chose not to use like InterpreterPoolExecutor which don't take advantage of shared objects, or Interpreter.call which places limitations on the function called.

But something didn't sit right with me, why would there not be an easy to use implementation for users?

So I gave it another look and reimplemented my solution with the standard executor here:

candidates = ((start, node, grid) for node in path if node != start)
with InterpreterPoolExecutor() as executor:
    sum(1 for res in executor.map(solve, candidates) if res)

... and the results indeed came out a lot slower. Around 4 seconds compared to the 0.6 seconds before (13 threads on my M3 Macbook).

So that seems to prove my point then? Well not exactly, I tried rewriting my own implementation again like:

    candidates = ((start, node, grid) for node in path if node != start)
    executor = Executor("implementations.d6", "solve", workers=workers)
    sum(1 for res in executor.map(solve) if res)

it was just as slow. So what's going on here?

It turns out that whether using the standard library or hand writing my own executor, we're both using the same 'sharing' mechanism to move the arguments and results in and out of the interpreter.

But I had assumed that when we shared the grid of type tuple[tuple[bool, ...], ...] we simply share the reference of the object. But this is not actually the case, the docs don't go into a lot of detail in actual fact, most objects are copied between interpreters.

This doesn't explain why my initial implementation was so fast. After rechecking my original code, I found out that I had gotten quite lucky and used interpreters.prepare_main to pass the grid to each interpreter at the start. This still perform the copy but only once for each worker, on the other hand, my newer implementations copy grid for each invocation.

To show this more clearly, we can use memoryview instead. memoryview is one of the few exceptions that do not require copying as it shares mutable data. The way it works is that the memoryview object itself is copied each time but the underlying data is simply referenced.

Changing the grid to a memoryview backed by a bytearray we get back our performance we expect. See the code below:

Efficient function passing

In Subinterpreters and Asyncio, I tried to make subinterpreters work more generally with asyncio and created aiointerpreters.

Here again I made some incorrect claims:

In order to pass the function itself, the arguments and return values are pickled before they are passed to the interpreters. Which will impact performance and may cancel out a lot of the performance gains over other parallelism mechanisms like free-threading and multi-processing.

Which as we know now, the only way to avoid copying the data is using memoryview or to reduce the amount of copying using something interpreters.prepare_main.

Another thing I got wrong is:

Then there's the problem of loading functions into the interpreters. As far as I can tell, the best way to do so without pickle is to import the functions inside the interpreters. This is likely the mechanism behind call_in_thread.

At the time, I had not realised that pickle works on functions the following way:

Note that functions (built-in and user-defined) are pickled by fully qualified name, not by value. This means that only the function name is pickled, along with the name of the containing module and classes. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

So it was ultimately unnecessary to create a custom function loading mechanism to load the function into the interpreter as pickle was already doing the same thing (oops!).

This means that there's no discernable performance benefits in my Runner implementation over InterpreterPoolExecutor.

How should we use interpreters then?

In short just use InterpreterPoolExecutor.

with InterpreterPoolExecutor() as executor:
    executor.map(function, args)

    # or 
    executor.submit(function, single_arg)
    executor.submit(function, single_arg)
    ...

Results can be accessed easily via futures. The standard lib also provides some basic way to synchronise futures via wait or as_completed

Asyncio

What if you need more complex synchronisation? We can run Executors in asyncio using loop.run_in_executor(...).

with InterpreterThreadPoolExecutor() as executor:
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(executor, function, argument)

here's an example using the excellent asyncio.TaskGroup:

async def to_coro[T](aw: Awaitable[T]) -> T:
    return await aw

with InterpreterPoolExecutor() as executor:
    loop = asyncio.get_event_loop()
    async with asyncio.TaskGroup() as tg:
        for _ in range(runs):
            # NOTE: a task must be created from a coroutine, run_in_executor returns an `asyncio.Future`
            tg.create_task(
                to_coro(
                    loop.run_in_executor(executor, function, argument)
                )
            )

What about `aiointerpreters`?

I'm hoping by this point, I've successfully convinced you that aiointerpreters is redundant and the standard library functionality is enough.

I can now safely say that I'm wrong about the out of the box usability of InterpreterPoolExecutor. That being said I don't feel too silly making those mistakes.

With subinterpreters still a very new and niche feature, there are still a lot of sharp edges. One thing that led to my confusions is that there are not many resources on how it works, there are some difference between the information from the PEP and the final implementation. So I do hope the details I share here can save people some time.

As for aiointerpreters, going forward the focus will be more on enhancing the features the standard library offers rather than completely reinventing the wheel (again). This process is already underway ...

InterpreterThreadPoolExecutor

As I've been going through the code in the standard library I've discovered that the current executor implementation is written on top of ThreadPoolExecutor. Once a task is given to a thread it then get's sent to an interpreter within the thread.

This gives us an opportunity to create a hybrid executor that can dispatch to an interpreter conditionally. I created InterpreterThreadExecutor:

from aiointerpreters.executors import InterpreterThreadPoolExecutor, interpreter


with InterpreterThreadPoolExecutor() as executor:
    executor.map(interpreter(cpu_bound), (argument for _ in range(runs)))
    executor.map(io_bound, (argument for _ in range(runs)))

When the user want to request a dedicated interpreter they simply need to wrap the function in the interpreter decorator.

I think this makes a lot of sense as threads and interpreters both have their trade offs. Interpreters are isolated and allow parallel memory access but passing arguments needs a lot more care and could be slower. On the other hand, threads (GIL enabled) cannot run cpu bound code in parallel.

We could create 2 thread pools but I think it's a lot easier and resource efficient to maintain a single thread pool.

This is particularly useful for asyncio.to_thread():

We can set the default executor at a loop level:

asyncio.get_running_loop().set_default_executor(InterpreterThreadPoolExecutor())

And then whenever we call to_thread the correct executor will be chosen based on what the user specified.

async with asyncio.TaskGroup() as tg:
    tg.create_task(asyncio.to_thread(interpreter(cpu_bound), argument))
    tg.create_task(asyncio.to_thread(io_bound, argument))

More complex Executors and Runner

Whilst the default aiointerpreters.runner.Runner is a little redundant now, it doesn't mean there are no need for making custom versions of Executor or Runner. There is no one size fits all solution to concurrency and parallelism, so there's plenty of opportunity to create different versions of Executor that behave differently and solve different problems that the user face.

I'm currently working on a Runner that distributes async tasks to coroutines running in separate interpreters similar to aiomultiprocess.

There's also opportunity to take advantage of prepare_main in an executor to reduce the number of copying required.

Finally, running an executor and integrating it with asyncio is not always straightforward, and I'm sure there's some interesting abstractions that can be created on this front.

Closing words

Apologies if this all reads like a massive dump of information, I've realised how tricky the subject of concurrency is and I found that it's important here to provide the most accurate details.

True parallelism in Python is nascent and there simply isn't an obvious way to do things right now and maybe there never will be. With that comes some difficulty in knowing what do do. The most crucial thing here is to get your hands dirty and try different things for yourself and slowly a clearer understanding will form.

p.s. If you're more interested in free-threading, please look at Python 3.14: State of free threading and How free are threads in Python now? my opinions on it have largely remained unchanged.

Trying out marimo notebooks

2025-12-03T00:00:00+00:00

As we approach the holiday seasons, we get to "enjoy" another 12 days of Advent of Code. Every year I try to do something a little different, last year I had a lot of fun solving it in Ocaml, prior years I've tried the latest Python features. This year I've decided to use marimo notebooks.

Marimo is a jupyter competitor, providing an interactive environment to develop code and visualise data. As the new kid in the block, it has a very sleek look and good support for a lot of different visualisation frameworks. There is already a very good article on realpython about it, so I won't go into too much detail on its features.

Instead I'll talk about the feature that makes marimo truly standout: builtin reactivity! When a cell in marimo creates a variable, marimo tracks it. When the variable changes, any cell that references it will update automatically. For example, try changing my_value in the second cell below:

If you're not familiar with embedded notebooks, you can have a look at my past blog post which uses jupyter-lite. Marimo has even better support for wasm notebooks.

Honestly, when you first try it everything feels like magic. But there are some quirks the more you get into it. In order for marimo to effectively track the variables, it places restrictions on where variables can be used. And cyclical dependencies are possible here.

The good thing is marimo will let you know immediately why, so you won't need to look up why something is not working. Overall I don't find it too difficult to use but it'll be good to hear what others think.

To me marimo is just much more fun than jupyter, I can essentially write little reactive web UI in pure Python. I think that's important, I'm not someone who has enjoyed writing UIs, and I think it's because I just don't find javascript fun. Of course, we're not going to be able to create particularly complex and performant UIs but I think fun matters a lot too.

In any case, here's a neat little app demonstrating the solution for today's advent of code, I've included some input UI elements like the slider. Changing the inputs will automatically cause the output to regenerate:

So what do you think? If it looks cool to you, you can try it out at marimo.app!

Fixing lazy imports: Generating Static Types Dynamically

2025-10-29T00:00:00+00:00

So I've just released a package called lazy-helper. This comes as lazy loading has been a hot topic once again due to the proposal PEP-810.

Using my package we can define lazy loaded dependencies nicely:

Coincidentally Brian Okken has recently written a blog post covering the topic of lazy loading where he goes over how it works. This is the same mechanism my package uses, essentially defining a module level __getattr__ with a cache. So I won't go into too much detail there.

Obscured static information

What I want to focus on discussing is one of the downsides of this approach. Since we're using strings to represent our imports, what is imported and where it points to is now obscured. This means when you use an import like the following:

from . import lazy


lazy.typing

Your editor or tools like mypy simply does not know where that leads to. Even worse, it doesn't even know if it's correct at all. If we have a typo like lazy.tpying we won't find out until runtime.

One way to provide this information is to manually specify this in a block like this:

if typing.TYPE_CHECKING:
    ...
    from textual.widget import Widget
    from textual.widgets._button import Button
    from textual.widgets._checkbox import Checkbox

This is an example taken directly from textual which Brian discussed in his blog. Now your tooling understands the imports but at the cost of having to write this out manually. And we're still susceptible to typos and mistakes. So is there a better way to solve this?

Providing static information dynamically

This may sound a bit contradictory but we can leverage runtime information to solve our lack of static information.

Let's look at how Lazy is actually implemented:

from dataclasses import dataclass, field


type Alias = str
type ModuleName = str
type AttributeName = str
type LazyImport = tuple[ModuleName, AttributeName | None]


@dataclass
class Lazy:
    imports: dict[Alias, LazyImport] = field(default_factory=dict)

    def __call__(...): 
        """Register the import"""
        ...

    def __getattr__(self, name):
        """Run the import associated with the name"""
        ...

We can see that at its core, we are wrapping a dictionary which defines all the imports we are interested in. All the information is there when we load all the lazy imports. We just need a way to expose it to the static tooling.

Enter `pyi` files

.pyi files are what's known as stub files. They contain only typing information and used by static type checkers to infer the types without all the logic.

.pyi files takes precedence over the .py files they represent. So they can be used to type something that cannot be typed directly. It is quite often used for extension modules but also complex projects with a lot of legacy like boto3-stubs.

Similar to the if typing.TYPE_CHECKING: approach we can create a stub file with the correct imports:

# lazy.pyi
import typing
from typing import TypedDict
import typing as tp
from typing import TypedDict as TD

The difference is that we can now have the static typing information completely separate from the actual implementation. Making it a lot easier to programmatically generate these stub files.

We can generate the pyi files for the imports with a method like the following:

    def stubgen(self):
        for alias, (module, attribute) in self.imports.items():
            full_path = f"{module}.{attribute}" if attribute else module
            from_, _, import_ = full_path.rpartition(".")
            import_ = f"{import_} as {alias}" if import_ != alias else import_

            if from_:
                print(f"from {from_} import {import_}")
            else:
                print(f"import {import_}")

        print()

We'll then create a file with the printed code:

Since the stub file is generated from the runtime information. Your editor should pick up any issues typos or mistake in declaring the imports. It also saves a lot of effort as you don't have to write the stub manually.

For this reason, I've packaged a cli to simplify the process:

lazy-stubgen src/lazy.py

Stub generation is useful generally

Python is a very dynamic language, the language places very few limitations on what you can do. On the other hand, the static typing ecosystem is relatively new with a lot of limitations.

When the type system is inadequate for a problem, there are 3 things people usually try:

Try to change the code and not use an advance feature.
Try very hard to represent the typing as best as possible. This is sometimes known as type gymnastics.
Or simply give up on trying to type the code.

I want to be clear here, all of these are valid. Stub generators are yet another way to solve this problem, It has the potential to provide typing with very little manual effort. What's important is to keep understand all the techniques available to you and to apply then when they are needed.

Here are some other examples in the wider ecosystem you should check out:

MonkeyType a very tool that stores typing information as the code executes, to then generate static types afterwards.
protoc protocol buffers are compiled into Python code with a mess of descriptors that is not type friendly, the compiler has a --pyi_out option to create the type stubs to solve this issue.

Python 3.14: 3 Asyncio Changes

2025-10-09T00:00:00+01:00

Python 3.14 was officially released on October 7th. There are a lot of new features and I've covered some of them before in:

What I haven't covered here are any of the asyncio changes. There just happened to also be 3 changes to Asyncio this release ...

Asyncio Debugger

Python's builtin debugger pdb may seem basic but one of its most powerful features is to run python code at a breakpoint:

def my_func(data: dict) -> None:
    breakpoint()  # or import pdb; pdb.set_trace()
    ...

Which enters the debugger at the stack frame where breakpoint is called and allows us to interact with objects in scope.

>>> my_func({})
> /Users/jamie.chang/projects/Jamie-Blog/<stdin>-0(2)my_func()
-> import sys
(Pdb) data
{}
(Pdb)

However when it comes to asyncio commands it gets tricky:

class MyClient:
    async def now(self):
        return time.time()


async def main():
    client = MyClient()
    breakpoint()
    ...

(Pdb) client.now()
<coroutine object MyClient.now at 0x102ad16c0>

We cannot await at all in pdb:

(Pdb) await client.now()
*** SyntaxError: 'await' outside function

Nor can we run it with asyncio.run:

(Pdb) asyncio.run(client.now())
*** RuntimeError: asyncio.run() cannot be called from a running event loop

Now this is possible with pdb.set_trace_async

async def main():
    client = MyClient()
    await pdb.set_trace_async()
    ...

Note that set_trace_async() is awaited here. As I understand it, this is a necessary compromise to get this feature working.

And ... Success!

(Pdb) print(await client.now())
1759933736.955492

This might be a niche feature, but it's crucial for debugging things like async ORMs.

If you just need to interact with async resources in a REPL, you can already start an async REPL since Python 3.8:

$ python -m asyncio
asyncio REPL 3.9.6 (default, Nov 11 2024, 03:15:38) 
[Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(1)

Asyncio Introspection

Whether I'm teaching asyncio to newcomers or debugging complex concurrent programs myself. I've always found it difficult to visualise what's happening.

When teaching asyncio we often just time the difference in performance, explaining it properly is hard.

It is now possible with introspection. We can see the current call graph at runtime by:

python -m asyncio pstree 12345

where 12345 is the process id.

Note on macOS it needs to be run using sudo.

The graph looks like:

└── (T) Task-1
    └──  main /Users/jamie.chang/projects/concurreny-bench/backpressure/semaphore3.py:25
        └──  Semaphore.acquire /Users/jamie.chang/.local/share/uv/python/cpython-3.14.0rc3-macos-aarch64-none/lib/python3.14/asyncio/locks.py:407
            ├── (T) Task-30376
            │   └──  process_url /Users/jamie.chang/projects/concurreny-bench/backpressure/semaphore3.py:15
            │       └──  sleep /Users/jamie.chang/.local/share/uv/python/cpython-3.14.0rc3-macos-aarch64-none/lib/python3.14/asyncio/tasks.py:702
            ├── (T) Task-30389
            │   └──  process_url /Users/jamie.chang/projects/concurreny-bench/backpressure/semaphore3.py:15
            │       └──  sleep /Users/jamie.chang/.local/share/uv/python/cpython-3.14.0rc3-macos-aarch64-none/lib/python3.14/asyncio/tasks.py:702
            ├── (T) Task-30393
            │   └──  process_url /Users/jamie.chang/projects/concurreny-bench/backpressure/semaphore3.py:15
            │       └──  sleep /Users/jamie.chang/.local/share/uv/python/cpython-3.14.0rc3-macos-aarch64-none/lib/python3.14/asyncio/tasks.py:702
            ├── (T) Task-30408
            │   └──  process_url /Users/jamie.chang/projects/concurreny-bench/backpressure/semaphore3.py:15
            │       └──  sleep /Users/jamie.chang/.local/share/uv/python/cpython-3.14.0rc3-macos-aarch64-none/lib/python3.14/asyncio/tasks.py:702

And make it a lot clearer what's going on inside the event loop.

Since it can be triggered on a currently running process, we can even use this to debug a live running application that may be stuck on some async resources.

I've been using this with watch to see a live view of the graph.

You can also get the information in table form by:

python -m asyncio ps 12345

Finally, there are new python apis for this to get the information at a specific point in the code.

Asyncio in multiple free threads

Not to be confused with Free Threaded Python With Asyncio. Where we run specific blocking code in threads using asyncio.to_thread.

Prior to 3.14 event loops were not thread safe in the free-threaded builds of Python. This means you can't run asyncio in a separate thread. In Python 3.14 Kumar Aditya updated the event loop implementation maing it thread safe whilst also improving performance. You can see his blog post here.

But if we are already doing things concurrently why would threads matter? Well this actually depends on how much blocking work your coroutines are doing.

Let's say we have N coroutines all fetching data from a web page somewhere. This is perfect for asyncio, as http requests can be made without blocking the event loop.

async with TaskGroup() as tg:
    for url in urls:
        tg.create_task(scrape(url))

But what do we do with the scraped data? We need to parse this, so this will block the event loop as it's CPU bound.

async def process(url):
    data = await scrape(url)
    parse(data)  # CPU bound
    ...

...

async with TaskGroup() as tg:
    for url in urls:
        tg.create_task(process(url))

You will ind that the CPU bound tasks will impact the performance of the event loop. This is actually a situation that I've run into a couple times already. This can be solved with asyncio.to_thread here but sharing a lot of data right now may have a performance impact.

Instead we can now spawn multiple free-threads with event loops. Each thread is still impacted by the CPU bound blocking tasks, however as we are processing the CPU-bound work concurrently, the impact is lowered.

We will need a way to pass data between the threads, the best way I found is using queue.Queue:

async with asyncio.TaskGroup() as tg:
    while True:
        try:
            url = queue.get_nowait()
        except Empty:
            break
        task = tg.create_task(process(url))
        task.add_done_callback(lambda _: queue.task_done())

For benchmark performances it's better to check Kumar's blog post. He posts a very impressive performance improvement, which I also see in my own examples. He also shows a much higher raw TCP performance so there's potential to increase IO bound workloads as well.

On the other hand, I found free-threading memory sharing awkward right now in Python and that impacts performance. My preferred method is using my very own aiointerpreters to launch the tasks to interpreter threads with subinterpreter's powerful shared memory functionality.

As with a lot of concurrency problems, there is not a single way to solve these. This is fine, and you should try different approaches, run benchmarks see how they compare to pick the best method for your problem.

Finally trying out Mojo 🔥

2025-10-02T00:00:00+01:00

Mojo has been on my radar for a while. In fact, I heard about it on launch back in May 2023. It was touted as a super set of Python with 10000s of times the performance. It's main focus is on AI related workloads with strong built in support of GPU programming.

I've used Advent of code 2024 day 6 as a benchmark for in Python before in Python 3.14: State of free threading. So I figured why not solve the same puzzle one more time.

Implementation

Implementation can be found here:

I won't go into all the details instead I'll choose to highlight parts of the code that are interesting.

Python like syntax

Python's syntax can be divisive, I happened to really like it (I'm biased). Mojo's syntax definitely carries over the spirit of Python's.

@fieldwise_init
struct Grid[T: Movable & ImplicitlyCopyable](Copyable, Movable):
    var data: List[List[T]]

    fn __contains__(self, pos: Pair) -> Bool:
        return 0 <= pos.first < len(self.data) and 0 <= pos.second < len(
            self.data[pos.first]
        )

    fn __getitem__(self, pos: Pair) -> T:
        return self.data[pos.first][pos.second]

    fn __setitem__(mut self, pos: Pair, value: T):
        self.data[pos.first][pos.second] = value

The above code is very similar to a dataclass I would create in Python:

@dataclass
class Grid[T]:
    data: list[list[T]]

    def __contains__(self, pos: Pair) -> bool:
        return 0 <= pos.first < len(self.data) and 0 <= pos.second < len(
            self.data[pos.first]
        )

    def __getitem__(self, pos: Pair) -> T:
        return self.data[pos.first][pos.second]

    def __setitem__(self, pos: Pair, value: T) -> None:
        self.data[pos.first][pos.second] = value

So far so good, @fieldwise_init on a struct in mojo is pretty much equivalent to @dataclass in Python. The function syntax is pretty much the same. We need to declare traits similar to rust but otherwise this all makes sense.

Ergonomics

Whilst the syntax has the look and feel of Python. It's a little sad that a lot of python's conveniences haven't arrived in mojo yet.

The biggest being the lack of generators. As a Python programmer I'm yielding whenever I can. It's a quick and easy way to define iteration behaviour. Mojo let's you define iterators as structs but it's pretty tough, see how I defined an iterator for how a guard should move:

struct Guard(Copyable, Iterable, Iterator, Movable):
    alias Element = Pair
    alias IteratorType[
        iterable_mut: Bool, //, iterable_origin: Origin[iterable_mut]
    ]: Iterator = Self

    var grid: ArcPointer[Grid[Bool]]
    var position: Pair
    var direction: Pair
    var obstacle: Optional[Pair]

    fn __init__(
        out self,
        position: Pair,
        read grid: ArcPointer[Grid[Bool]],
        obstacle: Optional[Pair] = None,
    ):
        self.position = position
        self.grid = grid
        self.direction = Pair(-1, 0)
        self.obstacle = obstacle

    fn __has_next__(self) -> Bool:
        return self.position in self.grid[]

    fn __iter__(ref self) -> Self.IteratorType[__origin_of(self)]:
        return self.copy()

    fn __next__(mut self) -> Pair:
        var old = self.position
        candidate = self.position + self.direction
        while (
            candidate in self.grid[]
            and self.grid[][candidate]
            or (self.obstacle and candidate == self.obstacle.value())
        ):
            self.direction = self.direction.rotate()
            candidate = self.position + self.direction

        self.position = candidate
        return old

There is a lot of boiler plate needed for this, though it's not totally unreasonable. Using yield in Python would of course be a lot easier. But if we were forced to make a class based iterator then it'll look similar.

Another feature I missed is pattern matching, but it's hopefully coming soon.

Traits

Traits are used to define some behaviour an object should have in mojo. Traits were made popular in rust. But is basically like an interface in Java but can have default method implementations. In Python this is closest to an abstract base class.

struct Guard(Copyable, Iterable, Iterator, Movable):
    ...

Here Copyable, Iterable, Iterator, Movable are all built in traits, for example Iterable trait is needed so we can use Guard in a for loop:

for positions in guard:
    ...

And traits can be more powerful than Python types because they can be composed using &. Where as Python currently doesn't have type intersections.

As someone who does a lot of type annotations in Python, I like the power traits can give you here. But there are some sharp corners:

Some things are still not representable: Tuple in mojo is not Hashable this is because whether it's hashable or not depends on the child types, and there is no way in mojo to represent that.
There are a lot of traits to remember, it can get old writing Copyable and Movable over and over again. That's not to mention we have Implicit versions of some traits too.

Memory Ownership

Mojo is a statically typed compiled language, it has an memory ownership model similar to rust and zig.

Essentially a value is associated with the scope it's defined in, you can transfer this ownership by using ^ either when you're passing arguments to a function and therefore new scope, or when you are returning from a scope.

fn parse(path: Path) raises -> Tuple[Grid[Bool], Pair]:
    matrix: List[List[Bool]] = []  # list defined here owned by the current scope
    ...
    return Grid(matrix^), start.value()  # passing the ownership to Grid struct

This takes some getting used to, but it does often boil down adding ^ when the compiler yells at you.

Lifetimes

The ownership model necessitates the parametrisation of lifetimes, when a value is passed to a struct the compiler needs to know how long the variable is alive for and adjust the struct's lifetime accordingly.

This is done in mojo via origin

An example when defining an iterator:

struct ResultSetIter[mut: Bool, //, origin: Origin[mut]](
    Copyable, Iterator, Movable
):
    alias Element = Bool

    var result_set: Pointer[ResultSet, origin]

Here we parameterise the origin to match the origin for the pointer.

In my little example, the lifetimes have been the most challenging. I'm aware that lifetimes are also one of the nastiest parts of rust as well. In my case, having very few examples of lifetimes being used with structs available to me really made it more difficult.

In one case, I gave up and just used an ARC (Atomic Reference Counting) Pointer. Which allows me to manage the value without lifetime, at the cost of unnecessary memory indirection.

Concurrency

Mojo doesn't currently have a lot of utilities for concurrency, it doesn't let you create threads directly.

I've found parallelize which is similar to a thread pool.

    var inputs = [elem for elem in positions]
    var result_set = ResultSet(len(inputs))

    @parameter
    fn worker(row: Int):
        var guard = Guard(start, grid, inputs[row])
        loop = has_loop(guard)
        result_set[row] = loop

    parallelize[worker](len(inputs))

To collect the results I've implemented a wrapper around some memory. This allows each item to be stored in a separate piece of memory. Avoiding the need for thread safe data structures that mojo currently doesn't have.

struct ResultSet(Copyable, Movable):
    var data: UnsafePointer[Bool]
    var size: Int

    fn __init__(out self, size: Int):
        self.data = UnsafePointer[Bool].alloc(size)
        self.size = size

    fn __copyinit__(out self, existing: Self):
        self.data = UnsafePointer[Bool].alloc(existing.size)
        self.size = existing.size
        memcpy(dest=self.data, src=existing.data, count=existing.size)

    fn __del__(deinit self):
        self.data.free()

    fn __setitem__(self, index: Int, value: Bool):
        (self.data + index)[] = value

    fn __getitem__(self, index: Int) -> Bool:
        return (self.data + index)[]

This has been a lot more effort due to the lack of general support for threading. Ideally there's a way to collect the result in shorter batches, and an API that's similar to Python's ThreadPoolExecutor. It looks like there might be async support coming soon, so there is something to look forward to!

Performance

I've measured the following performance:

Single core: 1.037 s Multi core: 0.273 s

This is around 3x the performance of my best python implementations. It'll be interesting to see how this would compare with other system programming languages like rust.

One of mojo's big features is its interoperability with Python. So there's a lot of opportunity to speed up your slow python code with mojo and this is before we even look at SIMD and GPUs.

My feeling around Mojo

Some people have voiced concerns over the fact that mojo is backed by VCs and is mostly closed sourced.

Personally, I don't have any problems with this, we can't be too idealistic here. Software like uv might not exist without companies or investors funding it. And there is commitment to open source it eventually.

I think what mojo is trying to achieve is certainly interesting. Whilst it started out as a "Python Superset" it's not turned into a mix of Zig, Rust + Python syntax + inbuilt GPU/SIMD support.

If you think that's a lot, you'd be right! You may have noticed as we walk through mojo's features, I find myself comparing to rust most of the time and not Python. People who don't know an existing systems programming language may find it too hard. People with experience in these languages may decide to stay in those languages.

But on the flip side, if they can achieve this then we may have a really powerful language.

Lastly I want to comment on the documentation around mojo. Though the tutorial is fairly good, if you veer just off the beaten track, then you get very very basic docs with no examples. For this I had to reference the source code for the standard library, which is luckily the part of mojo that is open source.

Next steps

There are 2 aspects of mojo I've yet to try:

GPU programming: I just don't have the hardware to take advantage of it
Python interop: As I mentioned earlier, this could be a big win for mojo. Certainly something I will put some time to in the future.

Blog 1 year in

2025-09-24T00:00:00+01:00

It's been just over a year since I started my blog. In fact, this is my 27th blog post. I thought it'd be a good time for me to share some insights on blogging.

Motivation

Recently I've been asked by some people on why I do this. I think for people here there are a lot of reasons for writing blog posts. For example, it may increase your public profile and help you land a job.

Though I believe the blog has helped me in my job hunting process, I don't think it's what motivates me. For me there are two aspects that gives me a big dopamine hit. One is to come up with ideas and concepts that are somewhat unique and interesting. And the second is knowing that people can learn a little from the content I produce.

My personal favourites include:

I love that both of these articles had aspects that weren't well explored in Python before. For the TypedDict one I had an idea to use it for dependency injection. And the pattern matching article explores the structure of iterators in Python and uses it in conjunction with pattern matching.

Honestly the process of experimenting and researching these topics were rewarding on their own. However normally I wouldn't spend this much effort when I don't have an immediate use of it professionally.

Whilst I'm sure there are individuals that write for the sake of writing, I don't I can keep up writing without knowing people are reading it. Knowing that an article is being read by people gives me a lot of satisfaction and keeps me going.

I've tracked the traffic through google search console and cloudflare web analytics.

The numbers are not particularly impressive but I can see it's growing over time. I'm also able to use this information to put more time into topics people are genuinely interested in.

For example, I noticed a few months ago that searches for sub-interpreters were quietly increasing. In all likelihood, this is a combination of its inclusion in Python 3.14, and people generally being unfamiliar with it. This inspired me to research and write a blog post about subinterpreters and asyncio.

Where I share my posts

I started off first sharing my posts on LinkedIn. LinkedIn provided basic metrics already and I probably had the most connections there.

Blusky on the other hand has not been very successful for me, being a relatively new site it has been hard to build up engagement from scratch. However I love using it to follow other people in the Python space.

Finally pycoder's weekly has been simply fantastic. It's a very good source of dedicated Python article and news and my posts fit well with the newsletter. Posts featured there include:

Python 3.14: 3 smaller features
Subinterpreters and Asyncio
TypedDicts Are Better Than You Think
Free Threaded Python With Asyncio which has a benefit of always being referenced by Simon Willison in his blog post

It's then no surprise that these are also my top performing posts over the past year.

What else should I be doing?

I know some of my posts have been linked on hackernews and have had some engagement there. It sounds a little silly but I've consciously avoided posting myself because I know I'd get hung up over the comments.

That being said, platforms such as reddit and hackernews are just great platforms to have engage in discussions. Since one of my goals is to explore new or obscure features in Python, it is a great place to host these discussions.

Another suggestion I received is to add comments. There are a few good free options available:

https://utteranc.es/
https://giscus.app/

They all use github issues or discussion to power the comment section.

Recently I've had some fruitful discussions in github issues from readers, so I'll look to add this in the near future.

Finally

I've personally found the process of writing blogs immensely rewarding, I hope to keep this up for as long as possible. Communication is not everybody's strong suit, and it can be daunting. But I think sharing ideas is really important to keep the ecosystem of Python and engineering fresh and exciting, and I hope more people start writing.

Asyncio backpressure - follow up

2025-09-14T00:00:00+01:00

Previously when discussing asyncio backpressure I've made some claims that were not necessarily complete.

I said:

It works well for 100s of urls but when we hit a big number like 10000s we have a problem.

The program seemingly hangs. This is because all the tasks are being created first and only then to do allow the tasks to start executing. The program will also use much more memory than it needs and generally might slow down due to more context switching. Definitely not what we want

There are 2 issues here:

First about the program hanging, it's actually quite hard to observe that behaviour.
Second about the numbers, 10000s of tasks is actually not a big number for asyncio.

This was pointed out in a Github issue by Misha Behersky. Additionally I had not made it very clear why I proposed using semaphore like:

async def process_all(urls: Iterable[str]) -> None:
    semaphore = asyncio.Semaphore(200)
    async with asyncio.TaskGroup() as tg:
        for url in urls:
            await semaphore.acquire()
            tg.create_task(process_one(url)).add_done_callback(lambda _: semaphore.release())

As opposed to using it in a more standard way:

def process_one(url, semaphore):
    async with semaphore:
        ...

In order to properly explain the behaviour, I needed to construct a much better example so that I can benchmark both memory and speed.

Simulation

First and foremost, I've been testing back pressure by making concurrent calls to fetch wikipedia articles.

async def request(client: httpx.AsyncClient, url: str) -> None:
    await client.get(url)
    ...

This is a good real world test, but when we are experimenting with thousands of concurrent calls, it adds load to unsuspecting servers.

I could set up my own servers, but we have a simpler choice.

One of the benefits of asyncio is that it provides primitives for concurrent operations. We can in that case easily simulate the network latency using asyncio.sleep:

async def request(url: str) -> None:
    await asyncio.sleep(random.uniform(0.1, 0.2))  # simulate the time delay for getting requests

Here we assume the latency is between 100 and 200 ms. But we can also extend this to match a distribution observe in the real world. Similarly we can use this to simulate load with databases and message queues.

Measuring the speed and memory

Testing the speed or duration is simple, the best way is using time.perf_counter. We can compose this into a context manager like this:

@contextmanager
def timer(message: str) -> Iterator[None]:
    start = time.perf_counter()
    try:
        yield
    finally:
        print(message, time.perf_counter() - start, "s")

Similarly we can track the peak memory used by using tracemalloc:

@contextmanager
def memory_profiler(message: str) -> Iterator[None]:
    tracemalloc.start()

    try:
        yield
        _, peak_memory = tracemalloc.get_traced_memory()
        print(f"Peak memory usage: {peak_memory:,} bytes")
    finally:
        tracemalloc.stop()

Testing different implementations

In the original article, I proposed both batching and semaphore as methods of back pressure. So our test scenarios are as follows

batched execution.
acquiring semaphore inside the task as Python intended.
acquiring semaphore before task creation and releasing with callback as I proposed.

Finally a bonus implementation suggested in the issue thread to release the semaphore at the end of the function call:

async def main():
    semaphore = Semaphore(...)

    async with TaskGroup() as tg:
        for url in urls:
            await semaphore.acquire()
            tg.create_task(process(url, semaphore))


async def process(url, semaphore):
    try:
        ...
    finally:
        semaphore.release()

The code can be found here. We run 100,000 inputs for each implementation limiting concurrency to 500 at a time.

Results

File	Description	Memory usage in bytes	Duration in seconds
batching.py	Using batched to process things in batches as opposed to using semaphores	821,692	41.7
semaphore1.py	Traditional pythonic way of using semaphore, does not limit task creation	152,154,848	30.9
semaphore2.py	Semaphore limiting task creation with callback	1,018,796	30.2
semaphore3.py	Semaphore limiting task creation with release called in the task function	839,176	30.2

Analysis

Traditional use of semaphore does not limit the number of tasks being created, we must first create all 100,000 tasks and then start processing the tasks. As a result, the amount of memory requires is many times higher. Though the difference in speed is fairly small, it is consistently observable as we are not able to start processing during the creation phase.

Another interesting result is that the semaphore2 with the callback uses more memory than just passing the semaphore to release in semaphore3. This is a quirk of constructing an extra lambda as a callback. Additionally, it may be more obvious than using the callback, so this method is definitely worth considering.

Finally I want to talk about batching. Batching is very memory efficient as it doesn't need any extra objects however it does take a lot longer. This is explained in my original post:

The problem is if we have a small amount of tasks with long wait times then it'll slow down the whole batch.

Notice that our distributions has a lot of variance:

random.uniform(0.1, 0.2)

if we reduce this to random.uniform(0.1, 0.11) we have closer results

File	Duration in seconds
batching.py	24.65
semaphore1.py	22.0
semaphore2.py	21.1
semaphore3.py	21.1

My view is batching and the 2 semaphore solutions are all very good solutions. Batching is simple, but you can get more performance out of using semaphores just not in the obvious way.

On the other hand using semaphore normally is good but you should be aware of the memory implication.

Finally

I've spent a lot of time investigating asyncio back pressure here. The point is to provide the full context of the different options to use.

I also hope that I've provided some ideas around how you might investigate different design patterns yourself.

Simplify lambda deployments with UV

2025-09-01T00:00:00+01:00

The python packaging landscape and developer experience has shifted dramatically in the past year or so with uv's launch marked a pivotal moment. But behind the scenes many PEPs have worked to get us to this point.

One such PEP was PEP 723 – Inline script metadata which we discussed in why you should write your tools in Python Again. This PEP combined with support from uv allow us to write single file scripts whilst also handling dependencies in the same script.

I've been thinking about other places this might be useful outside of command line apps. Having recently used a lot of lambdas, I believe lambdas are the perfect place to use inline metadata and UV.

Building a single file lambda with UV

Lambdas most often run a small amount of simple code meaning the code is usually a single file. This fits particularly well with inline-metadata as we can keep everything compact and simple.

So let's give it a go, using the commands in uv's docs.

`uv init`

Start by creating a file to write our code, we can define which python version we want here:

uv init --script main.py --python 3.13

This creates a file:

# /// script
# requires-python = ">=3.13"
# dependencies = []
# ///


def main() -> None:
    print("Hello from main.py!")


if __name__ == "__main__":
    main()

We can now add our code to it:

# /// script
# requires-python = ">=3.13"
# dependencies = []
# ///

from pydantic import BaseModel


class Body(BaseModel):
    message: str


def lambda_handler(event, context):
    return {"statusCode": 200, "body": Body(message="Hello, World!").model_dump_json()}

`uv add`

Then we add any dependencies we need:

uv add --script main.py pydantic

This adds the dependency to the metadata of the file:

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "pydantic",
# ]
# ///
...

Optional: `uv lock`

uv lock is a relatively new feature for scripts, for those who want reproducible builds using lock files.

uv lock --script main.py

Produces a lock file at main.py.lock. When the lock file is present uv will then respect the lockfile's dependency.

Develop code and `uv run` locally

After you develop your code, you can run the script without any virtual environment setup using:

uv run --script main.py

Packaging the app

So far we've used relatively well known workflows within uv. Now comes the tricky part, we need to download the dependencies and package it alongside the code.

The simplest way I found is to first export the deps as a requirements file.

uv export --script main.py --output-file requirements.txt

And then install using uv pip install

uv pip install \
    -r requirements.txt \
    --target package/ \
    --python-platform x86_64-manylinux2014 \
    --python-version 3.13 \
    --link-mode copy \
    --only-binary=:all: \
    --upgrade

Now this is a complex command! Let's break it down:

--target to install the package to a directory instead of a virtualenv
--python-platform is needed as we need to install packages for the lambdas specific. The options are essentially just arm or x86. -- --link-code copy to produce a hard copy of the package if it comes from the uv cache -- --only-binary to prevent uv from building a wheel from sdist, the resulting wheel may not work on the target platform.

We need this complexity because pydantic contains platform specific native code. This is also a problem for pip.

Creating the zip file and uploading it

Creating the zip file now is simple:

cd package
zip -r ../deployment.zip .
cd ..
zip -g deployment.zip main.py

Finally we can use aws cli to upload the zip file:

aws lambda update-function-code \
  --function-name <YOUR_FUNCTION_NAME> \
  --zip-file fileb://deployment.zip \

We now have a workflow that uses a single file for both code and dependencies, which is convenient and then leverages uv for its dependency management and faster workflow.

Making things easier

But we can do a bit better, in our current workflow we still need to know which platform we need to install the dependencies for. To avoid this, we can simply just look up the platform and python versoin from AWS using boto3:

def get_lambda_info(client: LambdaClient, name: str) -> dict:
    response = client.get_function_configuration(FunctionName=name)
    match response:
        case {"Runtime": runtime, "Architectures": ["x86_64"]}:
            return {
                "python-version": runtime.remove_prefix("python"),
                "architecture": "x86_64-manylinux2014",
            }
        case {"Runtime": runtime, "Architectures": ["arm64"]}:
            return {
                "python-version": runtime.remove_prefix("python"),
                "architecture": "aarch64-manylinux2014",
            }

        case other:
            raise Exception(f"Unexpected response: {other}")

Putting things together

Since get_lambda_info is a python function, and if I'm honest I am terrible at bash scripting. I've added everything in a Python script and published it as simple-lambda.

I won't include the full source here, you can check it out on github.

In my script I ended up using the following builtin libraries:

pathlib to handle file and directory paths.
tempfile to create a temporary directory that gets deleted afterwards
zipfile to bundle everything in a .zip file at the end

Using builtin python modules doesn't only save you from installing extra dependencies but also ensures that you code is platform-agnostic. This is a very desirable feature of a cli.

How to use the script

It's simple, just write your single file lambda function as before. and run:

uvx simple-lambda deploy <your_function_name> main.py

We're using uvx here which will dynamically install simple-lambda and run the script of the same name (by default). It also caches the package to avoid reinstalling it in the future.

If you want to explicitly keep a version of simple-lambda on your machine then you can install it as a tool:

uv tool install simple-lambda
simple-lambda deploy <your_function_name> main.py

You do have the option to install with pip of course but uv is really just more convenient.

Finally

My hope is that either you find the script itself useful, or you've learned a bit more about tooling in Python.

I wrote this script to solve a real problem for myself, but I'm constantly amazed at the progress being made to improve Python packaging and tooling.

p.s. If you want to see another great use case of inline metadata, check out pydantic.run.

Dynamic config part 1: Pydantic and file watchers

2025-08-26T00:00:00+01:00

Feature flags or dynamic configuration is something that I find very useful, however I've never had the chance to use them. This is for a lack of options launchdarkly, flagsmith and unleash to name a few.

SaaS options can be amazing with a full array of features, but I would love a simple solution to start with that can be set up in your own cluster.

I don't have a full solution in my head, instead I'm going to try and explore the space and work towards a solution over the next few posts.

pydantic-settings

pydantic-settings would probably be the go to solution in Python for config management. Example configuration will look something like this:

class Settings(BaseSettings):
    setting1: str
    setting2: str

    model_config = {"toml_file": Path("config.toml")}

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        *args: object,
        **kwargs: object,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (TomlConfigSettingsSource(settings_cls),)


settings = Settings()

We have a pydantic model with the fields "setting1" and "setting2". There's a model_config that contains metadata which acts as the "settings of settings". The settings_customise_sources method defines the sources used to populate the and in what order.

I do have my misgivings how this works, mainly around the way the sources and tied to the object model. Something more functional would be clearer:

settings = load(
    Settings,
    toml_loader("config.toml"),
    env_loader(".env"),
    ...,
)

But that's a topic for another day.

Config from file

Environment variables are undoubtedly the easiest way to configure an application. However there is no good method to update the envvar during a process' execution and have process reload it.

Another approach is to load configuration from network resources such as another service or databases. This is likely how existing SaaS solutions work. This is a viable choice that offers a ton of flexibility, the potential operational overhead is worrying, for example, what happens if we fail to load the config from the remote? We likely need to handle many different failure nodes in our application.

That's why I've chosen to configure the app a file as in the above example. Which is simple and more reliable than the network method. The file content can easily be changed by another process or manually. Likewise it can easily be viewed for debugging purposes.

Note that I've chosen to use toml in the above example, but pydantic-settings includes loaders for almost all formats. My personal preference is toml for small amount of config and yaml for potentially large config files.

Reloading the file dynamically

That leaves us with the biggest question, how will we reload the file dynamically?

Luckily this is a question that already has well established answers: file watchers. File watchers are often used for local development to allow for dynamic reloading of servers. inotify is one such exapmle on linux where a lot of server code is being run.

There are already many packages that make use of inotify, I've chosen watchfiles for its broad compatibility across multiple platforms and generally great performance.

The api is incredibly straightforward:

async for _ in awatch(CONFIG_PATH):
    ...  # handle the file change

We enter a new iteration every time there's file change, during this iteration we can reload the settings. There are sync apis available as well but I'll stick to async for now.

Putting things together

I've added everything together in a FastAPI service. The service has a single endpoint to fetch the current settings values. Django and flask services can also use something similar to this, which I may cover in the near future.

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "fastapi",
#     "pydantic-settings",
#     "uvicorn",
#     "watchfiles",
# ]
# ///
import asyncio
from contextlib import asynccontextmanager
from pathlib import Path
from typing import AsyncIterator, Self

import uvicorn
from fastapi import FastAPI
from pydantic_settings import (
    BaseSettings,
    PydanticBaseSettingsSource,
    TomlConfigSettingsSource,
)
from watchfiles import awatch


CONFIG_PATH = Path("config.toml")


class Settings(BaseSettings):
    setting1: str
    setting2: str

    model_config = {"toml_file": CONFIG_PATH}

    @classmethod
    def load(cls) -> Self:
        return cls()  # type: ignore

    def reload(self) -> None:
        self.__init__()  # type: ignore

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        *args,
        **kwargs,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (TomlConfigSettingsSource(settings_cls),)


settings = Settings.load()


async def auto_reload_settings():
    async for _ in awatch(CONFIG_PATH):
        settings.reload()


@asynccontextmanager
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
    async with asyncio.TaskGroup() as tg:
        task = tg.create_task(auto_reload_settings())
        try:
            yield
        finally:
            task.cancel()



app = FastAPI(lifespan=lifespan)


@app.get("/settings")
def get_settings() -> Settings:
    return settings


if __name__ == "__main__":
    uvicorn.run(app)

A few notes

Reloading and loading the settings

The documentation for inplace reloading can be found here. The direct invocation of __init__ is a little odd so I've decided to encapsulate it as reload but otherwise this is just following the official documentation.

Running a background asyncio task

lifespan is the best way to add startup and teardown logic to a fastapi app.

asyncio.TaskGroup is used here to handle the task future and avoid not await warnings.

Next steps

I think what I've put together here is a very simple solution, but simple solutions can work more reliably.

This is of course only the first step in hopefully a series of posts.In the near future I'll explore:

Connecting the code with the infrastructure (e.g. kubernetes) we're running the code on.
Solving the observability challenge that comes with dynamically changing configuration.

Subinterpreters and Asyncio

2025-08-05T00:00:00+01:00

PEP-734 subinterpreters in the stdlib has officially been included in the Python 3.14 as a very late addition. subinterpreters now has a new home in the standard library module called concurrent.interpreters.

If you've been following my blog posts you'll know that I'm particularly excited about this feature.

My little advent of code example has been my goto benchmark of parallelism in Python. And multiple interpreters have consistently beaten free-threading in performance. Though I've always found the usability a little lacking.

Usability

Using only the standard library, there are two high level ways to access interpreters for parallel execution.

InterpreterPoolExecutor

The new InterpreterPoolExecutor is similar to ProcessPoolExecutor.

from concurrent.futures import InterpreterPoolExecutor


def sums(num: int) -> int:
    return sum(i * i for i in range(num + 1))


with InterpreterPoolExecutor() as executor:
    print(list(executor.map(sums, [100_000] * 4)))

In order to pass the function itself, the arguments and return values are pickled before they are passed to the interpreters. Which will impact performance and may cancel out a lot of the performance gains over other parallelism mechanisms like free-threading and multi-processing.

Interpreter's `call_in_thread`

call_in_thread will call the given function but it'll use shared memory to call the thread.

from concurrent.interpreters import create


def sums(num: int) -> int:
    return sum(i * i for i in range(num + 1))

threads = [create().call_in_thread(sums, 100_000) for _ in range(4)]
for t in threads:
    t.join()

This will be much faster than pickling, but it's still a little awkward to use.

First issue is that the function does not abstract the thread creation nor does it provide any mechanism to retrieve the results.

The bigger issue for some of my use cases are that we're not reusing interpreters. Suppose we want to call something 1000s of times over the program's life cycle, creating 1000s of interpreters will not be feasible. This issue alone may turn people away from using the faster shared memory mechanism.

Using Asyncio

My solution for concurrency is almost always async, I've even made a similar case in Free Threaded Python With Asyncio.

I am fully aware of how many people in the community dislike async. Unfortunately this is an opinion that could divide the community and is definitely a topic for another day.

To marry subinterpreters with asyncio I created the package aiointerpreters. Here's a very basic example:

with Runner(workers=5).start() as runner:
    await asyncio.gather(runner.run(cpu_bound_function, "arg"), runner.run(cpu_bound_function, "other arg"))

With the following restrictions:

function must be a module level function.
function can only take in 'shareable' types.
function can only return 'shareable' types.

Shareable types are str | bytes | int | float | bool | None | tuple[Shareable, ...] | Queue | memoryview

These caveats are directly linked to the design of subinterpreters, which you can read more about it below and in the original PEP.

Architecture

The architecture of the system can be represented by the following sequence diagram

We spawn multiple worker(interpreter) threads and a single coordinator thread. The worker thread consume the task queue and put results on the result queue.

The coordinator thread then picks up the results and sets the results in the future. Any awaiting tasks on the main thread will then resolve. Here's a simplified snippet of code to give you some idea of how this looks:

class Runner:
    def __init__(self, *, workers: int) -> None:
        self._tasks = create_queue()
        self._results = create_queue()
        self._futures: dict[int, asyncio.Future] = {}
        self._code = dedent("""
            while True:
                task = tasks.get()
                # run function
                results.put(result)
        """)
        self.workers = workers

    def _worker(self) -> None:
        interp = create()
        interp.prepare_main(tasks=self._tasks, results=self._results)
        interp.exec(self._code)
        interp.close()

    @contextmanager
    def start(self) -> Iterator[Self]:
        """Start the runner in a `with` block.

        This will create the workers eagerly.
        """
        threads = [
            Thread(target=self._coordinator, daemon=True),
            *(Thread(target=self._worker, daemon=True) for _ in range(self.workers)),
        ]
        for t in threads:
            t.start()
        try:
            yield self
        finally:
            for _ in range(self.workers):
                self._tasks.put(None)

            for t in threads:
                t.join()

    def _coordinator(self) -> None:
        workers = self.workers
        while workers > 0:
            match self._results.get():
                case None:
                    # Interpreter closed
                    workers -= 1
                case int(i), False, str(reason):
                    future = self._futures.pop(i)
                    future.set_exception(InterpreterError(reason))
                case int(i), True, result:
                    future = self._futures.pop(i)
                    future.set_result(result)
                case other:
                    raise InterpreterError("Unexpected queue value: ", other)

    async def run[**P, R](self, fn: Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R:
        future = asyncio.Future()
        id_ = id(future)
        self._futures[id_] = future
        self._tasks.put((id_, args, tuple(kwargs.items())))

Note that the actual working example uses loop.call_soon_threadsafe(...) the event loop is not thread safe.

Function loading

Then there's the problem of loading functions into the interpreters. As far as I can tell, the best way to do so without pickle is to import the functions inside the interpreters. This is likely the mechanism behind call_in_thread.

So inside the interpreter's code, I added the following utility to load functions:

import importlib
import importlib.util
from functools import cache

@cache
def load_entry_point(entry_point_type, path_or_module, name):
    if entry_point_type == "module":
        return getattr(importlib.import_module(path_or_module), name)

    elif entry_point_type == "path":
        spec = importlib.util.spec_from_file_location("my_module", path_or_module)
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        return getattr(module, name)

    else:
        assert False

There are two ways to load functions, one using the module name. For example, from package.sub_package.module import function. The second way by the module's file path.

The module method is used by default as this is generally how Python will do imports. Paths are used in cases where the module in question is __main__ which the subinterpreters will have no concept of.

Crawler example

To demonstrate the advantage of multiple interpreters, I created the following crawler example.

import asyncio
import time
from contextlib import contextmanager
from typing import cast
from urllib.parse import urljoin

from bs4 import BeautifulSoup
from httpx import AsyncClient

from aiointerpreters.runner import Runner

BASE_WIKI_URL = "https://en.wikipedia.org"
urls = [...]


def parse(content: bytes) -> int:
    soup = BeautifulSoup(content, "html.parser")
    urls = (
        urljoin(BASE_WIKI_URL, cast(str, a_tag.get("href", "")))
        for a_tag in soup.select("div#bodyContent a[href]")
    )
    return sum(1 for _ in urls)


async def run_all(urls: list[str]) -> None:
    async with AsyncClient(timeout=30) as client:
        with Runner(workers=12).start() as executor:
            semaphore = asyncio.Semaphore(200)
            async with asyncio.TaskGroup() as tg:
                for url in urls:
                    await semaphore.acquire()
                    tg.create_task(
                        fetch_and_count(executor, client, url),
                        name=url,
                    ).add_done_callback(lambda _: semaphore.release())


async def fetch_and_count(runner: Runner, client: AsyncClient, url: str) -> None:
    response = await client.get(url)
    response.raise_for_status()
    links = await runner.run(parse, content=response.content)
    print(f"Found {url = } {links = }")


if __name__ == "__main__":
    asyncio.run(run_all(urls))

The crawler example loads wikipedia pages in concurrently using asyncio and then delegates the parsing task to subinterpreters.

This combines of asyncio's fast IO bound operations and subinterpreters parallel parsing. Using the different technologies where they are most performant. The fast communication between the main thread's event loop and interpreters means performance degradation is fairly minimal.

I would encourage people to try it on their own machines first. I have not had the chance to benchmark things rigorously across different architectures.

In contrast, when trying the same with free-threading on my machine I observed significant slowdowns.

Parting words

My goal with this package is to give more people the chance to leverage interpreters. I know this is probably not how everyone will prefer to use it and that's fine. I hope I've at least created a starting point for people to experiment more.

I look forward to seeing how this feature evolves, whether people use it with asyncio, or go with channels and CSP, or maybe we create something entirely different.

Python 3.14: 3 smaller features

2025-07-11T00:00:00+01:00

Python 3.14 is just around the corner and it's jampacked with huge updates:

But as with any release, there are many nice smaller and less noticeable features. Features you won't see unless you comb through the entire release notes. Luckily I am boring enough to do just that.

contextvars.Token is now a contextmanager

As Python embraces different forms of concurrency, contextvars has become extremely important.

A very common example is logging.

That being said, I've always found resetting ContextVars a bit of a pain, as the a token needs to be kept around.

A = ContextVar("A")


token = A.set("val")

...  # other code
...

A.reset(token)

So much so I've written the below function many times:

@contextmanager
def auto_reset[T](context: ContextVar[T], token: Token[T]) -> Iterator[None]:
    try:
        yield
    finally:
        try:
            context.reset(token)
        except Exception:
            ...


def set_context[T](context: ContextVar[T], value: T) -> ContextManager[None]:
    token = context.set(context.get() | kwargs)
    return auto_reset(context, token)


with set_context(A, "val"):
    ...

The function sets a context and allows it to be reset by a context manager instead.

Well now you won't have to, in 3.14 Token is now a ContextManager:

A = ContextVar("A")


with A.set('val'):
    ...

The new code speaks for itself, it's much more concise and just makes sense.

`functools.Placeholder`

Suppose you have a function like this:

def div(a: int, b: int) -> int:
    ...

and want to create a partial function that always divides by a certain number:

div_by_2 = partial(b=2)

This actually can be typed checked as both mypy and pyright support partials as a special rule (example).

There are situations where partial couldn't work though, suppose our function doesn't take in keyword arguments:

def div(a: int, b: int, /) -> int:
    ...

This could be done with a lambda a: div(a, 2) but we lose typing information. Or we can create a named function but that can be quite verbose.

I have a rather complicated solution in better-functools for this exact scenario. But it pushes the limits of the current capabilities of typing in Python and I ended up resorting to specifying the types explicitly.

In Python 3.14 however this is completely possible with partial.Placeholder:

div_by_2 = partial(Placeholder, 2)

The placeholder simply marks any positional arguments we are not specifying yet.

Caveat: as far as I can tell, none of the type checkers have updated their rules yet. Given that partial itself works I think it's only a matter of time.

UUID v6 v7 and v8 are added to Python 3.14.

More specifically I'm very excited about v7.

As a rule, I use uuid4 as my primary keys in my database. This is because compared to auto-incrementing integer ids, the key generation process is a lot simpler and can be done either by the database server or at the client side. The randomness of the UUID is also a good security measure as nothing can be inferred by just knowing its value.

The problem? They cannot be used to sort.

It turns out having some ordering built into the ids can be extremely helpful. For most if not all of my tables, I end up with an extra increment id or timestamp to provide the order.

V4 ids when indexed by the database do not offer anything but lookup. But this is where uuid7 comes in, uuid7 has a time component builtin to it, and can be used to lexicographically sort in chronological order. This is all whilst still providing a good amount of randomness. The best of both worlds.

from uuid import uuid7
from datetime import UTC, datetime

uuid7() < uuid7()  # uuid7 is ordered chronologically

u = uuid7()
u.time  # time in ms since epoch
datetime.fromtimestamp(u.time / 1000, UTC)

V6 is similar in that it can be sorted chronologically, but it is less random than V7. In fact, in RFC 9562 it is explicitly stated that:

Implementations SHOULD utilize UUIDv7 instead of UUIDv1 and UUIDv6 if possible

Finally, version 8 is a standard that puts very few limits on the data inside the UUID, allowing vendors to create their own UUIDs based on custom standards. It is not something I personally have a use case for.

My Chinese birthdays, time keeping is hard!

2025-06-25T00:00:00+01:00

I grew up in both the UK and China. So I'd like to think I have a little understanding of both cultures. China uses both the western gregorian calendar and a version of lunar calendar called 农历. This means I have a Chinese lunar birthday as well as a western one.

I did find a small library called lunardate:

Don't go all out on presents though, there's another one in exactly a month.

If you run LunarDate.fromSolarDate(2025, 7, 25) you get LunarDate(2025, 6, 1, 1) which is the same date as LunarDate.fromSolarDate(2025, 6, 25) barring the last parameter 1 which indicates a leap month. Unlike leap days a leap month is a 'repeated' month. Happy birthday to me again!

A leap month is caused by the difference between a solar year (365 ish days) and a lunar year 354 ish days. 3 months are added every 19 years. The maths checks out but the exact mechanism depends on some astrological observations.

Fun fact, I wondered how the library knows when to add a leap month and the answer is quite simple! It has simply stored all leap months between 1900 and 2099.

Back to the point

Apologies for the long winded story about my extra birthdays, now back to the regularly scheduled Python content

I started thinking about how the chinese calendar is complicated syncing moon cycles to solar years. Trying to sync two cycles that don't directly relate to each other has necessitated the concept of leap months.

But then it occurred to me that's exactly what almost all calendars are doing! Leap days itself are necessitated by the ~8hr error trying to synchronise days with years. And gregorian months are really all over the place. Timekeeping in Python and other languages also has its quirks.

Leap Seconds

Leap months are actually a very similar concept to leap seconds. Where occasionally due to various drifts in our time keeping we add an extra second. Like leap months, the date is determined by measurements, which is to say it's not deterministic when the leap second will be added. Though there is usually half a year of warning before the introduction.

In Python as in many programming languages and platforms, leap seconds are not explicitly tracked or represented due to its non-deterministic nature. Operating systems will either repeat the second deliberately slowdown the clock to accommodate for the extra second.

Though it's just a single second, this does have the ability cause issues. In python to track a leap second you can use astropy which accounts for leap seconds

Whilst it's unlikely that you'll need to regularly reach for this, it is good to know what to do when the issue of leap seconds becomes relevant.

Datetime and timezones

NOTE: the pyodide editors may not load timezones properly, apologies in advance

I get reminded (almost) every year on new quirks about it. For the most part the only reasonable thing to do is to use UTC whenever possible:

datetime.now(UTC)

if you use datetime.now() without timezones please don't, it's bad, you'll regret it sooner or later. datetime.utcnow() is really not much better and has in fact been deprecated.

Now just because you use timezone aware datetime doesn't mean you're safe. If you use a timezone with daylight saving time, when the times go back 1hr we get an overlapping interval (a folded datetime):

This returns datetime.datetime(2025, 10, 26, 1, 0, fold=1, tzinfo=zoneinfo.ZoneInfo(key='Europe/London'))

Returns the non-folded datetime.datetime(2025, 10, 26, 1, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/London'))

So far so good, but what if we compare them like this:

We get True which is not at all what I expected. This is because for same zone comparison only the wall clock time is used to preserve backwards compatibility, you can read more about it in PEP-495

But the weirdness doesn't end here:

Normally the above script will return True for any datetime, DST or otherwise. But during folded time (whichever side of the fold you're on), this will always return False.

The reason is is explained in the footnote:

This exception is designed to preserve the hash and equivalence invariants in the face of paradoxes of inter-zone arithmetic

If datetime(2025, 10, 26, 1, tzinfo=UTC)).astimezone(LONDON) were to equal datetime(2025, 10, 26, 1, tzinfo=UTC) then the hash must be equal. But due to backwards compatibility the hash is not equal there. So we don't allow the value to be equal either.

This is so baffling, it took me several tries to understand it.

Finally

Honestly I learned a lot more about time keeping than I expected when researching this topic. I believe no matter the language or tools, it will always be a difficult problem. And as such, we must treat it with the care it deserves when we're building a system that is sensitive to these timestamps.

Asyncio backpressure - Processing lots of tasks in parallel

2025-06-20T00:00:00+01:00

There's mixed feedback to Asyncio in the community. Some people passionately hate it whilst others believe "writing async make program go fast". This debate is way too much for me to cover here right now. Though maybe I'll look at it in the future.

For me I use asyncio a lot, and it's genuinely a useful tool but not without issues. I have had a problem for a while, it involves fetching a large amount of urls:

async def process_one(url: str) -> None:
    data = await fetch(url)
    await upload(data)


async def process_all(urls: Iterable[str]) -> None:
    for url in urls:
        await process_one(url)

But since it's asyncio we're using then we want to parallelise the tasks:

async def process_all(urls: Iterable[str]) -> None:
    async with asyncio.TaskGroup() as tg:
        for url in urls:
            tg.create_task(process_one(url))

Now here's where the problem starts. It works well for 100s of urls but it doesn't work well when we are using a very large number of urls.

The issue is that we must create all the tasks before any tasks starts running in our current approach. In my testing, this starts to become a problem when we have 10M urls as the program will hang too long whilst creating tasks.

The even bigger issue is that the memory usage will be a lot higher, as all 10M tasks will need to be stored somewhere in memory.

Edit Notes:

Previously I quoted the following behaviour and numbers that were not correct:

Now here's where the problem starts. It works well for 100s of urls but when we hit a big number like 10000s we have a problem.

The program seemingly hangs. This is because all the tasks are being created first and only then to do allow the tasks to start executing. The program will also use much more memory than it needs and generally might slow down due to more context switching. Definitely not what we want!

Thanks to @bmwant for pointing this out.

See my update post for details. In short the memory implications are far greater than the speed impact.

sleep

At a first glance, we could instead add a bit of a sleep in the for loop allowing some tasks to start. And easing the pressure on the system.

async def process_all(urls: Iterable[str]) -> None:
    async with asyncio.TaskGroup() as tg:
        for url in urls:
            await asyncio.sleep(0)
            tg.create_task(process_one(url))

Though we can adjust the sleep according to the speed of processing, in practice the number is quite hard to pick. Too much sleep and you slow down too much.

batch

So rethinking the problem, we can act more directly. What we really want is to simply reduce the amount of tasks that can be created in parallel.

A simple way to do so is by batching. itertools.batched makes it easy to split the urls into manageable batches:

async def process_all(urls: Iterable[str]) -> None:
    for batch in batched(urls, 200):
        async with asyncio.TaskGroup() as tg:
            for url in batch:
                tg.create_task(process_one(url))

We allow 200 tasks to run in parallel and wait until all tasks are complete before starting the next 200.

This is simple and easy to understand, as a results it's been my go to method for problems like this. The problem is if we have a small amount of tasks with long wait times then it'll slow down the whole batch.

Semaphore

Finally that leads me to asyncio.Semaphore. Semaphore is a common tool to limit the number of concurrent accesses to a resource. Generally the usecase is to add it inside of the task:

semaphore = asyncio.Semaphore(200)


async def process_one(url: str) -> None:
    async with semaphore:
        data = await fetch(url)
        await upload(data)

This will limit the number of concurrent fetches and uploads. But this doesn't protect us from creating the tasks in the first place. So we need to use it during task creation:

async def process_all(urls: Iterable[str]) -> None:
    semaphore = asyncio.Semaphore(200)
    async with asyncio.TaskGroup() as tg:
        for url in urls:
            await semaphore.acquire()
            tg.create_task(process_one(url)).add_done_callback(lambda _: semaphore.release())

The semaphore.acquire call will block us from creating the tasks before the one of the tasks finishes and releases the semaphore. So we'll have a pretty constant 200 concurrent tasks until we start exhausting the urls.

Backpressure

The techniques mentioned above are all a form of backpressure. We're trying to ease the memory pressure of the system by either sleeping to slow down the rate of task creation or to place some physical limits on the number of tasks that can be created.

The exact mechanism will differ by use case and runtime. For example, we don't need to explicitly create backpressure for threadpools as the concurrency is already limited by the pool size:

def process_one(url):
    ...


def process_all(urls: Iterable[str]) -> None:
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        pool.map(process_one, urls)

Queues with max_size are another method that comes to mind. And generally your use case might differ and require other mechanisms. Backpressure is a topic that's not covered particularly well and it's definitely worth playing around with different methods for your own code.

Docker's MCP toolkit First Look

2025-06-12T00:00:00+01:00

Short update since First Look at MCP. Docker has released it's MCP toolkit which includes a catalog of MCP servers as well as helpers to connect to the MCP clients.

It is exceeding easy to use it, docker is registered as a single MCP server for your client. And tools from the MCP catalog can be added to it by click from the Docker UI. If credentials are needed you'll be prompted to fill it in.

Under the hood

Docker boasts some security features which claims to scan for malicious images and filter out sensitive information sent by the mcp client. It remains to be seen how these claims hold up.

All the MCP servers are running inside of a container, which means the 3rd party MCP servers can't access your filesystem.

Not quite it

Whilst this is a step in the right direction, it's still not hitting the mark for me. There's a reason why we don't run all our desktop applications in docker. Docker is still a bit too expensive to run in the background and a little hard to use in the first place.

Further to that docker MCP is still just a local MCP server. I firmly believe that remote MCP servers are where its potential lies. And though security and sandboxing is important it's a little odd that a local MCP server has no access to the local filesystem.

First Look at MCP

2025-06-03T00:00:00+01:00

Previously I've played with tool calling in Langchain and Python sandboxes. But recently MCP (Model Context Protocol) is front and center. So I tried to create my own github MCP in Python to integrate cursor with the github cli.

What is MCP?

Before we get started, its good to familiarise yourself with the definition of MCP. The idea is that you define MCP servers that expose data (via resources) and actions (via tools). The tool definitions can be inspected via reflection by the MCP client and then it can call on the server when it needs to, the so called USB-C of LLMs.

So in my case cursor will act as a MCP client, and I'll write a local MCP server to connect cursor to github via tool calls.

My local github MCP server

Here I've implemented a MCP server with git and github tools:

I'm using FastMCP which has the look and feel of FastAPI endpoints.

@mcp.tool()
def multiply(a: float, b: float) -> float:
    """Multiplies two numbers."""
    return a * b

Then using uv with inline metadata (see my previous post) allows us to hook it up to MCP clients without worrying about dependencies:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "fastmcp",
# ]
# ///
...

There was a bit of awkwardness since a lot of the commands depended on being run in the right directory. So we must set the cwd (current working directory) when invoking terminal commands.

Likewise I had setup some error handling and result handling:

async def run(*commands, cwd: str | None = None) -> str:
    """Run a terminal command and return the output."""
    process = await asyncio.create_subprocess_exec(
        *commands,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        cwd=cwd,
    )
    await process.wait()
    assert process.stdout
    assert process.stderr
    if process.returncode != 0:
        raise Exception(await process.stderr.read())
    return (await process.stdout.read()).decode()

Finally we can add the MCP server to cursor:

{
  "mcpServers": {
    "pull_request": {
      "command": "/Users/jamie.chang/github.py"
    }
  }
}

And now I have a way to interact with my github PRs right inside cursor

This has now opened up the door to adding other MCP server such as slack and JIRA, allowing me improve my workflow.

Why not use the official MCP server?

You might have seen: official github MCP server and wondered why I don't just use this. Well there are several reasons:

The setup cost of the official MCP server is quite high, requiring docker and credentials.
Writing my own tools allow a much greater degree of customisation. Allowing me to write my own templates and structure git commands the way I want.
Creating my own MCP server allows me to understand MCP to a greater degree.

As I was drafting this post, I also came across Simon Willison's post on an exploitation of the official MCP server. Exploits like this are also possible with your own MCP servers but you can refine your tools to cater to your own needs and in the process this can limit the attack surface.

Things that bugged me.

As MCP is a new concept, it is still very rough. This applies to both the server and the client. As I was implementing the MCP server I discovered cursor had some bugs that prevented me from using enums in my tool definition.

On the server side, I had trouble installing some MCP servers locally. This may be solved by remote MCP servers that would live in the cloud and allow you to simply connect to it. The trouble is I couldn't find any SaaS they were available. This is perhaps due to the lack of maturity of MCP especially when it comes authentication.

Another problem is that cursor only supports MCP tools, it doesn't support other MCP features such as resource and prompts. The general support for the MCP protocol is not very consistent across the board.

Looking ahead: Sampling

The MCP protocol has recently defined a new concept called sampling. I think it's perhaps the most telling when it comes to the intention of MCP.

The concept sampling allows the server to invoke a LLM during a tool invocation, seeding some of the control to the server. An example lifted from fastMCP's docs:

@mcp.tool
async def analyze_sentiment(text: str, ctx: Context) -> dict:
    """Analyze the sentiment of a text using the client's LLM."""
    # Create a sampling prompt asking for sentiment analysis
    prompt = f"Analyze the sentiment of the following text as positive, negative, or neutral. Just output a single word - 'positive', 'negative', or 'neutral'. Text to analyze: {text}"

    # Send the sampling request to the client's LLM (provide a hint for the model you want to use)
    response = await ctx.sample(prompt, model_preferences="claude-3-sonnet")

    # Process the LLM's response
    sentiment = response.text.strip().lower()

    # Map to standard sentiment values
    if "positive" in sentiment:
        sentiment = "positive"
    elif "negative" in sentiment:
        sentiment = "negative"
    else:
        sentiment = "neutral"

    return {"text": text, "sentiment": sentiment}

I was a little perplexed by this initially, in the above example we could probably have just split up the tool call. But the point of this is to allow server developers to have more control and involve LLM in the workflow.

And the more I look at it, the more I think it looks like a system call:

Though it's seeding some of the control to the server, the sampling call is designed with security in mind and supports human in the loop controls. This is to me is analogous to how the user program can request operating system resources via system calls.

This positions the LLM at the core, the CPU in our analogy. The servers are the programs running on the hypothetical operating system. Looking at the bigger picture this is exactly what the creators of MCP Anthropic envisions with their LLMs making it the star of the show.

I'm not fully convinced on MCP by any means, but for the first time in a long while, it felt like I got some control in how I want the LLM to perform.

Python 3.14: State of free threading

2025-05-26T00:00:00+01:00

In my posts earlier this year I talked about the parallelism performance on 3.13 free-threaded builds. In particular I looked at solving an advent of code problem. In How free are threads in Python now? I discovered significant performance penalties for using free-threading and a lack of tooling available to debug these issues. In a later post I compared free-threading with the yet unreleased subinterpreters.

The conclusion being that subinterpreters are often overlooked but they provide far more predictable scaling characteristics. And with explicit memory management the performance is better than free-threading.

3.14 Updates

Python 3.14 is still in beta so everything should be taken with a grain of salt here. But there are good news, in 3.14 many of the "TODO"s have been completed. This should mean better performance and fewer bugs.

So I reran my code here with 3.14:

Raw data:

Worker Count	3.14t threads	3.13t threads	3.14t subinterpreters	3.13t subinterpreters	3.14 subinterpreters	3.13 subinterpreters
1	2.920236	3.591980	2.848517	3.581693	3.285170	3.616936
2	1.694204	2.322360	1.673501	1.801588	1.696457	1.837193
3	1.289783	1.678028	1.042733	1.262571	1.212331	1.285815
4	1.157097	1.387227	0.828986	0.976082	0.897258	0.972027
5	1.295146	1.356140	0.704804	0.820581	0.825292	0.800690
6	2.109609	2.562865	0.719364	0.783600	0.724958	0.741971
7	2.687946	3.488201	0.642275	0.778133	0.661795	0.688237
8	3.101248	4.438853	0.645746	0.735963	0.634333	0.648535
9	3.634345	5.401229	0.649026	0.732540	0.614862	0.729464
10	4.235933	6.152377	0.712666	0.789292	0.612465	0.593892
11	4.315591	6.908039	0.682817	0.721079	0.586761	0.586010
12	4.361890	6.873490	0.653831	0.772412	0.590408	0.588454
13	4.597265	6.815938	0.644710	0.753303	0.605583	0.603435
14	4.582979	7.090147	0.751220	0.775110	0.612578	0.588413
15	4.501879	6.522076	0.725688	0.859769	0.610816	0.594174

Better but is it enough?

Performance numbers of free-threading in 3.14 are definitely better then 3.13, but the scaling is definitely still disappointing.

I also ran different versions of subinterpreters, which were pretty much the same with small but consistent variations. Subinterpreters still generally beat free-threading.

Additionally there is still no way to profile and debug free threading performance which is a bit disappointing.

Let's look forward

The update may not be a game changer yet, but the trend is still positive.

There's also been talk of more radical changes to memory ownership and data structures see Project Verona.

I have experienced cases where free-threading hits the mark of performance whilst being easier to use compared to subinterpreters. Though the community hasn't quite worked out the direction of free-threading, there's a good chance these incremental improvements will add up in the future allowing free-threading to become a no-brainer.

For now the best thing to do is to experiment and find examples where free-threading helps and examples where it doesn't.

t-strings: the good and the ugly

2025-05-07T00:00:00+01:00

This one's hot off the press as the first beta for Python 3.14 (aka. π-thon) has hit. We're looking at a chunky release with a lot of new features. But all I can think about are these new template strings (officially t-strings).

PEP-750 officially introduces the concept. The idea is the syntax of f-strings but allowing customised behaviours.

Quick Recap of f-strings

F-strings were introduced in Python 3.7. The allow strings to formatted for concisely:

name = "world"
f'Hello {name}'

# instead of 
'Hello {name}'.format(name=name)

There are also some lesser known but very useful features.

f-string with `=`

Very useful for debugging, you can render both the name of the variable and the value with an = between.

name = "Jamie"
assert f'User {name = }' ==  'User name = Jamie'

f-string with formatting spec

We can also quickly format objects with f-strings, this is especially useful for formatting datetimes:

cost = 15
assert f'{cost:.2f}' == '15.00'
assert f'{datetime(2025, 1, 1):%Y/%m/%d}' == '2025/01/01'

Why t-strings?

For me there are two big reasons for t-strings to exist.

f-string syntax but with custom rendering logic.
f-string but with deferred rendering.

In the PEP there are proposals to use this for html rendering and escaping or sql parameter substitution.

evil = "<script>alert('evil')</script>"
template = t"<p>{evil}</p>"
assert html(template) == "<p>&lt;script&gt;alert('evil')&lt;/script&gt;</p>"

Where the t-string returns a string.templatelib.Template object and the html function contains the custom logic for rendering the template. In this case we delay rendering the string and then our custom logic escapes any potentially harmful variables in the string.

How does it work?

According to the PEP, the Template object looks like:

class Template:
    ...  # Omitted a bunch of other fields

    def __iter__(self) -> Iterator[str | Interpolation]:
        ...

We can iterate the template for all the parts of the template. For a template of t'Hello {name}!' becomes:

('Hello', Interpolation('jamie', 'name', None, ''), '!')

My use cases

I can see a lot of potential use cases many of them outlined in the PEP already.

Logging - a decent use case

My favourite use case so far is to solve a bit of a nit I have. The following code looks pretty innocent:

logging.debug(f"{user} - Something happened")

f-strings are eager, so user is stringified immediately. This is fine for simple variables but might be expensive for large nested objects. Then consider running this code in production where we turn off debug logging, we would still need to pay the cost of calling str(user) even if we don't log. This is the rationale behind ruff's G004.

t-strings by nature is 'deferred', so as long as we teach logging how to handle them they will work better than f-string here.

The following snippet does exactly that:

Now we can directly use t-strings for logging as below:

initialize()

name = "Jamie"
logging.info(t'{name = }')

A little ugly: shorthand kwargs

So this is where I think we can take this syntax a little too far. I had this idea a few weeks ago when PEP-736 was rejected. PEP-736 comes from the observation that there are often redundant writing when it comes to invocation by keyword arguments:

User(name=name, birthday=birthday, ...)

This is actually something our t-strings can help with. The name of the variable and the value are captured simultaneously, so we just need to then turn the t-string into a dictionary:

from string.templatelib import Interpolation, Template


def kw(*templates: Template) -> dict[str, object]:
    return {
        inter.expression: inter.value 
        for template in templates 
        for inter in template.interpolations
    }

Then we can apply it to a function call:

@dataclass
class User:
    name: str
    birthday: date

User(**kw(t"{birthday} {name}"))
User(**kw(
    t"{birthday=}",
    t"{name=}",
))

Honestly, I'm a little proud of this, but equally repulsed. I think it demonstrates one of the problems with t-strings. It's the fact that they are really not strings to begin with.

They are an intermediate representation that can turn into strings, but there's no reason it has to be.

Fringe use cases like this are usually my forte, but here it looks kind of ugly and awkward to me. This coupled with the fact that we lose all typing information when we build our 't-string` make it hard to recommend here.

Should we be using t-strings then?

I think t-strings are amazing and generally will be beneficial, I was definitely hyped about them.

I have recently shifted my opinions on new syntax in Python. I used to hate any weird syntax, going so far as to prefer dict.update(new_dict) over the new dict |= new_dict. But my recent experiments with DSLs showed me that done right, there is a place for new syntax to be very beneficial.

I do also see the argument against new syntax. Before writing this post, I watched anthonywritescode's video. He actually goes very in depth on t-strings made a good argument about the added mental overhead of this.

Therefore I can only say that we need to find our balance here. For me, I'll pursue opportunities to use t-string to generate strings. But will tread more carefully when going off-piste to create complex objects.

Why you should write your tools in Python Again

2025-04-08T00:00:00+01:00

You're probably thinking that there are already plenty of tools written in Python.

But I see that most of the popular tools like mypy and flake8 are built for development environments. In contrast, general purpose cli tools tend to be built in other languages, for example most of the docker/kubernetes clis are built in go.

This wasn't always the case, let's set our mind to the era of Python 2. 2.7 enjoyed an almost 10 year lifespan whilst the Python community was working on Python 3. The stability of 2.7 meant that it was easy to distribute tools as Python scripts.

Then Python 3 arrived with lot's of nice stuff: f-strings, dataclasses, pattern-matching etc.. But no more stability, a script working on 3.5 might have problems on 3.7 and so on. 3rd party dependencies adding extra dimensions to the problem.

Compiled languages like go and rust don't tend to have this problem as the binary executable is distributed, the executable no longer depends on the language itself, but on the platform it's running on. So tools must be built for every platform it's expected to run on. However in many cases this is more manageable than Python's versions and dependencies.

Writing a terminal app

Fast forward to the present. I've been keen on writing scripts to automate some of the more mundane tasks. I wanted to write a terminal app to display my JIRA tickets, so I don't have to deal with the clunky web UI.

I've' heard a lot of good things about textual, not having the patience to learn rust, I gave textual a go.

I proudly present my little TUI (Terminal UI):

Source code here

The app itself is very simple, we're just displaying the tickets and title. Nothing fancy, speedy and to the point.

I won't go into too much depth on the code itself, textual worked well and was fun to use. Instead let's dive into how we distribute and run the script.

So how has Python fixed its problems?

In short, uv!

uv has been all the rage lately, for good reasons. Thanks to uv package management in Python got a lot better.

In my specific case I used uv for the following.

manage dependencies such as Python version and packages
execute the script in an isolated Python environment without any hassle.

Let me elaborate, at the top of my script we have:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "requests",
#     "textual",
#     "platformdirs",
#     "tomli-w",
#     "keyring",
# ]
# [tool.uv]
# exclude-newer = "2025-04-09T00:00:00Z"
# ///

PEP-723 inline metadata creates a standard to setting the metadata like dependencies at the top of the file.

Breaking down each line we have requires-python = ">=3.12" for the python version and then the list of dependencies.

Finally a uv specific exclude-newer that acts as a pseudo lockfile, limiting dependencies by date instead of by version.

This is a whole Python project in one, and uv will happily run it with a single command:

$ uv run --script jira-tui.py

Note the shebang #!/usr/bin/env -S uv run --script, a neat trick from Rob Allen. On linux the shebang line will save us from typing uv run --script every time.

Essentially uv has become the one stable runtime, as long as the user has uv scripts may be distributed easily.

Python's Broad Compatibility

Just distributing the script is not enough, we also need to make sure that the end user's platform can execute the script.

Owing to Python's popularity, most packages support all major OS's and CPU architectures out of the box.

In fact, it's so comfortable you hardly have to think about it:

textual: the main framework was designed from the ground up to be as compatible as possible
keyring: used to store credentials uses different secure storage based on the OS.
standard lib packages like pathlib and webbrowser all work cross platform.

The one time I actively considered platforms was where to store the config file, which was simply solved using the platformdirs package.

How does python compare to other languages?

The rust and go tools are definitely here to stay. In fact, uv is almost completely written in rust. The main reason to use other languages are performance and concurrency. In my (biased) experience performance is not a priority for terminal applications, but your experiences may vary.

Python's strength lies in the fact that it's simple, easy to get started with and iterate on. This applies even more so with cli scripting, where we generally one quick changes, small feedback loop and to not think about releases.

The takeaway here is that whilst Python doesn't fit every use case, uv has helped it level the playing field and made scripting fun again!

Building a DSL with Python Operators

2025-03-12T00:00:00+00:00

I've been a little obsessed with operator overloading lately. First using |= in sqlalchemy-builder and then using | and @ in better-functools.

I didn't actually know that these qualify as DSLs, specifically what's known as "internal DSLs". Funnily enough, I'm usually not a fan of DSLs. A few reasons come to mind:

DSLs are most often half baked, pretty much every CI-CD pipeline uses a YAML based DSL that's frustrating to work with.
DSLs lack the IDE support of a proper language.
DSLs are another thing to learn, internal DSLs are especially annoying as you need to learn the host language too.

But I realise now that DSLs are also an opportunity to introduce something new in a familiar environment. To demonstrate how and why, I've created another cookbook that builds a functional DSL from the ground up.

better-functools: Python functional fun

2025-03-09T00:00:00+00:00

I recently put some effort into creating better-functools. It's a package that adds some tooling for functional programming in Python. And allows us to write some unique looking code in a manner similar to functional languages:

Try in sandbox

Why?

At a glance this doesn't look very "Pythonic". More complex expressions and syntax are often controversial in Python, but I don't think it's a good enough reason to avoid it. Python is a multi-paradigm language, but it can be awkward to do anything but the expected OOP. New syntax can really help with expanding what's possible with Python. A good example of new syntax is pattern matching, which is a feature that came from functional languages. I'm keen on using this to make functional programming more available to a wider user base, as opposed to the stereotypical functional bros.

Inspiration

If you can't tell already, this is largely inspired by ML specifically OCaml. Last Christmas I decided to solve Advent of Code in OCaml and I loved it.

The biggest takeaway from this experience is that you don't need to understand monads or the other mathematical concepts to program functionally. Those are probably things you come around later but it's not make functional programming fun.

What I loved most is the the system and ergonomics in OCaml.

Note: The ML family are not the only programming languages, other languages may have a stronger focus on ergonomics than type safety.

OCaml is strongly typed but does not require explicit annotations, instead types are inferred. For example, the signature of add in the following snippet is automatically inferred as + operator only takes integer values.

let add a b = a + b
(* val add : int -> int -> int = < fun > *)

This greatly reduces the burden on programmers.

On the topic of ergonomics, OCaml like many other functional languages has a lot of features to make composing functions and chaining function calls easier. The most notable of which are currying:

let add a b = a + b
let inc = add 1

And pipeline:

[1; 2; 3; 4] |> map inc |> sum |> string_of_int |> print_endline

The implication of this is that you rarely need to abstract out functions because you can easily compose and chain them inline when you need them.

Functional patterns with `better-functools` and Python

So how can we use these ideas in Python?

Currying

We can also do currying in Python, but it's awkward as Python has many different types of arguments: defaulted, positional, keyword and combinations of all of them.

Instead I've added @bind(...) as a short hand to bind an argument and always return a partial.

inc = add @ bind(1)

The resulting expression is a bit more verbose than currying but has the same look and feel.

pipeline

The 'pipeline' operator |> in OCaml chains operations together in a clear and concise way. And I'm definitely not the only person to think this way:

TC39 is a proposal to add |> in JS.
There are other python libraries that implement similar ideas e.g. pipe.
In golang a rejected proposal

My approach uses the | operator in python and you must explicitly start and end a pipeline

Pipeline(inputs) | ... | Pipeline.unwrap

The specific reason why I chose to make the Pipeline explicit but not @ operations is mainly due to how common | is vs @ in Python. Something I'll go in more depth in a future post.

Other Cool Patterns

# Binding partially to the second argument not the first
func(itertools.combinations @ func.arg(Iterable[int]) @ bind(2))

# Null coalescing 
nvl(count_or_none @ nvl @ apply(add @ bind(1)))  # increment count or return null

# Composing functions inline
sum @ compose(eq @ bind(2020))  # check sum equal to 2020 

# Mixing `|` and `@`
_ = (
    Pipeline(inputs)
    | map @ bind(mul @ 2)  # multiply all inputs by 2
    | sum  # add them up
    | eq @ bind(2020)  # check if they sum to 2020
    | print  # print the result
)

Sqlalchemy Footgun: Discarding the statement

2025-02-02T00:00:00+00:00

This one has frustrated me for a while.

It starts off with a REST API route. For example in fastAPI

@app.get("/")
def search_users(session: Session) -> list[User]:
    """Finds users optionally filter by user_id"""
    statement = select(User).order_by(User.name)
    return session.execute(statement).scalars().all()

Then we get asked to filter by a User.name or something else:

@app.get("/")
def search_users(session: Session, name: str | None) -> list[User]:
    """Finds users optionally filter by user_id"""
    statement = select(User).order_by(User.name)
    if name:
        statement.where(User.name == name)
    return session.execute(statement).scalars().all()

And we're done! Right?

Nope, statement.where creates a new Select object, but since that object is not assigned to anything it is never used. The statement we execute with is the original select(User).order_by(User.name). We then get all the users in the database as opposed to the specific user we were looking for.

Solutions

This has happened to me enough times that I wanted to do something about it.

Testing

Well it goes without saying, go and test your endpoint with all the possible parameters. Whilst this should always be done, it takes a longer time for problems to be discovered.

For this problem I will focus on other approaches that may allow it to be caught sooner.

New Type Annotation

To me this issue should be caught by static type checkers, we want to catch this issue as soon as possible so something that works as an LSP is ideal.

This is a potential type annotation for it:

class Select:
    def where(self, *value) -> NoDiscard[Self]:
        ...

References:

I think this definitely should be added and still hope that this gets turned into a PEP and added to Python. But unfortunately there's just too many directions this ideas is pulled in, so we will need to wait and see.

Linting and Type Checking

I found out that Pyright actually offers the following configuration:

reportUnusedCallResult [boolean or string, optional]: Generate or suppress diagnostics for call statements whose return value is not used in any way and is not None. The default value for this setting is "none".

This only checks for functions whose return value is not None so at least it won't flag functions like print.

But there are still functions that this will unnecessarily flag up. For example,

async with asyncio.TaskGroup() as tg:
    tg.create_task(some_task(...))

create_task returns a Future but it's not necessary to use it. The over all noise on my code base was just too much here. But it's worth a try to see how many false positive you get on your code base.

The Builder Pattern

Looking at the problem from a different angle.

The issue is Select.where creates a brand new Select object so has no side effects (pure function), and I'm introducing bugs by confusing a pure function with one that has side effects.

So a simple fix is to make all operations impure.

We can wrap the Select object using the following class:

class Builder:
    def __init__(self, statement: Select) -> None:
        self.statement = statement

    def where(self, *conditions) -> None:
        self.statement = self.statement.where(*conditions)

Incidentally this is known as the builder pattern, methods on an object build up a mutable internal state and finally when we're done building we extract the internal state.

The Operator Approach

The approach I went with is a bit more "interesting" but it's perhaps not as pythonic.

Consider the corrected code from my example:

statement = statement.where(User.name == name)

One big reason I forget the statement = is that it's just a big more verbose. If only there was a short hand to call .where and immediately assign it.

That's exactly how inplace operator in python work:

Instead of

value = value + 1

We write:

value += 1

Almost all the operators can be overridden at an class level. Libraries like pipe overrides __or__ (more precisely __ror__) and let's users use | to chain functions.

So we can wrap any callable that takes a single argument with: Code sample in pyright playground

from dataclasses import dataclass
from typing import Callable

@dataclass
class Pipeable[T, R]:
    """Wrap a callable and allow it to be involked with `|`.
    """

    fn: Callable[[T], R]

    def __call__(self, value: T) -> R:
        """Call the wrapped function."""
        return self.fn(value)

    def __ror__(self, value: T) -> R:
        """Use pipeline to call the wrapped function."""
        return self.fn(value)


def _square(x: int) -> int:
    return x * x


squared = Pipeable(_square)
value = 2
value | squared
value |= squared

This allows us to use the shorthand |= with any function, making our code more concise and less error prone.

The added benefit with Pyright is that it treats use | operator as an creating an expression, so value | squared will produce the "Expression value is unused" error. No analogous feature exist in MyPy yet but this is great for people who use Pyright.

Putting this together, we can do something like this:

statement = select(MyModel).order_by(MyModel.created_at)
if name:
    statement |= where(MyModel.name == name)

if reverse:
    statement |= order_by(MyModel.created_at.desc())

If you think this is useful or just want to try it for fun, I've built a package called sqlalchemy-builder.

If you don't want to install it, the recently released pydantic.run is perfect for playing around with the code in your browser!

What have we learned?

From this experience, I'm seeing that there are some gaps in typing in Python, and it's unclear what changes are prioritise and what changes are ultimately left out of the language. I think a proper typing support is still preferred compared to the other methods which are more like workarounds.

The discovery of operator overloading and Pyright as way to detect unused return value is a pleasant surprise and I hope to find other use cases for it. Using more syntax in Python might become controversial but I think it depends on the use case, for example, pathlib makes heavy use of / operator.

I'd love to hear feedback on sqlalchemy-builder, this solves a real problem for me, but I'm not sure if other people run into this issue.

Jamie's Pattern Matching Cookbook

2025-01-16T00:00:00+00:00

Structural pattern matching is probably the coolest new syntax introduced to Python. Added in 3.10, it's been a few years now and more people are writing apps in 3.10* than any other version now.

Though even with the wide adoption of 3.10 and more people being exposed to the match and case syntax, I find still some resistance towards more advanced usage of it. This is natural as new syntax will always take time. Even something as simple as := walrus operator took some time for the community to warm up to it.

Given the massive potential of pattern matching, I thought it be good to compile a cookbook for it so that people can familiarise themselves with it. I once again created the cookbook using JupyterLite, and surprisingly I found a lot holes in my own knowledge of pattern matching.

So please read though it, play with it, copy it, create your own. It's okay to decide not to use all the features of pattern matching, but it's certainly good to build a knowledge base for yourself.

Cookbook

Same as before I've created a cookbook using JupyterLite in the web. You can interact with it here,

Browse my Cookbooks

My cookbooks are hosted statically here.

The source code can be found here.

Downloading the notebook

Click the "JupyterLab" button above.
In the file view, right click and select "Download"

How good are sub-interpreters in Python now?

2025-01-03T00:00:00+00:00

Introduction

I wrote about free-threading recently in How free are threads in Python now?. Where I found some unexpected difficulties parallelising my solution.

In the conclusion section I wrote:

I'm starting to think that the approach sub-interpreters is taking, with brand new concurrency primitives and shared memory data structures has a lot of merits. I like the idea of adding a new dimension of concurrency as opposed to modifying the current. Perhaps I'll try to implement my AOC solution using sub-interpreters.

This week I tried sub-interpreters for the first time and was pleasantly surprised.

Sub-interpreters

The PEP to look at is PEP 734 - Multiple Interpreters in the Stdlib. But this is only possible because of the continued work in PEP 684 - A Per-Interpreter GIL.

In short, instead of achieving parallelism by removing the GIL, we create isolated memory spaces in the same process each paired with a GIL. The isolated memory spaces are the interpreter state, and memory access within the interpreter state is subject to the interpreter's GIL. Different interpreters are then able to run in parallel as they access independent memory spaces.

Certain objects can be shared via sharable objects. This is only possible because the interpreters all live in the same process. In multi-processing this is much harder to achieve.

PEP-734 is not officially accepted for 3.13, but its features are usable via Eric Snow's interpreters package.

AOC Implementation

Full implementation can be found here. I'll focus on the parts that specifically use the specification in the PEP.

Features like channels that are outside of the most recent specification though available are not used.

Types

First I looked at all the types that can be shared between interpreters and it's not so hard to define them as an annotation:

# Create a type so that we can type check what we pass into the shared state.
type Shareable = (
    str | bytes | int | float | bool | None | tuple[Shareable, ...] | Queue | memoryview
)

Most shareable types are immutable with the exception of Queue and memoryview. They are the primary ways of passing data in and out of interpreters. Queue.get can also act as a mechanism to synchronise between interpreters.

Interpreter

Interpreters can also be initialised with some global shareable state. For example:

interp.prepare_main(queue=Queue(), value=5)

This is how we pass the Queue or memoryview objects into the interpreters in the first place.

Running code in interpreters requires using Interpreter.exec:

interp = interpreters.create()
interp.exec("print(\"hello world\")")

This is analogous to exec but in the specified interpreter.

There are also other ways to execute code I chose not to use like InterpreterPoolExecutor which don't take advantage of shared objects, or Interpreter.call which places limitations on the function called.

Note: Interpreter.exec executes code in the foreground, so we would also need to wrap it in a thread.

Calling a function

Many of the examples in the PEP use an import inside the interpreter:

    interp.exec("""if True:
        from mymodule import edit_data
        while True:
            token = control.get()
            edit_data(data)
            control.put(token)
        """)

This is because functions are objects that are not shareable. So it's often just easier to have the interpreter recreate the function by importing.

My wrapper

Putting this all together, I wrote a wrapper to:

create interpreters inside thread
import a function inside the interpreters to execute on input from the task queue
collect the results and close the interpreters

class Executor:
    def __init__(
        self,
        module: str,
        entrypoint: str,
        *,
        workers: int,
        **bind: Shareable,
    ) -> None:
        self.tasks = interpreters.create_queue()
        self.results = interpreters.create_queue()
        self.bind = bind

        self.code = dedent(f"""
            from {module} import {entrypoint} as entrypoint
            from interpreters_backport import interpreters

            tasks = interpreters.Queue({self.tasks.id})
            results = interpreters.Queue({self.results.id})

            while True:
                req = tasks.get()
                if req is None:
                    # Stop!
                    break
                try:
                    res = entrypoint(*req, {", ".join(f"{name}={name}" for name in bind)})
                except Exception as e:
                    results.put((False, repr(e)))
                else:
                    results.put((True, res))
        """)
        self.workers = workers

    def worker(self) -> None:
        interp = interpreters.create()
        if self.bind:
            interp.prepare_main(**self.bind)
        interp.exec(self.code)
        interp.close()

    @contextmanager
    def _get_results(self, expected: int) -> Iterator[Iterator[Shareable]]:
        def _iter():
            for _ in range(expected):
                success, res = cast(
                    tuple[Literal[True], Shareable] | tuple[Literal[False], str],
                    self.results.get(),
                )
                if not success:
                    raise InterpreterError(res)

                yield res

        it = _iter()

        try:
            with closing(it):
                yield _iter()
        finally:
            for _ in range(self.workers):
                self.tasks.put(None)  # Signal for interpreters to finish

    def map(self, its: Iterable[Shareable]) -> Iterator[Shareable]:
        threads = [Thread(target=self.worker) for _ in range(self.workers)]
        for thread in threads:
            thread.start()

        tasks = 0
        for elem in its:
            tasks += 1
            self.tasks.put(elem)

        with self._get_results(tasks) as results:
            yield from results

        for t in threads:
            t.join()

This is derived from a lot of the examples in the PEP. However, there are a few quirks.

There's a bug preventing me from passing queues inside the interpreters directly, so I used a workaround by passing Queue.id inside the interpreter.

In _get_results we signal termination using self.tasks.put(None). This is done after collecting the results to prevent the interpreter from terminating before we collect objects returned from it.

The following example demonstrates what happens when the interpreters close too soon:

from textwrap import dedent
from interpreters_backport import interpreters

interp = interpreters.create()
queue = interpreters.create_queue()
interp.exec(
    dedent(f"""
        from interpreters_backport import interpreters
        queue = interpreters.Queue({queue.id})
        queue.put((1, 2, 3))
    """)
)
interp.close()
print(queue.get())

You receive the output interpreters_backport.interpreters.queues.UNBOUND., where (1, 2, 3) is expected.

Running the executor

Modifying my implementation we can run the executor:

candidates = ((start, node) for node in path if node != start)
executor = Executor("implementations.d6", "solve", workers=workers, grid=grid)
with timer(f"{workers = }"):
    print("part2", sum(1 for res in executor.map(candidates) if res), end="; ")

where the solve function is located in a module implementations.d6.

Since grid is a constant, it's injected into each interpreter at the start too.

The Results

Results running using 1 to 15 workers. As a reminder we have 11 threads available on my machine.

➜  subinterpreters git:(main) ✗ uv run d6.py
Reading inline script metadata from `d6.py`
part1 4883
part2 1655; workers = 1 3.575728 s elapsed
part2 1655; workers = 2 1.858705 s elapsed
part2 1655; workers = 3 1.284574 s elapsed
part2 1655; workers = 4 0.977416 s elapsed
part2 1655; workers = 5 0.8245 s elapsed
part2 1655; workers = 6 0.739645 s elapsed
part2 1655; workers = 7 0.692572 s elapsed
part2 1655; workers = 8 0.68285 s elapsed
part2 1655; workers = 9 0.623754 s elapsed
part2 1655; workers = 10 0.625284 s elapsed
part2 1655; workers = 11 0.600622 s elapsed
part2 1655; workers = 12 0.627131 s elapsed
part2 1655; workers = 13 0.578773 s elapsed
part2 1655; workers = 14 0.586851 s elapsed
part2 1655; workers = 15 0.600652 s elapsed

This surprised me a lot. After my efforts with free-threading, I was definitely expecting a lot more tweaks before getting anywhere.

Compared to my results of free threading:

The speedup is much more predictable, where free-threading performance bottlenecks very quickly.

My Overall Experience

There's a lot to unpack here, not just on sub-interpreters but also free-threading.

Sub-interpreters are unfinished/unpolished

It's no surprise here. PEP-734 is not yet accepted, so the bugs are understandable.

There's definitely a lack of examples and documentation, but hopefully it's something that the community can help with.

I also think that maybe there's a high level abstraction missing here. The API is inspired by Golang and CSP in general. It's probably not what Pythonistas are most familiar with. Using exec also feels a bit strange. Hopefully the wider community can start creating abstractions that make it more usable.

Free-threading is deceptively hard

In my last post I wrote:

However I'm a bit worried about existing multi-threaded code. Before free-threaded python, threads were used for IO bound operations when the GIL was released. It's going to be a big challenge if one day the GIL is disabled and the performance may become many times slower.

This worries me the most, people are used to threads and that's supposed to be a positive, but in reality they are just used to the GIL.

Sub-interpreters are a new feature so it has an advantage here.

Memory management is key

Despite the awkwardness working with interpreters, when it finally ran it was achieved the goal of speeding up execution. By exposing mechanisms to explicitly manage memory the execution becomes a lot more predictable.

When it comes to free-threading, this is definitely missed. My hope is that over the next few releases we'll see frameworks work on memory management making free threading more viable.

Finally

I wasn't originally planning on using sub-interpreters in the first place, but I'm now fully behind the idea.

I hope it gathers a lot more interest this year and we'll see more interesting use cases and examples.

If this post piqued your interest then there's a lot more resources out there:

Anthony Shaw's post
Yury Selivanov's talk which also mentions a new framework called memhive

How free are threads in Python now?

2024-12-31T00:00:00+00:00

Free-threaded Python (PEP-703) was released in October 2024. It enables true multi-threaded execution without the restriction of the GIL.

I previously covered this in Free Threaded Python With Asyncio, the example used there was selected because it very clearly demonstrates the performance increase, or moreover, the threads "running free".

But that's not really code that solves a problem. It's simply repeating the same calculation. As I've been having a lot of fun with advent of code this December, it provided a very good testbed for free threading.

AOC Day 6

Spoilers for day 6 2024 ahead! full source code here

Day 6 was a good example of a problem that can be solved using parallelisation.

Part 1 involved tracing the path of a guard through a map. Part 2 then asks for locations where we could place a single obstacle and cause the guard to form a loop.

Here's my full solution:

from __future__ import annotations

from contextlib import contextmanager
from dataclasses import dataclass
from itertools import pairwise
from pathlib import Path
from typing import Hashable, Iterable, Iterator, Self, assert_never


type Pair = tuple[int, int]


up = 0
right = 1
down = 2
left = 3


def turn(direction: int) -> int:
    return (direction + 1) % 4


@dataclass(slots=True)
class Ranges:
    """Traversable location in the map."""
    rows: range
    cols: range

    def __getitem__(self, key: tuple[slice, int] | tuple[int, slice]) -> Iterator[Pair]:
        match key:
            case int() as r, slice() as cols:
                return ((r, c) for c in self.cols[cols])
            case slice() as rows, int() as c:
                return ((r, c) for r in self.rows[rows])
            case _ as other:
                assert_never(other)

    def walk(self, start: Pair, direction: int) -> Iterator[Pair]:
        match direction:
            case 0:
                return ((row, start[1]) for row in self.rows[start[0] :: -1])
            case 2:
                return ((row, start[1]) for row in self.rows[start[0] :: 1])
            case 3:
                return ((start[0], col) for col in self.cols[start[1] :: -1])
            case 1:
                return ((start[0], col) for col in self.cols[start[1] :: 1])
            case _:
                assert False


@dataclass(slots=True)
class Grid:
    start: Pair
    tiles: list[list[bool]]
    ranges: Ranges

    @classmethod
    def from_lines(cls, lines: Iterable[str]) -> Self:
        start = None
        coord = (0, 0)
        rows = []
        for r, line in enumerate(lines):
            row: list[bool] = []
            for c, char in enumerate(line):
                coord = (r, c)

                if char == "^":
                    start = coord
                row.append(char == "#")
            rows.append(row)

        assert start is not None
        return cls(start, rows, Ranges(range(len(rows)), range(len(rows[0]))))

    def __getitem__(self, key: Pair) -> bool:
        return self.tiles[key[0]][key[1]]


@contextmanager
def read_lines(path: Path) -> Iterator[Iterator[str]]:
    with path.open() as f:
        yield (line.rstrip() for line in f)


def walk(grid: Grid, obstruction: Pair | None = None) -> Iterator[Pair]:
    direction = up
    start = grid.start

    yield start

    while True:
        walk = pairwise(grid.ranges.walk(start, direction))

        for prev, curr in walk:
            if grid[curr] or curr == obstruction:
                direction = turn(direction)
                start = prev
                break

            yield curr
        else:
            return


def loops[T: Hashable](it: Iterator[T]) -> bool:
    visited = set()
    for e in it:
        if e in visited:
            return True
        visited.add(e)
    return False


if __name__ == "__main__":
    with read_lines(Path("inputs") / "d6.txt") as lines:
        grid = Grid.from_lines(lines)

    path = set(walk(grid))
    print("part1", len(path))

    candidates = (node for node in path if node != grid.start)
    print(
        "part2",
        sum(1 for node in candidates if loops(pairwise(walk(grid, node)))),
    )

A few details I want to highlight here: - Grid implements __getitem__ so we can access locations on a map more directly e.g. grid[1, 2] will access location (1, 2) - Ranges represents the full range of locations in the map and allows us to easily trace paths in a single direction - walk brings everything together and handles the logic when we hit an obstruction.

Part 2 is what we're more interested in here. We iterate for all locations on the path (around 5000 in my case) and check if adding an obstruction there forms a loop.

Parallelisation

I timed just part 2 and it takes around 4 seconds on my 11 core M3 Mac. However it should be simple to parallelise, as each of these walks are independent and the grid is only being read not modified.

Using ThreadPoolExecutor, we can change the last part of the code to run in parallel:

with ThreadPoolExecutor() as executor:
    results = executor.map(
        lambda node: loops(pairwise(walk(grid, node))),
        candidates,
    )
    print("part2", sum(1 for r in results if r))

I was then shocked to find that the execution was almost twice as slow at almost 10 seconds.

Hypothesis

There are several potential reasons to explain this kind of behaviour.

Specialization

Python 3.11 released with performance enhancements via Specialization.

Free-threaded Python has specialisation disabled in threaded code, see https://peps.python.org/pep-0703/#improved-specialization for more details

The performance overhead should however be fairly small, according to the PEP the slowdown should be within 10%.

Lock contention

The GIL's job was to protect objects from concurrent access. This is still important in free-threaded python, so there must be per-object locks in place of the GIL.

These locks can still cause threads to block waiting on them.

Thread start up time

This is more of an issue associated with processes, sub-interpreters but not threads. See Anthony Shaw's post. So the startup time cannot explain the slowness.

Profiling

To test my theories, my immediate thought was just run cProfile.

python -m cProfile d6.py

... and nothing! The process hangs, and no output is shown. It turns out there's a bug in cProfile for free-threaded builds of Python 🤦.

Well this made the problem exponentially harder. I've tried a few other profiler options without success. I briefly considered looking into sys.monitoring but it would probably take me a lot more research to write a profiler.

Experiment, Research, Repeat

Without good profiling the only thing to do is to understand the problem better and try a bunch of things.

Time per thread count

First thing I checked the performance with respect to the thread count, by default, python sets the max_worker count to n + 4 where n is the number of cores in the system.

for workers in range(1, 16):
    candidates = (node for node in path if node != grid.start)

    with timer(f"{workers = }: "):
        with ThreadPoolExecutor(max_workers=workers) as executor:
            results = executor.map(
                lambda node: loops(pairwise(walk(grid, node))),
                candidates,
            )
        print("part2", sum(1 for r in results if r), end="; ")

(advent2024-python) ➜  advent2024-python git:(main) ✗ python d6.py
part1 4883
part2 1655; workers = 1:  4.060725 s elapsed
part2 1655; workers = 2:  4.789434 s elapsed
part2 1655; workers = 3:  5.203642 s elapsed
part2 1655; workers = 4:  5.516271 s elapsed
part2 1655; workers = 5:  6.114952 s elapsed
part2 1655; workers = 6:  7.461776 s elapsed
part2 1655; workers = 7:  7.562252 s elapsed
part2 1655; workers = 8:  8.026832 s elapsed
part2 1655; workers = 9:  8.335291 s elapsed
part2 1655; workers = 10:  8.731357 s elapsed
part2 1655; workers = 11:  9.152564 s elapsed
part2 1655; workers = 12:  9.476967 s elapsed
part2 1655; workers = 13:  9.589462 s elapsed
part2 1655; workers = 14:  9.955716 s elapsed
part2 1655; workers = 15:  9.728167 s elapsed

So more thread means less performance here, this strongly suggests some sort of lock contention.

How locking works

Built-in types use internal locks to guarantee thread safety.

I read the PEP and it appears that the locking is rather complicated. For the most part read access is "optimistic". Since we only read the list it's unlikely to be the culprit. However, to eliminate any doubts I've switched to the immutable tuple, which does not require locking:

@dataclass(slots=True)
class Grid:
    start: Pair
    tiles: tuple[tuple[bool, ...], ...]
    ranges: Ranges

As expected this did not solve the issue and the performance degradation did not go away.

Reference counting

This is going very deep into Python internals, not something I'm particularly comfortable with. I did get a bit of a clue here where someone else also ran into performance issues.

Reference counting is the method used by Python for garbage collection. The idea is that each object stores a counter for the number of references which is then used for garbage collection when the references hit 0.

The bigger implication here is that referencing an object will require the counter to be incremented, which requires locking when the object is referenced from multiple threads.

A form of reference counting called "biased reference counting" is used in free-threaded Python. This allows faster reference count access to objects created within the thread than objects created from different threads.

Overall, this means that even read-only access can be slowed down due to reference counting.

Immortal objects

See PEP-683, this is a recent change too that skips reference counting by marking certain objects as immortal. Immortalization is implemented in free-threaded python as well. It's only really applied to True, False, None as well as classes and top level functions.

Here my grid is already using booleans so it's already using immortal objects. I did use Enum in an earlier version and there was definitely an improvement of the performance here. But it wasn't very significant. Nevertheless it might solve performance issues in other situations.

The Breakthrough

After many iterations, I finally looked at the following line:

if grid[current] or current == obstruction:

This uses the magic method __getitem__

    def __getitem__(self, key: Pair) -> bool:
        return self.tiles[key[0]][key[1]]

which references the Grid object and gets the in turn then references the tiles attribute.

The code also lives in the walk function inside the double loop so it's called frequently!

def walk(grid: Grid, obstruction: Pair | None = None) -> Iterator[Pair]:
    ...
    while True:
        ...
        for prev, curr in walk:
            if grid[curr] or curr == obstruction:
                ...
            ...
        ...

I'd like to claim that I intuitively looked here, but really I looked in a lot of different places before finding this.

In any case we can bypass the __getitem__ call by

for prev, (r, c) in walk:
    if grid.tiles[r][c] or (r, c) == obstruction:

Running the code again:

(advent2024-python) ➜  advent2024-python git:(main) ✗ python d6_threads.py
part1 4883
part2 1655; workers = 1:  3.801581 s elapsed
part2 1655; workers = 2:  3.10458 s elapsed
part2 1655; workers = 3:  2.925348 s elapsed
part2 1655; workers = 4:  2.944652 s elapsed
part2 1655; workers = 5:  3.200314 s elapsed
part2 1655; workers = 6:  4.337672 s elapsed
part2 1655; workers = 7:  4.507129 s elapsed
part2 1655; workers = 8:  4.739261 s elapsed
part2 1655; workers = 9:  4.932432 s elapsed
part2 1655; workers = 10:  5.219045 s elapsed
part2 1655; workers = 11:  5.798776 s elapsed
part2 1655; workers = 12:  6.20557 s elapsed
part2 1655; workers = 13:  6.459673 s elapsed
part2 1655; workers = 14:  6.571967 s elapsed
part2 1655; workers = 15:  6.590083 s elapsed

This is actually the first time we've seen positive performance gain. Curiously, the higher thread counts still result in negative performance.

Taking this further we still reference grid inside the loop, so we can assign tiles to the outside instead:

    tiles = grid.tiles 

    while True:
        walk = pairwise(ranges.walk(start, direction))

        for prev, (r, c) in walk:
            if tiles[r][c] or (r, c) == obstruction:
                ...

Once again:

(advent2024-python) ➜  advent2024-python git:(main) ✗ python d6_threads.py
part1 4883
part2 1655; workers = 1:  3.572743 s elapsed
part2 1655; workers = 2:  2.257732 s elapsed
part2 1655; workers = 3:  1.665534 s elapsed
part2 1655; workers = 4:  1.38678 s elapsed
part2 1655; workers = 5:  1.383013 s elapsed
part2 1655; workers = 6:  2.540588 s elapsed
part2 1655; workers = 7:  3.520335 s elapsed
part2 1655; workers = 8:  4.365446 s elapsed
part2 1655; workers = 9:  5.072404 s elapsed
part2 1655; workers = 10:  5.722957 s elapsed
part2 1655; workers = 11:  6.087521 s elapsed
part2 1655; workers = 12:  6.179896 s elapsed
part2 1655; workers = 13:  6.174642 s elapsed
part2 1655; workers = 14:  6.194982 s elapsed
part2 1655; workers = 15:  6.100845 s elapsed

A decent performance when using 3, 4 or 5 workers.

Worker count

It seems to me like there's a sweet spot for the number of cores vs the reference contention. 4 seems to be quite consistently good on my machine too cores then the reference count contention starts dominating the timing.

It's also possible that it's an indication of performance cores vs efficiency cores on the M3 processor, but I believe this is less likely as we've already demonstrated how dramatic the differences are when it comes to reference count contention.

Key Takeaways

When looking at free threading in Python here are the following key things to look for.

Threads are not free

Threads are not free in terms of performance, you definitely have to pay for it. And concurrent access means you can't really be free of locks.

Avoid shared mutable writes

In this example we didn't have any shared writes, however in based on my previous testing, this can be extremely slow.

Try to structure objects so that they don't need to be mutated across threads.

Minimise shared reads

I don't believe shared reads that can be avoided, the ease of memory access is the biggest advantage of multi-threading.

Worker count may be important

A direct takeaway from the idea that "threads are not free" is that you may need to reduce the number of threads based on the situation.

It's important to test the code with different number of worker threads.

Profiling is important

Profiling is extremely important, I hope cProfile gets fixed or some other profiler can help with this situation.

Conclusion

There's a lot to unpack here, but my biggest thought is that multi-threading in Python is actually deceptively difficult. Simple Python operations may have an amplified performance impact.

Debugging performance problems should get easier when we have profiling working again, so this shouldn't be a big issue for engineers looking to solve parallel problems.

However I'm a bit worried about existing multi-threaded code. Before free-threaded python, threads were used for IO bound operations when the GIL was released. It's going to be a big challenge if one day the GIL is disabled and the performance may become many times slower.

I'm starting to think that the approach sub-interpreters is taking, with brand new concurrency primitives and shared memory data structures has a lot of merits. I like the idea of adding a new dimension of concurrency as opposed to modifying the current. Perhaps I'll try to implement my AOC solution using sub-interpreters.

Finally, despite the effort it took me here, I can see that a lot of work has gone into Python to make the GIL optional. From immortalization to biased reference counting and many more I haven't covered. There is definitely a lot of space for performance improvement. I can't wait to see how the framework authors take advantage of free-threading.

Customising Pattern Matching Behaviour

2024-12-09T00:00:00+00:00

I've been doing advent of code again this year. There are two Python features I always rely on, iterators and pattern matching. Iterators allow for operations on each of its elements without allocating memory for a collection. Ever since pattern matching was introduced in Python 3.10, it's been particularly useful to unpack nested object structure.

Unfortunately pattern matching and iterators don't work well together. By definition elements of an iterator do not necessarily exist so there's nothing to match against. Often to pattern match we must first consume the iterator:

match range(10): 
    case [0, *_]:  # Won't match of course
        ...

match list(range(10)):
    case [0, *_]:  # matches now since iterator is consumed
        ...

Ideally we'd be able to consume the head elements of the iterator on demand, but this isn't possible directly with iterators.

Official Support for Custom Matching

In the original PEP, there's mention of the idea of custom matchers. But the idea was ultimately deferred due to the lack of a clear use case.

Matching Properties

But it turns out we don't really need any of this. We can match any named attributes of an object even a property.

@dataclass
class Repeat:
    value: int
    times: int

    @property
    def repeated(self) -> list[int]:
        return [self.value] * self.times


match Repeat("abcd", 2):
    case Repeat(times=times, repeated=list() as repeated):
        print(f"After {times} times", repeated)

This is really useful as we need to dynamically unpack the iterator if we want to match the iterator. We can create an object with the next property that evaluates the next elements on demand.

@dataclass
class Node:
    value: int
    it: Iterator[int]

    @classmethod
    def from_iter(cls, it: Iterator[int]) -> Node:
        return cls(next(it), it)

    @property
    def next(self) -> Node:
        return Node(next(self.it), self.it)


match Node.from_iter(iter(range(10))):
    case Node(value=0, next=Node(value=1, _)):
        print("Starts with 0 and 1")

There is a bug in the code, consider the following match statement:

match Node.from_iter(iter(range(10))):
    case Node(value=9, next=Node(value=8, _)):
        print("Starts with 9 and 8")
    case Node(value=0, next=Node(value=1, _)):
        print("Starts with 0 and 1")

The looks like it'll match in the second case, but it can't. This is because the first case checks the first 2 elements by consuming them.

But the fix is actually really simple, we simple use cached_property to cache the result of next:

@dataclass
class Node:
    value: int
    it: Iterator[int]

    @classmethod
    def from_iter(cls, it: Iterator[int]) -> Node:
        return cls(next(it), it)

    @cached_property
    def next(self) -> Node:
        return Node(next(self.it), self.it)

Matching positionally using `__match_args__`

Matching named attributes can be very verbose. We can specify which arguments to match sequentially by adding __match_args__ to the class:

@dataclass
class Node:
    value: int
    it: Iterator[int]

    __match_args__ = ("value", "next")

    @classmethod
    def from_iter(cls, it: Iterator[int]) -> Node:
        return cls(next(it), it)

    @cached_property
    def next(self) -> Node:
        return Node(next(self.it), self.it)


match Node.from_iter(iter(range(10))):
    case Node(9, Node(value=8, _)):
        print("Starts with 9 and 8")
    case Node(0, Node(value=1, _)):
        print("Starts with 0 and 1")

Matching the end of iteration

Finally we need to consider the StopIteration at the end of the iterator.

@dataclass
class Node:
    value: int
    it: Iterator[int]

    __match_args__ = ("value", "next")

    @classmethod
    def from_iter(cls, it: Iterator[int]) -> Node | None:
        try:
            return cls(next(it), it)
        except StopIteration:
            return None

    @cached_property
    def next(self) -> Node | None:
        try:
            return Node(next(self.it), self.it)
        except StopIteration:
            return None


match Node.from_iter(iter(range(1))):
    case Node(9, Node(value=8, _)):
        print("Starts with 9 and 8")
    case Node(0, None):
        print("Starts with 0")

Conclusion

You might still be wondering if this is actually useful. I've published this as pattern-utils so you can try it out. Beware the implementation is slightly different to facilitate matching generator with return, for example,

from pattern_utils import generator as gen


def example_generator():
    yield "some resource"
    return "done"


match gen.matcher(example_generator()):
    case gen.Node(resource, gen.Empty(end_result)):
        print(resource, end_result)

I think more important is using the same technique of writing a wrapper with properties to better exploit pattern matching. If I discover other common use cases I will update the library.

Python-in-Python Sandboxing LLM Generated Code

2024-12-02T00:00:00+00:00

I've been experimenting with Langchain for GPT based queries. One problem we often encounter with GPT is hallucinations. This makes certain classes of problems unsuited to GPT, one example is maths and statistics. Whilst there are improvements for recent models often the maths cannot be trusted.

When I try to ask a data heavy question on https://chatgpt.com/, it generates and runs Python code and then returns the answer. I wanted to find a way to do this when using the API ideally as a langchain tool.

Can we just run the code?

One solution is to run the code in a subprocess. A simple tool implementation might be:

@tool
def run_python(python_code: str) -> str:
    """Run Python code and capture the results printed to stdout.

    numpy np is not available, no other 3rd party libraries are available.
    """
    return subprocess.run(["python", "-c", python_code], capture_output=True, text=True).stdout.strip()

So this definitely works! But it makes me feel very uncomfortable. Allowing arbitrary code to execute on a system level can be a huge security issue. If the prompt comes from a user, it's possible they can make LLM generate code that can take control or expose files on the system.

Even if there is strict control of the prompt, we would need to manage the lifecycle of the process to make sure the computation doesn't take too long or too much resources.

Builtin Langchain Solutions

Looking at langchain for some help, I can see that if offers a suite of code interpreter tools.

The interpreters are hosted as separate services, some are available only in the cloud and some can be self hosted. For cloud based interpreter there's still the question of whether you can trust the service. Especially since we're likely going to send data over. There's also the matter of cost.

The self-hosting options are certainly viable, but it's still going to require spinning up a separate Docker container.

Python Sandbox

I wanted to find Python sandboxes that were simple to setup.

PyPy sandbox

The first sandbox I came across is from PyPy. With this we can apply the same subprocess.run command but just invoke the sandboxed version of PyPy as opposed to the regular Python interpreter.

I've ran into problems running it:

➜  sandbox git:(main) ./pypy_interact.py     
  File "/Users/jamie.chang/personal-projects/pypy/pypy/sandbox/./pypy_interact.py", line 58
    'pypy-c': RealFile(self.executable, mode=0111),
                                             ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

A closer look at the docs however reveal that this might not be as actively maintained as I had hoped. There might be newer efforts of this, but I'm not sure how to access them.

WASM

In the past I've heard about WASM being used as a runtime which can offer isolation. For example, cloudflare's edge workers. WASI was introduced in large part to define and restrict what system calls are available to the wasm process.

When researching this I came across an article by Simon Willison, with working code examples on how to achieve this using wasmtime and VMWare's build of python.wasm.

That's pretty much all the hard work done for us!

So I've done some minor modification to Simon's code, using the latest release of python.wasm found here the code is otherwise unchanged.

def run_python_code(code: str, fuel: int = 400_000_000) -> str:
    engine_cfg = Config()
    engine_cfg.consume_fuel = True
    engine_cfg.cache = True

    linker = Linker(Engine(engine_cfg))
    linker.define_wasi()

    python_module = Module.from_file(linker.engine, "python-3.12.0.wasm")

    config = WasiConfig()

    config.argv = ("python", "-c", code)
    config.preopen_dir(".", "/")

    with NamedTemporaryFile() as out:
        config.stdout_file = out.name

        store = Store(linker.engine)

        # Limits how many instructions can be executed:
        store.set_fuel(fuel)
        store.set_wasi(config)
        instance = linker.instantiate(store, python_module)

        # _start is the default wasi main function
        start = instance.exports(store)["_start"]
        start(store)
        return out.read().decode()


@tool
def run_python(python_code: str) -> str:
    """Run Python code capturing the stdout.

    numpy np is not available, no other 3rd party libraries are available.
    """
    return run_python_code(python_code)

This feels much closer to the correct solution and not just because the code is already written for me.

I like that the solution relies on the WASI standard. This is currently a rather trendy technology and the support for it is increasing. By default WASI programs do not have access to anything on the filesystem, and extra permissions must be requested.

Wasmtime has also made things lot easier. The Python bindings mean that we can run the WASM binary inside the same Python process, without a need to create subprocesses, hence the title "Python-in-Python" or more accurately (but not as catchy) "Python-in-WASM-in-Python".

Wasmtime provides a fuel mechanism which limits the number of wasm instructions that runs. This is a good way to limit the processing for each Python call and prevent denial of service attacks.

The portability of WASM means that the same method can be used to run Python sandbox in other languages and other platforms. There's even competing runtimes for WASM like wasmer.

Also I found out later that the Riza Interpreter with a supported langchain tool is also WASM based. Though it's still hosted in a separate container.

Further Work

There are a few small caveats here. For one I only have surface knowledge when it comes to WASM and wasmer. There might be security concerns I haven't thought about.

Also you might have noticed that for the tool implementation, I've stated in the description that

"numpy np is not available, no other 3rd party libraries are available."

This is because GPT has a tendency to use numpy whenever possible. I've not worked out exactly how to make libraries like numpy available to python.wasm, there is some mention of how to do this in the wasmlabs article but it means a more complex setup process and I'm uncertain that this will work for numpy which contains native binary.

There's definitely more work that can be done to simplify the setup process even more. Maybe it can even be made pip installable. It might also be good to investigate working with other languages.

My SQLAlchemy Cookbook

2024-11-25T00:00:00+00:00

I've worked with SQLAlchemy for a while now, and in my opinion it's the best ORM in Python. It's feature rich with strong support for all major databases. And it maintains the SQL feel without losing things like typing.

However there are some challenges here. Despite having very nice documentation and good abstractions for beginners, there can sometimes be an abundance of choice, which can make it more difficult to start on a problem and make it harder to collaborate.

Library

It's tempting to write a library on top of SQLAlchemy that includes the patterns you want to use. However it's hard to find any other reason to do so. Code in SQLAlchemy is already very concise, further abstractions won't yield any tangible benefits to size and complexity of the user code.

There are also many downsides to writing a library namely maintenance and development velocity.

Cookbook

My preferred way is to have a single place to keep all the patterns I use.

I was inspired by the official Python Logging Cookbook. It manages to provide code examples and explanations for logging without needing to bloat the Python standard library more than necessary.

The cookbook is written using JupyterLite in the web, this is a nice trick I stole from DuckDB's blog.

TypedDicts are better than you think

2024-10-01T00:00:00+01:00

TypedDict was introduced in PEP-589 which landed in Python 3.8.

The primary use case was to create type annotations for dictionaries. For example,

class Movie(TypedDict):
    title: str


movie: Movie = {"title": "Avatar"}

I remember thinking at the time that this was pretty neat, but I tend to use dataclass or pydantic to represent 'record' type data. Instead I use dictionaries more as a collection, so the standard dict[KT, VT] annotation is enough.

Non-totality

I revisited typeddicts when I looked at implementing a HTTP patch endpoint.

Let's suppose I have a data structure represented by the following dataclass:

@dataclass
class User:
    id: UUID
    name: str
    subscription: str | None = None

Where subscription = None means no subscription.

Let's say we want to option to patch name, subscription. You might define the patch body using dataclass:

@dataclass
class PatchUser:
    name: str | None = None
    subscription: str | None = None

Here we have a problem, for subscription does None mean don't change or remove subscription.

We can fix this a number of ways, for example, we can take the string 'none' to mean no subscription instead, or make a new sentinel value called NoChange to indicate no changes.

These solutions all feel a little awkward, this is because dataclasses don't have a concept of a field being missing. But this is where dictionaries shine. Dictionaries are not general expected to have all the fields available. We get a KeyError if a field is missing and there are convenience methods such as .get(key, [default]) to fetch a key that is not guaranteed to be present.

This makes TypedDict the ideal data structure in this scenario:

class PatchUser(TypedDict, total=False):
    name: str | None
    subscription: str | None = None

Since total is False here (by default it is set to True), name or subscription can be absent from the dictionary. Which represents the PATCH operation much better than a dataclass or Pydantic model.

Further additions in PEP-655 allows us to mark individual fields as Required or NotRequired which further increases its flexibility.

If you're wondering about FastAPI support for TypedDict, Pydantic supports it out of the box. So your TypedDict can be used in a FastAPI endpoint.

Using `TypedDict` as `**kwargs`

PEP-692 introduced the ability to type variadic keyword arguments using TypedDict.

So the following two snippets are equivalent. Without TypedDict:

def my_function(*, option1: int, option2: str) -> None:
    ...

Using TypedDict:

from typing import TypedDict, Unpack


class Options(TypedDict):
    option1: int
    option2: str


def my_function(**options: Unpack[Options]) -> None:
    ...

At a glance I can say that the TypedDict option is rather verbose. Though it does become more useful if Options were used in multiple function definitions.

def my_function2(**options) -> None:
    ...


def my_function3(*, other_option: str, **options) -> None:
    ...

Where it truely shines is once again with non-totality.

Suppose we have the following scenario, where we want to create a custom version of pytest.fixture, but still pass through some arguments.

def fixture(scope: str = "module", autouse: bool = False):
    return pytest.fixture(scope, autouse)

Here to get the typing right I not only have to find the type of each argument but also the default value. It would be better if we use **kwargs so we can just avoid passing the arguments through. And to keep type information we just need to use our trusty TypedDict once more:

class FixtureOptions(TypedDict, total=False):
    scope: str
    autouse: bool


def fixture(**options: Unpack[FixtureOptions]):
    # Some custom implementations
    ...
    return pytest.fixture(**options)

Non-totallity means that we don't have to pass in scope and autouse. We can just have the default.

Sentinels

We can achieve similar behaviour with sentinels:

UNSPECIFIED: Any = object()  # Has to be Any type so it could be set as default for other types.

def my_func(option1: bool = UNSPECIFIED, ...) -> ...:
    if option1 is UNSPECIFIED:
        ...
    ...

Sentinels work well enough here, but we have to remember to handle them. Additionally type annotations for sentinels can be a bit awkward, here we made UNSPECIFIED an Any type, but it means that inside the function option1 is only typed as bool. There are options to expose the sentinel type but they may add even more confusion.

Using `TypedDict` to pass in dependencies

We can do even more with PEP-692! When I first learned about the PEP, I thought it was only about function signature. But reading through it more thoroughly, I discovered that another consequence of the PEP is that type checkers can now check for function invocation when using TypedDicts:

def purge(queue: str, timeout: float) -> ...:
    ...


class Options(TypedDict):
    queue: str
    timeout: float


class WrongOptions(TypedDict):
    queue: str
    timeout: timedelta


options: Options = ...
purge(**options)  # ✅

wrong_options: WrongOptions = ...
purge(**wrong_options)  # ❌

This feature is necessary in many situations such as cases where we pass through the kwargs. For example, in the fixture example, when we invoke pytest.fixture(**options) the type checker will perform proper type checking.

But we can use it in more creative ways.

Dependency Injection

Let's consider a situation where we have many resources that share some dependencies.

class UserClient:
    def __init__(self, db: Engine, user_service: APIClient) -> None:
        ...


class ProjectClient:
    def __init__(self, db: Engine, user_service: APIClient, project_service: APIClient) -> None:
        ...

We want a way to create all the dependencies in one place and pass in the dependencies.

Essentially we need something that is the union of all kwargs of the resources. That suddernly sounds a lot like a TypedDict:

class Dependencies(TypedDict):
    db: Engine
    user_service: APIClient
    project_service: APIClient


def create_deps(...) -> Dependencies:
    ...

Unfortunately this won't work since UserClient can't take project_service as a kwarg.

To fix this, we need to rewrite the resources such that we accept arbitrary arguments.

class UserClient:
    def __init__(self, ..., **_) -> None:
        ...
...

And then we can do the injection like this:

class ResourceWithMissing:
    def __init__(self, other: Any, **_) -> None:
        ...


def inject(deps: Dependencies):
    UserClient(**deps)  # ✅
    ProjectClient(**deps)  # ✅
    ResourceWithMissing(**deps)  # ❌
    ...


inject(create_deps(...))

With the solution complete, we can now rely on the type system to check the dependency injection to see if any arguments are incorrect or missing.

I will admit that changing resource signature with **_ is not ideal, but this is a smaller change than most dependency injection frameworks. And we get static type checking which a lot of the frameworks won't support.

Upcoming Features

PEP-728 will allow types of extra items to be defined, and a typed dict to be closed meaning no extra items can be defined.

This new change looks like it'll help us define record types more precisely.

I personally haven't thought of many other use cases for it, but as I've demonstrated above it's always worth reading through the PEP and experimenting with the new change.

PEP-705 might already be out by the time you read this. This will allow for read only items to be specified.

This is primarily intended for situations where different typed dicts intuitively should be compatible but potential mutations (deletions) can create problems.

Closing Python Iterators

2024-09-28T00:00:00+01:00

I recently came across a video by mcoding about Python iterators' unfortunate closing behaviour.

To sum up the video, when an iterator is interrupted we intuitively we would expect the exit of a context manager or a finally block to be called but that's not necessarily the case. For example,

def generator_with_close():
    print("Starting")
    try:
        yield
    finally:
        print("Closing")

if we then have the following code interrupt the generator:

for _ in generator_with_close():
    break

We actually get what we expected:

Starting
Closing

But this is due to Python's garbage collection implicitly calling the close method on the generator.

And as the video pointed out, this won't work if we keep any reference to the iterator. Nor is it necessarily the case for all Python implementation that we always garbage collect immediately.

Additionally the asyncio version of this is even more problematic as we won't be able to run the async aclose during garbage collection.

The PEP-533 proposal for fixing this ultimately cannot be accepted due to backwards compatibility concerns.

Calling `.close()`

The best way to resolve this issue is to call .close() explicitly. Or with contextlib.closing. This is guaranteed to work and be correct but places the responsibility on the caller of the generator to close the iterator.

The issue is, whether the iterator needs to be closed is implementation specific. So the caller either has to step into the implementation or verbosely call close for every generator.

Solutions

I believe there to be two correct ways to solve this issue, in lew of the PEP solution. Both method change the callee's signature to avoid any ambiguity.

Option 1

Never use finally or with inside generator functions.

This sounds pretty terrible, but in practice is probably not that big of an issue. The reasons to use with/finally tend to be for closing resources, so we can just shift where we define and close the resource:

with create_db() as db:
    for row in query(db);
        ...

Generally, I believe this is a better pattern. We often use dependencies more than once. And this is a case where explicit is better.

Option 2

I will concede that not all cases are covered by Option 1. For example, I have this following snippet of code to fetch messages from a broker:

def messages():
    for message in broker.messages():
        with set_context(id=message.id):
            yield message.json()

I can lift set_context out of the generator. But it certainly makes more sense to set the id inside as the id is not needed outside.

So I think it's better to have an explicit way to force people to close the generator.

We can achieve this by converting the generator function to a contextmanager function:

from functools import wraps
from contextlib import closing
from typing import Callable, ContextManager, Generator 


def contextgenerator[T: Generator, **P](gen: Callable[P, T]) -> Callable[P, ContextManager[T]]:
    """
    Wraps generators in a context manager.

    This ensures generators are closed properly.
    """
    @wraps(gen)
    def closing_gen(*args: P.args, **kwargs: P.kwargs) -> ContextManager[T]:
        return closing(gen(*args, **kwargs))

    return closing_gen

Then we just need to wrap the generator in a @contextgenerator like this:

@contextgenerator
def messages():
    ...

And called this way:

with messages() as message_iterator:
    for message in message_iterator:
        ...

Here we rely on the decorator to communicate the change in signature to the end user.

But the type checker will also pick up the fact that messages now returns a context manager as opposed to an iterator.

The benefit here is that there is no way to start the generator without entering the context manager, thus ensuring that the generator closes.

The caller can choose to use this when closing behaviour is required, or use a plain generator when not.

For asyncio we have a similar solution:

from functools import wraps
from contextlib import aclosing
from typing import Callable, AsyncContextManager, AsyncGenerator


def asynccontextgenerator[T: AsyncGenerator, **P](gen: Callable[P, T]) -> Callable[P, AsyncContextManager[T]]:
    """
    Wraps generators in a context manager.

    This ensures generators are closed properly.
    """
    @wraps(gen)
    def closing_gen(*args: P.args, **kwargs: P.kwargs) -> AsyncContextManager[T]:
        return aclosing(gen(*args, **kwargs))

    return closing_gen

Conclusion

This issue is something that would catch me out on occasion, thanks to mcoding's video I finally have a good understanding on what's going on.

I think in leu of a more sophisticated fix like PEP-533, we could use a combination of the two approaches I've given to mitigate the issue. This can be done by library authors or developers when defining the generators and doesn't rely on the caller to close them explicitly.

Free Threaded Python With Asyncio

2024-09-19T00:00:00+01:00

With the imminent release of Python 3.13, I wanted to look at the biggest changes coming to Python. I think by far the most exciting feature is free-threaded Python from PEP-703.

As I'm quite late to the party, there's already a lot of articles talking about it. I came accross an excellent article from Simon Willison, which successfully demonstrated parallelism for pure Python functions. Building on top of this I wanted to look at ways to synchronise threads beyond using ThreadPoolExecutor.map.

Prior to Python 3.13 threads were used for IO bound tasks due to the GIL, Asyncio is also used for IO (duh...) and we can wrap threads using asyncio.to_thread. For example,

await asyncio.to_thread(io_bound_task, "first_arg", optional="optional")

Can we use it for CPU bound tasks? Here's a quote lifted directly from Asyncio docs:

Note Due to the GIL, asyncio.to_thread() can typically only be used to make IO-bound functions non-blocking. However, for extension modules that release the GIL or alternative Python implementations that don’t have one, asyncio.to_thread() can also be used for CPU-bound functions. The only thing stopping us was the GIL, so CPU bound tasks shouldn't be a problem. Though it still feels a bit silly given the name AsyncIO.

I updated Simon's test script with AsyncIO modifications:

import argparse
import os
import sys
import time
from asyncio import get_running_loop, run, to_thread, TaskGroup
from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager


@contextmanager
def timer():
    start = time.time()
    yield
    print(f"Elapsed time: {time.time() - start}")


def cpu_bound_task(n):
    """A CPU-bound task that computes the sum of squares up to n."""
    return sum(i * i for i in range(n))


async def main():
    parser = argparse.ArgumentParser(description="Run a CPU-bound task with threads")
    parser.add_argument("--threads", type=int, default=4, help="Number of threads")
    parser.add_argument("--tasks", type=int, default=10, help="Number of tasks")
    parser.add_argument(
        "--size", type=int, default=5000000, help="Task size (n for sum of squares)"
    )
    args = parser.parse_args()

    get_running_loop().set_default_executor(ThreadPoolExecutor(max_workers=args.threads))

    with timer():
        async with TaskGroup() as tg:
            for _ in range(args.tasks):
                tg.create_task(to_thread(cpu_bound_task, args.size))


if __name__ == "__main__":
    print("Parallel with Asyncio")
    print(f"GIL {sys._is_gil_enabled()}")  # type: ignore
    run(main())

And I ran it with and without GIL on my M3 Macbook Pro:

➜ python parallel_asyncio.py
Parallel with Asyncio
GIL False
Elapsed time: 0.5552260875701904

Without free-threading:

➜  python parallel_asyncio.py
Parallel with Asyncio
GIL True
Elapsed time: 1.6787209510803223

The results are as expected, when we use AsyncIO to run our code concurrently we observe the speed-up we'd expect from parallel execution.

But why do this?

Generally when it comes to Asyncio, the discussion around it is always about the performance or lack there of. Whilst performance is certainly important, the ability to reason about concurrency is the biggest benefit.

I personally think the addition of TaskGroup makes AsyncIO concurrent tasks rather easy to reason about, we can use this to sychronise the results of threaded tasks.

Depending on your familiarity with AsyncIO, it might actually be the simplest way to start a thread. This is kind of what makes go routines so convenient in golang.

There's also the possibility to mix IO bound async tasks and CPU bound tasks this way. Something like this:

async with TaskGroup() as tg:
    io_task_future = tg.create_task(fetch(url))
    tg.create_task(to_thread(cpu_bound_task))
    tg.create_task(to_thread(cpu_bound_task))

await to_thread(compute_results, await io_task_future)

Concrete Examples

Right now it's too hard to tell if this is what we want, I believe having some more concrete examples would give us a better idea. I will try to follow up on this if I think of anything.

My First Blog Post

2024-09-18T00:00:00+01:00

My name is Jamie Chang, I'm a software engineer in London. I like all things Python, but I'll try to mix it up sometimes. I decided to start this blog to get some of the ideas out from my head and to practice communicating technical topics.

For this site, I'm starting off with a Python static site generator called Pelican and Github pages to host the site.

Jamie's Blog

Benchmarking Free-threading Performance with Tachyon

Enter Tachyon

Verifying the fix

Closing Thoughts

Free-threading is slowly getting there

Datetime Arithmetic in Python

Basic Arithmetic

Advanced Arithmetic

Closing Thoughts

Caching an Asyncio Function the Easy Way

Closing thoughts

Counting down Chinese New Year with Marimo

Replacing tox with UV

Existing Tooling

uv to rule them all

Extras

Overriding Package Versions

Using an isolated venv

Conclusion

Webasgi - Run python web apps in a browser

Vibe coding

Pyodide

ASGI (Asynchronous Server Gateway Interface)

Browser Emulation

The hard part ...

Everything put together

Why?

Writing a SQS task framework from scratch

Why SQS?

Why not use ...?

Creating my abstractions

Writing the code

Identifying Trade-offs

Creating the client

Error Handling

Extending keep alive

Abstracting consumption

Finally putting everything together

Scaling it

Closing Words

Asyncio is neither fast nor slow

Reviewing the code

Recreating the benchmark

Results

The dilemma of benchmarks

Critical analysis

The new results

Digging even deeper

Conclusion

Future Steps

I was wrong about Subinterpreters

Efficient data sharing between interpreters

Efficient function passing

How should we use interpreters then?

Asyncio

What about aiointerpreters?

InterpreterThreadPoolExecutor

More complex Executors and Runner

Closing words

Trying out marimo notebooks

Fixing lazy imports: Generating Static Types Dynamically

Obscured static information

Providing static information dynamically

Enter pyi files

Stub generation is useful generally

Python 3.14: 3 Asyncio Changes

Asyncio Debugger

Asyncio Introspection

Asyncio in multiple free threads

Finally trying out Mojo 🔥

Implementation

Python like syntax

Ergonomics

Traits

Memory Ownership

Lifetimes

Concurrency

Performance

My feeling around Mojo

What about `aiointerpreters`?

Enter `pyi` files

`uv init`

`uv add`

Optional: `uv lock`

Develop code and `uv run` locally

Interpreter's `call_in_thread`

`functools.Placeholder`