PyMacaron

Star
Logo

A python microservice framework

Reference:

Overview
Get started
Write an API specification
Implement your API
Docker packaging
Deploy to AWS Beanstalk
JWT authentication
Configuration
Error handling
Asynchronous execution
Database serialisation
Testing
Monitoring

Asynchronous Execution

pymacaron-async is a an extension of pymacaron that seamlessly adds asynchronous task execution based on celery/Redis to your API endpoints.

Install

Install redis in your dev environment:

apt-get install redis-server
pip install redis

Then install ‘pymacaron-async’:

pip install pymacaron-async

And add it to your microservice dependencies:

echo 'pymacaron-async' >> requirements.txt

Usage

Instruct the pymacaron server to start celery workers by adding the following line in ‘pym-config.yaml’:

with_async: true

And make python methods asynchronous by decorating them as follows:

from pymacaron_async import asynctask
from pymacaron_core.swagger.apipool import ApiPool

# Make send_email_async() into an asynchronously
# executable celery task, executed separately
# a celery worker spawned by the PyMacaron framework
@asynctask()
def send_email_async(title):
    # Call 3-rd party emailing API pass
    pass

# API endpoint, defined in your swagger API spec
def do_signup_user():
    do_stuff()

    # Schedule a task sending this email
    # and go on, not waiting for the result
    send_email_async('Welcome !')

    return ApiPool.myapi.model.Ok()

NOTE: it is a good idea to have a naming convention for methods decorated with @asynctask, such as appending ‘_async’ to their names. This will make your code easier to understand.

If you want to delay execution of the asynchronous task:

# Delay 1min (60secs)
@asyntask(delay=60)

That’s all.

Limitations of argument types

Arguments to the asynchronous methods should not be classes instances, since celery won’t serialize them correctly when passing them to the asynchronous task. Dicts, arrays and native python types work fine.

Why and how?

It is considered good behavior for a REST api endpoint to return as quickly as possible. Any long running task it needs to perform, such as image manipulation, 3rd party calls whose replies are not immediately needed, logging or analytics, should be executed asynchronously so as to not delay the HTTP response to the api caller.

Unfortunately, this is not trivially done. Except with pymacaron-async :-)

pymacaron-async adds asynchronous execution capability to pymacaron servers, by spawning in the background a celery worker in which all your api modules are loaded. The celery worker loads your swagger api files in just the same way as the pymacaron server, imports all the modules containing your endpoint implementations and emulates a Flask context including the current user’s authentication token. That way, code executed asynchronously in a celery task sees exactly the same context as code executing synchronously in the endpoint method.