So far the options we decided are
- Rewrite python3 multiprocessing with trio or asyncio
- Use something like https://github.com/erdewit/distex
- Rewrite celery billiard with trio/distex/asyncio
- Open to suggestions
What do you need this multiprocessing replacement to do?
To be a better replacement of billiard in celery. so initially atleast the feature parity with billiard 3.6.x or may be python multiprocessing with async capacity from trio/asyncio. Mainly for the distributed process poll etc. But there are some more refactoring and new features are under design phase.
The multiprocessing module has a huge feature set, that tries to provide every kind of IPC primitive and support every possible IPC pattern. Arguably it’s too huge, and promises more than it (or anyone) can deliver… and most people I know have only used, like, 5% of its features 
If you really need all the features of multiprocessing, at the same level of abstraction, then that suggests one kind of advice. If you really have a more specific use case in mind, and would be happy with a library that targets that use case more specifically (or even happier, since a library that’s not trying to be all things to all people can often make better trade-offs), then my advice would be very different. But I need to know which
What features in billiard/multiprocessing does celery currently use?
2 Likes
we decided not to rely on multiprocessing anymore for celery 5! trio & amqp 1.0 will be the core of celery 5. though we are still analyzing what to use to replace multiprocessing.
1 Like