# Help celery 5 to choose a better trio/async based alternative of python multiprocessing

**URL:** <https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146>\
**Category:** Help and advice\
**Created:** [March 30, 2019, 3:16am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146 "2019-03-30T03:16:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![auvipy](https://yyz2.discourse-cdn.com/free1/user_avatar/trio.discourse.group/auvipy/32/27_2.png) [@auvipy](https://trio.discourse.group/u/auvipy)\
**Post date:** [March 30, 2019, 3:16am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146/1 "2019-03-30T03:16:00Z")

</div>

So far the options we decided are

- Rewrite python3 multiprocessing with trio or asyncio
- Use something like [https://github.com/erdewit/distex](https://github.com/erdewit/distex)
- Rewrite celery billiard with trio/distex/asyncio
- Open to suggestions

---

<div class="post-metadata">

**Author:** ![njs](https://yyz2.discourse-cdn.com/free1/user_avatar/trio.discourse.group/njs/32/11_2.png) [@njs](https://trio.discourse.group/u/njs)\
**Post date:** [March 30, 2019, 4:36am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146/2 "2019-03-30T04:36:57Z")

</div>

What do you need this multiprocessing replacement to do?

---

<div class="post-metadata">

**Author:** ![auvipy](https://yyz2.discourse-cdn.com/free1/user_avatar/trio.discourse.group/auvipy/32/27_2.png) [@auvipy](https://trio.discourse.group/u/auvipy)\
**Post date:** [March 30, 2019, 8:20am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146/3 "2019-03-30T08:20:56Z")

</div>

To be a better replacement of billiard in celery. so initially atleast the feature parity with billiard 3.6.x or may be python multiprocessing with async capacity from trio/asyncio. Mainly for the distributed process poll etc. But there are some more refactoring and new features are under design phase.

---

<div class="post-metadata">

**Author:** ![njs](https://yyz2.discourse-cdn.com/free1/user_avatar/trio.discourse.group/njs/32/11_2.png) [@njs](https://trio.discourse.group/u/njs)\
**Post date:** [March 31, 2019, 3:38am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146/4 "2019-03-31T03:38:44Z")

</div>

The `multiprocessing` module has a _huge_ feature set, that tries to provide every kind of IPC primitive and support every possible IPC pattern. Arguably it’s too huge, and promises more than it (or anyone) can deliver… and most people I know have only used, like, 5% of its features 🙂

If you really need _all_ the features of `multiprocessing`, at the same level of abstraction, then that suggests one kind of advice. If you really have a more specific use case in mind, and would be happy with a library that targets that use case more specifically (or even happier, since a library that’s not trying to be all things to all people can often make better trade-offs), then my advice would be very different. But I need to know which 🙂

What features in `billiard`/`multiprocessing` does `celery` currently use?

---

<div class="post-metadata">

**Author:** ![auvipy](https://yyz2.discourse-cdn.com/free1/user_avatar/trio.discourse.group/auvipy/32/27_2.png) [@auvipy](https://trio.discourse.group/u/auvipy)\
**Post date:** [September 30, 2019, 4:34am UTC](https://trio.discourse.group/t/help-celery-5-to-choose-a-better-trio-async-based-alternative-of-python-multiprocessing/146/5 "2019-09-30T04:34:44Z")

</div>

we decided not to rely on multiprocessing anymore for celery 5! trio & amqp 1.0 will be the core of celery 5. though we are still analyzing what to use to replace multiprocessing.