So I’ve been struggling a bit trying to think about the big picture. There’s lots of great stuff happening in Trio-land; we’re fixing bugs, landing features, and I’m barely keeping up with PRs (which is great, please keep sending PRs :-)). But, are we focusing on the right things? We love cool features but if the project as a whole doesn’t get traction then eventually that work will be wasted. And it seemed like a good excuse to try out this fancy new forum thing .
I’m mostly thinking about this for prioritizing my own work; as volunteers you all can work on whatever you want :-). But maybe you’ll find it useful too. And I’d certainly appreciate help thinking about it.
What could we do?
Here’s a big brain dump of “strategically relevant” projects. This doesn’t include, like, fixing bugs or general maintenance; I assume we’ll keep doing that. I’m trying to think of stuff that might affect the overall survival and success of the Trio project. I’m probably missing things.
exceptiongroup: This will be a major API break, and likely to be a bumpy ride. We really want to get it done ASAP before we get more popular.
trio-asyncio: This is a huge enabler for making Trio useful now, and bootstrapping past the cycle of no libraries -> no users -> no libraries. The core implementation is great, but to reach its potential we need to do some significant work on the API and docs (see)
more usable threads (#810): Like trio-asyncio, this is really important for bootstrapping.
trimeter: Everyone who starts using Trio wants this in the first five minutes. We should at least get it to the point where there’s a usable version on PyPI.
sansio tooling for writing protocols: People want to know how to implement protocols on top of Trio. People want to do basic obvious stuff like read lines. We need to figure out how this is going to look. It will probably also influence our understanding of the Channel interface.
PEP for preventing yield inside cancel scopes/nurseries: A common issue that’s really hard to explain or catch. There’s a deadline if we want to get this in for 3.8.
interpreter changes to catch unawaited coroutines (bpo-30491): also a common problem, same deadline issue.
docs reorg: Our manual is still organized the way I threw it together when first starting out. It’s straining under the load. We should rework the tutorial, add a cookbook section, and split up the reference section into more pages. Probably the tutorial is the last part (it gets easier to write as exceptiongroup lands, as new libraries appear to use in examples, etc.), but the other parts we can/should do any time.
I particularly like the idea of getting the cookbook section set up ASAP, because once the basic framework is there then contributing examples is highly parallelizable, and when we’re implementing the other features here we can write the cookbook entry in parallel to help us figure out the right API.
write up feedback on ASGI spec: I have a bunch of thoughts here, and the longer I wait, the more people will start using ASGI, and the harder it will be to change anything.
write up “how to port” guide (and/or talk?), reviewing the different approaches: Lots of people are wondering this.
write up “Trio in production at PyPI” case study for my blog: This will be a fabulous marketing tool.
write post on asyncio vs trio for my blog: People want to know what’s going on here. We want to be able to point to this.
systematically try to close potential API
breaker
issues: We need to end up stable. The more deprecations we can get out of the way now, the less bumpy the ride will be for future users. Some of these issues are basically ready to go we just need to do it and get them off our stack.
sphinx theme: This is a bit silly, but I once heard the flask folks cite “having a custom sphinx theme” as one of the things they think was helpful for growing their ecosystem, and it seems surprisingly plausible. And we already have a bunch of sphinx tweaks. We should make our theme a bit more visually distinctive (maybe just what we have now + changing the background colors?), and package it up as a trio-sphinx-theme for Trio projects to use.
auto-release tooling: This keepscomingup. As a whole, the Trio project is super-ambitious and there’s way too much to do and not enough of us to do it, so we need to be efficient or we’ll drown in trivialities. Auto-release tooling removes a major source of trivialities. But setting it up is itself a distracting triviality :-).
linting tools: We should have pylint and mypy plugins (#671). It’s a good selling point (“look, we’re more serious than asyncio” ;-)), and I think it would be surprisingly easy to get a MVP working.
GUI integration: See #399. People often ask about this. It’s one of the few “core” features we’re missing. I think we could do something pretty slick, better than any other popular event loop. Precondition: rewrite the Windows I/O code.
figure out cancel ordering and/or graceful cancellation (#147): Graceful cancellation is an incredibly common issue; having a solid answer here could be a compelling selling point. The cancel ordering part we haven’t dug into as much, but is probably the more important part, since it might change the basic nursery semantics.
Trio-ific ways to access stdin/stdout/stderr: Beginners seem to want to use input() a lot. I don’t blame them! Being able to just say await trio.input() would be super helpful for examples, teaching, etc. Implementing it will be pretty complicated, but then, we’re actually running out of complicated things to add to the core library, so why not?
What did I miss?
What should we do?
Well. You can see why I’m overwhelmed :-). But we have to make some choices; doing everything at once is the same as doing nothing. Any thoughts on how I should pick?
Hi njs. For me, the number one missing feature is the support for http clients and servers, with websocket support. I’m still using aiohttp, tieing me to ayncio for new projects, and this is probably mostly due to lack of examples. Would love a best-practise/example guide showing how to bring up a http server within trio, attach some static files, dynamic content, and a websocket handler at a particular URI within the same server. This has always been the ‘can I use this stack’ checkbox for me. Venerable old twisted, and asyncio+aiohttp pass this test, therefore the ‘other’ protocols of whatever my gateway does gets implemented in those primatives. Last time I looked the handoff from h11 parser to a resource tree, and passing the sockets over to websocket library after the protocol upgrade was still sketchy. I don’t expect to be able to use ASGI or anything thats got a synchronous mindset, but being able to attach await-able handers in some resource tree is kind of the big hurdle.
I’m also a bit weary of using the various stack-adapters to mix the aiohttp stack in the same process, there’s likely loads of edge cases. I’d love to be involved in a trio-first http-out of the box effort.
@shuckc I’m currently using Hypercorn in combination with Quart-Trio (you have to google that url, discourse only allows me to post 2 links :-/) to serve HTTP (it seems to support Websockets, but I haven’t used it yet) and Asks as a client.
It’s still a bit iffy, but it works, you might want to experiment with it and see if it fits your needs.
That being said, I agree that trio needs some kind of “official” way to deal with HTTP easily.
I’m looking at this from my personal perspective as a Trio consumer—so, what I would like to see first, rather than what I would work on. Taking items exclusively from the original list, this a (very) roughly prioritized list of what right now I believe is more important to me:
exceptiongroup: mostly because MultiErrors right now are a bit awkward and I tend to work around them rather than with them
cancel ordering and/or graceful cancellation: my code is full of ad hoc solutions and I would very gladly migrate to a fully supported idiomatic solution to graceful cancellation
docs reorg: I love Trio’s doc but it seems to have grown a bit too fast and information is not always where you expect it to be.
trimeter: again, lot of ad hoc solutions to common problems which probably deserve a common idiom.
sansio tooling: ditto
linting tools: more a nice-to-have thing
This probably doesn’t reflect the well-being of the project or “marketing” needs. Also I’m less interested in http stuff than I used to be. And some very things that might be useful for a lot of people (like asyncio interop) are not mentioned because they don’t fit in my plans now. Still, looking at my subset, I think the common value pattern is basic ergonomics and having this stuff in place would do great to Trio’s value proposition of “async programming for humans”
We love cool features but if the project as a whole doesn’t get traction then eventually that work will be wasted.
This is my main concern, too. I’m going to arbitrarily divide the audience into 2 groups: people who have heard of Trio and want to try it, and people who have not. In the first category, I think the ecosystem and stability are the most important factors, but you’re already well aware so I won’t dwell on that. Everything below is IMHO, of course.
For people in the second category, the target audience is much bigger than just users of other async frameworks; it includes potentially anybody interested in concurrent programming in Python. Concurrency is a very confusing topic for programmers that haven’t encountered it before, and the Python docs don’t do really do anything to explain what the pros and cons of different options are. (And inexplicably don’t mention asyncio at all? Am I missing it?).
As somebody who spent a lot of time teaching this to myself and trying (and failing) with a lot of different projects, I immediately recognized the appeal of Trio’s design because I had just finished wrestling with a complicated asyncio project.
Getting people to want to use Trio requires getting them to understand the nature of their concurrency requirements (CPU bound or I/O bound?), the pros and cons of explicit coroutines vs greenlets/callbacks/futures, the pros and cons of structured concurrency, etc. That’s really a big lift. I don’t have any solution, except that I always thought it would be cool to publish a comprehensive and holistic “Concurrency In Python” manual.[1] If there was any interest in this idea, I would be willing to contribute some time to it.
The things that stand out to me are a mix of things that would be great for me and ones that seem very important to the development of Trio:
urllib3 port: Porting urllib3 would be huge for me and my company. We would be able to use Trio in so many more places through our codebase
trimeter: As said in the original post, this is something basically everyone wants
docs reorg: Easy to read documentation complete with good tutorials and an easy to consult cookbook make for an easy to adopt library. Would be helpful both as someone who’s read a lot of the docs, but also someone trying to get their teammates to use Trio
ASGI feedback: This seems like one of the most important items on the list. If the ASGI spec needs some work and you have ideas, then it’s much better to get those comments/proposals out as soon as possible so they have the best chance of being adopted/implemented
linting tools: I’m a sucker linting/formatting and being able to demonstrate that you pass basic checks when people ask questions about your codebase. It might not actually mean that much, but it certainly makes a lot of people feel better
Really looking forward to seeing what happens with all this!
@mehaase your proposal for a comprehensive guide to concurrency seems really interesting and I might be interested in helping with that as well.
@shuckc per brakhane’s comments, Asks is easy to use with Trio and is a good replacement for aiohttp
in modern applications, you probably want to use asyncio for everything.
It would be great to have access to Prompt Toolkit in Trio apps (and vice versa). A lot of really nice developer-facing apps are made with Prompt Toolkit (including IPython), so bringing it into the Trio ecosystem could introduce a whole new crowd of developers to Trio.
Relatedly, a curses library like Blessed, and a widget library like Urwid or Picotui.
For me, I think the urllib3/requests port will have the biggest impact.
On the HTTP server side I plan on giving hypercorn/quart-trio a go but my concerns are that they’re not battle-hardened / production ready just yet. Only time (and usage) will solve that though so I’m not sure there’s much trio can do to to improve that.
After that, trimeter looks like a very interesting project that I could make use of, but again, it’s too new and doesn’t have the momentum and community behind it to ensure it’ll stick around so its not something I could use for anything other than tinkering just yet.
Given finite resources I’d probably focus on getting the core rock-solid and then I think the urllib3/requests port would have the broadest impact.
It really would! Ideally native support, but in the mean time, FYI, Martijn Faassen was looking at getting it working under trio-asyncio and ran into a bug that needs fixing but I think had some success?
Since I, personally, hardly know how to write readable code, I regret that I won’t become a contributor to this project.
Anyway, I would love to learn how to wrap os.scandir, until someone (who knows how to write readable code) has incorporated it into trio.
The way trio.Path.iterdir works would IMHO fit perfectly (given that I understand it right.) It seems to be an async constructor that calls os.listdir in a worker thread and then returns a regular generator (that does the actual yielding of results).
The main point is the DirEntry objects.
I guess the main problem is the caching of DirEntry.stat(). If the data is in cache, you don’t really wish to await the result. But if not, you wouldn’t wish to block the execution.
I wasn’t sure what your Github username was, so I didn’t CC you, but if you click the [Subscribe] button on the right then that will subscribe you to the discussion.
Having considered the issue a few hours more, I might rephrase myself:
In this particular case, I longed for DirEntries. But the matter is more general.
Maybe there’s a need for something like a cookbook that explains what one must do and think of in the Trio world, when one wishes to await a synchronous function running in another thread.
The one item I rather urgently need is cancellation propagation through threads. I.e. I run something in a thread, the thread calls back to Trio, the called-back code gets cancelled, nothing happens.
GUI integration: See #399 . People often ask about this. It’s one of the few “core” features we’re missing. I think we could do something pretty slick, better than any other popular event loop. Precondition: rewrite the Windows I/O code .
The current leaders in this area include QT and GTK. But I’m optimistic about a future involving http://pybee.org whose bold vision includes native interfaces on “iOS, Android, Windows, MacOS, Linux, Web, and tvOS”. Since the Pybee project is rapidly growing, it might be worthwhile to make sure Trio gets a foothold in that space earlier rather than later.
.
