I wish #Python developers didn’t write their libraries with asyncio.get_event_loop() and loop.add_signal_handler everywhere like it’s JavaScript.
I know that managing the lifecycle of your own loops can be a hassle in Python (and I wish that the asyncio API did a better job at it), but writing Python libraries in the async JS flavour is basically a guarantee of being thread-unsafe and thread-incompatible.
add_signal_handler uses the set_wakeup_fd C API under the hood, which only works if the file descriptors are initialized in the interpreter’s main thread.
I’m now working on rewriting the #Telegram integration in #Platypush (it’s been broken since the Telegram bot API migrated to the new asyncio paradigm). Since the new code uses get_event_loop and add_signal_handler everywhere, I have to come up with a complex architecture that involves at least a separate process for the Telegram listener (with its own event loop which uses the bot API), another thread as a bridge to run commands on the loop using run_coroutine_threadsafe, and at least two inter-process queues to send commands and receive responses.
To all the Python library developers out there: I know that it’s tempting to use the asyncio API like it’s JavaScript, and forget that your code may run in multi-threaded contexts. But please, try and go the extra step and manage the lifecycle of your own loops. Prefer new_event_loop+set_event_loop over get_event_loop. Avoid add_signal_handler if possible, instead opt for asyncio.Event with an option to specify a compatible event class (like threading.Event or multiprocess.Event), so the library user can adapt the synchronization mechanism to their code’s concurrency model rather than the other way around.
Many users of your libraries who want to run them in a separate thread will very very grateful for helping you keep their code simpler.
We may still have to use the *_coroutine_threadsafe APIs as a proxy, but at least we won’t have to come up with multi-process architectures and communication protocols to accommodate your library’s constraints.
Fabio Manganiello /
npub13u…dpvgs
2024-03-02 15:51:19
Author Public Key
npub13uunvh7djw9ep54nswkuxlneyee7ehcpc7e53t68krykrdeg6j4qrdpvgsPublished at
2024-03-02 15:51:19Event JSON
{
"id": "b0eba8cd7a069afe254a61188f266bf61017aea77ce361e72672058ca7631707",
"pubkey": "8f39365fcd938b90d2b383adc37e792673ecdf01c7b348af47b0c961b728d4aa",
"created_at": 1709391079,
"kind": 1,
"tags": [
[
"t",
"platypush"
],
[
"t",
"python"
],
[
"t",
"telegram"
],
[
"proxy",
"https://manganiello.social/objects/771a9f7b-87a4-4d28-a47b-2dca2a771e29",
"activitypub"
]
],
"content": "I wish #Python developers didn’t write their libraries with asyncio.get_event_loop() and loop.add_signal_handler everywhere like it’s JavaScript.\n\nI know that managing the lifecycle of your own loops can be a hassle in Python (and I wish that the asyncio API did a better job at it), but writing Python libraries in the async JS flavour is basically a guarantee of being thread-unsafe and thread-incompatible.\n\nadd_signal_handler uses the set_wakeup_fd C API under the hood, which only works if the file descriptors are initialized in the interpreter’s main thread.\n\nI’m now working on rewriting the #Telegram integration in #Platypush (it’s been broken since the Telegram bot API migrated to the new asyncio paradigm). Since the new code uses get_event_loop and add_signal_handler everywhere, I have to come up with a complex architecture that involves at least a separate process for the Telegram listener (with its own event loop which uses the bot API), another thread as a bridge to run commands on the loop using run_coroutine_threadsafe, and at least two inter-process queues to send commands and receive responses.\n\nTo all the Python library developers out there: I know that it’s tempting to use the asyncio API like it’s JavaScript, and forget that your code may run in multi-threaded contexts. But please, try and go the extra step and manage the lifecycle of your own loops. Prefer new_event_loop+set_event_loop over get_event_loop. Avoid add_signal_handler if possible, instead opt for asyncio.Event with an option to specify a compatible event class (like threading.Event or multiprocess.Event), so the library user can adapt the synchronization mechanism to their code’s concurrency model rather than the other way around.\n\nMany users of your libraries who want to run them in a separate thread will very very grateful for helping you keep their code simpler.\n\nWe may still have to use the *_coroutine_threadsafe APIs as a proxy, but at least we won’t have to come up with multi-process architectures and communication protocols to accommodate your library’s constraints.",
"sig": "8d3e6055c62999dbca61f188cdd96699408cf254873107b7c9f8a0b17cef278d5aa97ff4f48e2a94c36ed3d6c3fb50d6b03e3a4fe01dcbcaa310653379c4b0dc"
}