arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krisztián Szűcs <>
Subject Re: [Rust] [DataFusion] Preferences on futures / threading crates?
Date Mon, 04 Mar 2019 12:53:17 GMT
On Mon, Mar 4, 2019 at 5:55 AM Andy Grove <> wrote:

> I have been working on a PoC of parallel query execution and it is working
> well, and I am now starting to create PRs for the various refactors
> necessary for this in DataFusion.
> I haven't been following the async/await and futures/tokio developments
> lately but for the PoC I used tokio-threadpool which seems simple to use.

> I just wanted to give everyone a chance to give their thoughts on this
> before I get too far with my batch of PRs. Is anyone opposed to using
> tokio-threadpool?
DataFusion's tasks should be CPU bound and according to tokio-threadpool's
documentation [1], it is more suitable for event loops:
"It is optimized for the primary Tokio use case of many independent tasks
 with limited computation and with most tasks waiting on I/O."

Rayon seems to follow different semantics, but depending on futures-rs is
considerable, especially because it is maintained by the rust lang nursery.


Cheers, Krisztian

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message