arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krisztián Szűcs <szucs.kriszt...@gmail.com>
Subject Re: [Rust] [DataFusion] Preferences on futures / threading crates?
Date Mon, 04 Mar 2019 12:53:17 GMT
On Mon, Mar 4, 2019 at 5:55 AM Andy Grove <andygrove73@gmail.com> wrote:

> I have been working on a PoC of parallel query execution and it is working
> well, and I am now starting to create PRs for the various refactors
> necessary for this in DataFusion.
>
> I haven't been following the async/await and futures/tokio developments
> lately but for the PoC I used tokio-threadpool which seems simple to use.


> I just wanted to give everyone a chance to give their thoughts on this
> before I get too far with my batch of PRs. Is anyone opposed to using
> tokio-threadpool?
>
DataFusion's tasks should be CPU bound and according to tokio-threadpool's
documentation [1], it is more suitable for event loops:
"It is optimized for the primary Tokio use case of many independent tasks
 with limited computation and with most tasks waiting on I/O."

Rayon seems to follow different semantics, but depending on futures-rs is
considerable, especially because it is maintained by the rust lang nursery.

[1] https://docs.rs/tokio-threadpool/0.1.12/tokio_threadpool/

Cheers, Krisztian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message