arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neville Dipale <nevilled...@gmail.com>
Subject Re: [Rust] [DataFusion] Preferences on futures / threading crates?
Date Mon, 04 Mar 2019 17:44:59 GMT
I'm a fan of using Rayon. Perhaps if it's not too much work, we could
compare the two

On Mon, 4 Mar 2019 at 15:04, Krisztián Szűcs <szucs.krisztian@gmail.com>
wrote:

> On Mon, Mar 4, 2019 at 5:55 AM Andy Grove <andygrove73@gmail.com> wrote:
>
> > I have been working on a PoC of parallel query execution and it is
> working
> > well, and I am now starting to create PRs for the various refactors
> > necessary for this in DataFusion.
> >
> > I haven't been following the async/await and futures/tokio developments
> > lately but for the PoC I used tokio-threadpool which seems simple to use.
>
>
> > I just wanted to give everyone a chance to give their thoughts on this
> > before I get too far with my batch of PRs. Is anyone opposed to using
> > tokio-threadpool?
> >
> DataFusion's tasks should be CPU bound and according to tokio-threadpool's
> documentation [1], it is more suitable for event loops:
> "It is optimized for the primary Tokio use case of many independent tasks
>  with limited computation and with most tasks waiting on I/O."
>
> Rayon seems to follow different semantics, but depending on futures-rs is
> considerable, especially because it is maintained by the rust lang nursery.
>
> [1] https://docs.rs/tokio-threadpool/0.1.12/tokio_threadpool/
>
> Cheers, Krisztian
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message