arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jed Brown <...@jedbrown.org>
Subject Re: [DISCUSS][C++][Proposal] Threading engine for Arrow
Date Fri, 03 May 2019 15:57:49 GMT
Antoine Pitrou <antoine@python.org> writes:

> Hi Jed,
>
> Le 03/05/2019 à 05:47, Jed Brown a écrit :
>> I would caution to please not commit to the MKL/BLAS model in which the
>> library creates threads internally.  It's a disaster for managing
>> oversubscription and affinity issues among groups of threads and/or
>> multiple processes (e.g., MPI).
>
> Implicit multi-threading is important for user-friendliness reasons
> (especially in higher-level bindings such as the Python-bindings).

I would argue that can be tucked into bindings versus making it
all-or-nothing in the C++ interface.  It's at least worthy of
discussion.

>> The library is then free to use constructs like omp taskgroup/taskloop
>> as granularity warrants; it will never utilize threads that the
>> application didn't explicitly give it.
>
> I don't think we're planning to use OpenMP in Arrow, though Wes probably
> has a better answer.

I was just using it to demonstrate the semantic.  Regardless of what
Arrow uses internally, there will be a cohort of users who are
interested in using Arrow with OpenMP.

Mime
View raw message