arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uwe Korn <uw...@xhochy.com>
Subject Re: Arrow Board Report: Request for Feedback
Date Mon, 11 Jul 2016 07:42:50 GMT
Hello Jacques,

we could also add the Feather format [1] from Wes and Hadley Wickham to 
the list. It uses the Arrow spec to provide a common columnar 
interchange format for Python and R DataFrames. With speed and 
interoperability it has the same core values as Arrow itself. With it we 
have already got a userbase with a native (non-JVM ;)) implementation of 
Arrow. Through its buzz it also spurred some interest in Arrow in the 
Julia community (though I'm not clear how far they actually got with 
their prototype implementation).

Uwe

[1] https://github.com/wesm/feather

On 11.07.16 07:27, Jacques Nadeau wrote:
> An update incorporating your feedback:
>
> ## Description:
>
> Arrow is a columnar in-memory analytics layer designed to accelerate big
> data.
> It houses a set of canonical in-memory representations of flat and
> hierarchical data along with multiple language-bindings for structure
> manipulation. It also provides IPC and common algorithm implementations.
>
> ## Issues:
>
> - There are no issues requiring board attention at this time.
>
> ## Activity:
> - Awareness continues to increase with the community having done
> presentations
>    at various meetups as well as the following conferences: Pydata Paris,
> Hadoop
>    Summit Ireland, Hadoop Summit San Jose and Berlin Buzzwords.
> - The CPP work has made good progress.
> - The cross-project work with Parquet has seen substantial work (both in
> the
>    Parquet project and the Arrow project). This should be a great first
> example
>    proof-of-concept integration showing the benefits of an in-memory
> columnar layer.
> - There has been substantial progress on development of for the IPC /
> memory sharing.
> - Java development has slowed some but appears to be picking up again.
>
> ## Health report:
> - We've seen good discussion and development activity since the last
> report.
> - We need to get to a first release.
>    - Prior to doing so, the community is working on rudimentary integration
> tests between
>      Java and C++ and more formal format specification.
> - More work can be done to make the project approachable to newly
> interested parties by
>    creating additional documentation and quickstart. A sample application
> will also help.
>
> ## PMC changes:
>
> - Currently 17 PMC members.
> - No new PMC members added in the last 3 months.
> - Last PMC addition was Abdel Hakim Deneche on Tue Jan 20 2016
>
> ## Committer base changes:
>
> - Currently 20 committers.
> - No new committers added in the last 3 months
> - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016
>
> ## Releases:
>
> - No releases yet.
>
> ## JIRA activity:
>
> - 71 JIRA tickets created in the last 3 months
> - 40 JIRA tickets closed/resolved in the last 3 months
>
> On Sun, Jul 10, 2016 at 10:07 PM, Wes McKinney <wesmckinn@gmail.com> wrote:
>
>> hi Jacques,
>>
>> I would mention there's been a significant synergy between C++ efforts
>> in Apache Parquet to build a bridge between the projects (which will
>> be a nice proof-of-concept of the benefits of the common in-memory
>> columnar layer).  Uwe Korn has been really active here with 33 commits
>> to Parquet and 21 to Arrow.
>>
>> Micah Kornfield has been contributing significant to the reification
>> process of the specs and IPC / memory sharing procedure. We should
>> prioritize assembling a more fully formed first-cut metadata spec and
>> getting rudimentary integration tests working between the Java and C++
>> implementations before we make a release.
>>
>> - Wes
>>
>>
>> On Sun, Jul 10, 2016 at 10:00 PM, Jacques Nadeau <jacques@apache.org>
>> wrote:
>>> Hello All,
>>>
>>> Can I get some feedback and additional details to add to the Arrow Board
>>> Report. See my draft below:
>>>
>>> thanks,
>>> Jacques
>>>
>>>
>>> ## Description:
>>>
>>> Arrow is a columnar in-memory analytics layer designed to accelerate big
>>> data.
>>> It houses a set of canonical in-memory representations of flat and
>>> hierarchical data along with multiple language-bindings for structure
>>> manipulation. It also provides IPC and common algorithm implementations.
>>>
>>> ## Issues:
>>>
>>> - There are no issues requiring board attention at this time.
>>>
>>> ## Activity:
>>> - Awareness continues to increase with the community having done
>>> presentations
>>>    at various meetups as well as the following conferences: Pydata Paris,
>>> Hadoop
>>>    Summit Ireland, Hadoop Summit San Jose and Berlin Buzzwords.
>>> - The CPP and Python work has seen steady progress and development of a
>>>    Parquet <> Arrow interchange layer is improving. A small number of
>> people
>>> are
>>>    actively moving that forward.
>>> - Java development has slowed some but appears to be picking up again.
>>>
>>> ## Health report:
>>> - Discussion and code activity has been sporadic. We saw a good initial
>>> flurry
>>>    of activity but need to make the project more approachable for new
>> users.
>>> We
>>>    can do this by doing the following:
>>> - We need to get a first release done.
>>> - We need to put together a quickstart and demo application for new
>> users.
>>> ## PMC changes:
>>>
>>> - Currently 17 PMC members.
>>> - No new PMC members added in the last 3 months.
>>> - Last PMC addition was Abdel Hakim Deneche on Tue Jan 20 2016
>>>
>>> ## Committer base changes:
>>>
>>> - Currently 20 committers.
>>> - No new committers added in the last 3 months
>>> - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016
>>>
>>> ## Releases:
>>>
>>> - No releases yet.
>>>
>>> ## JIRA activity:
>>>
>>> - 71 JIRA tickets created in the last 3 months
>>> - 40 JIRA tickets closed/resolved in the last 3 months


Mime
View raw message