incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Hausenblas <michael.hausenb...@gmail.com>
Subject Re: meeting notes 10/22/13
Date Tue, 22 Oct 2013 17:03:01 GMT

> Here are the notes from todays hangout. Michael, can you copy them into the google doc?


Thanks & done.

Cheers,
		Michael

--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/

On 22 Oct 2013, at 17:49, Jason Altekruse <altekrusejason@gmail.com> wrote:

> Hello All,
> 
> Here are the notes from todays hangout. Michael, can you copy them into the
> google doc?
> 
> participants: Jacques, Micheal hausenblas, Lisen Mu, Yash Sharma, Jinfeng,
> Jason Altekruse, Harri, Steven Phillips, Timothy Chen, Julien Hyde
> 
> New employee at MapR: Jinfeng
>    - couple more in the next month
> 
> Jacques:
>    - merged limit
>    - clarify VVs
>        - never access internal state of VV when it is invalid
>    - release notes
> 
> Steven:
>    - ordered partitioner
>        - abstract out distributed cache interface
>    - continue to work on spooling to disk
> Jason:
>    -semi-blocking
>        - look at sort and ordered hash partitioner
> 
> Yash
>    - name of functions
>        - separate class for operators and functions for more clarity
>            - different operators have their own class files
> 
> Lisen
>    - fork of Drill
>        - data pushed form leaves rather than pulled from root
>        - we have been thinking about this same problem
>            - don't want to wait for IO all the time
>            - pre-fetch rather than push
>            - in a join you might get pushed a huge amount of data when you
> aren't ready for it
>            - stream processing
>                - alternative concept around foreman
>                - not quite right for streams
>                - resource allocation
>                    - not as much for resource requirements
>        -HyperLogLog
>            - space saving
>            - acceptable - not precise
>        - data assembly - business logic
>            - approximations will be important to drill
>            - no serious thinking about sampling
>            - certain types of scanners should support sampling
>                - hard with some without reading all data anyway
>                - Hbase might be easier to do a scan
>            - doing it with their own business logic and statistics
>                - hard to generalize
> 
> Hari
>    - not much for updates
>    - pick up with amazon ec2 docs
>        - had problem where we need 8 gigs
>        - cannot get it running on free micro instance
>        - got it working removing the direct memory flag in POM
>        - tim - out of memory exception right away
>            - was this with or without changing the option for direct
> memory?
> 
> Tim
>    - wir patch in
>    - amp labs big data benchmark
>        - having numbers for performance evaluation
>        - set up on their repo for drill datasets
>        - installing HDFS to all of the nodes
>        - doesn't look to complicated
>    - cannot submit sql in distributed mode because of bad optimizer
>    - recent review board patches
>        - describe code more completely
>        - hard to review without docs
>        - Julien - single powerpoint slide per operator
>        - google doc? like the logical plan doc
> 
> 
> Ben
>    - code gen portion of merging receiver
>    - no blockers
>        - getting to code review soon
> 
> Julian
>    - joined hortonworks
>    - working on optiq
>    - helping hive, but also working on Drill
>    - making optiq everything it can be
>    - splitting JDBC into thin client
>        - thinking about it, no implementation yet
>        - right now pushing sorts down to Mongo
>    - jacques - session next week on JDBC?
>    - roadmap on optiq
>        - commit logs tell some of the story
>        - roadmap would be helpful
>        - will put out call for optiq users like drill
>        - put together feature list for next release(s)
>        - next 6 months, want to be agile, but wants to be more predictable
>        - Jinfeng will be working with optimizer and optiq


Mime
View raw message