hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: DISCUSSION: lets do a developer workshop on near-term work
Date Sat, 18 Jul 2015 18:24:58 GMT
That's not a realistic or useful test scenario, unless the goal is to accelerate queries where
all cells are filtered at the server. 



> On Jul 18, 2015, at 11:02 AM, Anoop John <anoop.hbase@gmail.com> wrote:
> 
> No Andy. 11425 having doc attached to it. At the end of it, we have added
> perf numbers in a cluster testing.  This was done using PE get and scan
> tests with filtering all cells at server (to not consider n/w bandwidth
> constraints)
> 
> -Anoop-
> 
> On Sat, Jul 18, 2015 at 9:30 PM, Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
> 
>> We have some microbenchmarks, not evidence of differences seen from a
>> client application. I'm not saying that microbenchmarks are not totally
>> necessary and a great start - they are - but that they don't measure an end
>> goal. Furthermore unless I've missed one somewhere we don't have a JIRA or
>> design doc that states a clear end goal metric like the strawman I threw
>> together in my previous mail. A measurable system level goal and some data
>> from full cluster testing would go a lot further toward letting all of us
>> evaluate the potential and payoff of the work. In the meantime we should
>> probably be assembling these changes on a branch instead of in trunk, for
>> as long as the goal is not clearly defined and the payoff and potential for
>> perf regressions is untested and unknown.
>> 
>> 
>>> On Jul 18, 2015, at 8:05 AM, Anoop John <anoop.hbase@gmail.com> wrote:
>>> 
>>> Thanks Andy and Lars.  The parent jira has doc attached which contains
>> some
>>> perf gain numbers..  We will be doing more tests in next 2 weeks (before
>>> end of this month) and will publish them.   Yes it will be great if it is
>>> more IST friendly time :-)
>>> 
>>> -Anoop-
>>> 
>>> On Fri, Jul 17, 2015 at 9:44 PM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>>> wrote:
>>> 
>>>>> I can represent your side Ram (and Anoop). I've been known always argue
>>>> both side of a discussion and to never take sides easily (drives some
>> folks
>>>> crazy).
>>>> 
>>>> I can vouch for this (smile)
>>>> 
>>>> I also can offer support for off heaping there. At the same time we do
>>>> have a gap where we can't point to a timeline of improvements (yet,
>> anyway)
>>>> with benchmarks showing gains where your goals need them. For example,
>>>> stock HBase in one JVM can address max N GB for response time
>> distribution
>>>> D; dev version of HBase in off heap branch can address max N' GB for
>>>> distribution D', where N' > N and D > D' (distribution D' statistically
>>>> shows better/lower response times).
>>>> 
>>>> 
>>>> 
>>>>> On Jul 17, 2015, at 6:56 AM, lars hofhansl <larsh@apache.org> wrote:
>>>>> 
>>>>> I'm in favor of anything that improves performance (and preferably
>>>> doesn't set us back into a world that's worse than C due to the lack of
>>>> pointers in Java).Never said "I don't like it", it's just that I'm
>> perhaps
>>>> asking for more numbers and justification in weighing the pros and cons.
>>>>> I can represent your side Ram (and Anoop). I've been known always argue
>>>> both side of a discussion and to never take sides easily (drives some
>> folks
>>>> crazy). And Stack's there too, he yell at me where needed :)
>>>>> 
>>>>> Perhaps we can do it a bit later in the evening so there is a fighting
>>>> chance that folks on IST can participate. I know that some of our folks
>> on
>>>> IST would love to participate in the backup discussion).
>>>>> 
>>>>> Like Enis, I'm also happy to host. We're in Downtown SF. I'd just need
>>>> an approx. number of folks.
>>>>> 
>>>>> -- Lars
>>>>> 
>>>>>    From: ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
>>>>> To: "dev@hbase.apache.org" <dev@hbase.apache.org>; lars hofhansl
<
>>>> larsh@apache.org>
>>>>> Sent: Wednesday, July 15, 2015 10:10 AM
>>>>> Subject: Re: DISCUSSION: lets do a developer workshop on near-term work
>>>>> 
>>>>> Hi
>>>>> What time will it be on August 26th?
>>>>> @LarsYa. I know that you are not generally in favour of this offheaping
>>>> stuff.  May be if we (from India) can attend this meeting remotely your
>>>> thoughts can be discussed and also the current state of this work.
>>>>> RegardsRam
>>>>> 
>>>>> 
>>>>> On Wed, Jul 15, 2015 at 9:28 PM, lars hofhansl <larsh@apache.org>
>> wrote:
>>>>> 
>>>>> Works for me. I'll be back in the Bay Area the week of August 9th.
>>>>> We have done a _lot_ of work on backups as well - ours are more
>>>> complicated as we wanted fast per-tenant restores, so data is "grouped"
>> by
>>>> tenant. Would like to sync up on that (hopefully some of the folks who
>>>> wrote most of the code will be in town, I'll check).
>>>>> 
>>>>> Also interested in the "Time" and "offheap" parts (although you folks
>>>> usually do not like what I think about the offheap efforts :) ).
>>>>> Would like to add the following topics:
>>>>> 
>>>>> 
>>>>> - "Timestamp Resolution". Or making space for more bits in the
>>>> timestamps (happy to cover that, unless it's part of the "Time" topic)
>>>>> 
>>>>> 
>>>>> - "Replication". We found that replication cannot keep up with high
>>>> write loads, due to the fact that replicated is strictly single threaded
>>>> per regionserver (even though we have multiple region servers on the
>> sink
>>>> side)
>>>>> 
>>>>> 
>>>>> - "Spark integration" (Ted Malaska?)
>>>>> 
>>>>> 
>>>>> OK... Out now to make a "bullshit hat".
>>>>> 
>>>>> -- Lars
>>>>> 
>>>>> ________________________________
>>>>> From: Sean Busbey <busbey@cloudera.com>
>>>>> To: dev <dev@hbase.apache.org>
>>>>> Sent: Tuesday, July 14, 2015 7:11 PM
>>>>> Subject: Re: DISCUSSION: lets do a developer workshop on near-term work
>>>>> 
>>>>> 
>>>>> I'm planning to be in the Bay area the week of the 24th of August.
>>>>> 
>>>>> --
>>>>> Sean
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jul 14, 2015 7:53 PM, "Andrew Purtell" <apurtell@apache.org>
>> wrote:
>>>>>> 
>>>>>> I can be up in your area in August.
>>>>>> 
>>>>>>>> On Tue, Jul 14, 2015 at 5:31 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>> 
>>>>>>>> On Tue, Jul 14, 2015 at 3:39 PM, Enis Söztutar <enis.soz@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Sounds good. It has been a while we did the talk-aton.
>>>>>>>> 
>>>>>>>> I'll be off starting 25 of July, so I prefer something next
week if
>>>>>>>> possible.
>>>>>>>> 
>>>>>>>> You ever coming back? If so, when? I'm back on 10th of August
>> (Mikhail
>>>>>> on
>>>>>>> the 20th).
>>>>>>> St.Ack
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> Enis
>>>>>>>> 
>>>>>>>>> On Tue, Jul 14, 2015 at 3:18 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>>> 
>>>>>>>>> Matteo and I were thinking it time devs got together
for a pow-wow.
>>>>>>> There
>>>>>>>>> is a bunch of stuff in flight at the moment (see below
list) and it
>>>>>>> would
>>>>>>>>> be good to meet and whiteboard, surface goodo ideas that
have gone
>>>>>>>> dormant
>>>>>>>>> in JIRA, or revisit designs/proposals out in JIRA-attached
google
>> doc
>>>>>>>> that
>>>>>>>>> need socializing.
>>>>>>>>> 
>>>>>>>>> You can only come if you are wearing your bullshit hat.
>>>>>>>>> 
>>>>>>>>> Topics we'd go over could include:
>>>>>>>>> 
>>>>>>>>> + Our filesystem layout will not work if 1M regions (Matteo/Stack)
>>>>>>>>> + Current state of the offheaping of read path and alternate
>> KeyValue
>>>>>>>>> implementation (Anoop/Ram)
>>>>>>>>> + Append rejigger (Elliott)
>>>>>>>>> + A Pv2-based Assign (Matteo/Steven)
>>>>>>>>> + Splitting meta/1M regions
>>>>>>>>> + The revived Backup (Vladimir)
>>>>>>>>> + Time (Enis)
>>>>>>>>> + The overloaded SequenceId (Stack)
>>>>>>>>> + Upstreaming IT testing (Dima/Sean)
>>>>>>>>> + hbase-2.0.0
>>>>>>>>> 
>>>>>>>>> I put names by folks I know could talk to the topic.
If you want to
>>>>>>> take
>>>>>>>>> over a topic or put your name by one, just say.  Suggest
that
>>>>>>> discussion
>>>>>>>>> lead off with a 5-10minute on current state of
>>>>>>>>> thought/design/implementation.
>>>>>>>>> 
>>>>>>>>> What do others think?
>>>>>>>>> 
>>>>>>>>> What date would suit folks?
>>>>>>>>> 
>>>>>>>>> Anyone want to host?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Matteo and St.Ack
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Best regards,
>>>>>> 
>>>>>>   - Andy
>>>>>> 
>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>>>>>> (via Tom White)
>> 

Mime
View raw message