hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: "Added a key not lexically larger than previous key" when using KeyValueSortReducer
Date Thu, 21 Jan 2010 21:34:37 GMT
Thanks for writing back to the list...  I was wondering.
St.Ack

On Thu, Jan 21, 2010 at 1:29 PM, Menno Luiten <mluiten@artifix.net> wrote:
> Turns out to be a major screw-up on my behalf :)
>
> Passed line number as key instead of actual row key, ofcourse resulting
> in reducing and ordering 1 line at a time. Woops..
>
> It's working fine now :) thanks for your time..
> Menno
>
> On do, 2010-01-21 at 20:35 +0100, Menno Luiten wrote:
>> Manually added the "kv.clone()"-patch and it did not solve the issue,
>> unfortunately.
>>
>> Will most certainly dig deeper as to why this is happening and will post
>> the results here or in JIRA.
>>
>> Thanks
>>
>> On do, 2010-01-21 at 10:47 -0800, stack wrote:
>> > On Thu, Jan 21, 2010 at 9:48 AM, Menno Luiten <mluiten@artifix.net> wrote:
>> >
>> > > ...Then I use the KeyValueSortReducer as Reducer, and
>> > > limit myself to 1 reducer only, to prevent writing a custom Partitioner.
>> > >
>> > >
>> > This all sounds like you are setting things up properly.
>> >
>> >
>> >
>> > > java.io.IOException: Added a key not lexically larger than previous
>> > > key=im2.jpgaspectcolor/blue1---some-color/p.995�������,
>> > > lastkey=im2.jpgaspectcolor/cyan---some-color/p.995�������
>> > >
>> > > This is a complaint that comes up out of hfile if keys are not in strictly
>> > increasing lexicographical order.
>> >
>> >
>> >
>> > > I thought the KeyValueSortReducer would sort the output to prevent these
>> > > errors from occuring.
>> >
>> >
>> > Yes.  Me too.  There is a bug in hbase-0.20.2 KVSR -- hbase-2101 -- but it
>> > would not explain what you are seeing, as far as I can tell.
>> >
>> >
>> > > Am I doing something wrong or should I write my
>> > > own comparator which compares actual row,column pairs?
>> > >
>> > > You shouldn't have to.  See the comparator passed to the TreeMap used
>> > internally in KVSR.  It should do the compare properly.
>> >
>> > I'm not sure whats going wrong.  You seem to have it set up right yet keys
>> > are going into hfile out of order.  Any chance of your digging in to figure
>> > what is going awry?
>> >
>> > Thanks,
>> > St.Ack
>>
>>
>
>
>

Mime
View raw message