hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Menno Luiten <mlui...@artifix.net>
Subject Re: "Added a key not lexically larger than previous key" when using KeyValueSortReducer
Date Thu, 21 Jan 2010 21:29:48 GMT
Turns out to be a major screw-up on my behalf :)

Passed line number as key instead of actual row key, ofcourse resulting
in reducing and ordering 1 line at a time. Woops..

It's working fine now :) thanks for your time..
Menno

On do, 2010-01-21 at 20:35 +0100, Menno Luiten wrote:
> Manually added the "kv.clone()"-patch and it did not solve the issue,
> unfortunately.
> 
> Will most certainly dig deeper as to why this is happening and will post
> the results here or in JIRA.
> 
> Thanks
> 
> On do, 2010-01-21 at 10:47 -0800, stack wrote:
> > On Thu, Jan 21, 2010 at 9:48 AM, Menno Luiten <mluiten@artifix.net> wrote:
> > 
> > > ...Then I use the KeyValueSortReducer as Reducer, and
> > > limit myself to 1 reducer only, to prevent writing a custom Partitioner.
> > >
> > >
> > This all sounds like you are setting things up properly.
> > 
> > 
> > 
> > > java.io.IOException: Added a key not lexically larger than previous
> > > key=im2.jpgaspectcolor/blue1---some-color/p.995�������,
> > > lastkey=im2.jpgaspectcolor/cyan---some-color/p.995�������
> > >
> > > This is a complaint that comes up out of hfile if keys are not in strictly
> > increasing lexicographical order.
> > 
> > 
> > 
> > > I thought the KeyValueSortReducer would sort the output to prevent these
> > > errors from occuring.
> > 
> > 
> > Yes.  Me too.  There is a bug in hbase-0.20.2 KVSR -- hbase-2101 -- but it
> > would not explain what you are seeing, as far as I can tell.
> > 
> > 
> > > Am I doing something wrong or should I write my
> > > own comparator which compares actual row,column pairs?
> > >
> > > You shouldn't have to.  See the comparator passed to the TreeMap used
> > internally in KVSR.  It should do the compare properly.
> > 
> > I'm not sure whats going wrong.  You seem to have it set up right yet keys
> > are going into hfile out of order.  Any chance of your digging in to figure
> > what is going awry?
> > 
> > Thanks,
> > St.Ack
> 
> 



Mime
View raw message