hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From steven zhuang <steven.zhuang.1...@gmail.com>
Subject Re: multiple scanners on same table/region
Date Tue, 27 Apr 2010 14:13:58 GMT
thank you guys,
   I think Stack is right, what the ticket described is exactly what I met,
the scanner close silently, there is no exception thrown out.
   And hi, Tim, I think you can reproduce my case by sleep 1 or 2 minutes
between two next() calls. :)

   Again,  thanks you guys.

   Looking forward to release 0.20.4. :D


On Tue, Apr 27, 2010 at 9:39 PM, Tim Robertson <timrobertson100@gmail.com>wrote:

> Hi,
>
> I missed that comment about not being able to see my test code.  Sorry
> about
> that, and here it is:  https://dl.dropbox.com/u/608155/HBase/Test.java
>
> It's just a quick hack and nothing clever, but has 10000 threads (500
> running concurrently) running the scan and I have repeated it many times
> with no errors (it is only 10 rows, but easy to modify).
>
> Cheers,
> Tim
>
>
>
>
>
>
> On Tue, Apr 27, 2010 at 3:31 PM, Stack <stack@duboce.net> wrote:
>
> > You may be running into HBASE-2481?
> > St.Ack
> >
> > On Tue, Apr 27, 2010 at 1:30 AM, steven zhuang
> > <steven.zhuang.1984@gmail.com> wrote:
> > > the first thread can be found at:
> > > http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/10074
> > >
> > >           After some dig, it seems that the problem is caused by long
> > pause
> > > between two "scanner.next()" call.
> > >
> > >           In my case the program has to spent a relatively long while
> to
> > > process one row, when it calls "scanner.next()" again, seems that the
> > > returned Result will be null even if there should be more rows in the
> > > tables.  The rowcaching is set to 1.
> > >            I have checked some of the source code, seems there is some
> > > mechanism which will call the close() method of the ClientScanner, but
> I
> > am
> > > still checking.
> > >           I don't know if there is a certain timeout on
> > > ClientScanner/ScannerCallable after a row has been successfully
> returned,
> > > seems that timeout cause  my problem here.
> > >
> > >          Any reply is appreciated.
> > >
> > >
> > > On Fri, Apr 23, 2010 at 11:10 AM, steven zhuang <
> > > steven.zhuang.1984@gmail.com> wrote:
> > >
> > >> hi,
> > >>           sorry I start another thread here.  This mail is actually
> > answer
> > >> to a previous thread "multiple scanners on same table will cause
> > problem?
> > >> Scan results change among different tries.".
> > >>           the mail system kept saying that I am spamming, now it seems
> > that
> > >> it's right! :)
> > >>
> > >> here is my reply to people in that thread:
> > >>
> > >>       I don't know if there is a limit on reads to a single row/region
> > in
> > >> HBase, but if there is, I might have exceeded that limit.   :(
> > >>       in my case, there are hundreds of rows, with dozens of kilos of
> > cells
> > >> in a row(a 256 MB region may contain 10- rows). for each row, I
> started
> > a
> > >> thread on each CF, there are 8 of them, so there might be dozens of
> > scanners
> > >> on the same region.
> > >>       and, to Tim, I could not see your attached mail, my test code is
> > >> pasted below, it just iterate on the rows and column families, output
> > all
> > >> the cells.
> > >>
> > >>   private void doScan() throws Exception {
> > >>  if (null == CopyOfTestTTT234.table) {
> > >> return;
> > >> }
> > >> Scan s = new Scan();
> > >>  s.setStartRow("aaa".getBytes());
> > >> s.setStopRow("ccc".getBytes());
> > >> s.setCaching(CopyOfTestTTT234.ROWCACHING);  //it's 1 here.
> > >>  ResultScanner scanner = CopyOfTestTTT234.table.getScanner(s);
> > >> while (true) {
> > >> Result row = scanner.next();
> > >>  if(null==row) break;
> > >> String rowKey = new String(row.getRow());
> > >> NavigableMap<byte[], NavigableMap<byte[], byte[]>> fm = row
> > >>  .getNoVersionMap();
> > >> while (fm.size() > 0) {
> > >> Entry<byte[], NavigableMap<byte[], byte[]>> ee = fm
> > >>  .pollFirstEntry();
> > >> String fName = new String(ee.getKey());
> > >> NavigableMap<byte[], byte[]> ff = ee.getValue();
> > >>  while (ff.size() > 0) {
> > >> Entry<byte[], byte[]> cell = ff.pollFirstEntry();
> > >> String key = new String(cell.getKey());
> > >>  String val = new String(cell.getValue());
> > >> System.out.println(Thread.currentThread().hashCode() + "\t"
> > >>  + rowKey + "\t" + fName + "\t" + key + "\t" + val);
> > >> }
> > >> }
> > >>  }
> > >> }
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message