hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Robertson <timrobertson...@gmail.com>
Subject Re: multiple scanners on same table/region
Date Tue, 27 Apr 2010 13:39:19 GMT
Hi,

I missed that comment about not being able to see my test code.  Sorry about
that, and here it is:  https://dl.dropbox.com/u/608155/HBase/Test.java

It's just a quick hack and nothing clever, but has 10000 threads (500
running concurrently) running the scan and I have repeated it many times
with no errors (it is only 10 rows, but easy to modify).

Cheers,
Tim






On Tue, Apr 27, 2010 at 3:31 PM, Stack <stack@duboce.net> wrote:

> You may be running into HBASE-2481?
> St.Ack
>
> On Tue, Apr 27, 2010 at 1:30 AM, steven zhuang
> <steven.zhuang.1984@gmail.com> wrote:
> > the first thread can be found at:
> > http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/10074
> >
> >           After some dig, it seems that the problem is caused by long
> pause
> > between two "scanner.next()" call.
> >
> >           In my case the program has to spent a relatively long while to
> > process one row, when it calls "scanner.next()" again, seems that the
> > returned Result will be null even if there should be more rows in the
> > tables.  The rowcaching is set to 1.
> >            I have checked some of the source code, seems there is some
> > mechanism which will call the close() method of the ClientScanner, but I
> am
> > still checking.
> >           I don't know if there is a certain timeout on
> > ClientScanner/ScannerCallable after a row has been successfully returned,
> > seems that timeout cause  my problem here.
> >
> >          Any reply is appreciated.
> >
> >
> > On Fri, Apr 23, 2010 at 11:10 AM, steven zhuang <
> > steven.zhuang.1984@gmail.com> wrote:
> >
> >> hi,
> >>           sorry I start another thread here.  This mail is actually
> answer
> >> to a previous thread "multiple scanners on same table will cause
> problem?
> >> Scan results change among different tries.".
> >>           the mail system kept saying that I am spamming, now it seems
> that
> >> it's right! :)
> >>
> >> here is my reply to people in that thread:
> >>
> >>       I don't know if there is a limit on reads to a single row/region
> in
> >> HBase, but if there is, I might have exceeded that limit.   :(
> >>       in my case, there are hundreds of rows, with dozens of kilos of
> cells
> >> in a row(a 256 MB region may contain 10- rows). for each row, I started
> a
> >> thread on each CF, there are 8 of them, so there might be dozens of
> scanners
> >> on the same region.
> >>       and, to Tim, I could not see your attached mail, my test code is
> >> pasted below, it just iterate on the rows and column families, output
> all
> >> the cells.
> >>
> >>   private void doScan() throws Exception {
> >>  if (null == CopyOfTestTTT234.table) {
> >> return;
> >> }
> >> Scan s = new Scan();
> >>  s.setStartRow("aaa".getBytes());
> >> s.setStopRow("ccc".getBytes());
> >> s.setCaching(CopyOfTestTTT234.ROWCACHING);  //it's 1 here.
> >>  ResultScanner scanner = CopyOfTestTTT234.table.getScanner(s);
> >> while (true) {
> >> Result row = scanner.next();
> >>  if(null==row) break;
> >> String rowKey = new String(row.getRow());
> >> NavigableMap<byte[], NavigableMap<byte[], byte[]>> fm = row
> >>  .getNoVersionMap();
> >> while (fm.size() > 0) {
> >> Entry<byte[], NavigableMap<byte[], byte[]>> ee = fm
> >>  .pollFirstEntry();
> >> String fName = new String(ee.getKey());
> >> NavigableMap<byte[], byte[]> ff = ee.getValue();
> >>  while (ff.size() > 0) {
> >> Entry<byte[], byte[]> cell = ff.pollFirstEntry();
> >> String key = new String(cell.getKey());
> >>  String val = new String(cell.getValue());
> >> System.out.println(Thread.currentThread().hashCode() + "\t"
> >>  + rowKey + "\t" + fName + "\t" + key + "\t" + val);
> >> }
> >> }
> >>  }
> >> }
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message