accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Accumulo on MapR Continued - LargeRowTest
Date Tue, 17 Apr 2012 22:42:11 GMT
I think it was failing for 84.  It get some data for row 84 from a
block, but its not sure if the next block contains more data for that
row (it does not) so it looks and when it does that fails.

The fact that you can query all rows in the local fs show that the
file in DFS is sounds.  Also, the fact that you can query the
individual row in DFS shows the file is sound.  The file is not
corrupt, but corruption seems to occur under certain circumstances
when reading the file.

The rows that are being looked up in the file are not in sorted order.
 So we are reading randomly from the file.  So it seems that seeking
the file around alot randomly seems to eventually leave something in
MapR in a bad state.  At this point I feel the rfile code is ok and
that it seems like the problem is w/ MapR, because it worked in local
fs and w/ single row in dfs.  What do you think, does this deduction
sound reasonable?

There is one thing that is not being printed out, that is when it
reads the rfile index.  In this case that is there its failing.  The
following information comes from the index, but we are not printing
any debug about reading the index blocks.

  .... DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 .....

You can see in the stack trace that failed that its was trying to read
an index block.

  at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)

Some of the previous failures you sent me occurred when it was trying
to read a data block, not an index block.

At this point I am not sure what else I can do.  Let me know if you
have any more questions.

Keith


On Tue, Apr 17, 2012 at 6:19 PM, Keys Botzum <kbotzum@maprtech.com> wrote:
> Keith,
>
> Great idea. Here are the results.
>
> Run with just record 84 against Hadoop file system:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
84  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
> 17 12:22:22,428 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:22:22,685 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:22:22,690 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:22:22,718 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>
> Notice no exception. When I ran it against the entire file, here's the last bit of output:
>
> 17 12:21:39,605 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:21:39,605 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:21:39,612 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: invalid distance too far back
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method)
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
>        at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
>        at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:96)
>        ... 6 more
>
>
> That seems to imply that the read error is just after row #84.
>
> I then copied the file out of the file system onto a normal linux file system and ran
the test requested:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ cp /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
/tmp
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
file:///tmp/F000000w.rf
> 17 12:24:02,893 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:24:03,039 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
> 17 12:24:03,042 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,049 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: len = 1
> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: data = 1
> row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
> 17 12:24:03,055 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
> 17 12:24:03,125 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 1
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 9
> row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x...
TRUNCATED : [] 1334170440109 false
> 17 12:24:03,129 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:24:03,160 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
> 17 12:24:03,160 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,161 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,163 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: data = 12
> row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ...
TRUNCATED : [] 1334170440112 false
> 17 12:24:03,165 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
> 17 12:24:03,190 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
> 17 12:24:03,190 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,193 [rfile.RelativeKey] DEBUG: data = 15
> row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj...
TRUNCATED : [] 1334170440115 false
> 17 12:24:03,193 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
> 17 12:24:03,240 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
> 17 12:24:03,240 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: data = 32
> row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
> 17 12:24:03,243 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
> 17 12:24:03,267 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
> 17 12:24:03,267 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 35
> row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk...
TRUNCATED : [] 1334170440135 false
> 17 12:24:03,271 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
> 17 12:24:03,271 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
> 17 12:24:03,294 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
> 17 12:24:03,294 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 38
> row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2...
TRUNCATED : [] 1334170440138 false
> 17 12:24:03,298 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
> 17 12:24:03,316 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093
entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED
: [] 1334170440146 false
> 17 12:24:03,316 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,317 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,319 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 46
> row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
> 17 12:24:03,321 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
> 17 12:24:03,352 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
> 17 12:24:03,352 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: data = 58
> row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(...
TRUNCATED : [] 1334170440158 false
> 17 12:24:03,355 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:24:03,373 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
> 17 12:24:03,373 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,375 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 61
> row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M...
TRUNCATED : [] 1334170440161 false
> 17 12:24:03,383 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
> 17 12:24:03,405 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
> 17 12:24:03,406 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 69
> row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]...
TRUNCATED : [] 1334170440169 false
> 17 12:24:03,411 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
> 17 12:24:03,426 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
> 17 12:24:03,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 72
> row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
> 17 12:24:03,432 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,434 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
> 17 12:24:03,462 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:24:03,462 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:24:03,486 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
> 17 12:24:03,491 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
> 17 12:24:03,504 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 95
> row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
> 17 12:24:03,516 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,519 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
> 17 12:24:03,535 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
> 17 12:24:03,535 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 98
> row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5...
TRUNCATED : [] 1334170440198 false
> 17 12:24:03,545 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
> 17 12:24:03,545 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>
> No error. Output looks the same until the exception.
>
> Out of curiosity I took advantage of MapR's NFS access and diff'ed the two files:
> diff /tmp/F000000w.rf /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>
> No difference.
>
> Since we know there is a record 92 after 84, I also tried this just to see what happens:
>
> ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92 file:///tmp/F000000w.rf
> 17 12:32:36,433 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:32:36,664 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
> 17 12:32:36,669 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
> 17 12:32:36,696 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>
> And using the file within the Hadoop file system:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
92  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
> 17 12:33:38,830 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
> 17 12:33:39,104 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
> 17 12:33:39,108 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:33:39,113 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:33:39,114 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
> 17 12:33:39,127 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:33:39,134 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>
> Looks the same to me.
>
> Your standalone test client is very useful. I've run it with MapR client file system
tracing enabled to see if that helps to clarify the issue. I'm sharing that with MapR Engineering
- the trace isn't secret or anything, I just don't think it would mean much to you.
>
> Also, what sleep in the test can I increase to avoid the race condition that leads to
the "table split" error? I ask because I'm trying to run the test w/o compression hoping that
might clarify a few things but the false error just confuses things.
>
> If you have any additional ideas or things you want to try, please let me know. I'm more
than willing to run additional tests.  I will pursue in parallel with MapR engineering.
>
> Once again, thanks for your help,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 17, 2012, at 2:57 PM, Keith Turner wrote:
>
>> Ok,
>>
>> I modified LargeRowDirectQuery to support the local fs and optionallly
>> lookuping an individual row like it used to.  I pushed theses changes
>> to github.  Can you try running the following two test?
>>
>>  * Lookup just row 84    with   LargeRowDirectQuery 84
>> .../t-0000007/F000000w.rf
>>  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf
>>
>> The "table spit points out of range error" is just a timing issue w/
>> the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
>> and sometimes its not long enough.   Thats ok.
>>
>> Keith
>

Mime
View raw message