accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keys Botzum <kbot...@maprtech.com>
Subject Re: Accumulo on MapR Continued - LargeRowTest
Date Wed, 18 Apr 2012 15:00:15 GMT
Keith,


With MapR Engineering's assistance we've been able to determine that this is somehow related
to MapR buffering of file reads. We'll continue working to figure out where the issue really
lies and how to address it.

I'll post a followup here, hopefully soon.

Thanks again for all of your help. It is really appreciated,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com


On Apr 17, 2012, at 6:42 PM, Keith Turner wrote:

> I think it was failing for 84.  It get some data for row 84 from a
> block, but its not sure if the next block contains more data for that
> row (it does not) so it looks and when it does that fails.
> 
> The fact that you can query all rows in the local fs show that the
> file in DFS is sounds.  Also, the fact that you can query the
> individual row in DFS shows the file is sound.  The file is not
> corrupt, but corruption seems to occur under certain circumstances
> when reading the file.
> 
> The rows that are being looked up in the file are not in sorted order.
> So we are reading randomly from the file.  So it seems that seeking
> the file around alot randomly seems to eventually leave something in
> MapR in a bad state.  At this point I feel the rfile code is ok and
> that it seems like the problem is w/ MapR, because it worked in local
> fs and w/ single row in dfs.  What do you think, does this deduction
> sound reasonable?
> 
> There is one thing that is not being printed out, that is when it
> reads the rfile index.  In this case that is there its failing.  The
> following information comes from the index, but we are not printing
> any debug about reading the index blocks.
> 
>  .... DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 .....
> 
> You can see in the stack trace that failed that its was trying to read
> an index block.
> 
>  at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
> 
> Some of the previous failures you sent me occurred when it was trying
> to read a data block, not an index block.
> 
> At this point I am not sure what else I can do.  Let me know if you
> have any more questions.
> 
> Keith
> 
> 
> On Tue, Apr 17, 2012 at 6:19 PM, Keys Botzum <kbotzum@maprtech.com> wrote:
>> Keith,
>> 
>> Great idea. Here are the results.
>> 
>> Run with just record 84 against Hadoop file system:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
84  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 17 12:22:22,428 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:22:22,685 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:22:22,690 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:22:22,718 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 
>> Notice no exception. When I ran it against the entire file, here's the last bit of
output:
>> 
>> 17 12:21:39,605 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:21:39,605 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:21:39,612 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
>> java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: invalid distance too far back
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method)
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
>>        at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
>>        at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:96)
>>        ... 6 more
>> 
>> 
>> That seems to imply that the read error is just after row #84.
>> 
>> I then copied the file out of the file system onto a normal linux file system and
ran the test requested:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ cp /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
/tmp
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
file:///tmp/F000000w.rf
>> 17 12:24:02,893 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,039 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,042 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,049 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
>> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: len = 1
>> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: data = 1
>> row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,055 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
>> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
>> 17 12:24:03,125 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
>> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 1
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 9
>> row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x...
TRUNCATED : [] 1334170440109 false
>> 17 12:24:03,129 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
>> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:24:03,160 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
>> 17 12:24:03,160 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,161 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,163 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: data = 12
>> row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ...
TRUNCATED : [] 1334170440112 false
>> 17 12:24:03,165 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
>> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
>> 17 12:24:03,190 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
>> 17 12:24:03,190 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ
o+1M
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,193 [rfile.RelativeKey] DEBUG: data = 15
>> row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj...
TRUNCATED : [] 1334170440115 false
>> 17 12:24:03,193 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
>> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
>> 17 12:24:03,240 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,240 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: data = 32
>> row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,243 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
>> 17 12:24:03,267 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
>> 17 12:24:03,267 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 35
>> row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk...
TRUNCATED : [] 1334170440135 false
>> 17 12:24:03,271 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
>> 17 12:24:03,271 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 17 12:24:03,294 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
>> 17 12:24:03,294 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 38
>> row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2...
TRUNCATED : [] 1334170440138 false
>> 17 12:24:03,298 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
>> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
>> 17 12:24:03,316 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093
entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED
: [] 1334170440146 false
>> 17 12:24:03,316 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,317 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,319 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 46
>> row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
>> 17 12:24:03,321 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
>> 17 12:24:03,352 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
>> 17 12:24:03,352 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
>> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: data = 58
>> row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(...
TRUNCATED : [] 1334170440158 false
>> 17 12:24:03,355 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:24:03,373 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
>> 17 12:24:03,373 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,375 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 61
>> row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M...
TRUNCATED : [] 1334170440161 false
>> 17 12:24:03,383 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 17 12:24:03,405 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
>> 17 12:24:03,406 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 69
>> row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]...
TRUNCATED : [] 1334170440169 false
>> 17 12:24:03,411 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
>> 17 12:24:03,426 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 72
>> row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,432 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
>> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,434 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ
o+1M
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
>> 17 12:24:03,462 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,462 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,486 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
>> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
>> 17 12:24:03,491 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 17 12:24:03,504 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 95
>> row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,516 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
>> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,519 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
>> 17 12:24:03,535 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
>> 17 12:24:03,535 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 98
>> row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9'
/Ifd5... TRUNCATED : [] 1334170440198 false
>> 17 12:24:03,545 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
>> 17 12:24:03,545 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>> 
>> No error. Output looks the same until the exception.
>> 
>> Out of curiosity I took advantage of MapR's NFS access and diff'ed the two files:
>> diff /tmp/F000000w.rf /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 
>> No difference.
>> 
>> Since we know there is a record 92 after 84, I also tried this just to see what happens:
>> 
>> ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92 file:///tmp/F000000w.rf
>> 17 12:32:36,433 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:32:36,664 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
>> 17 12:32:36,669 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
>> 17 12:32:36,696 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
>> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 
>> And using the file within the Hadoop file system:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
92  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 17 12:33:38,830 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
>> 17 12:33:39,104 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
>> 17 12:33:39,108 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:33:39,113 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:33:39,114 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
>> 17 12:33:39,127 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
>> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:33:39,134 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 
>> Looks the same to me.
>> 
>> Your standalone test client is very useful. I've run it with MapR client file system
tracing enabled to see if that helps to clarify the issue. I'm sharing that with MapR Engineering
- the trace isn't secret or anything, I just don't think it would mean much to you.
>> 
>> Also, what sleep in the test can I increase to avoid the race condition that leads
to the "table split" error? I ask because I'm trying to run the test w/o compression hoping
that might clarify a few things but the false error just confuses things.
>> 
>> If you have any additional ideas or things you want to try, please let me know. I'm
more than willing to run additional tests.  I will pursue in parallel with MapR engineering.
>> 
>> Once again, thanks for your help,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> On Apr 17, 2012, at 2:57 PM, Keith Turner wrote:
>> 
>>> Ok,
>>> 
>>> I modified LargeRowDirectQuery to support the local fs and optionallly
>>> lookuping an individual row like it used to.  I pushed theses changes
>>> to github.  Can you try running the following two test?
>>> 
>>>  * Lookup just row 84    with   LargeRowDirectQuery 84
>>> .../t-0000007/F000000w.rf
>>>  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf
>>> 
>>> The "table spit points out of range error" is just a timing issue w/
>>> the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
>>> and sometimes its not long enough.   Thats ok.
>>> 
>>> Keith
>> 


Mime
View raw message