accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keys Botzum <kbot...@maprtech.com>
Subject Re: Accumulo on MapR Continued - LargeRowTest
Date Tue, 17 Apr 2012 18:25:06 GMT
Keith,

Once again I thank you for your help. It is much appreciated.  I ran the test you suggested
on the existing table and here is the outcome. I ran it three times and the result was always
exactly the same - an IOException trying to read block 84.

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
 /user/mapr/accumulo-SE-test-04-28878/tables/2/t-0000007/F000000w.rf 
17 10:10:43,981 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED
: [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED
: [] 1334170440184 false
17 10:10:44,171 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
17 10:10:44,176 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,184 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,185 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,193 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 10:10:44,193 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,193 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 1
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 1
row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
17 10:10:44,200 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
17 10:10:44,207 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,207 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,208 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 10:10:44,208 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 10:10:44,425 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
17 10:10:44,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,433 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 10:10:44,435 [rfile.RelativeKey] DEBUG: len = 1
17 10:10:44,435 [rfile.RelativeKey] DEBUG: data = 9
row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
17 10:10:44,435 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
17 10:10:44,441 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,441 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,442 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 10:10:44,442 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 10:10:44,605 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
17 10:10:44,606 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,612 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,612 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,613 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 12
row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ...
TRUNCATED : [] 1334170440112 false
17 10:10:44,615 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
17 10:10:44,621 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,621 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 10:10:44,622 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,622 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,623 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,623 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,623 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 10:10:44,678 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
17 10:10:44,678 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,685 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,685 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 15
row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj...
TRUNCATED : [] 1334170440115 false
17 10:10:44,687 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
17 10:10:44,688 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,688 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,688 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 10:10:44,688 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 10:10:44,816 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
17 10:10:44,816 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,820 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,821 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,823 [rfile.RelativeKey] DEBUG: data = 32
row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
17 10:10:44,823 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092
entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2...
TRUNCATED : [] 1334170440101 false
17 10:10:44,828 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,828 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,829 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 10:10:44,829 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,829 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,829 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,830 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,830 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,830 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,830 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 10:10:44,888 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
17 10:10:44,888 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,893 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,893 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,894 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 10:10:44,896 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,896 [rfile.RelativeKey] DEBUG: data = 35
row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk...
TRUNCATED : [] 1334170440135 false
17 10:10:44,897 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
17 10:10:44,901 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,901 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,902 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 10:10:44,902 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,902 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,903 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,903 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 10:10:44,958 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
17 10:10:44,958 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,963 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,964 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,964 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 10:10:44,964 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 38
row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
17 10:10:44,966 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
17 10:10:44,967 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,967 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 10:10:45,004 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093 entries=1
key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED :
[] 1334170440146 false
17 10:10:45,004 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,013 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,013 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
17 10:10:45,016 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,016 [rfile.RelativeKey] DEBUG: data = 46
row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0...
TRUNCATED : [] 1334170440146 false
17 10:10:45,017 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
17 10:10:45,021 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,021 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,022 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 10:10:45,097 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093
entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED
: [] 1334170440158 false
17 10:10:45,097 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,101 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,101 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,102 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,104 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 10:10:45,104 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,104 [rfile.RelativeKey] DEBUG: data = 58
row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(...
TRUNCATED : [] 1334170440158 false
17 10:10:45,105 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
17 10:10:45,110 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,110 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,112 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:10:45,151 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
17 10:10:45,152 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,156 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,157 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 10:10:45,158 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,158 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,159 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,159 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,159 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 10:10:45,159 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,159 [rfile.RelativeKey] DEBUG: data = 61
row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M...
TRUNCATED : [] 1334170440161 false
17 10:10:45,178 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
17 10:10:45,183 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,183 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,184 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 10:10:45,184 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,184 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,185 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,185 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 10:10:45,223 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093
entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED
: [] 1334170440169 false
17 10:10:45,224 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,228 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,228 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,229 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 10:10:45,229 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,229 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 69
row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]...
TRUNCATED : [] 1334170440169 false
17 10:10:45,240 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093
entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED
: [] 1334170440138 false
17 10:10:45,247 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,247 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,249 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,249 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 10:10:45,284 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093
entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
17 10:10:45,284 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,288 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,288 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,289 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 10:10:45,291 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,291 [rfile.RelativeKey] DEBUG: data = 72
row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
17 10:10:45,300 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
17 10:10:45,308 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,308 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,309 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:10:45,359 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
17 10:10:45,360 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,364 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,364 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: invalid distance too far back
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
	at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
	at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:83)
	... 6 more


Just to check for consistent behavior, I reran the large row test to generate a new set of
tables (failed in the usual place) and then ran your new test program to read through the
tablet file. It also failed in the exact same place:

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
 /user/mapr/accumulo-SE-test-04-13876/tables/2/t-0000007/F000000w.rf 

…..
17 10:20:12,078 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
17 10:20:12,086 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:20:12,086 [rfile.RelativeKey] DEBUG: len = 131072
17 10:20:12,087 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:20:12,087 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,087 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,087 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,088 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,088 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,088 [rfile.RelativeKey] DEBUG: data = 

17 10:20:12,088 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:20:12,134 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
17 10:20:12,134 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:20:12,139 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:20:12,139 [rfile.RelativeKey] DEBUG: len = 131072
17 10:20:12,140 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 2
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: invalid distance too far back
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
	at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
	at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:83)
	... 6 more



Out of curiosity I ran both tests again with compression in Accumulo disabled (it is already
disabled in MapR). My thought here is that perhaps compression obscures some of the issue.
I cannot explain this but when run this way, the large row test consistently fails with a
"table spit points out of range error" on the client side with nothing obvious on the server.
It doesn't look like it even created the tablets. What's odd is that we aren't seeing the
EOF error which is what we saw when running this test before without compression in Accumulo
1.4.0 (and I just reran 1.4.0 to validate I'm still getting EOF). 

The only difference I could think of other than debug information is that I did change the
java heap settings for the test driver by editing TestUtils.py per Eric's recommendation.
So just to be sure, I put those back in. And now I get the usual EOFException in readFully().
Interesting. This seems to imply some kind of very subtle race condition that is impacted
by the Java heap settings (likely GC behavior). Now here's the weird part. I then ran your
test program over the tablet you suggested and here's the result - notice it appears to succeed:

mapr@SE-test-04:/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
 /user/mapr/accumulo-SE-test-04-1054/tables/2/t-0000007/F000000w.rf 
….
row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><...
TRUNCATED : [] 1334170440172 false
17 10:53:48,483 [rfile.RFile] DEBUG: Getting block offset=524443 csize=131093 rsize=131093
entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED
: [] 1334170440115 false
17 10:53:48,487 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,487 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,487 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:53:48,487 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:53:48,545 [rfile.RFile] DEBUG: Getting block offset=4851096 csize=131093 rsize=131093
entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
17 10:53:48,545 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,551 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,551 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:53:48,553 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,553 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`...
TRUNCATED : [] 1334170440184 false
17 10:53:48,588 [rfile.RFile] DEBUG: Getting block offset=3015559 csize=131093 rsize=131093
entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED
: [] 1334170440192 false
17 10:53:48,588 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,594 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,594 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 92
row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%...
TRUNCATED : [] 1334170440192 false
17 10:53:48,605 [rfile.RFile] DEBUG: Getting block offset=3408897 csize=131093 rsize=131093
entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED
: [] 1334170440135 false
17 10:53:48,609 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 10:53:48,651 [rfile.RFile] DEBUG: Getting block offset=1966639 csize=131093 rsize=131093
entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
17 10:53:48,651 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,660 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,660 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 10:53:48,661 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,661 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,662 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,662 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,662 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 10:53:48,662 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,662 [rfile.RelativeKey] DEBUG: data = 95
row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c...
TRUNCATED : [] 1334170440195 false
17 10:53:48,688 [rfile.RFile] DEBUG: Getting block offset=2884467 csize=131092 rsize=131092
entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED
: [] 1334170440109 false
17 10:53:48,690 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,690 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,690 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 10:53:48,728 [rfile.RFile] DEBUG: Getting block offset=655536 csize=131093 rsize=131093
entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440198 false
17 10:53:48,728 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,733 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,733 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 10:53:48,735 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,735 [rfile.RelativeKey] DEBUG: data = 98
row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5...
TRUNCATED : [] 1334170440198 false
17 10:53:48,750 [rfile.RFile] DEBUG: Getting block offset=1311115 csize=131093 rsize=131093
entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED
: [] 1334170440112 false
17 10:53:48,754 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,754 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: Read ts 1334170440112

If you compare this "success" case to the earlier failure, all the outputs are identical except
for the block offset (which I think is related to compression) and size until the failure.
Our of curiosity I ran your test program over .../tables/2/t-000000[0-8]/* and all succeeded.

As another test, I left in the Java heap settings, but allowed compression to be used (by
editing TestUtils.py to remove the property I added to disable compression).  In this case
I get the usual IOException "invalid distance too far back" when I run the large row auto
test.  If I run your test client to scan the table it gives the same error as the first test
on row #84 - java.io.IOException: invalid distance too far back.



Something very strange is happening here. There is an overwhelming amount of information I
can send you, what would be useful? I can run any of the following tests against any build
you like:
- compression on, heap for tests large
- compression on, heap default (small) for tests
- compression off, heap for tests large
- compression on, heap default (small) for tests

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 17, 2012, at 12:37 PM, Keith Turner wrote:

> The tablet server failed when it tried to do that read.  One thing of
> interest is that the numbers match up.  In the test program and the
> tablet server, the block offset and size is the same : "offset=3456401
> csize=107952 rsize=131093".  This rules out that the tablet server is
> simply reading data from the wrong location in the file.  However the
> tablet server failed when it tried to read the block at location
> 3456401.  This seems to show that sometimes the data can be read but
> other times it can not.
> 
> I modified the test program to lookup all rows that exist in the file.
> I pushed the changes to github.  This is more similar to what is
> happening on the tablet server.  You no longer need to pass a row # to
> the test program, just a file name.  Just run LargeRowDirectQuery
> <file>.  Can you try running this and see if an exception occurs?
> 
> Keith
> 
> On Mon, Apr 16, 2012 at 9:27 PM, Keys Botzum <kbotzum@maprtech.com> wrote:
>> Keith,
>> 
>> As requested I have run the new test you provided.
>> 
>> Here is the output from ./run.py -t large row -v 10 -d
>> console:
>> 
>> 
>> 
>> 
>> logs:
>> 
>> 
>> 
>> 
>> Here is the output from the new test program:
>> /opt/keith-turner-accumulo-1.4.0-MapR-630654d/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
61 /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28878/tables/2/
>> 
>> 16 18:10:08,112 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 16 18:10:08,320 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093
entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED
: [] 1334170440161 false
>> 16 18:10:08,324 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: len = 131072
>> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 2
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data = 61
>> top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M...
TRUNCATED : [] 1334170440161 false
>> 16 18:10:08,369 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093
entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:...
TRUNCATED : [] 1334170440132 false
>> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: len = 131072
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 
>> Is this what you expected? It seems rather brief. Should I run it differently? Out
of curiosity I ran it using 60, 62, and 70 as input just to see what happens:
>> 
>> 60:
>> 
>> 16 18:13:34,076 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 62:
>> 
>> 16 18:14:07,656 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 70:
>> 
>> 16 18:15:49,995 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 
>> Not sure what this means.
>> 
>> 
>> 
>> 
>> Note that I ran the auto test twice. The reason I did so is because the first run
produced an error I've never seen before. Here's the error:
>> …. Verify Call Count 2
>> ….
>> Creating Range at row 96 initial bytes are: )L#W+688U29C81-#4okb#-liSsd[!MB7
>> key = )L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... TRUNCATED
: [] 1334170440096 false
>> Creating Range at row 97 initial bytes are: N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXL
>> key = N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=j+fK... TRUNCATED
: [] 1334170440097 false
>> Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED
: [] 1334170440098 false
>> Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n <('vgazD9
>> key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M...
TRUNCATED : [] 1334170440099 false
>> 16 17:58:24,809 [admin.TableOperations] INFO : Problem with metadata table, it has
a hole / != * ... retrying ...
>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest"
died null
>> java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.Exception: # of table splits points out of range, #splits=10
table=lr min=1 max=9
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.checkSplits(FunctionalTest.java:216)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test1(LargeRowTest.java:98)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:86)
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>        ... 6 more
>> 
>> Here's the same sequence from the run that yields the "expected" error:
>> 
>> …. Verify Call Count 2
>> ….
>> DEBUG:test.auto:out: Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> DEBUG:test.auto:out: key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9'
/Ifd5... TRUNCATED : [] 1334170440098 false
>> DEBUG:test.auto:out: Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n
<('vgazD9
>> DEBUG:test.auto:out: key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M...
TRUNCATED : [] 1334170440099 false
>> DEBUG:test.auto:out: Verify Call Count 3
>> 
>> 
>> There is no corresponding error in the server logs that I can see but I've save them
in case you are curious. I have the logs in case you are curious. This might be a spurious
error but I include it just in case it gives you an idea.
>> 
>> In both runs there are numerous "incorrect header check" messages in the logs. That
might be significant.
>> 
>> If you need anything, do not hesitate to ask.
>> 
>> Thanks again,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> On Apr 16, 2012, at 6:32 PM, Keith Turner wrote:
>> 


Mime
View raw message