Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ACFAA91DD for ; Tue, 17 Apr 2012 18:25:48 +0000 (UTC) Received: (qmail 80053 invoked by uid 500); 17 Apr 2012 18:25:46 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 80005 invoked by uid 500); 17 Apr 2012 18:25:46 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 79806 invoked by uid 99); 17 Apr 2012 18:25:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Apr 2012 18:25:46 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.41] (HELO mail-pz0-f41.google.com) (209.85.210.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Apr 2012 18:25:37 +0000 Received: by dajx4 with SMTP id x4so10188861daj.0 for ; Tue, 17 Apr 2012 11:25:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=1KmhD3sVO8Ag0Oc4Zb2PL0qKX1FG6fU8oBDJmMKpTOI=; b=EHrbrb/ftpelXcST0xSjzp6F1gZliRbn1ayHWANdVzNv7/hWdP5Iezi4dc18qj5o5L Cw4TQmIfAChelKn8sS3SHQZgS8O+H8nmxr/poZiJIx9QD/ti26wl2PH7gl7WzuEdF8xp 1/3CKG5nI7YLHfzvHbqMJj90E8VO0+bSsCOlkN6YjJyIE2bqljlmO8Kb02GCgcUWVDBe 5LezpTTQ+7An/myjYJLuyrF4lmL3zgRWsUfmkRYSHeP5B+rQqvsImdIUTHZK39I7YU+y fJW+inrmJTDc66pK6aRMJXSXfPs1gvybrQI8FO+H4JjzkDPTpi0b+ylWPU9r73xN3IXY Fr0A== Received: by 10.68.201.9 with SMTP id jw9mr17793159pbc.88.1334687114881; Tue, 17 Apr 2012 11:25:14 -0700 (PDT) Received: from [192.168.1.3] (c-68-34-238-6.hsd1.md.comcast.net. [68.34.238.6]) by mx.google.com with ESMTPS id vr9sm6795947pbc.38.2012.04.17.11.25.12 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Apr 2012 11:25:13 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Apple Message framework v1257) Subject: Re: Accumulo on MapR Continued - LargeRowTest From: Keys Botzum In-Reply-To: Date: Tue, 17 Apr 2012 14:25:06 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <6105A360-6FDE-4792-B5E5-61B0D23F7866@maprtech.com> References: <5D6F05D5-7779-40F8-9887-5557C94C8B5B@maprtech.com> <84C21276-7BA4-40BF-B34B-E963DE9F13A1@maprtech.com> <28448BFC-4882-4B60-87E8-F57F9A392C5E@maprtech.com> <802A3202-0BB3-458A-9A69-13E7E7B4A5BF@maprtech.com> <64C055E2-6590-4297-9A40-76679E133E3B@maprtech.com> To: user@accumulo.apache.org X-Mailer: Apple Mail (2.1257) X-Gm-Message-State: ALoCoQnnw33811O6QAJg6i+ttjVGdl4w61GdRGqA2cRntJ+wPHVcTBkNROC3RuYlozL15cS5QNx4 Keith, Once again I thank you for your help. It is much appreciated. I ran the = test you suggested on the existing table and here is the outcome. I ran = it three times and the result was always exactly the same - an = IOException trying to read block 84. /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo = org.apache.accumulo.server.test.functional.LargeRowDirectQuery = /user/mapr/accumulo-SE-test-04-28878/tables/2/t-0000007/F000000w.rf=20 17 10:10:43,981 [util.NativeCodeLoader] INFO : Loaded the native-hadoop = library first key : = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... = TRUNCATED : [] 1334170440146 false last key : = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... = TRUNCATED : [] 1334170440184 false 17 10:10:44,171 [rfile.RFile] DEBUG: Getting block offset=3D3564353 = csize=3D107995 rsize=3D131092 entries=3D1 = key=3Df,RZuff_>.36RjOcx05Y1^qA'g'$@Q: = 7CgMi)$q*Yz[30/HbC.36RjOcx05Y1^qA'g'$@Q:=20 17 10:10:44,193 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,193 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,194 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,194 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,194 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,194 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,194 [rfile.RelativeKey] DEBUG: Read ts 1334170440101 17 10:10:44,194 [rfile.RelativeKey] DEBUG: len =3D 1 17 10:10:44,194 [rfile.RelativeKey] DEBUG: data =3D 1 row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: = 7CgMi)$q*Yz[30/HbCWRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] = 1334170440158 false 17 10:10:44,207 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,207 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,208 [rfile.RelativeKey] DEBUG: data =3D = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl 17 10:10:44,208 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,209 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,209 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,209 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,209 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,209 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,209 [rfile.RelativeKey] DEBUG: Read ts 1334170440158 17 10:10:44,425 [rfile.RFile] DEBUG: Getting block offset=3D2376219 = csize=3D108006 rsize=3D131092 entries=3D1 = key=3DbO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... = TRUNCATED : [] 1334170440109 false 17 10:10:44,426 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:44,433 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,433 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,434 [rfile.RelativeKey] DEBUG: data =3D = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^ 17 10:10:44,434 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,434 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,434 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,434 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,434 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,434 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440109 17 10:10:44,435 [rfile.RelativeKey] DEBUG: len =3D 1 17 10:10:44,435 [rfile.RelativeKey] DEBUG: data =3D 9 row # 9 top key : = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... = TRUNCATED : [] 1334170440109 false 17 10:10:44,435 [rfile.RFile] DEBUG: Getting block offset=3D2484225 = csize=3D107991 rsize=3D131093 entries=3D1 = key=3Dc1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... = TRUNCATED : [] 1334170440192 false 17 10:10:44,441 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,441 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,442 [rfile.RelativeKey] DEBUG: data =3D = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx 17 10:10:44,442 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,443 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,443 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,443 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,443 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,443 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,443 [rfile.RelativeKey] DEBUG: Read ts 1334170440192 17 10:10:44,605 [rfile.RFile] DEBUG: Getting block offset=3D1080062 = csize=3D107959 rsize=3D131093 entries=3D1 key=3D_B = W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OFQIiL;j!AT([DfXJ = o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 = false 17 10:10:44,678 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:44,685 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,685 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,686 [rfile.RelativeKey] DEBUG: data =3D = ]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ o+1M 17 10:10:44,686 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,686 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,686 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,686 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,686 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,686 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,686 [rfile.RelativeKey] DEBUG: Read ts 1334170440115 17 10:10:44,686 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:44,686 [rfile.RelativeKey] DEBUG: data =3D 15 row # 15 top key : ]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ = o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 = false 17 10:10:44,687 [rfile.RFile] DEBUG: Getting block offset=3D539977 = csize=3D108001 rsize=3D131093 entries=3D1 = key=3D^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=3Dw9' = /Ifd5... TRUNCATED : [] 1334170440198 false 17 10:10:44,688 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,688 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,688 [rfile.RelativeKey] DEBUG: data =3D = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M 17 10:10:44,688 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,689 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,689 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,689 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,689 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,689 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,689 [rfile.RelativeKey] DEBUG: Read ts 1334170440198 17 10:10:44,816 [rfile.RFile] DEBUG: Getting block offset=3D3456401 = csize=3D107952 rsize=3D131093 entries=3D1 = key=3DeZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... = TRUNCATED : [] 1334170440132 false 17 10:10:44,816 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:44,820 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,821 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,822 [rfile.RelativeKey] DEBUG: data =3D = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic 17 10:10:44,822 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,822 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,822 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,822 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,822 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,822 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,822 [rfile.RelativeKey] DEBUG: Read ts 1334170440132 17 10:10:44,822 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:44,823 [rfile.RelativeKey] DEBUG: data =3D 32 row # 32 top key : = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... = TRUNCATED : [] 1334170440132 false 17 10:10:44,823 [rfile.RFile] DEBUG: Getting block offset=3D3564353 = csize=3D107995 rsize=3D131092 entries=3D1 = key=3Df,RZuff_>.36RjOcx05Y1^qA'g'$@Q: = 7CgMi)$q*Yz[30/HbC.36RjOcx05Y1^qA'g'$@Q:=20 17 10:10:44,829 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,829 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,829 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,830 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,830 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,830 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,830 [rfile.RelativeKey] DEBUG: Read ts 1334170440101 17 10:10:44,888 [rfile.RFile] DEBUG: Getting block offset=3D2808288 = csize=3D107987 rsize=3D131093 entries=3D1 key=3DcMCOJH I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false 17 10:10:44,901 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,901 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,902 [rfile.RelativeKey] DEBUG: data =3D = d_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/ 17 10:10:44,902 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,902 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,903 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,903 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,903 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,903 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,903 [rfile.RelativeKey] DEBUG: Read ts 1334170440161 17 10:10:44,958 [rfile.RFile] DEBUG: Getting block offset=3D1512018 = csize=3D107992 rsize=3D131093 entries=3D1 = key=3D`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... = TRUNCATED : [] 1334170440138 false 17 10:10:44,958 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:44,963 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,964 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,964 [rfile.RelativeKey] DEBUG: data =3D = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161 17 10:10:44,964 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,965 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,965 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,965 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,965 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,965 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,965 [rfile.RelativeKey] DEBUG: Read ts 1334170440138 17 10:10:44,965 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:44,965 [rfile.RelativeKey] DEBUG: data =3D 38 row # 38 top key : = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... = TRUNCATED : [] 1334170440138 false 17 10:10:44,966 [rfile.RFile] DEBUG: Getting block offset=3D1620010 = csize=3D108006 rsize=3D131093 entries=3D1 = key=3Da8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; = ];.A{tOZbnE-eRkIc:$3G'U=3DD'c... TRUNCATED : [] 1334170440195 false 17 10:10:44,967 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:44,967 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:44,968 [rfile.RelativeKey] DEBUG: data =3D = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4 17 10:10:44,968 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,968 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,968 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,968 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,968 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:44,968 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:44,968 [rfile.RelativeKey] DEBUG: Read ts 1334170440195 17 10:10:45,004 [rfile.RFile] DEBUG: Getting block offset=3D16 = csize=3D107979 rsize=3D131093 entries=3D1 = key=3D\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... = TRUNCATED : [] 1334170440146 false 17 10:10:45,004 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,013 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,013 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,015 [rfile.RelativeKey] DEBUG: data =3D = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc 17 10:10:45,015 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,015 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,015 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,015 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,015 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,015 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,015 [rfile.RelativeKey] DEBUG: Read ts 1334170440146 17 10:10:45,016 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,016 [rfile.RelativeKey] DEBUG: data =3D 46 row # 46 top key : = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... = TRUNCATED : [] 1334170440146 false 17 10:10:45,017 [rfile.RFile] DEBUG: Getting block offset=3D107995 = csize=3D107979 rsize=3D131093 entries=3D1 key=3D]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] = 1334170440172 false 17 10:10:45,021 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,021 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,022 [rfile.RelativeKey] DEBUG: data =3D ]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,F 17 10:10:45,023 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,023 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,023 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,023 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,023 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,023 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,023 [rfile.RelativeKey] DEBUG: Read ts 1334170440172 17 10:10:45,097 [rfile.RFile] DEBUG: Getting block offset=3D3888361 = csize=3D107960 rsize=3D131093 entries=3D1 key=3DfXK1$f#LZUJbYP@0AZ)1d = H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] = 1334170440158 false 17 10:10:45,097 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,101 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,101 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,102 [rfile.RelativeKey] DEBUG: data =3D = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl 17 10:10:45,103 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,103 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,103 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,103 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,103 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,103 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,104 [rfile.RelativeKey] DEBUG: Read ts 1334170440158 17 10:10:45,104 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,104 [rfile.RelativeKey] DEBUG: data =3D 58 row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d = H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] = 1334170440158 false 17 10:10:45,105 [rfile.RFile] DEBUG: Getting block offset=3D3996321 = csize=3D107990 rsize=3D131093 entries=3D1 = key=3Dgj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`..= . TRUNCATED : [] 1334170440184 false 17 10:10:45,110 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,110 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,111 [rfile.RelativeKey] DEBUG: data =3D = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3( 17 10:10:45,111 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,111 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,111 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,111 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,111 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,111 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,112 [rfile.RelativeKey] DEBUG: Read ts 1334170440184 17 10:10:45,151 [rfile.RFile] DEBUG: Getting block offset=3D2916275 = csize=3D108005 rsize=3D131093 entries=3D1 = key=3Dd_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/=3DO = vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false 17 10:10:45,152 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,156 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,157 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,158 [rfile.RelativeKey] DEBUG: data =3D = d_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/ 17 10:10:45,158 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,158 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,158 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,158 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,159 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,159 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,159 [rfile.RelativeKey] DEBUG: Read ts 1334170440161 17 10:10:45,159 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,159 [rfile.RelativeKey] DEBUG: data =3D 61 row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/=3DO = vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false 17 10:10:45,178 [rfile.RFile] DEBUG: Getting block offset=3D3456401 = csize=3D107952 rsize=3D131093 entries=3D1 = key=3DeZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... = TRUNCATED : [] 1334170440132 false 17 10:10:45,183 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,183 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,184 [rfile.RelativeKey] DEBUG: data =3D = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic 17 10:10:45,184 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,184 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,185 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,185 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,185 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,185 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,185 [rfile.RelativeKey] DEBUG: Read ts 1334170440132 17 10:10:45,223 [rfile.RFile] DEBUG: Getting block offset=3D1188021 = csize=3D107982 rsize=3D131093 entries=3D1 = key=3D`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=3D$ = -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false 17 10:10:45,224 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,228 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,228 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,229 [rfile.RelativeKey] DEBUG: data =3D = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs( 17 10:10:45,229 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,229 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,230 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,230 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,230 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,230 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,230 [rfile.RelativeKey] DEBUG: Read ts 1334170440169 17 10:10:45,230 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,230 [rfile.RelativeKey] DEBUG: data =3D 69 row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=3D$ = -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false 17 10:10:45,240 [rfile.RFile] DEBUG: Getting block offset=3D1512018 = csize=3D107992 rsize=3D131093 entries=3D1 = key=3D`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... = TRUNCATED : [] 1334170440138 false 17 10:10:45,247 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,247 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,248 [rfile.RelativeKey] DEBUG: data =3D = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161 17 10:10:45,248 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,248 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,248 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,248 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,248 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,249 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,249 [rfile.RelativeKey] DEBUG: Read ts 1334170440138 17 10:10:45,284 [rfile.RFile] DEBUG: Getting block offset=3D107995 = csize=3D107979 rsize=3D131093 entries=3D1 key=3D]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] = 1334170440172 false 17 10:10:45,284 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,288 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,288 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,289 [rfile.RelativeKey] DEBUG: data =3D ]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,F 17 10:10:45,290 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,290 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,290 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,290 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,290 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,290 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,290 [rfile.RelativeKey] DEBUG: Read ts 1334170440172 17 10:10:45,291 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,291 [rfile.RelativeKey] DEBUG: data =3D 72 row # 72 top key : ]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] = 1334170440172 false 17 10:10:45,300 [rfile.RFile] DEBUG: Getting block offset=3D431981 = csize=3D107996 rsize=3D131093 entries=3D1 = key=3D]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ = o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 = false 17 10:10:45,308 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,308 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,309 [rfile.RelativeKey] DEBUG: data =3D = ]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ o+1M 17 10:10:45,310 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,310 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,310 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,310 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,310 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,310 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,310 [rfile.RelativeKey] DEBUG: Read ts 1334170440115 17 10:10:45,359 [rfile.RFile] DEBUG: Getting block offset=3D3996321 = csize=3D107990 rsize=3D131093 entries=3D1 = key=3Dgj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`..= . TRUNCATED : [] 1334170440184 false 17 10:10:45,360 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:10:45,364 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:10:45,364 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:10:45,366 [rfile.RelativeKey] DEBUG: data =3D = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3( 17 10:10:45,366 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,366 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,366 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,366 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,366 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:10:45,366 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:10:45,366 [rfile.RelativeKey] DEBUG: Read ts 1334170440184 17 10:10:45,366 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:10:45,366 [rfile.RelativeKey] DEBUG: data =3D 84 row # 84 top key : = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... = TRUNCATED : [] 1334170440184 false Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" = died null java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.start.Main$1.run(Main.java:89) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: invalid distance too far back at = org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Nat= ive Method) at = org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompr= essor.java:221) at = org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorSt= ream.java:81) at = org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.j= ava:75) at = java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at = java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.FilterInputStream.read(FilterInputStream.java:66) at java.io.DataInputStream.readInt(DataInputStream.java:370) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(= MultiLevelIndex.java:256) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(M= ultiLevelIndex.java:657) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(Mult= iLevelIndex.java:430) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(Mul= tiLevelIndex.java:477) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400= (MultiLevelIndex.java:436) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLev= elIndex.java:665) at = org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.= java:700) at = org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.j= ava:616) at = org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026) at = org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeR= owDirectQuery.java:83) ... 6 more Just to check for consistent behavior, I reran the large row test to = generate a new set of tables (failed in the usual place) and then ran = your new test program to read through the tablet file. It also failed in = the exact same place: /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo = org.apache.accumulo.server.test.functional.LargeRowDirectQuery = /user/mapr/accumulo-SE-test-04-13876/tables/2/t-0000007/F000000w.rf=20 =85.. 17 10:20:12,078 [rfile.RFile] DEBUG: Getting block offset=3D431981 = csize=3D107996 rsize=3D131093 entries=3D1 = key=3D]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ = o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 = false 17 10:20:12,086 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:20:12,086 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:20:12,087 [rfile.RelativeKey] DEBUG: data =3D = ]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ o+1M 17 10:20:12,087 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,087 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,087 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,088 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,088 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,088 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,088 [rfile.RelativeKey] DEBUG: Read ts 1334170440115 17 10:20:12,134 [rfile.RFile] DEBUG: Getting block offset=3D3996321 = csize=3D107990 rsize=3D131093 entries=3D1 = key=3Dgj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`..= . TRUNCATED : [] 1334170440184 false 17 10:20:12,134 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:20:12,139 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:20:12,139 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:20:12,140 [rfile.RelativeKey] DEBUG: data =3D = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3( 17 10:20:12,141 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,141 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,141 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,141 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,141 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:20:12,141 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:20:12,141 [rfile.RelativeKey] DEBUG: Read ts 1334170440184 17 10:20:12,141 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:20:12,141 [rfile.RelativeKey] DEBUG: data =3D 84 row # 84 top key : = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... = TRUNCATED : [] 1334170440184 false Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" = died null java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.start.Main$1.run(Main.java:89) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: invalid distance too far back at = org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Nat= ive Method) at = org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompr= essor.java:221) at = org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorSt= ream.java:81) at = org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.j= ava:75) at = java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at = java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.FilterInputStream.read(FilterInputStream.java:66) at java.io.DataInputStream.readInt(DataInputStream.java:370) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(= MultiLevelIndex.java:256) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(M= ultiLevelIndex.java:657) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(Mult= iLevelIndex.java:430) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(Mul= tiLevelIndex.java:477) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400= (MultiLevelIndex.java:436) at = org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLev= elIndex.java:665) at = org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.= java:700) at = org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.j= ava:616) at = org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026) at = org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeR= owDirectQuery.java:83) ... 6 more Out of curiosity I ran both tests again with compression in Accumulo = disabled (it is already disabled in MapR). My thought here is that = perhaps compression obscures some of the issue. I cannot explain this = but when run this way, the large row test consistently fails with a = "table spit points out of range error" on the client side with nothing = obvious on the server. It doesn't look like it even created the tablets. = What's odd is that we aren't seeing the EOF error which is what we saw = when running this test before without compression in Accumulo 1.4.0 (and = I just reran 1.4.0 to validate I'm still getting EOF).=20 The only difference I could think of other than debug information is = that I did change the java heap settings for the test driver by editing = TestUtils.py per Eric's recommendation. So just to be sure, I put those = back in. And now I get the usual EOFException in readFully(). = Interesting. This seems to imply some kind of very subtle race condition = that is impacted by the Java heap settings (likely GC behavior). Now = here's the weird part. I then ran your test program over the tablet you = suggested and here's the result - notice it appears to succeed: mapr@SE-test-04:/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ = ./accumulo = org.apache.accumulo.server.test.functional.LargeRowDirectQuery = /user/mapr/accumulo-SE-test-04-1054/tables/2/t-0000007/F000000w.rf=20 =85. row # 72 top key : ]-e3O]_IM$p = D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] = 1334170440172 false 17 10:53:48,483 [rfile.RFile] DEBUG: Getting block offset=3D524443 = csize=3D131093 rsize=3D131093 entries=3D1 = key=3D]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ = o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 = false 17 10:53:48,487 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:53:48,487 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:53:48,487 [rfile.RelativeKey] DEBUG: data =3D = ]I=3D:DIg/1Yg>QIiL;j!AT([DfXJ o+1M 17 10:53:48,487 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,488 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,488 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,488 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,488 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,488 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440115 17 10:53:48,545 [rfile.RFile] DEBUG: Getting block offset=3D4851096 = csize=3D131093 rsize=3D131093 entries=3D1 = key=3Dgj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`..= . TRUNCATED : [] 1334170440184 false 17 10:53:48,545 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:53:48,551 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:53:48,551 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:53:48,552 [rfile.RelativeKey] DEBUG: data =3D = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3( 17 10:53:48,552 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,552 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,552 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,552 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,552 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,552 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,552 [rfile.RelativeKey] DEBUG: Read ts 1334170440184 17 10:53:48,553 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:53:48,553 [rfile.RelativeKey] DEBUG: data =3D 84 row # 84 top key : = gj<=3Da%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... = TRUNCATED : [] 1334170440184 false 17 10:53:48,588 [rfile.RFile] DEBUG: Getting block offset=3D3015559 = csize=3D131093 rsize=3D131093 entries=3D1 = key=3Dc1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... = TRUNCATED : [] 1334170440192 false 17 10:53:48,588 [rfile.RelativeKey] DEBUG: entering fastSkip() 17 10:53:48,594 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 17 10:53:48,594 [rfile.RelativeKey] DEBUG: len =3D 131072 17 10:53:48,595 [rfile.RelativeKey] DEBUG: data =3D = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx 17 10:53:48,595 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,595 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,595 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,595 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,595 [rfile.RelativeKey] DEBUG: len =3D 0 17 10:53:48,595 [rfile.RelativeKey] DEBUG: data =3D=20 17 10:53:48,595 [rfile.RelativeKey] DEBUG: Read ts 1334170440192 17 10:53:48,595 [rfile.RelativeKey] DEBUG: len =3D 2 17 10:53:48,595 [rfile.RelativeKey] DEBUG: data =3D 92 row # 92 top key : = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... = TRUNCATED : [] 1334170440192 false 17 10:53:48,605 [rfile.RFile] DEBUG: Getting block offset=3D3408897 = csize=3D131093 rsize=3D131093 entries=3D1 key=3DcMCOJH The tablet server failed when it tried to do that read. One thing of > interest is that the numbers match up. In the test program and the > tablet server, the block offset and size is the same : "offset=3D3456401= > csize=3D107952 rsize=3D131093". This rules out that the tablet server = is > simply reading data from the wrong location in the file. However the > tablet server failed when it tried to read the block at location > 3456401. This seems to show that sometimes the data can be read but > other times it can not. >=20 > I modified the test program to lookup all rows that exist in the file. > I pushed the changes to github. This is more similar to what is > happening on the tablet server. You no longer need to pass a row # to > the test program, just a file name. Just run LargeRowDirectQuery > . Can you try running this and see if an exception occurs? >=20 > Keith >=20 > On Mon, Apr 16, 2012 at 9:27 PM, Keys Botzum = wrote: >> Keith, >>=20 >> As requested I have run the new test you provided. >>=20 >> Here is the output from ./run.py -t large row -v 10 -d >> console: >>=20 >>=20 >>=20 >>=20 >> logs: >>=20 >>=20 >>=20 >>=20 >> Here is the output from the new test program: >> /opt/keith-turner-accumulo-1.4.0-MapR-630654d/bin$ ./accumulo = org.apache.accumulo.server.test.functional.LargeRowDirectQuery 61 = /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28878/tables/2/ >>=20 >> 16 18:10:08,112 [util.NativeCodeLoader] INFO : Loaded the = native-hadoop library >> 16 18:10:08,320 [rfile.RFile] DEBUG: Getting block offset=3D2916275 = csize=3D108005 rsize=3D131093 entries=3D1 = key=3Dd_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/=3DO = vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false >> 16 18:10:08,324 [rfile.RelativeKey] DEBUG: entering fastSkip() >> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 >> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: len =3D 131072 >> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: data =3D = d_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/ >> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: Read ts 1334170440161 >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len =3D 2 >> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =3D 61 >> top key : d_FH?P(Wl`V.D8A4Vyx=3D[L!u5V6"660/=3DO = vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false >> 16 18:10:08,369 [rfile.RFile] DEBUG: Getting block offset=3D3456401 = csize=3D107952 rsize=3D131093 entries=3D1 = key=3DeZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... = TRUNCATED : [] 1334170440132 false >> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: fieldsSame =3D 0 >> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: len =3D 131072 >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =3D = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len =3D 0 >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =3D >> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: Read ts 1334170440132 >>=20 >> Is this what you expected? It seems rather brief. Should I run it = differently? Out of curiosity I ran it using 60, 62, and 70 as input = just to see what happens: >>=20 >> 60: >>=20 >> 16 18:13:34,076 [util.NativeCodeLoader] INFO : Loaded the = native-hadoop library >>=20 >> 62: >>=20 >> 16 18:14:07,656 [util.NativeCodeLoader] INFO : Loaded the = native-hadoop library >>=20 >> 70: >>=20 >> 16 18:15:49,995 [util.NativeCodeLoader] INFO : Loaded the = native-hadoop library >>=20 >>=20 >> Not sure what this means. >>=20 >>=20 >>=20 >>=20 >> Note that I ran the auto test twice. The reason I did so is because = the first run produced an error I've never seen before. Here's the = error: >> =85. Verify Call Count 2 >> =85. >> Creating Range at row 96 initial bytes are: = )L#W+688U29C81-#4okb#-liSsd[!MB7 >> key =3D = )L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... = TRUNCATED : [] 1334170440096 false >> Creating Range at row 97 initial bytes are: = N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXL >> key =3D = N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=3Dj+fK... = TRUNCATED : [] 1334170440097 false >> Creating Range at row 98 initial bytes are: = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M >> key =3D ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=3Dw9' = /Ifd5... TRUNCATED : [] 1334170440098 false >> Creating Range at row 99 initial bytes are: '?[3F /4g=3D?/#-U3IFa;.n = <('vgazD9 >> key =3D '?[3F /4g=3D?/#-U3IFa;.n = <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y> 16 17:58:24,809 [admin.TableOperations] INFO : Problem with metadata = table, it has a hole / !=3D * ... retrying ... >> ERROR:test.auto:This looks like a stack trace: Thread = "org.apache.accumulo.server.test.functional.FunctionalTest" died null >> java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) >> at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.accumulo.start.Main$1.run(Main.java:89) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.lang.Exception: # of table splits points out of = range, #splits=3D10 table=3Dlr min=3D1 max=3D9 >> at = org.apache.accumulo.server.test.functional.FunctionalTest.checkSplits(Func= tionalTest.java:216) >> at = org.apache.accumulo.server.test.functional.LargeRowTest.test1(LargeRowTest= .java:98) >> at = org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.j= ava:86) >> at = org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalT= est.java:312) >> ... 6 more >>=20 >> Here's the same sequence from the run that yields the "expected" = error: >>=20 >> =85. Verify Call Count 2 >> =85. >> DEBUG:test.auto:out: Creating Range at row 98 initial bytes are: = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M >> DEBUG:test.auto:out: key =3D = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=3Dw9' /Ifd5... = TRUNCATED : [] 1334170440098 false >> DEBUG:test.auto:out: Creating Range at row 99 initial bytes are: = '?[3F /4g=3D?/#-U3IFa;.n <('vgazD9 >> DEBUG:test.auto:out: key =3D '?[3F /4g=3D?/#-U3IFa;.n = <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y> DEBUG:test.auto:out: Verify Call Count 3 >>=20 >>=20 >> There is no corresponding error in the server logs that I can see but = I've save them in case you are curious. I have the logs in case you are = curious. This might be a spurious error but I include it just in case it = gives you an idea. >>=20 >> In both runs there are numerous "incorrect header check" messages in = the logs. That might be significant. >>=20 >> If you need anything, do not hesitate to ask. >>=20 >> Thanks again, >> Keys >> ________________________________ >> Keys Botzum >> Senior Principal Technologist >> WW Systems Engineering >> kbotzum@maprtech.com >> 443-718-0098 >> MapR Technologies >> http://www.mapr.com >>=20 >>=20 >>=20 >> On Apr 16, 2012, at 6:32 PM, Keith Turner wrote: >>=20