Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Sat, 22 Aug 2015 23:33:45 +0000 (UTC)
From: "Josh Elser (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12857696.1440114716000.138503.1440286425993@Atlassian.JIRA>
In-Reply-To: <JIRA.12857696.1440114716000@Atlassian.JIRA>
References: <JIRA.12857696.1440114716000@Atlassian.JIRA>
 <JIRA.12857696.1440114716737@arcas>
Subject: [jira] [Commented] (ACCUMULO-3967) bulk import loses records when
 loading pre-split table
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708220#comment-14708220 ] 

Josh Elser commented on ACCUMULO-3967:
--------------------------------------

Looking at metadata, it appears we didn't get the reference for three tablets: {{4;02}}, {{4;11}}, {{4;23}}:

{noformat}
4;01 file:hdfs://localhost:8020/accumulo/tables/4/t-0000030/A000003n.rf []    776344,41660
4;03 file:hdfs://localhost:8020/accumulo/tables/4/t-000002v/A00000at.rf []    773215,41469
4;04 file:hdfs://localhost:8020/accumulo/tables/4/t-000002z/A000003m.rf []    775453,41598
4;05 file:hdfs://localhost:8020/accumulo/tables/4/t-0000038/A00000as.rf []    772770,41451
4;06 file:hdfs://localhost:8020/accumulo/tables/4/t-000002s/A00000ar.rf []    776667,41680
4;07 file:hdfs://localhost:8020/accumulo/tables/4/t-000002x/A000003o.rf []    779467,41810
4;08 file:hdfs://localhost:8020/accumulo/tables/4/t-0000035/A000003p.rf []    776883,41688
4;09 file:hdfs://localhost:8020/accumulo/tables/4/t-000002w/A000003q.rf []    775616,41611
4;10 file:hdfs://localhost:8020/accumulo/tables/4/t-000002y/A000003r.rf []    782617,41975
4;12 file:hdfs://localhost:8020/accumulo/tables/4/t-000002q/A00000au.rf []    772907,41461
4;13 file:hdfs://localhost:8020/accumulo/tables/4/t-0000034/A000003s.rf []    773722,41509
4;14 file:hdfs://localhost:8020/accumulo/tables/4/t-000003b/A00000av.rf []    773786,41518
4;15 file:hdfs://localhost:8020/accumulo/tables/4/t-000002t/A00000aw.rf []    778756,41789
4;16 file:hdfs://localhost:8020/accumulo/tables/4/t-0000033/A000003t.rf []    772805,41459
4;17 file:hdfs://localhost:8020/accumulo/tables/4/t-000003c/A00000ax.rf []    776262,41643
4;18 file:hdfs://localhost:8020/accumulo/tables/4/t-000002r/A00000ay.rf []    778681,41774
4;19 file:hdfs://localhost:8020/accumulo/tables/4/t-0000031/A000003u.rf []    774599,41555
4;20 file:hdfs://localhost:8020/accumulo/tables/4/t-0000036/A00000az.rf []    781791,41936
4;21 file:hdfs://localhost:8020/accumulo/tables/4/t-000002u/A00000b0.rf []    785018,42116
4;22 file:hdfs://localhost:8020/accumulo/tables/4/t-0000032/A000003v.rf []    780546,41877
4< file:hdfs://localhost:8020/accumulo/tables/4/default_tablet/A00000b1.rf []    778454,41757
{noformat}

.. given 24 tablets in the table:

{noformat}
root@accumulo17 accumulo.metadata> getsplits -t loadtest.T_Test
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{noformat}

> bulk import loses records when loading pre-split table
> ------------------------------------------------------
>
>                 Key: ACCUMULO-3967
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3967
>             Project: Accumulo
>          Issue Type: Bug
>          Components: client, tserver
>    Affects Versions: 1.7.0
>         Environment: generic hadoop 2.6.0, zookeeper 3.4.6 on redhat 6.7
> 7 node cluster
>            Reporter: Edward Seidl
>            Priority: Blocker
>             Fix For: 1.7.1, 1.8.0
>
>
> I just noticed that some records I'm loading via importDirectory go missing.  After a lot of digging around trying to reproduce the problem, I discovered that it occurs most frequently when loading a table that I have just recently added splits to.  In the tserver logs I'll see messages like 
> 20 16:25:36,805 [client.BulkImporter] INFO : Could not assign 1 map files to tablet 1xw;18;17 because : Not Serving Tablet .  Will retry ...
>  
> or
> 20 16:25:44,826 [tserver.TabletServer] INFO : files [hdfs://xxxx:54310/accumulo/tables/1xw/b-00jnmxe/I00jnmxq.rf] not imported to 1xw;03;02: tablet 1xw;03;02 is closed
> these appear after messages about unloading tablets...it seems that tablets are being redistributed at the same time as the bulk import is occuring.
> Steps to reproduce
> 1) I run a mapreduce job that produces random data in rfiles
> 2) copy the rfiles to an import directory
> 3) create table or deleterows -f
> 4) addsplits
> 5) importdirectory
> I have also performed the above completely within the mapreduce job, with similar results.  The difference with the mapreduce job is that the time between adding splits and the import directory is minutes rather than seconds.
> my current test creates 1000000 records, and after the importdirectory returns a count of rows will be anywhere from ~800000 to 1000000.
> With my original workflow, I found that re-importing the same set of rfiles three times would eventually get all rows loaded.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)