hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerry He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10622) Improve log and Exceptions in Export Snapshot
Date Sun, 02 Mar 2014 21:01:32 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917556#comment-13917556
] 

Jerry He commented on HBASE-10622:
----------------------------------

A few more comments (since you are doing the improvement ...)

{code}
-        // Verify that the written size match
-        if (totalBytesWritten != inputFileSize) {
-          String msg = "number of bytes copied not matching copied=" + totalBytesWritten
+
-                       " expected=" + inputFileSize + " for file=" + inputPath;
-          throw new IOException(msg);
{code}
You think this is unnecessary?

In the run(),  can we cleanup/delete snapshotTmpDir if Step 2 failed so that we don't ask
the user to manually clean it since it comes from our Step 1 copy?

Can we add a job counter say 'COPIES_FILES' to be along side with 'BYTES_COPIED'?

Another issue is probably more involved, and does not need to be covered in this JIRA. It
is the overall progress reporting of the ExportSnapshot job.
For example,  
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot1 -copy-to /user/biadmin/mysnapshots
-mappers 30

{code}
14/03/02 12:19:54 INFO mapred.JobClient:  map 0% reduce 0%
14/03/02 12:20:12 INFO mapred.JobClient:  map 6% reduce 0%
14/03/02 12:20:13 INFO mapred.JobClient:  map 44% reduce 0%
14/03/02 12:20:19 INFO mapred.JobClient:  map 83% reduce 0%
{code}
There is about 130G to export.  But it takes just a few secs to get to 83%, after the first
around of mappers are launched, and will stay there for a long time.
Similarly at the end it will show 100% for a long time while there are mappers still running.
he map progress percentage is quite inaccurate with regard to the over progress. 

> Improve log and Exceptions in Export Snapshot 
> ----------------------------------------------
>
>                 Key: HBASE-10622
>                 URL: https://issues.apache.org/jira/browse/HBASE-10622
>             Project: HBase
>          Issue Type: Bug
>          Components: snapshots
>            Reporter: Matteo Bertozzi
>            Assignee: Matteo Bertozzi
>             Fix For: 0.99.0
>
>         Attachments: HBASE-10622-v0.patch, HBASE-10622-v1.patch, HBASE-10622-v2.patch,
HBASE-10622-v3.patch
>
>
> from the logs of export snapshot is not really clear what's going on,
> adding some extra information useful to debug, and in some places the real exception
can be thrown



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message