hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cooper Bethea <co...@siftscience.com>
Subject Re: snapshot export failure
Date Tue, 29 Oct 2013 23:09:14 GMT
I re-ran ExportSnapshot; it completed successfully and I am now able to
clone the snapshot to a new table. No errors reported. Here's the output.
¯\_(ツ)_/¯


hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
grains_1383068263 -copy-to hdfs://experiment-b-name-1d:8020/hbase -mappers 6
2013-10-29/22:47:58.986 UTC [main] org.apache.hadoop.conf.Configuration
[WARN] - hadoop.native.lib is deprecated. Instead, use
io.native.lib.available
2013-10-29/22:48:09.157 UTC [main] org.apache.hadoop.hbase.util.FSVisitor
[INFO] - No logs under
directory:hdfs://production-b/hbase/.snapshot/grains_1383068263/.logs
2013-10-29/22:49:37.642 UTC [main]
org.apache.hadoop.io.compress.zlib.ZlibFactory [INFO] - Successfully loaded
& initialized native-zlib library
2013-10-29/22:49:37.644 UTC [main] org.apache.hadoop.io.compress.CodecPool
[INFO] - Got brand-new compressor [.deflate]
2013-10-29/22:49:38.210 UTC [main] org.apache.hadoop.mapred.JobClient
[WARN] - Use GenericOptionsParser for parsing the arguments. Applications
should implement Tool for the same.
2013-10-29/22:49:38.432 UTC [main]
org.apache.hadoop.mapreduce.lib.input.FileInputFormat [INFO] - Total input
paths to process : 6
2013-10-29/22:49:38.805 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - Running job: job_201310241813_0018
2013-10-29/22:49:39.811 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 0% reduce 0%
2013-10-29/22:50:18.891 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 1% reduce 0%
2013-10-29/22:50:48.937 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 2% reduce 0%
2013-10-29/22:51:21.986 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 3% reduce 0%
2013-10-29/22:51:59.037 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 4% reduce 0%
2013-10-29/22:52:34.088 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 5% reduce 0%
2013-10-29/22:53:04.128 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 6% reduce 0%
2013-10-29/22:53:30.164 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 7% reduce 0%
2013-10-29/22:53:46.185 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 8% reduce 0%
2013-10-29/22:53:56.201 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 9% reduce 0%
2013-10-29/22:54:05.215 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 10% reduce 0%
2013-10-29/22:54:14.228 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 11% reduce 0%
2013-10-29/22:54:23.242 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 12% reduce 0%
2013-10-29/22:54:29.250 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 13% reduce 0%
2013-10-29/22:54:32.254 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 14% reduce 0%
2013-10-29/22:54:38.264 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 15% reduce 0%
2013-10-29/22:54:41.268 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 16% reduce 0%
2013-10-29/22:54:44.276 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 17% reduce 0%
2013-10-29/22:54:47.280 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 19% reduce 0%
2013-10-29/22:54:50.284 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 20% reduce 0%
2013-10-29/22:54:53.289 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 21% reduce 0%
2013-10-29/22:54:56.294 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 23% reduce 0%
2013-10-29/22:54:59.298 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 25% reduce 0%
2013-10-29/22:55:02.303 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 27% reduce 0%
2013-10-29/22:55:05.308 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 29% reduce 0%
2013-10-29/22:55:08.313 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 31% reduce 0%
2013-10-29/22:55:11.318 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 34% reduce 0%
2013-10-29/22:55:14.322 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 36% reduce 0%
2013-10-29/22:55:16.326 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 37% reduce 0%
2013-10-29/22:55:17.328 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 39% reduce 0%
2013-10-29/22:55:20.333 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 41% reduce 0%
2013-10-29/22:55:23.337 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 44% reduce 0%
2013-10-29/22:55:26.342 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 46% reduce 0%
2013-10-29/22:55:29.347 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 49% reduce 0%
2013-10-29/22:55:32.352 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 52% reduce 0%
2013-10-29/22:55:35.357 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 54% reduce 0%
2013-10-29/22:55:37.360 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 55% reduce 0%
2013-10-29/22:55:38.362 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 56% reduce 0%
2013-10-29/22:55:40.365 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 57% reduce 0%
2013-10-29/22:55:41.367 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 59% reduce 0%
2013-10-29/22:55:44.372 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 62% reduce 0%
2013-10-29/22:55:47.377 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 65% reduce 0%
2013-10-29/22:55:50.381 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 68% reduce 0%
2013-10-29/22:55:53.386 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 71% reduce 0%
2013-10-29/22:55:55.390 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 72% reduce 0%
2013-10-29/22:55:58.394 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 73% reduce 0%
2013-10-29/22:56:08.408 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 74% reduce 0%
2013-10-29/22:56:11.412 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 75% reduce 0%
2013-10-29/22:56:14.417 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 76% reduce 0%
2013-10-29/22:56:16.420 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 77% reduce 0%
2013-10-29/22:56:19.425 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 79% reduce 0%
2013-10-29/22:56:23.434 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 81% reduce 0%
2013-10-29/22:56:26.439 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 84% reduce 0%
2013-10-29/22:56:29.444 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 86% reduce 0%
2013-10-29/22:56:32.449 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 89% reduce 0%
2013-10-29/22:56:35.454 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 92% reduce 0%
2013-10-29/22:56:38.459 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 94% reduce 0%
2013-10-29/22:56:41.464 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 97% reduce 0%
2013-10-29/22:56:44.468 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 99% reduce 0%
2013-10-29/22:56:47.474 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] -  map 100% reduce 0%
2013-10-29/22:56:48.477 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - Job complete: job_201310241813_0018
2013-10-29/22:56:48.494 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Counters: 25
2013-10-29/22:56:48.498 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-   File System Counters
2013-10-29/22:56:48.501 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     FILE: Number of bytes read=0
2013-10-29/22:56:48.501 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     FILE: Number of bytes written=1307244
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     FILE: Number of read operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     FILE: Number of large read operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     FILE: Number of write operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     HDFS: Number of bytes read=466490
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     HDFS: Number of bytes written=0
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     HDFS: Number of read operations=43906
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     HDFS: Number of large read operations=0
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     HDFS: Number of write operations=0
2013-10-29/22:56:48.506 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-   Job Counters
2013-10-29/22:56:48.507 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Launched map tasks=6
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Data-local map tasks=4
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Rack-local map tasks=2
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Total time spent by all maps in occupied slots (ms)=2218597
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Total time spent by all reduces in occupied slots (ms)=0
2013-10-29/22:56:48.509 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Total time spent by all maps waiting after reserving slots (ms)=0
2013-10-29/22:56:48.509 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Total time spent by all reduces waiting after reserving slots (ms)=0
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-   Map-Reduce Framework
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Map input records=5010
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Map output records=0
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Input split bytes=990
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Spilled Records=0
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     CPU time spent (ms)=134140
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Physical memory (bytes) snapshot=1306460160
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Virtual memory (bytes) snapshot=9209536512
2013-10-29/22:56:48.514 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
-     Total committed heap usage (bytes)=2658074624


On Tue, Oct 29, 2013 at 3:35 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>wrote:

> could you re-run the ExportSnapshot and post the output?
> also could you check the exit code? echo $?
>
>
> Matteo
>
>
>
> On Tue, Oct 29, 2013 at 10:29 PM, Cooper Bethea <coops@siftscience.com
> >wrote:
>
> > Ted, thanks for responding.
> >
> > I've checked the namenode log. There are no messages at greater than INFO
> > level emitted by the namenode during the period when the copy ran.
> >
> > The Export job tasks likewise emitted no errors.
> >
> >
> > On Tue, Oct 29, 2013 at 3:09 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > bq. There are no errors in the logs.
> > >
> > > I assume you have checked master log / region server logs.
> > > How about namenode log ?
> > >
> > >
> > > On Tue, Oct 29, 2013 at 3:06 PM, Cooper Bethea <coops@siftscience.com
> > > >wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > I've been attempting to export a snapshot from one of my HBase
> clusters
> > > to
> > > > the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
> > > > successfully export a very small collection (few MB), but when I
> > attempt
> > > to
> > > > export a larger collection something odd happens. The Export MR
> > finishes
> > > > successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
> > > > instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the
> > > logs.
> > > >
> > > > Is this familiar to anyone?
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message