hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Ervits <are9...@nyp.org>
Subject Re: ExportSnapshot webhdfs problems
Date Mon, 06 Oct 2014 13:24:17 GMT
Does it work with a small table? I prefer to use hftp instead of webhdfs.

From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: Friday, October 03, 2014 11:01 AM
To: user@hadoop.apache.org <user@hadoop.apache.org>
Subject: ExportSnapshot webhdfs problems

I posted this on users@hbase, but got not response, so I thought I’d try here:

I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster
using the webhdfs protocol.
I’ve done this successfully before, though there are always mapper failures and retries
in the job log. However, I’m not
having success with a rather large table due to an excessive number of failures. The exceptions
in the job log are always:

14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_000005_1, Status
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146)

So I presume the real exception is taking place on the target system. However, examining the
namenode logs and a handful of
the datanode logs has not revealed any exceptions that correlate with those in the job log.
Is there some other log I should be looking at?
I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard
to believe its a capacity problem.


This electronic message is intended to be for the use only of the named recipient, and may
contain information that is confidential or privileged.  If you are not the intended recipient,
you are hereby notified that any disclosure, copying, distribution or use of the contents
of this message is strictly prohibited.  If you have received this message in error or are
not the named recipient, please notify us immediately by contacting the sender at the electronic
mail address noted above, and delete and destroy all copies of this message.  Thank you.
View raw message