hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manjeet Singh <manjeet.chand...@gmail.com>
Subject Re: hbase data migration from one cluster to another cluster on different versions
Date Thu, 26 Oct 2017 05:27:58 GMT
Furthermore, clarity why I used scp command is :

I have copy source cluster files to destination cluster by using scp
command and put them into destination cluster HDFS (It's because of two
different version of Haddop  destination cluster hadoop is 1.2.1 and
destination is having Hadoop 2.0 ) First I get HDFS files to local linux
and use scp command to put them into destination cluster.

Thanks
Manjeet Singh

On Thu, Oct 26, 2017 at 10:26 AM, Manjeet Singh <manjeet.chandhok@gmail.com>
wrote:

> Hi Yung,
>
> First thanks for reply
> The link provided by you is for upgrading the Hbase version and problem
> statement is different
> Problem is when I am trying to export hbase data from one cluster to
> another cluster in same N/W, but with a different hbase version  i.e.
> 0.94.27 (source cluster hbase) and another is destination cluster hbase
> version is 1.2.1.
> So this link shall be refer
> http://hbase.apache.org/0.94/book/ops_mgt.html#export
>
>
> for the second point which I forget to mention in mail, I did copy contents
> of /data/ExportedFiles
> in destination cluster which is having HBase 1.2.1 but not with
> distcp instead of I used scp command
> and when I am trying to import data I am getting below error
>
> 17/10/23 16:13:50 INFO mapreduce.Job: Task Id :
> attempt_1505781444745_0070_m_000003_0, Status : FAILED
> Error: java.io.IOException: keyvalues=NONE read 2 bytes, should read 121347
>         at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(
> SequenceFile.java:2306)
>         at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordRead
> er.nextKeyValue(SequenceFileRecordReader.java:78)
>         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nex
> tKeyValue(MapTask.java:556)
>         at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue
> (MapContextImpl.java:80)
>         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.ne
> xtKeyValue(WrappedMapper.java:91)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
> upInformation.java:1693)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
>
> can you please elaborate more about  "Is the environment ready for the
> upgrade?"
>
> Thanks
> Manjeet Singh
>
>
>
> On Thu, Oct 26, 2017 at 8:32 AM, Yung-An He <mathstana@gmail.com> wrote:
>
>> Hi,
>>
>> Have you seen the reference guide
>> <http://hbase.apache.org/book.html#_upgrade_paths> to make sure that the
>> environment is ready for the upgrade?
>> Perhaps you could try to copy the contents of /data/ExportedFiles to the
>> HBase 1.2.1 cluster using distcp before import data instead of using
>> "hdfs://<IP>:8020/data/ExportedFiles" directly.
>> Then create the table on the HBase 1.2.1 cluster using HBase Shell. Column
>> families must be identical to the table on the old one.
>> Finally, import data from /data/ExportedFiles on the HBase 1.2.1 cluster.
>>
>>
>> Best Regards.
>>
>> 2017-10-24 1:27 GMT+08:00 Manjeet Singh <manjeet.chandhok@gmail.com>:
>>
>> > Hi All,
>> >
>> > Can anyone help?
>> >
>> > adding few more investigations I have move all files to the destination
>> > cluster hdfs and I have run below command:-
>> >
>> > sudo -u hdfs hbase org.apache.hadoop.hbase.mapreduce.Import test_table
>> > hdfs://<IP>:8020/data/ExportedFiles
>> >
>> > I am getting below error
>> >
>> > 17/10/23 16:13:50 INFO mapreduce.Job: Task Id :
>> > attempt_1505781444745_0070_m_000003_0, Status : FAILED
>> > Error: java.io.IOException: keyvalues=NONE read 2 bytes, should read
>> 121347
>> >         at
>> > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.
>> > java:2306)
>> >         at
>> > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.
>> > nextKeyValue(SequenceFileRecordReader.java:78)
>> >         at
>> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.
>> > nextKeyValue(MapTask.java:556)
>> >         at
>> > org.apache.hadoop.mapreduce.task.MapContextImpl.
>> > nextKeyValue(MapContextImpl.java:80)
>> >         at
>> > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.
>> > nextKeyValue(WrappedMapper.java:91)
>> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:
>> 787)
>> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> >         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>> >         at java.security.AccessController.doPrivileged(Native Method)
>> >         at javax.security.auth.Subject.doAs(Subject.java:422)
>> >         at
>> > org.apache.hadoop.security.UserGroupInformation.doAs(
>> > UserGroupInformation.java:1693)
>> >         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>> >
>> >
>> >
>> >
>> > can anyone suggest how to migrate data?
>> >
>> > Thanks
>> > Manjeet Singh
>> >
>> >
>> >
>> >
>> >
>> > Hi All,
>> >
>> > I have query regarding hbase data migration from one cluster to another
>> > cluster in same N/W, but with a different version of hbase one is
>> 0.94.27
>> > (source cluster hbase) and another is destination cluster hbase version
>> is
>> > 1.2.1.
>> >
>> > I have used below command to take backup of hbase table on source
>> cluster
>> > is:
>> >  ./hbase org.apache.hadoop.hbase.mapreduce.Export SPDBRebuild
>> > /data/backupData/
>> >
>> > below files were genrated by using above command:-
>> >
>> >
>> > drwxr-xr-x 3 root root        4096 Dec  9  2016 _logs
>> > -rw-r--r-- 1 root root   788227695 Dec 16  2016 part-m-00000
>> > -rw-r--r-- 1 root root  1098757026 Dec 16  2016 part-m-00001
>> > -rw-r--r-- 1 root root   906973626 Dec 16  2016 part-m-00002
>> > -rw-r--r-- 1 root root  1981769314 Dec 16  2016 part-m-00003
>> > -rw-r--r-- 1 root root  2099785782 Dec 16  2016 part-m-00004
>> > -rw-r--r-- 1 root root  4118835540 Dec 16  2016 part-m-00005
>> > -rw-r--r-- 1 root root 14217981341 Dec 16  2016 part-m-00006
>> > -rw-r--r-- 1 root root           0 Dec 16  2016 _SUCCESS
>> >
>> >
>> > in order to restore these files I am assuming I have to move these
>> files in
>> > destination cluster and have to run below command
>> >
>> > hbase org.apache.hadoop.hbase.mapreduce.Import <tablename>
>> > /data/backupData/
>> >
>> > Please suggest if I am on correct direction, second if anyone have
>> another
>> > option.
>> > I have tryed this with test data but above command took very long time
>> and
>> > at end it gets fails
>> >
>> > 17/10/23 11:54:21 INFO mapred.JobClient:  map 0% reduce 0%
>> > 17/10/23 12:04:24 INFO mapred.JobClient: Task Id :
>> > attempt_201710131340_0355_m_000002_0, Status : FAILED
>> > Task attempt_201710131340_0355_m_000002_0 failed to report status for
>> 600
>> > seconds. Killing!
>> >
>> >
>> > Thanks
>> > Manjeet Singh
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > luv all
>> >
>>
>
>
>
> --
> luv all
>



-- 
luv all

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message