hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Not able to copy one HDFS data to another HDFS location using distcp
Date Mon, 07 Sep 2015 08:14:14 GMT
I am trying to copy one HDFS data to another HDFS location
I am able to achieve the same using "distcp" command
hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest

But I want to try the same using Java Api
After a long search I found one code and executed . But it didnt copied my
src file to destination.


*public class TouchFile {*

* /***
* * @param args*
* * @throws Exception *
* */*
* public static void main(String[] args) throws Exception {*
* // TODO Auto-generated method stub*
* //create configuration object*
* Configuration config = new Configuration();*
* config.set("fs.defaultFS", "hdfs://mySrcip:8020/");*
* config.set("hadoop.job.ugi", "hdfs");*
* /**
* * Distcp*
* */*
* String sourceNameNode = "hdfs://mySrcip:8020/copyDev";*
*    String destNameNode = "hdfs://myDestip:8020/copyTest";*
*    String fileList = "myfile.txt";*
* distFileCopy(config,sourceNameNode,destNameNode,fileList);*
* }*
* /***
*     * Copies files from one cloud to another using Hadoop's distributed
copy features. Uses*
*     * input to build DISTCP configuration settings. *
*     **
*     * param config Hadoop configuration*
*     * param sourceNameNode full HDFS path to parent source directory*
*     * param destNameNode full HDFS path to parent destination directory*
*     * param fileList Comma separated string of file names in
sourceNameNode to be copied to destNameNode*
*     * returns Elapsed time in milliseconds to copy files*
*     */*
*    public static long distFileCopy( Configuration config, String
sourceNameNode, String destNameNode, String fileList ) throws Exception {*
*            System.out.println("In dist copy");*

*        StringTokenizer tokenizer = new StringTokenizer(fileList,",");*
*        ArrayList<String> list = new ArrayList<>();*

*        while ( tokenizer.hasMoreTokens() ){*
*            String file = sourceNameNode + "/" + tokenizer.nextToken();*
*            list.add( file );*
*        }*

*        String[] args = new String[list.size() + 1];*
*        int count = 0;*
*        for ( String filename : list ){*
*            args[count++] = filename;*
*        }*

*        args[count] = destNameNode;*

*        System.out.println("args------>"+Arrays.toString(args));*
*        long st = System.currentTimeMillis();        *
*        DistCp distCp=new DistCp(config,null);*
*        distCp.run(args);   *
*        return System.currentTimeMillis() - st;*

*    }*

*}*



Am I doing anything wrong.
Please suggest

-- 
*Thanks & Regards *


*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
http://www.unmeshasreeveni.blogspot.in/

Mime
View raw message