Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D28A6DF62 for ; Mon, 19 Nov 2012 15:35:05 +0000 (UTC) Received: (qmail 36820 invoked by uid 500); 19 Nov 2012 15:35:05 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 36681 invoked by uid 500); 19 Nov 2012 15:35:05 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 36529 invoked by uid 99); 19 Nov 2012 15:35:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Nov 2012 15:35:04 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of drkashif8310@gmail.com designates 209.85.214.179 as permitted sender) Received: from [209.85.214.179] (HELO mail-ob0-f179.google.com) (209.85.214.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Nov 2012 15:34:59 +0000 Received: by mail-ob0-f179.google.com with SMTP id eq6so4611950obc.38 for ; Mon, 19 Nov 2012 07:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=A9FVup4jXuZq0mUnp3R6H2MkzuJ+l+8UlzsozAYYVlA=; b=Y/G/k5NwStjmlZzjS0DtpJ2X/yGf+TFueg/EaG0Ns+R6vA/WGl6735Jbxf8MzrzHWo SLX4qy5j2pXApkNv3wuKig/ziWLmVxIt8gdpCG7dw/4MBlqheo4d91109zcHFG0EH4A7 jsClqZSsNGcmO7+rU8Nhj5h8QDibH0JjqykCRedxVHobzUdTaZVex+AQQ7TgQQKOopi8 gBSk+8goTDGfaLZuub0kb4cfu7c1GiLuFKEa9RRhelOgKmq6viCP/8/bWKa5XLhvrr8R 7Q85XRTR7JdGiKRMH2KJNG+Wtb6EOrluoSdSYh6tydMU45QjeuJMJnZ0cPXG8K4P2qq2 iBVQ== MIME-Version: 1.0 Received: by 10.60.24.169 with SMTP id v9mr10957453oef.117.1353339278986; Mon, 19 Nov 2012 07:34:38 -0800 (PST) Received: by 10.76.150.34 with HTTP; Mon, 19 Nov 2012 07:34:38 -0800 (PST) In-Reply-To: References: <6E9BC887-DB1E-49ED-A38B-11E3342F436B@gmail.com> Date: Mon, 19 Nov 2012 15:34:38 +0000 Message-ID: Subject: Re: Automatically upload files into HDFS From: kashif khan To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e89a8fb1f812fc0fbf04cedada2b X-Virus-Checked: Checked by ClamAV on apache.org --e89a8fb1f812fc0fbf04cedada2b Content-Type: text/plain; charset=ISO-8859-1 The input path is fine. Problem in output path. I am just wonder that it copy the data into local disk (/user/root/) not into hdfs. I dont know why? Is it we give the correct statement to point to hdfs? Thanks On Mon, Nov 19, 2012 at 3:10 PM, Mohammad Tariq wrote: > Try this as your input file path > Path inputFile = new Path("file:///usr/Eclipse/Output.csv"); > > Regards, > Mohammad Tariq > > > > On Mon, Nov 19, 2012 at 8:31 PM, kashif khan wrote: > >> when I am applying the command as >> >> $ hadoop fs -put /usr/Eclipse/Output.csv /user/root/Output.csv. >> >> its work fine and file browsing in the hdfs. But i dont know why its not >> work in program. >> >> Many thanks for your cooperation. >> >> Best regards, >> >> >> >> >> On Mon, Nov 19, 2012 at 2:53 PM, Mohammad Tariq wrote: >> >>> It would be good if I could have a look on the files. Meantime try some >>> other directories. Also, check the directory permissions once. >>> >>> Regards, >>> Mohammad Tariq >>> >>> >>> >>> On Mon, Nov 19, 2012 at 8:13 PM, kashif khan wrote: >>> >>>> >>>> I have tried through root user and made the following changes: >>>> >>>> >>>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>>> Path outputFile = new Path("/user/root/Output1.csv"); >>>> >>>> No result. The following is the log output. The log shows the >>>> destination is null. >>>> >>>> >>>> 2012-11-19 14:36:38,960 INFO FSNamesystem.audit: allowed=true ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo src=/user dst=null perm=null >>>> 2012-11-19 14:36:38,977 INFO FSNamesystem.audit: allowed=true ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus src=/user dst=null perm=null >>>> 2012-11-19 14:36:39,933 INFO FSNamesystem.audit: allowed=true ugi=hbase (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus src=/hbase/.oldlogs dst=null perm=null >>>> 2012-11-19 14:36:41,147 INFO FSNamesystem.audit: allowed=true ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo src=/user/root dst=null perm=null >>>> 2012-11-19 14:36:41,229 INFO FSNamesystem.audit: allowed=true ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus src=/user/root dst=null perm=null >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Nov 19, 2012 at 2:29 PM, kashif khan wrote: >>>> >>>>> Yeah, My cluster running. When brows http://hadoop1.example.com: >>>>> 50070/dfshealth.jsp. I am getting the main page. Then click on Brows file >>>>> system. I am getting the following: >>>>> >>>>> hbase >>>>> tmp >>>>> user >>>>> >>>>> And when click on user getting: >>>>> >>>>> beeswax >>>>> huuser (I have created) >>>>> root (I have created) >>>>> >>>>> Would you like to see my configuration file. As did not change any >>>>> things, all by default. I have installed CDH4.1 and running on VMs. >>>>> >>>>> Many thanks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Nov 19, 2012 at 2:04 PM, Mohammad Tariq wrote: >>>>> >>>>>> Is your cluster running fine? Are you able to browse Hdfs through the >>>>>> Hdfs Web Console at 50070? >>>>>> >>>>>> Regards, >>>>>> Mohammad Tariq >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Nov 19, 2012 at 7:31 PM, kashif khan wrote: >>>>>> >>>>>>> Many thanks. >>>>>>> >>>>>>> I have changed the program accordingly. It does not show any error >>>>>>> but one warring , but when I am browsing the HDFS folder, file is not >>>>>>> copied. >>>>>>> >>>>>>> >>>>>>> public class CopyData { >>>>>>> public static void main(String[] args) throws IOException{ >>>>>>> Configuration conf = new Configuration(); >>>>>>> //Configuration configuration = new Configuration(); >>>>>>> //configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>>> //configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>>> >>>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>>> conf.addResource(new Path >>>>>>> ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>>> FileSystem fs = FileSystem.get(conf); >>>>>>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>>>>>> Path outputFile = new Path("/user/hduser/Output1.csv"); >>>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>>> fs.close(); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> 19-Nov-2012 13:50:32 org.apache.hadoop.util.NativeCodeLoader >>>>>>> WARNING: Unable to load native-hadoop library for your platform... >>>>>>> using builtin-java classes where applicable >>>>>>> >>>>>>> Have any idea? >>>>>>> >>>>>>> Many thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 19, 2012 at 1:18 PM, Mohammad Tariq wrote: >>>>>>> >>>>>>>> If it is just copying the files without any processing or change, >>>>>>>> you can use something like this : >>>>>>>> >>>>>>>> public class CopyData { >>>>>>>> >>>>>>>> public static void main(String[] args) throws IOException{ >>>>>>>> >>>>>>>> Configuration configuration = new Configuration(); >>>>>>>> configuration.addResource(new >>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>>>> configuration.addResource(new >>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>>>> FileSystem fs = FileSystem.get(configuration); >>>>>>>> Path inputFile = new >>>>>>>> Path("/home/mohammad/pc/work/FFT.java"); >>>>>>>> Path outputFile = new Path("/mapout/FFT.java"); >>>>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>>>> fs.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> Obviously you have to modify it as per your requirements like >>>>>>>> continuously polling the targeted directory for new files. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mohammad Tariq >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Nov 19, 2012 at 6:23 PM, kashif khan < >>>>>>>> drkashif8310@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks M Tariq >>>>>>>>> >>>>>>>>> As I am new in Java and Hadoop and have no much experience. I am >>>>>>>>> trying to first write a simple program to upload data into HDFS and >>>>>>>>> gradually move forward. I have written the following simple program to >>>>>>>>> upload the file into HDFS, I dont know why it does not working. could you >>>>>>>>> please check it, if have time. >>>>>>>>> >>>>>>>>> import java.io.BufferedInputStream; >>>>>>>>> import java.io.BufferedOutputStream; >>>>>>>>> import java.io.File; >>>>>>>>> import java.io.FileInputStream; >>>>>>>>> import java.io.FileOutputStream; >>>>>>>>> import java.io.IOException; >>>>>>>>> import java.io.InputStream; >>>>>>>>> import java.io.OutputStream; >>>>>>>>> import java.nio.*; >>>>>>>>> //import java.nio.file.Path; >>>>>>>>> >>>>>>>>> import org.apache.hadoop.conf.Configuration; >>>>>>>>> import org.apache.hadoop.fs.FSDataInputStream; >>>>>>>>> import org.apache.hadoop.fs.FSDataOutputStream; >>>>>>>>> import org.apache.hadoop.fs.FileSystem; >>>>>>>>> import org.apache.hadoop.fs.Path; >>>>>>>>> public class hdfsdata { >>>>>>>>> >>>>>>>>> >>>>>>>>> public static void main(String [] args) throws IOException >>>>>>>>> { >>>>>>>>> try{ >>>>>>>>> >>>>>>>>> >>>>>>>>> Configuration conf = new Configuration(); >>>>>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>>>>> conf.addResource(new Path ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>>>>> FileSystem fileSystem = FileSystem.get(conf); >>>>>>>>> String source = "/usr/Eclipse/Output.csv"; >>>>>>>>> String dest = "/user/hduser/input/"; >>>>>>>>> >>>>>>>>> //String fileName = source.substring(source.lastIndexOf('/') + >>>>>>>>> source.length()); >>>>>>>>> String fileName = "Output1.csv"; >>>>>>>>> >>>>>>>>> if (dest.charAt(dest.length() -1) != '/') >>>>>>>>> { >>>>>>>>> dest = dest + "/" +fileName; >>>>>>>>> } >>>>>>>>> else >>>>>>>>> { >>>>>>>>> dest = dest + fileName; >>>>>>>>> >>>>>>>>> } >>>>>>>>> Path path = new Path(dest); >>>>>>>>> >>>>>>>>> >>>>>>>>> if(fileSystem.exists(path)) >>>>>>>>> { >>>>>>>>> System.out.println("File" + dest + " already exists"); >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> FSDataOutputStream out = fileSystem.create(path); >>>>>>>>> InputStream in = new BufferedInputStream(new >>>>>>>>> FileInputStream(new File(source))); >>>>>>>>> File myfile = new File(source); >>>>>>>>> byte [] b = new byte [(int) myfile.length() ]; >>>>>>>>> int numbytes = 0; >>>>>>>>> while((numbytes = in.read(b)) >= 0) >>>>>>>>> >>>>>>>>> { >>>>>>>>> out.write(b,0,numbytes); >>>>>>>>> } >>>>>>>>> in.close(); >>>>>>>>> out.close(); >>>>>>>>> //bos.close(); >>>>>>>>> fileSystem.close(); >>>>>>>>> } >>>>>>>>> catch(Exception e) >>>>>>>>> { >>>>>>>>> >>>>>>>>> System.out.println(e.toString()); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> KK >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Nov 19, 2012 at 12:41 PM, Mohammad Tariq < >>>>>>>>> dontariq@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> You can set your cronjob to execute the program after every 5 >>>>>>>>>> sec. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mohammad Tariq >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Nov 19, 2012 at 6:05 PM, kashif khan < >>>>>>>>>> drkashif8310@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Well, I want to automatically upload the files as the files are >>>>>>>>>>> generating about every 3-5 sec and each file has size about 3MB. >>>>>>>>>>> >>>>>>>>>>> Is it possible to automate the system using put or cp command? >>>>>>>>>>> >>>>>>>>>>> I read about the flume and webHDFS but I am not sure it will >>>>>>>>>>> work or not. >>>>>>>>>>> >>>>>>>>>>> Many thanks >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Nov 19, 2012 at 12:26 PM, Alexander Alten-Lorenz < >>>>>>>>>>> wget.null@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Why do you don't use HDFS related tools like put or cp? >>>>>>>>>>>> >>>>>>>>>>>> - Alex >>>>>>>>>>>> >>>>>>>>>>>> On Nov 19, 2012, at 11:44 AM, kashif khan < >>>>>>>>>>>> drkashif8310@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> > HI, >>>>>>>>>>>> > >>>>>>>>>>>> > I am generating files continuously in local folder of my base >>>>>>>>>>>> machine. How >>>>>>>>>>>> > I can now use the flume to stream the generated files from >>>>>>>>>>>> local folder to >>>>>>>>>>>> > HDFS. >>>>>>>>>>>> > I dont know how exactly configure the sources, sinks and hdfs. >>>>>>>>>>>> > >>>>>>>>>>>> > 1) location of folder where files are generating: >>>>>>>>>>>> /usr/datastorage/ >>>>>>>>>>>> > 2) name node address: htdfs://hadoop1.example.com:8020 >>>>>>>>>>>> > >>>>>>>>>>>> > Please let me help. >>>>>>>>>>>> > >>>>>>>>>>>> > Many thanks >>>>>>>>>>>> > >>>>>>>>>>>> > Best regards, >>>>>>>>>>>> > KK >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Alexander Alten-Lorenz >>>>>>>>>>>> http://mapredit.blogspot.com >>>>>>>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --e89a8fb1f812fc0fbf04cedada2b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The input path is fine. Problem in output path. I am just wonder that it co= py the data into local disk=A0 (/user/root/) not into hdfs. I dont know why= ? Is it we give the correct statement to point to hdfs?

Thanks

On Mon, Nov 19, 2012 at 3:10 PM, Mohammad Ta= riq <dontariq@gmail.com> wrote:
Try this as your input file path=A0
Path inputFile =3D new Path("file:///usr/Ecli= pse/Output.csv");

Regards,
=A0=A0 =A0Mohamma= d Tariq



On Mon, Nov 19, 2= 012 at 8:31 PM, kashif khan <drkashif8310@gmail.com> wr= ote:
when I am applying the command as

$ hadoop fs -put /usr/Eclipse/Outp= ut.csv /user/root/Output.csv.

its work fine and file browsing in the= hdfs. But i dont know why its not work in program.

Many thanks for = your cooperation.

Best regards,




On Mo= n, Nov 19, 2012 at 2:53 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
It would be good if I could have a look on t= he files. Meantime try some other directories. Also, check the directory pe= rmissions once.

Regards,
=A0=A0 =A0Mohammad Tariq



On Mon, Nov 19, 2012 at 8:13 PM, kashif = khan <drkashif8310@gmail.com> wrote:

I have tried through root user and made the following changes:

=
Path inputFile =3D new Path("/usr/Eclipse/Output.csv");
Path outputFile =3D new Path("/user/root/Output1.csv");

No result. The following is the log output. The log shows the destinati= on is null.
=A0
2012-11-19 14:36:38,960 INFO FSNamesystem.audit: allowed=3Dtrue=
	ugi=3Ddr.who (auth:SIMPLE)	ip=3D/134.91.36.41	cmd=3Dgetfileinfo	src=3D/user	dst=3Dnull	perm=3Dn=
ull
2012-11-19 14:36:38,977 INFO FSNamesystem.audit: allowed=3Dtrue	ugi=3Ddr.wh=
o (auth:SIMPLE)	ip=3D/134=
.91.36.41	cmd=3DlistStatus	src=3D/user	dst=3Dnull	perm=3Dnull
2012-11-19 14:36:39,933 INFO FSNamesystem.audit: allowed=3Dtrue	ugi=3Dhbase=
 (auth:SIMPLE)	ip=3D/134.=
91.36.41	cmd=3DlistStatus	src=3D/hbase/.oldlogs	dst=3Dnull	perm=3Dnull
2012-11-19 14:36:41,147 INFO FSNamesystem.audit: allowed=3Dtrue	ugi=3Ddr.wh=
o (auth:SIMPLE)	ip=3D/134=
.91.36.41	cmd=3Dgetfileinfo	src=3D/user/root	dst=3Dnull	perm=3Dnull
2012-11-19 14:36:41,229 INFO FSNamesystem.audit: allowed=3Dtrue	ugi=3Ddr.wh=
o (auth:SIMPLE)	ip=3D/134=
.91.36.41	cmd=3DlistStatus	src=3D/user/root	dst=3Dnull	perm=3Dnull
Thanks














On Mon, Nov 19, 2012 at 2:29 PM, kashif khan=
 <drkashif8310@gmail.com> wrote:







Yeah, My cluster running. When brows http://hadoop1.example.com: 50070/dfshealth.jsp. I a=
m getting the main page. Then click on Brows file system. I am getting the =
following:









hbase
tmp
user
=A0
And when click on user getting:

beesw=
ax
huuser (I have created)
root (I have created)

Would you lik=
e to see my configuration file. As did not change any things, all by defaul=
t. I have installed CDH4.1 and running on VMs.









Many thanks





On =
Mon, Nov 19, 2012 at 2:04 PM, Mohammad Tariq <dontariq@gmail.com>=
 wrote:









Is your cluster running fine? Are you able t=
o browse Hdfs through the Hdfs Web Console at 50070?









Regards,
=A0=A0 =A0Mohammad Tariq




On Mon, Nov 19, 2012 at 7:31 PM, kashif =
khan <drkashif8310@gmail.com> wrote:










Many thanks.

I have changed the program accordingly. It does not sho=
w any error but one warring , but when I am browsing the HDFS folder, file =
is not copied.=A0 


public class CopyData {


public static void main(String[] args) throws IOException{

=A0=A0=A0 =A0=A0=A0 Configuration conf =3D new Configuration();
=A0=A0=
=A0=A0=A0=A0=A0 //Configuration configuration =3D new Configuration();
=
=A0=A0=A0=A0=A0=A0=A0 //configuration.addResource(new Path("/home/moha=
mmad/hadoop-0.20.205/conf/core-site.xml"));











=A0=A0=A0=A0=A0=A0=A0 //configuration.addResource(new Path("/home/moha=
mmad/hadoop-0.20.205/conf/hdfs-site.xml"));

=A0=A0=A0=A0=A0=A0=
=A0 conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));=

=A0=A0=A0 =A0=A0=A0 conf.addResource(new Path ("/etc/hadoop/conf/h=
dfs-site.xml"));












=A0=A0=A0=A0=A0=A0=A0 FileSystem fs =3D FileSystem.get(conf);
=A0=A0=A0=
=A0=A0=A0=A0 Path inputFile =3D new Path("/usr/Eclipse/Output.csv"=
;);
=A0=A0=A0=A0=A0=A0=A0 Path outputFile =3D new Path("/user/hduse=
r/Output1.csv");
=A0=A0=A0=A0=A0=A0=A0 fs.copyFromLocalFile(inputFi=
le, outputFile);











=A0=A0=A0=A0=A0=A0=A0 fs.close();
=A0=A0=A0 }
}

19-Nov-2012 13:50:32 org.apache.hadoop.util.NativeCode=
Loader <clinit>

WARNING: Unable to load native-hadoop li=
brary for your platform... using builtin-java classes where applicable


Have any idea?

Many thanks






On Mon, Nov 19, 2012 at=
 1:18 PM, Mohammad Tariq <dontariq@gmail.com> wrote:












If it is just copying the files without any =
processing or change, you can use something like this :











public class CopyData {


=A0 =A0=A0
=A0 =A0 public static void main(String[] arg=
s) throws IOException{



=A0 =A0 =A0 =A0=A0
=A0 =A0 =A0 =A0 Configuration confi=
guration =3D new Configuration();
=A0 =A0 =A0 =A0 configuration.a=
ddResource(new Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml=
"));
=A0 =A0 =A0 =A0 configuration.addResource(new Path(&quo=
t;/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml"));














=A0 =A0 =A0 =A0 FileSystem fs =3D FileSystem.get(configuration);
=

=A0 =A0 =A0 =A0 Path inputFile =3D new Path("/home/mohammad/pc/wo=
rk/FFT.java");
=A0 =A0 =A0 =A0 Path outputFile =3D new Path(=
"/mapout/FFT.java");














=A0 =A0 =A0 =A0 fs.copyFromLocalFile(inputFile, outputFile);
=A0 =A0 =A0 =A0 fs.close();
=A0 =A0 }
}

Obviously you have to modify it as per your requirem=
ents like continuously polling the targeted directory for new files.















Regards,
=A0=A0 =A0Mohamma=
d Tariq




On Mon, Nov 19, 2012 at 6:23 PM, kashif =
khan <drkashif8310@gmail.com> wrote:













Thanks M=A0 Tariq

As I am new in=A0 Java and Hadoop and have no much=
 experience. I am trying to first write a simple program to upload data int=
o HDFS and gradually move forward. I have written the following simple prog=
ram to upload the file into HDFS, I dont know why it does not working.=A0 c=
ould you please check it, if have time.















import java.io.BufferedInputStream;
import java.io.BufferedOutputStr=
eam;
import java.io.File;
import java.io.FileInputStream;
import j=
ava.io.FileOutputStream;
import java.io.IOException;
import java.io.I=
nputStream;














import java.io.OutputStream;
import java.nio.*;
//import java.nio.fil=
e.Path;

import org.apache.hadoop.conf.Configuration;
import org.a=
pache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOut=
putStream;














import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path=
;
public class hdfsdata {


public static void main(String [] a=
rgs) throws IOException
{
=A0=A0=A0 try{
=A0=A0=A0 =A0=A0=A0 
=
=A0=A0=A0 
=A0=A0=A0 Configuration conf =3D new Configuration();














=A0=A0=A0 conf.addResource(new Path("/etc/hadoop/conf/core-site.xml&qu=
ot;));
=A0=A0=A0 conf.addResource(new Path ("/etc/hadoop/conf/hdfs-=
site.xml"));
=A0=A0=A0 FileSystem fileSystem =3D FileSystem.get(con=
f);
=A0=A0=A0 String source =3D "/usr/Eclipse/Output.csv";














=A0=A0=A0 String dest =3D "/user/hduser/input/";
=A0=A0=A0 =A0=A0=A0 //String fileName =3D source.substring(source.lastIndexOf('/=
') + source.length());
=A0=A0=A0 String fileName =3D "Output1.c=
sv";
=A0=A0=A0 
=A0=A0=A0 if (dest.charAt(dest.length() -1) !=3D=
 '/')














=A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 dest =3D dest + "/" +fileName;=

=A0=A0=A0 }
=A0=A0=A0 else
=A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 des=
t =3D dest + fileName;
=A0=A0=A0 =A0=A0=A0 
=A0=A0=A0 }
=A0=A0=A0 =
Path path =3D new Path(dest);
=A0=A0=A0 =A0=A0=A0 
=A0=A0=A0 
=A0=
=A0=A0 if(fileSystem.exists(path))














=A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 System.out.println("File" + de=
st + " already exists");
=A0=A0=A0 }
=A0=A0=A0 

=A0=
=A0 FSDataOutputStream out =3D fileSystem.create(path);
=A0=A0 InputStre=
am in =3D new BufferedInputStream(new FileInputStream(new File(source)));













=A0=A0 File myfile =3D new File(source);
=A0=A0 byte [] b =3D new byte [=
(int) myfile.length() ];
=A0=A0 int numbytes =3D 0;
=A0=A0 while((num=
bytes =3D in.read(b)) >=3D 0)
=A0=A0=A0 =A0=A0 
=A0=A0 {
=A0=A0=
=A0 =A0=A0 out.write(b,0,numbytes);
=A0=A0 }














=A0=A0 in.close();
=A0=A0 out.close();
=A0=A0 //bos.close();
=A0=
=A0 fileSystem.close();
=A0=A0=A0 }
=A0=A0=A0 catch(Exception e)
=
=A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 
=A0=A0=A0 =A0=A0=A0 System.out.print=
ln(e.toString());
=A0=A0=A0 }
=A0=A0=A0 }
=A0=A0=A0 
}
















Thanks again,

Best regards,

KK



On Mon, Nov 19, 2012 at 12:41 PM, Mohammad Tariq <donta=
riq@gmail.com> wrote:















You can set your cronjob to execute the prog=
ram after every 5 sec.=A0

Regar=
ds,














=A0=A0 =A0Mohammad Tariq




On Mon, Nov 19, 2012 at 6:05 PM, kashif =
khan <drkashif8310@gmail.com> wrote:
















Well, I want to automatically upload the files as=A0 the files are generati=
ng about every 3-5 sec and each file has size about 3MB.

=A0Is it po=
ssible to automate the system using put or cp command?

I read about =
the flume and webHDFS but I am not sure it will work or not.


















Many thanks

Best regards




On Mon, Nov 19, 2012 at 12:26 PM, Alexander Alten-Lorenz <w=
get.null@gmail.com> wrote:


















Hi,



Why do you don't use HDFS related tools like put or cp?



- Alex




On Nov 19, 2012, at 11:44 AM, kashif khan <drkashif8310@gmail.com> wrote:



> HI,

>

> I am generating files continuously in local folder of my base machine.=
 How

> I can now use the flume to stream the generated files from local folde=
r to

> HDFS.

> I dont know how exactly configure the sources, sinks and hdfs.

>

> 1) location of folder where files are generating: /usr/datastorage/
> 2) name node address: htdfs://hadoop1.example.com:8020

>

> Please let me help.

>

> Many thanks

>

> Best regards,

> KK




--

Alexander Alten-Lorenz

http://mapredit.=
blogspot.com

German Hadoop LinkedIn Group: http://goo.gl/N8pCF






--e89a8fb1f812fc0fbf04cedada2b--