flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Automatically upload files into HDFS
Date Mon, 19 Nov 2012 15:41:40 GMT
It should work. Same code is working fine for me. Try to create some other
directory in your Hdfs and use it as your output path. Also see if you find
something in datanode logs.

Regards,
    Mohammad Tariq



On Mon, Nov 19, 2012 at 9:04 PM, kashif khan <drkashif8310@gmail.com> wrote:

> The input path is fine. Problem in output path. I am just wonder that it
> copy the data into local disk  (/user/root/) not into hdfs. I dont know
> why? Is it we give the correct statement to point to hdfs?
>
> Thanks
>
>
>
> On Mon, Nov 19, 2012 at 3:10 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>
>> Try this as your input file path
>> Path inputFile = new Path("file:///usr/Eclipse/Output.csv");
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Mon, Nov 19, 2012 at 8:31 PM, kashif khan <drkashif8310@gmail.com>wrote:
>>
>>> when I am applying the command as
>>>
>>> $ hadoop fs -put /usr/Eclipse/Output.csv /user/root/Output.csv.
>>>
>>> its work fine and file browsing in the hdfs. But i dont know why its not
>>> work in program.
>>>
>>> Many thanks for your cooperation.
>>>
>>> Best regards,
>>>
>>>
>>>
>>>
>>> On Mon, Nov 19, 2012 at 2:53 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>
>>>> It would be good if I could have a look on the files. Meantime try some
>>>> other directories. Also, check the directory permissions once.
>>>>
>>>> Regards,
>>>>     Mohammad Tariq
>>>>
>>>>
>>>>
>>>> On Mon, Nov 19, 2012 at 8:13 PM, kashif khan <drkashif8310@gmail.com>wrote:
>>>>
>>>>>
>>>>> I have tried through root user and made the following changes:
>>>>>
>>>>>
>>>>> Path inputFile = new Path("/usr/Eclipse/Output.csv");
>>>>> Path outputFile = new Path("/user/root/Output1.csv");
>>>>>
>>>>> No result. The following is the log output. The log shows the
>>>>> destination is null.
>>>>>
>>>>>
>>>>> 2012-11-19 14:36:38,960 INFO FSNamesystem.audit: allowed=true	ugi=dr.who
(auth:SIMPLE)	ip=/134.91.36.41	cmd=getfileinfo	src=/user	dst=null	perm=null
>>>>> 2012-11-19 14:36:38,977 INFO FSNamesystem.audit: allowed=true	ugi=dr.who
(auth:SIMPLE)	ip=/134.91.36.41	cmd=listStatus	src=/user	dst=null	perm=null
>>>>> 2012-11-19 14:36:39,933 INFO FSNamesystem.audit: allowed=true	ugi=hbase
(auth:SIMPLE)	ip=/134.91.36.41	cmd=listStatus	src=/hbase/.oldlogs	dst=null	perm=null
>>>>> 2012-11-19 14:36:41,147 INFO FSNamesystem.audit: allowed=true	ugi=dr.who
(auth:SIMPLE)	ip=/134.91.36.41	cmd=getfileinfo	src=/user/root	dst=null	perm=null
>>>>> 2012-11-19 14:36:41,229 INFO FSNamesystem.audit: allowed=true	ugi=dr.who
(auth:SIMPLE)	ip=/134.91.36.41	cmd=listStatus	src=/user/root	dst=null	perm=null
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 19, 2012 at 2:29 PM, kashif khan <drkashif8310@gmail.com>wrote:
>>>>>
>>>>>> Yeah, My cluster running. When brows http://hadoop1.example.com:
>>>>>> 50070/dfshealth.jsp. I am getting the main page. Then click on Brows
file
>>>>>> system. I am getting the following:
>>>>>>
>>>>>> hbase
>>>>>> tmp
>>>>>> user
>>>>>>
>>>>>> And when click on user getting:
>>>>>>
>>>>>> beeswax
>>>>>> huuser (I have created)
>>>>>> root (I have created)
>>>>>>
>>>>>> Would you like to see my configuration file. As did not change any
>>>>>> things, all by default. I have installed CDH4.1 and running on VMs.
>>>>>>
>>>>>> Many thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 19, 2012 at 2:04 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>>
>>>>>>> Is your cluster running fine? Are you able to browse Hdfs through
>>>>>>> the Hdfs Web Console at 50070?
>>>>>>>
>>>>>>> Regards,
>>>>>>>     Mohammad Tariq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 19, 2012 at 7:31 PM, kashif khan <drkashif8310@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Many thanks.
>>>>>>>>
>>>>>>>> I have changed the program accordingly. It does not show
any error
>>>>>>>> but one warring , but when I am browsing the HDFS folder,
file is not
>>>>>>>> copied.
>>>>>>>>
>>>>>>>>
>>>>>>>> public class CopyData {
>>>>>>>> public static void main(String[] args) throws IOException{
>>>>>>>>         Configuration conf = new Configuration();
>>>>>>>>         //Configuration configuration = new Configuration();
>>>>>>>>         //configuration.addResource(new
>>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml"));
>>>>>>>>         //configuration.addResource(new
>>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml"));
>>>>>>>>
>>>>>>>>         conf.addResource(new
>>>>>>>> Path("/etc/hadoop/conf/core-site.xml"));
>>>>>>>>         conf.addResource(new Path
>>>>>>>> ("/etc/hadoop/conf/hdfs-site.xml"));
>>>>>>>>          FileSystem fs = FileSystem.get(conf);
>>>>>>>>         Path inputFile = new Path("/usr/Eclipse/Output.csv");
>>>>>>>>         Path outputFile = new Path("/user/hduser/Output1.csv");
>>>>>>>>         fs.copyFromLocalFile(inputFile, outputFile);
>>>>>>>>         fs.close();
>>>>>>>>     }
>>>>>>>> }
>>>>>>>>
>>>>>>>> 19-Nov-2012 13:50:32 org.apache.hadoop.util.NativeCodeLoader
>>>>>>>> <clinit>
>>>>>>>> WARNING: Unable to load native-hadoop library for your platform...
>>>>>>>> using builtin-java classes where applicable
>>>>>>>>
>>>>>>>> Have any idea?
>>>>>>>>
>>>>>>>> Many thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 19, 2012 at 1:18 PM, Mohammad Tariq <dontariq@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> If it is just copying the files without any processing
or change,
>>>>>>>>> you can use something like this :
>>>>>>>>>
>>>>>>>>> public class CopyData {
>>>>>>>>>
>>>>>>>>>     public static void main(String[] args) throws IOException{
>>>>>>>>>
>>>>>>>>>         Configuration configuration = new Configuration();
>>>>>>>>>         configuration.addResource(new
>>>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml"));
>>>>>>>>>         configuration.addResource(new
>>>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml"));
>>>>>>>>>         FileSystem fs = FileSystem.get(configuration);
>>>>>>>>>         Path inputFile = new
>>>>>>>>> Path("/home/mohammad/pc/work/FFT.java");
>>>>>>>>>         Path outputFile = new Path("/mapout/FFT.java");
>>>>>>>>>         fs.copyFromLocalFile(inputFile, outputFile);
>>>>>>>>>         fs.close();
>>>>>>>>>     }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Obviously you have to modify it as per your requirements
like
>>>>>>>>> continuously polling the targeted directory for new files.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>     Mohammad Tariq
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 19, 2012 at 6:23 PM, kashif khan <
>>>>>>>>> drkashif8310@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks M  Tariq
>>>>>>>>>>
>>>>>>>>>> As I am new in  Java and Hadoop and have no much
experience. I am
>>>>>>>>>> trying to first write a simple program to upload
data into HDFS and
>>>>>>>>>> gradually move forward. I have written the following
simple program to
>>>>>>>>>> upload the file into HDFS, I dont know why it does
not working.  could you
>>>>>>>>>> please check it, if have time.
>>>>>>>>>>
>>>>>>>>>> import java.io.BufferedInputStream;
>>>>>>>>>> import java.io.BufferedOutputStream;
>>>>>>>>>> import java.io.File;
>>>>>>>>>> import java.io.FileInputStream;
>>>>>>>>>> import java.io.FileOutputStream;
>>>>>>>>>> import java.io.IOException;
>>>>>>>>>> import java.io.InputStream;
>>>>>>>>>> import java.io.OutputStream;
>>>>>>>>>> import java.nio.*;
>>>>>>>>>> //import java.nio.file.Path;
>>>>>>>>>>
>>>>>>>>>> import org.apache.hadoop.conf.Configuration;
>>>>>>>>>> import org.apache.hadoop.fs.FSDataInputStream;
>>>>>>>>>> import org.apache.hadoop.fs.FSDataOutputStream;
>>>>>>>>>> import org.apache.hadoop.fs.FileSystem;
>>>>>>>>>> import org.apache.hadoop.fs.Path;
>>>>>>>>>> public class hdfsdata {
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public static void main(String [] args) throws IOException
>>>>>>>>>> {
>>>>>>>>>>     try{
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     Configuration conf = new Configuration();
>>>>>>>>>>     conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
>>>>>>>>>>     conf.addResource(new Path ("/etc/hadoop/conf/hdfs-site.xml"));
>>>>>>>>>>     FileSystem fileSystem = FileSystem.get(conf);
>>>>>>>>>>     String source = "/usr/Eclipse/Output.csv";
>>>>>>>>>>     String dest = "/user/hduser/input/";
>>>>>>>>>>
>>>>>>>>>>     //String fileName = source.substring(source.lastIndexOf('/')
>>>>>>>>>> + source.length());
>>>>>>>>>>     String fileName = "Output1.csv";
>>>>>>>>>>
>>>>>>>>>>     if (dest.charAt(dest.length() -1) != '/')
>>>>>>>>>>     {
>>>>>>>>>>         dest = dest + "/" +fileName;
>>>>>>>>>>     }
>>>>>>>>>>     else
>>>>>>>>>>     {
>>>>>>>>>>         dest = dest + fileName;
>>>>>>>>>>
>>>>>>>>>>     }
>>>>>>>>>>     Path path = new Path(dest);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     if(fileSystem.exists(path))
>>>>>>>>>>     {
>>>>>>>>>>         System.out.println("File" + dest + " already
exists");
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    FSDataOutputStream out = fileSystem.create(path);
>>>>>>>>>>    InputStream in = new BufferedInputStream(new
>>>>>>>>>> FileInputStream(new File(source)));
>>>>>>>>>>    File myfile = new File(source);
>>>>>>>>>>    byte [] b = new byte [(int) myfile.length() ];
>>>>>>>>>>    int numbytes = 0;
>>>>>>>>>>    while((numbytes = in.read(b)) >= 0)
>>>>>>>>>>
>>>>>>>>>>    {
>>>>>>>>>>        out.write(b,0,numbytes);
>>>>>>>>>>    }
>>>>>>>>>>    in.close();
>>>>>>>>>>    out.close();
>>>>>>>>>>    //bos.close();
>>>>>>>>>>    fileSystem.close();
>>>>>>>>>>     }
>>>>>>>>>>     catch(Exception e)
>>>>>>>>>>     {
>>>>>>>>>>
>>>>>>>>>>         System.out.println(e.toString());
>>>>>>>>>>     }
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks again,
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> KK
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 19, 2012 at 12:41 PM, Mohammad Tariq
<
>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> You can set your cronjob to execute the program
after every 5
>>>>>>>>>>> sec.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>     Mohammad Tariq
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 19, 2012 at 6:05 PM, kashif khan
<
>>>>>>>>>>> drkashif8310@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Well, I want to automatically upload the
files as  the files
>>>>>>>>>>>> are generating about every 3-5 sec and each
file has size about 3MB.
>>>>>>>>>>>>
>>>>>>>>>>>>  Is it possible to automate the system using
put or cp command?
>>>>>>>>>>>>
>>>>>>>>>>>> I read about the flume and webHDFS but I
am not sure it will
>>>>>>>>>>>> work or not.
>>>>>>>>>>>>
>>>>>>>>>>>> Many thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 19, 2012 at 12:26 PM, Alexander
Alten-Lorenz <
>>>>>>>>>>>> wget.null@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why do you don't use HDFS related tools
like put or cp?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Alex
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 19, 2012, at 11:44 AM, kashif
khan <
>>>>>>>>>>>>> drkashif8310@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> > HI,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I am generating files continuously
in local folder of my
>>>>>>>>>>>>> base machine. How
>>>>>>>>>>>>> > I can now use the flume to stream
the generated files from
>>>>>>>>>>>>> local folder to
>>>>>>>>>>>>> > HDFS.
>>>>>>>>>>>>> > I dont know how exactly configure
the sources, sinks and
>>>>>>>>>>>>> hdfs.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1) location of folder where files
are generating:
>>>>>>>>>>>>> /usr/datastorage/
>>>>>>>>>>>>> > 2) name node address: htdfs://hadoop1.example.com:8020
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Please let me help.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Many thanks
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Best regards,
>>>>>>>>>>>>> > KK
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Alexander Alten-Lorenz
>>>>>>>>>>>>> http://mapredit.blogspot.com
>>>>>>>>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message