hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manish Bhoge" <manishbh...@rocketmail.com>
Subject Re: How to load raw log file into HDFS?
Date Tue, 15 May 2012 01:45:27 GMT
You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and
Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS.
If you want to run(test) PIG in local then in that case you don't to copy data to HDFS
Sent from my BlackBerry, pls excuse typo

-----Original Message-----
From: Michael Wang <Michael.Wang@meredith.com>
Date: Mon, 14 May 2012 18:43:47 
To: common-user@hadoop.apache.org<common-user@hadoop.apache.org>
Reply-To: common-user@hadoop.apache.org
Subject: RE: How to load raw log file into HDFS?

I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>. 
Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first,
and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs
from Local_file_path directly to pig or hive? After I load the files to hive, I found it is
put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
How do I know what <HDFS_FILE_PATH> are available? 

-----Original Message-----
From: Alexander Fahlke [mailto:alexander.fahlke.mailinglists@googlemail.com] 
Sent: Monday, May 14, 2012 1:53 AM
To: common-user@hadoop.apache.org
Subject: Re: How to load raw log file into HDFS?


the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:


Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org

On Mon, May 14, 2012 at 7:24 AM, Amith D K <amithdk@huawei.com> wrote:

> U can even use put/copyFromLocal
> both are similar and does the job via terminal.
> Or u can write a simple client program to do the job :)
> Amith
> ________________________________________
> From: samir das mohapatra [samir.helpdoc@gmail.com]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: common-user@hadoop.apache.org
> Subject: Re: How to load raw log file into HDFS?
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <satishthakre@gmail.com>
> wrote:
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >

This electronic message, including any attachments, may contain proprietary, confidential
or privileged information for the sole use of the intended recipient(s). You are hereby notified
that any unauthorized disclosure, copying, distribution, or use of this message is prohibited.
If you have received this message in error, please immediately notify the sender by reply
e-mail and delete it.

View raw message