hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriele Kahlout <gabri...@mysimpatico.com>
Subject How to -moveToLocal to local machine instead of remote machine?
Date Sun, 31 Jul 2011 18:41:48 GMT
I have sort of the same problem I feel.
I've a large segment I cannot index because there's not enough room, nor
-copyToLocal, local being the server.
How can I instead -copyToLocal where local is my local terminal machine,
instead of the server?


On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt <joe.greenawalt@gmail.com>wrote:

> Thanks, and I've seen this example.  I think after I can connect, i'm ok,
> but i'm not sure how to do it remotely.  I'm writing a groovy script just
> to
> test connection and i'll paste it below so you can see what i'm trying to
> do.
>
> @GrabResolver(name='org.apache.mahout.hadoop', root='
> > http://mymavenrepo/nexus/content/repositories/thirdparty')
> > @Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
> > version='0.20.203.0')
> > import org.apache.hadoop.hdfs.DFSClient
> > import org.apache.hadoop.hdfs.DistributedFileSystem
> > import org.apache.hadoop.conf.Configuration
> > org.apache.hadoop.hdfs.protocol.HdfsFileStatus
> > def DIR_HADOOP = "1.1.1.1";
> > def PORT_HADOOP = "9000";
> > def config = new Configuration()
> > config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP)  //got
> this
> > from from some site
> > def dfs = new DistributedFileSystem()
> > def dfsClient = dfs.getClient()
> > def fileInfo = dfsClient.getFileInfo("/DEV")
> > println fileInfo.isDir()
>
>
> On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
> William.Habermaas@fatwire.com> wrote:
>
> > Joe,
> >
> > Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
> >
> > It should give you an idea of how to read and write HDFS files. This page
> > is somewhat old and the package names have changed a bit between versions
> > but I hope it will get you on the right track. If you don't want to write
> > code there are HDFS copy utilities that you can use instead from shell
> > scripts.
> >
> > Bill
> >
> > -----Original Message-----
> > From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> > Sent: Tuesday, June 07, 2011 1:38 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: remotely downloading file
> >
> > Bill,
> > thanks for the reply, is there a resource that you have available that i
> > can
> > look at the correct way to connect remotely?
> > I seem to be seeing conflicting ways on doing that.
> >
> > I'm looking at:
> >
> >
> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
> >
> >
> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
> >
> > But the examples i'm seeing are using the Configuration but i don't see
> > that
> > being used in those classes.
> >
> > Thanks again,
> >
> > Joe
> >
> > On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
> > William.Habermaas@fatwire.com> wrote:
> >
> > > You can access HDFS for reading and writing from other machines. The
> API
> > > works through the HDFS client which can be anywhere on the network and
> > not
> > > just on the namenode. You just have to have the Hadoop core jar with
> your
> > > application wherever it is going to run.
> > >
> > > Bill
> > >
> > > -----Original Message-----
> > > From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> > > Sent: Friday, June 03, 2011 4:55 PM
> > > To: common-user@hadoop.apache.org
> > > Subject: remotely downloading file
> > >
> > > Hi,
> > > We're interested in using HDFS to store several large file sets to be
> > > available for download from our customers  in the following paradigm:
> > >
> > > Customer  <-  | APPSERVER-CLUSTER {app1,app2,app3} |  <-  | HDFS |
> > >
> > > I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
> > > could
> > > be done program-ably remotely after browsing the documentation.  But
> > after
> > > reading the API, it seems that writing Java code to interface with HDFS
> > > needs to happen locally?  Is that correct?
> > >
> > > If it is correct, what is the best/recommended way to
> > > deliver downloadables to the APPSERVERS (and vice versa) which are
> hosted
> > > in
> > > the same network but on different machines?
> > >
> > > Thanks,
> > > Joe
> > >
> >
>



-- 
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message