Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Appropriate for Hadoop?
Date: Wed, 29 Apr 2009 09:36:27 +0100
Message-ID: 
 <E5AF08CA89940F45AE064CAFF25C8A900A6482A6@EXMXCLUSTER.corp.edrlandmark.net>
In-Reply-To: <49efc3330904281224r36738921qbf994b12fddb5da7@mail.gmail.com>
Thread-Topic: Appropriate for Hadoop?
Thread-Index: AcnINxuspsInbDjlTw6+IZYoeQhXjQAblCEg
References: 
 <2C52DBBEC4855C438BB330CB0D3B46590131C8FB@SNV-EXVS01.ds.corp.yahoo.com>
 <E5AF08CA89940F45AE064CAFF25C8A900A5E9D32@EXMXCLUSTER.corp.edrlandmark.net>
 <49efc3330904281224r36738921qbf994b12fddb5da7@mail.gmail.com>
From: "Adam Retter" <Adam.Retter@landmark.co.uk>
To: <core-user@hadoop.apache.org>

I was more concerned that the input to our input is from SQL databases
and a proprietary EMC document store. And that our output is to a
different SQL database.

I don't want to use any sort of file system at all.

=20
Adam Retter
Software Developer
Landmark Information Group
=20
T: 01392 685403 (x5403)=20
=20
5-7 Abbey Court, Eagle Way, Sowton,
Exeter, Devon, EX2 7HY
=20
www.landmark.co.uk
=20
-----Original Message-----
From: Chuck Lam [mailto:chuck.lam@gmail.com]=20
Sent: 28 April 2009 20:25
To: core-user@hadoop.apache.org
Subject: Re: Appropriate for Hadoop?

HDFS is designed with Hadoop in mind, so there are certain advantages
(e.g.
performance, reliability, and ease of use) to using HDFS for Hadoop.
However, it's not required. For example, when you run Hadoop in
standalone
mode, it just uses the file system on your local machine. When you run
it on
Amazon AWS, it can use S3 as a file system.


On Tue, Apr 28, 2009 at 6:15 AM, Adam Retter
<Adam.Retter@landmark.co.uk>wrote:

>
> > Each document processing is independent and can be processed
> > parallelly, so that part could be done in a map reduce job.
> > Now whether it suits this use case depends on rate at which new
> > URI's are discovered for processing and acceptable delay in
processing
> > of a document. The way I see it you can batch the URI's
> > and input that to mapreduce job. Each mapper can work on sublist of
> URIs.
> > You can choose to make DB inserts from mapper itself. In that case
> > you can set no of reducers to 0. Otherwise if batching of the
queries
> > is an option then you can consider making batch inserts in reducer.
It
> > will help in reducing load on DB.
>
> So I don't have to use HDFS at all when using Hadoop?
>
>
>
> Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon,
EX2 7HY
> Registered Number 2892803 Registered in England and Wales
>
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
>
> The information contained in this e-mail is confidential and may be
subject
> to
> legal privilege. If you are not the intended recipient, you must not
use,
> copy,
> distribute or disclose the e-mail or any part of its contents or take
any
> action in reliance on it. If you have received this e-mail in error,
please
> e-mail the sender by replying to this message. All reasonable
precautions
> have
> been taken to ensure no viruses are present in this e-mail. Landmark
> Information
> Group Limited cannot accept responsibility for loss or damage arising
from
> the
> use of this e-mail or attachments and recommend that you subject these
to
> your virus checking procedures prior to use.
>
>

Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY
Registered Number 2892803 Registered in England and Wales=20

This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email=20

The information contained in this e-mail is confidential and may be subject=
 to=20
legal privilege. If you are not the intended recipient, you must not use, c=
opy,=20
distribute or disclose the e-mail or any part of its contents or take any=
=20
action in reliance on it. If you have received this e-mail in error, please=
=20
e-mail the sender by replying to this message. All reasonable precautions h=
ave=20
been taken to ensure no viruses are present in this e-mail. Landmark Inform=
ation
Group Limited cannot accept responsibility for loss or damage arising from =
the=20
use of this e-mail or attachments and recommend that you subject these to=
=20
your virus checking procedures prior to use.