Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of yaron.gonen@gmail.com
 designates 74.125.83.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <504F35EF.8050702@amd.com>
References: 
 <CAKj4Onxp9LSOozrN2R71sjqxBnZ7=Obi6YDb-D65RAX7Cj3npg@mail.gmail.com>
	<504F35EF.8050702@amd.com>
Date: Tue, 11 Sep 2012 17:25:25 +0300
Message-ID: 
 <CAKj4OnxCU9UVWsuyZPBuvWJUOhz-VknSCnrg3nL-jfWsb9Kk5Q@mail.gmail.com>
Subject: Re: Some general questions about DBInputFormat
From: Yaron Gonen <yaron.gonen@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151747b49a5910d004c96dd8bb

--00151747b49a5910d004c96dd8bb
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the fast response.
Nick, regarding locking a table: as far as I understood from the code, each
mapper opens its own connection to the DB. I didn't see any code such that
the job creates a transaction and passes it to the mapper. Did I
miss something?
again, thanks!


On Tue, Sep 11, 2012 at 4:00 PM, Nick Jones <nick.jones@amd.com> wrote:

> Hi Yaron
>
> Replies inline below.
>
>
> On 09/11/2012 07:41 AM, Yaron Gonen wrote:
>
>> Hi,
>> After reviewing the class's (not very complicated) code, I have some
>> questions I hope someone can answer:
>>
>>   * (more general question) Are there many use-cases for using
>>
>>     DBInputFormat? Do most Hadoop jobs take their input from files or DBs?
>>
>>  Bejoy's right, most jobs utilize data across HDFS or some other
> distributed architecture to feed M/R at a sufficient rate. DBInputFormat
> could be helpful in pulling pointers to other sources of data (e.g. file
> paths for filers where actual binary content is stored).
>
>>
>>   * What happens when the database is updated during mappers' data
>>
>>     retrieval phase? is there a way to lock the database before the
>>     data retrieval phase and release it afterwords?
>>
>>  The whole job creates a transaction against the RBDMS that ensures
> consistent state throughout the job.  Depending on the source and settings,
> this might entirely lock a table or lock the selected rows by the query.
>
>>
>>   * Since all mappers open a connection to the same DBS, one cannot
>>
>>     use hundreds of mapper. Is there a solution to this problem?
>>
>>  Depends on the connection limits and the number of rows requested. I've
> found that the server suffered other problems first before connection count
> limitations.
>
>>
>> Thanks,
>> Yaron
>>
>
>
>

--00151747b49a5910d004c96dd8bb
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks for the fast response.<div>Nick, regarding locking =
a table: as far as I understood from the code, each mapper opens its own co=
nnection to the DB. I didn&#39;t see any code such that the job creates a t=
ransaction and passes it to the mapper. Did I miss=A0something?</div>
<div>again, thanks!</div><div><br><br><div class=3D"gmail_quote">On Tue, Se=
p 11, 2012 at 4:00 PM, Nick Jones <span dir=3D"ltr">&lt;<a href=3D"mailto:n=
ick.jones@amd.com" target=3D"_blank">nick.jones@amd.com</a>&gt;</span> wrot=
e:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi Yaron<br>
<br>
Replies inline below.<div class=3D"im"><br>
<br>
On 09/11/2012 07:41 AM, Yaron Gonen wrote:<br>
</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex"><div class=3D"im">
Hi,<br>
After reviewing the class&#39;s (not very complicated) code, I have some qu=
estions I hope someone can answer:<br>
<br></div>
=A0 * (more general question) Are there many use-cases for using<div class=
=3D"im"><br>
=A0 =A0 DBInputFormat? Do most Hadoop jobs take their input from files or D=
Bs?<br>
<br>
</div></blockquote>
Bejoy&#39;s right, most jobs utilize data across HDFS or some other distrib=
uted architecture to feed M/R at a sufficient rate. DBInputFormat could be =
helpful in pulling pointers to other sources of data (e.g. file paths for f=
ilers where actual binary content is stored).<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<br>
=A0 * What happens when the database is updated during mappers&#39; data<di=
v class=3D"im"><br>
=A0 =A0 retrieval phase? is there a way to lock the database before the<br>
=A0 =A0 data retrieval phase and release it afterwords?<br>
<br>
</div></blockquote>
The whole job creates a transaction against the RBDMS that ensures consiste=
nt state throughout the job. =A0Depending on the source and settings, this =
might entirely lock a table or lock the selected rows by the query.<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<br>
=A0 * Since all mappers open a connection to the same DBS, one cannot<div c=
lass=3D"im"><br>
=A0 =A0 use hundreds of mapper. Is there a solution to this problem?<br>
<br>
</div></blockquote>
Depends on the connection limits and the number of rows requested. I&#39;ve=
 found that the server suffered other problems first before connection coun=
t limitations.<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
Yaron<br>
</blockquote>
<br>
<br>
</blockquote></div><br></div></div>

--00151747b49a5910d004c96dd8bb--