hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: IO Error when using multiple HBaseStorage in PIG
Date Sat, 21 May 2011 06:07:38 GMT
Digging into this bug a bit, I think I have a feel for what's happening, but
I want to check.

It seems like since the MR job is writing to two HBase tables, it's using
two instances of TableOutputFormat in the same thread. This means two
instances of HTable in the same thread. From previous discussions(1), HTable
shares connection state unless HTablePool is used. Since TableOutputFormat
doesn't use HTablePool though, it seems like the first call to
TableOutputFormat.close (which calls HTable.flushCommits) will close the
connection and the second call to it will then fail. Hence the second table
never gets written to

If this is the case, then the question becomes just how do you write to two
tables in one MR job using TableOutputFormat?

1- http://search-hadoop.com/m/IsdwtMF2pV/HTable+reuse/v=plain


On Fri, May 20, 2011 at 2:29 PM, Bill Graham <billgraham@gmail.com> wrote:

> Yes, that's what it seems. I've opened a Pig JIRA for it:
>
> https://issues.apache.org/jira/browse/PIG-2085
>
>
> On Thu, May 19, 2011 at 1:31 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> Your attachement didn't make it, it rarely does on the mailing lists.
>> I suggest you use a gist.github or a pastebin.
>>
>> Regarding the error, looks like something closed the HCM and someone
>> else is trying to use it. Since this is client side, it would point to
>> a Pig problem.
>>
>> J-D
>>
>> On Thu, May 19, 2011 at 12:28 PM, Keric Donnelly
>> <keric.donnelly@cbsinteractive.com> wrote:
>> > To All,
>> > I'm running into IO issues when trying to write to an Hbase table using
>> > multiple STORE commands in PIG script. I can comment out any 2 of the
>> STORE
>> > statements and run the script and then the data inserts fine. If I try
>> and
>> > run with all 3  get the following:
>> > java.io.IOException:
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@26c1186f
>> > closed
>> >
>> > I've attached a simplified version of the script to illustrate the
>> problem.
>> >
>> > Any thoughts or comments are appreciated.
>> >
>> > Thanks
>> >
>> > Keric
>> >
>> >
>> > --
>> >
>> > Keric Donnelly
>> > Senior Data Architect
>> > T 954-689-3291   C 954.683.5445
>> >
>> > 1401 West Cypress Creek Road, Fort Lauderdale, FL 33309
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message