hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Dorner <christopher.dor...@gmail.com>
Subject Re: Creation of Hfiles for multiple tables using Single Bulk Load Job?
Date Thu, 03 Nov 2011 16:53:22 GMT
Thanks for your reply.
Yes, it would be great if he could publish his version.


Am 02.11.2011 21:27, schrieb Ted Yu:
> If Arsalan publishes the modified version of MultiHFileoutputFormat, that
> would expedite adding support in 0.94
> After some validation, this can be backported.
>
> Cheers
>
> On Wed, Nov 2, 2011 at 10:14 AM, Christopher Dorner<
> christopher.dorner@gmail.com>  wrote:
>
>> Will HBase 0.92 support MultiHFileoutputFormat and IncrementalLoad for
>> different Tables?
>>
>> Is there a comfortable way to make it work for HBase 0.90.4 as well? I am
>> using Cloudera's CDH3u2.
>>
>>
>> Am 30.10.2011 12:57, schrieb Christopher Dorner:
>>
>>   Hi,
>>>
>>> i am facing a similar problem. I Need to read a large file to put into
>>> different hbase tables. Until now i have done it with
>>> MultiTableOutputFormat directly from the Mapper. Thats works ok, but i
>>> believe it will become quite slow when i try larger files. But I thought
>>> it is a good chance to try bulk load as well and compare them because i
>>> think it can be much faster in my scenario.
>>>
>>> It works fast and like a charm for one table. But i want to avoid
>>> reading the file a few times.
>>>
>>> Did you finally manage to make it work? Also with the lack of
>>> "configureIncrementalLoad" in the MultiHFileOutputFormat class? I am
>>> having only one family.
>>>
>>> Regards,
>>> Christopher
>>>
>>>
>>>
>>> Am -10.01.-28163 20:59, schrieb Arsalan Bilal:
>>>
>>>> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat"
>>>> code.
>>>> And it works fine for my solution. It generate data for multiple
>>>> families,
>>>> for multiple table and create multiple HFiles accordingly to related
>>>> Family
>>>> and save it at different paths.
>>>>
>>>> Thanks Andrew for indicating right direction and Thanks all community for
>>>> help
>>>>
>>>>
>>>>
>>>> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan
>>>> Bilal<charsalanbilal@gmail.com**>wrote:
>>>>
>>>>   Now this time, i am trying to achieve my objective by playing with
>>>>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my
>>>>> target.
>>>>> So i need your help
>>>>> I am using two
>>>>> context.write(key, kv1);
>>>>> context.write(key, kv2);
>>>>> then it should write/generate data/Hfile on two different paths for two
>>>>> different tables. I am trying to achieve this. Plz help.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<stack@duboce.net>  wrote:
>>>>>
>>>>>   Can you play around w/ the class to make it work for your case?
>>>>>> St.Ack
>>>>>>
>>>>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan
>>>>>> Bilal<charsalanbilal@gmail.com**>
>>>>>> wrote:
>>>>>>
>>>>>>> @Andrew
>>>>>>>
>>>>>>> Can u help me? U suggest that Try this:
>>>>>>> https://gist.github.com/**1237770<https://gist.github.com/1237770>,
>>>>>>> i
>>>>>>> saw it, How we can we use it actually?
>>>>>>> i try to use it as
>>>>>>>
>>>>>> job.setOutputFormatClass(**MultiHFileOutputFormat.class);
>>>>>>
>>>>>>> but it generate different column Families(not correct) inside
one
>>>>>>> folder(recursively). I need to generate different column families
in
>>>>>>> different folders.
>>>>>>> And there is also lack of configureIncrementalLoad Function in
your
>>>>>>>
>>>>>> pointed
>>>>>>
>>>>>>> file.
>>>>>>>
>>>>>>> Plz help me Andrew, Stack, i need your help.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<shujamughal@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>   @arslan,
>>>>>>>> I did not get chance to look at it.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal<
>>>>>>>>
>>>>>>> charsalanbilal@gmail.com>**wrote:
>>>>>>
>>>>>>>
>>>>>>>>   I am also facing same kind of problem, I am using one Map
only and
>>>>>>>>>
>>>>>>>> need to
>>>>>>
>>>>>>> generate multiple Hfiles for multiple table for insertion in
Hbase?
>>>>>>>>> Have you found any solution ? Can u help me to find solution
of this
>>>>>>>>>
>>>>>>>> one?
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<apurtell@apache.org
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>>   Try this: https://gist.github.com/**1237770<https://gist.github.com/1237770>
>>>>>>>>>>
>>>>>>>>>> See line 135.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Andy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Problems worthy of attack prove their worth by hitting
back. - Piet
>>>>>>>>>>
>>>>>>>>> Hein
>>>>>>
>>>>>>> (via Tom White)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>   ______________________________**__
>>>>>>>>>>> From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>>>>> To: user@hbase.apache.org; Andrew Purtell<apurtell@apache.org>
>>>>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM
>>>>>>>>>>> Subject: Re: Creation of Hfiles for multiple
tables using Single
>>>>>>>>>>>
>>>>>>>>>> Bulk
>>>>>>
>>>>>>> Load Job?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> andrew,
>>>>>>>>>>>
>>>>>>>>>>> can you guide me where to change and what to
change? it will help.
>>>>>>>>>>> thnx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman<
>>>>>>>>>>>
>>>>>>>>>> shujamughal@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you update it so I can use it? :)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell<
>>>>>>>>>>>>
>>>>>>>>>>> apurtell@apache.org>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>   From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>   I am using bulk load to insert data
into hbase. Its runs
>>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I
>>>>>>
>>>>>>> need to
>>>>>>>>>>
>>>>>>>>>>> insert in one table. But Now, I have the requirement
in which I
>>>>>>>>>>>>>>
>>>>>>>>>>>>> need
>>>>>>
>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> insert data into more than one table.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We started some support for this here:
>>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/**jira/browse/HBASE-3727<https://issues.apache.org/jira/browse/HBASE-3727>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> It needs to be updated for HFile v2,
I think, a couple of single
>>>>>>>>>>>>>
>>>>>>>>>>>> line
>>>>>>
>>>>>>> changes.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Andy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Problems worthy of attack prove their
worth by hitting back. -
>>>>>>>>>>>>>
>>>>>>>>>>>> Piet
>>>>>>
>>>>>>> Hein (via Tom White)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>
>>>>>>>>>>>>>> From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>>>> Cc:
>>>>>>>>>>>>>> Sent: Thursday, September 22, 2011
8:01 AM
>>>>>>>>>>>>>> Subject: Creation of Hfiles for multiple
tables using Single
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Bulk
>>>>>>
>>>>>>> Load Job?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using bulk load to insert data
into hbase. Its runs
>>>>>>>>>>>>>> fine if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I
>>>>>>
>>>>>>> need to
>>>>>>>>>>
>>>>>>>>>>> insert in one table. But Now, I have the requirement
in which I
>>>>>>>>>>>>>>
>>>>>>>>>>>>> need
>>>>>>
>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> insert data into more than one table. But it
seems that using
>>>>>>>>>>>>>>
>>>>>>>>>>>>> bulk
>>>>>>
>>>>>>> load, we
>>>>>>>>>>
>>>>>>>>>>> can generate hfiles for 1 table only. So the
question is that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>
>>>>>>> there any
>>>>>>>>>>
>>>>>>>>>>> solution which enable us to generate hfiles in
multiple output
>>>>>>>>>>>>>>
>>>>>>>>>>>>> paths
>>>>>>
>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> then we can load these different files in different
hbase
>>>>>>>>>>>>>>
>>>>>>>>>>>>> tables.
>>>>>>
>>>>>>>
>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>   <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards
>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Arsalan Bilal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards
>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Arsalan Bilal
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Arsalan Bilal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>


Mime
View raw message