hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Dorner <christopher.dor...@gmail.com>
Subject Re: Re: Creation of Hfiles for multiple tables using Single Bulk Load Job?
Date Sun, 30 Oct 2011 11:57:28 GMT
Hi,

i am facing a similar problem. I Need to read a large file to put into 
different hbase tables. Until now i have done it with 
MultiTableOutputFormat directly from the Mapper. Thats works ok, but i 
believe it will become quite slow when i try larger files. But I thought 
it is a good chance to try bulk load as well and compare them because i 
think it can be much faster in my scenario.

It works fast and like a charm for one table. But i want to avoid 
reading the file a few times.

Did you finally manage to make it work? Also with the lack of 
"configureIncrementalLoad" in the MultiHFileOutputFormat class? I am 
having only one family.

Regards,
Christopher



Am -10.01.-28163 20:59, schrieb Arsalan Bilal:
> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat" code.
> And it works fine for my solution. It generate data for multiple families,
> for multiple table and create multiple HFiles accordingly to related Family
> and save it at different paths.
>
> Thanks Andrew for indicating right direction and Thanks all community for
> help
>
>
>
> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan Bilal<charsalanbilal@gmail.com>wrote:
>
>> Now this time, i am trying to achieve my objective by playing with
>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my target.
>> So i need your help
>> I am using two
>> context.write(key, kv1);
>> context.write(key, kv2);
>> then it should write/generate data/Hfile on two different paths for two
>> different tables. I am trying to achieve this. Plz help.
>>
>>
>>
>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<stack@duboce.net>  wrote:
>>
>>> Can you play around w/ the class to make it work for your case?
>>> St.Ack
>>>
>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan Bilal<charsalanbilal@gmail.com>
>>> wrote:
>>>> @Andrew
>>>>
>>>> Can u help me? U suggest that Try this: https://gist.github.com/1237770,
i
>>>> saw it, How we can we use it actually?
>>>> i try to use it as
>>> job.setOutputFormatClass(MultiHFileOutputFormat.class);
>>>> but it generate different column Families(not correct) inside one
>>>> folder(recursively).  I need to generate different column families in
>>>> different folders.
>>>>   And there is also lack of configureIncrementalLoad Function in your
>>> pointed
>>>> file.
>>>>
>>>> Plz help me Andrew, Stack, i need your help.
>>>>
>>>>
>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<shujamughal@gmail.com>
>>> wrote:
>>>>
>>>>> @arslan,
>>>>> I did not get chance to look at it.
>>>>>
>>>>>
>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal<
>>> charsalanbilal@gmail.com>wrote:
>>>>>
>>>>>> I am also facing same kind of problem, I am using one Map only and
>>> need to
>>>>>> generate multiple Hfiles for multiple table for insertion in Hbase?
>>>>>> Have you found any solution ? Can u help me to find solution of this
>>> one?
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<apurtell@apache.org
>>>> wrote:
>>>>>>
>>>>>>> Try this: https://gist.github.com/1237770
>>>>>>>
>>>>>>> See line 135.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>>
>>>>>>>     - Andy
>>>>>>>
>>>>>>>
>>>>>>> Problems worthy of attack prove their worth by hitting back.
- Piet
>>> Hein
>>>>>>> (via Tom White)
>>>>>>>
>>>>>>>
>>>>>>>> ________________________________
>>>>>>>> From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>> To: user@hbase.apache.org; Andrew Purtell<apurtell@apache.org>
>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM
>>>>>>>> Subject: Re: Creation of Hfiles for multiple tables using
Single
>>> Bulk
>>>>>>> Load Job?
>>>>>>>>
>>>>>>>>
>>>>>>>> andrew,
>>>>>>>>
>>>>>>>> can you guide me where to change and what to change? it will
help.
>>>>>>>> thnx
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman<
>>> shujamughal@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Can you update it so I can use it? :)
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell<
>>> apurtell@apache.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I am using bulk load to insert data into hbase.
Its runs fine if
>>> I
>>>>>>> need to
>>>>>>>>>>> insert in one table. But Now, I have the requirement
in which I
>>> need
>>>>>>> to
>>>>>>>>>>> insert data into more than  one table.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We started some support for this here:
>>>>>>> https://issues.apache.org/jira/browse/HBASE-3727
>>>>>>>>>>
>>>>>>>>>> It needs to be updated for HFile v2, I think, a couple
of single
>>> line
>>>>>>> changes.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    - Andy
>>>>>>>>>>
>>>>>>>>>> Problems worthy of attack prove their worth by hitting
back. -
>>> Piet
>>>>>>> Hein (via Tom White)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: Shuja Rehman<shujamughal@gmail.com>
>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>> Cc:
>>>>>>>>>>> Sent: Thursday, September 22, 2011 8:01 AM
>>>>>>>>>>> Subject: Creation of Hfiles for multiple tables
using Single
>>> Bulk
>>>>>>> Load Job?
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> I am using bulk load to insert data into hbase.
Its runs fine if
>>> I
>>>>>>> need to
>>>>>>>>>>> insert in one table. But Now, I have the requirement
in which I
>>> need
>>>>>>> to
>>>>>>>>>>> insert data into more than  one table. But it
seems that using
>>> bulk
>>>>>>> load, we
>>>>>>>>>>> can generate hfiles for 1 table only.  So the
question is that
>>> is
>>>>>>> there any
>>>>>>>>>>> solution which enable us to generate hfiles in
multiple output
>>> paths
>>>>>>> and
>>>>>>>>>>> then we can load these different files in different
hbase
>>> tables.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards
>>>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>>
>>>>>>>>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards
>>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards
>>>>>>>> Shuja-ur-Rehman Baig
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Arsalan Bilal
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards
>>>>> Shuja-ur-Rehman Baig
>>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Arsalan Bilal
>>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Arsalan Bilal
>>
>>
>>
>
>


Mime
View raw message