hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy" <sa...@pearsonwholesale.com>
Subject Re: hbase master heap space
Date Sun, 30 Dec 2007 05:27:18 GMT
also noticed that's the return code should be 200 from the code but on wiki 
it shows it to be

http://wiki.apache.org/lucene-hadoop/Hbase/HbaseRest
HTTP 202 (Accepted) if it can be closed. HTTP 404 (Not Found) if the scanner 
id is invalid. HTTP 410 (Gone) if the scanner is already closed or the lease 
time has expired.

One other the other should be updated to match 202 or 200

Billy


"Bryan Duxbury" <bryan@rapleaf.com> wrote in 
message news:BE0CC670-92BE-41E4-9A94-EAAB72A09F02@rapleaf.com...
> I've created an issue and submitted a patch to fix the problem.
> Billy, can you download the patch and check to see if it works alright?
>
> https://issues.apache.org/jira/browse/HADOOP-2504
>
> -Bryan
>
> On Dec 29, 2007, at 3:36 PM, Billy wrote:
>
>> I checked and added the delete option to my code for the scanner
>> based on
>> the api from wiki but it looks like its not working at this time
>> basedo nthe
>> code and responce I got form the rest interfase. i get a "Not
>> hooked back up
>> yet" responce any idea on when this will be fixed?
>>
>> Thanks
>>
>> src/contrib/hbase/src/java/org/apache/hadoop/hbase/rest/
>> ScannerHandler.java
>>
>> public void doDelete(HttpServletRequest request, HttpServletResponse
>> response,
>> String[] pathSegments)
>> throws ServletException, IOException {
>> doMethodNotAllowed(response, "Not hooked back up yet!");
>> }
>>
>>
>> "Bryan Duxbury" <bryan@rapleaf.com> wrote in
>> message 
>> news:0459216F-F3F0-46C1-B7DB-57A6479BD809@rapleaf.com...
>>> Are you closing the scanners when you're done? If not, those might be
>>> hanging around for a long time. I don't think we've built in the
>>> proper
>>> timeout logic to make that work by itself.
>>>
>>> -Bryan
>>>
>>> On Dec 21, 2007, at 5:10 PM, Billy wrote:
>>>
>>>> I was thanking the same thing and been running REST outside of the
>>>> Master on
>>>> each server for about 5 hours now and used the master as a
>>>> backup  if
>>>> local
>>>> rest interface failed. You are right I seen a little faster
>>>> processing
>>>> time
>>>> from doing this vs. using just the master.
>>>>
>>>> Seams the problem is not with the master its self looks like
>>>> REST  is
>>>> using
>>>> up more and more memory not sure but I thank its to do with inserts
>>>> maybe
>>>> not but the memory usage is going up I an doing a scanner 2 threads
>>>> reading
>>>> rows and processing the data and inserting it in to a separate table
>>>> building a inverted index.
>>>>
>>>> I will restart everything when this job is done and try to do just
>>>> inserts
>>>> and see if its the scanner or inserts.
>>>>
>>>> The master is holding at about 75mb and the rest interfaces are
>>>> up  to
>>>> 400MB
>>>> and slowly going up on the ones running the jobs.
>>>>
>>>> I am still testing I will see what else I can come up with.
>>>>
>>>> Billy
>>>>
>>>>
>>>> "stack" <stack@duboce.net> wrote in 
>>>> message
>>>> news:476C1AA8.3030306@duboce.net...
>>>>> Hey Billy:
>>>>>
>>>>> Master itself should use little memory and though it is not out
>>>>> of  the
>>>>> realm of possibiliites, it should not have a leak.
>>>>>
>>>>> Are you running with the default heap size?  You might want to
>>>>> give it
>>>>> more memory if you are (See
>>>>> http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#3 for how).
>>>>>
>>>>> If you are uploading all via the REST server running on the
>>>>> master, the
>>>>> problem as you speculate, could be in the REST servlet itself
>>>>> (though
>>>>> it
>>>>> looks like it shouldn't be holding on to anything having given it a
>>>>> cursory glance).  You could try running the REST server
>>>>> independent of
>>>>> the
>>>>> master.  Grep for 'Starting the REST Server' in this page,
>>>>> http://wiki.apache.org/lucene-hadoop/Hbase/HbaseRest, for how
>>>>> (If  you
>>>>> are
>>>>> only running one REST instance, your upload might go faster if
>>>>> you  run
>>>>> multiple).
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>> Billy wrote:
>>>>>> I forgot to say that once restart the master only uses about
>>>>>> 70mb of
>>>>>> memory
>>>>>>
>>>>>> Billy
>>>>>>
>>>>>> "Billy" <sales@pearsonwholesale.com> 
>>>>>> wrote
>>>>>> in message news:fkejpo$u8c$1@ger.gmane.org...
>>>>>>
>>>>>>> I not sure of this but why does the master server use up so much
>>>>>>> memory.
>>>>>>> I been running an script that been inserting data into a
>>>>>>> table  for a
>>>>>>> little over 24 hours and the master crashed because of
>>>>>>> java.lang.OutOfMemoryError: Java heap space.
>>>>>>>
>>>>>>> So my question is why does the master use up so much memory
>>>>>>> at  most
>>>>>>> it
>>>>>>> should store the -ROOT-,.META. tables in memory and block to
>>>>>>> table
>>>>>>> mapping.
>>>>>>>
>>>>>>> Is it cache or a memory leak?
>>>>>>>
>>>>>>> I am using the rest interface so could that be the reason?
>>>>>>>
>>>>>>> I inserted according to the high edit ids on all the region
>>>>>>> servers
>>>>>>> about
>>>>>>> 51,932,760 edits and the master ran out of memory with a heap
of
>>>>>>> about
>>>>>>> 1GB.
>>>>>>>
>>>>>>> The other side to this is the data I inserted is only taking
up
>>>>>>> 886.61
>>>>>>> MB and that's with
>>>>>>> dfs.replication set to 2 so half that is only 440MB of data
>>>>>>> compressed
>>>>>>> at the block level.
>>>>>>> From what I understand the master should have lower memory
>>>>>>> and  cpu
>>>>>>> usage
>>>>>>> and the namenode on hadoop should be the memory hog it has to
>>>>>>> keep up
>>>>>>> with all the data about the blocks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
> 




Mime
View raw message