hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 宾莉金 <binli...@gmail.com>
Subject DFSClient Packet‘is not fully used,introduced Packet Pool and reuse Packet.
Date Sun, 29 Jul 2012 15:09:01 GMT
hi,  all devs
    Current when FSDataOutputStream's write and sync method are called a
new Packet may be created and put into dataQueue, the daemon DataStreamer
will get packet from the dataQueue and write to datanode.
    I find every Packet will new a byte array with size is 65557(more than
64K). but the pktLen is just a few hundreds or a few ten hunderds . So new
a byte array with big size but just use a very little of it. So the
utilization
rate is very low.
   If 1024 Packet are created one second,64M+ garbage is produced one
second.
   I rewrite the logic of new Packet with a PacketPool,I find the
hregionserver's gc frequency and gc total time reduce a lot,but a single gc
time increase a lot,so i reduce the jvm New gen size,but the same result
get,The average RT increase and TPS reduced. I cann't see why? And wish
some one can told me why? may be the reason is gc.
  Although average RT increase,but more request has little rt and also more
request has a big rt,so the  average RT increase.



-- 
*Best Regards,*
 lijin bin

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message