hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 宾莉金 <binli...@gmail.com>
Subject Re: DFSClient Packet‘is not fully used,introduced Packet Pool and reuse Packet.
Date Sun, 29 Jul 2012 15:26:57 GMT
  Packet utilization rate is low, because when sync a new packet will be
forced to be created. The smaller the key/value, the low the utilization
rate will be.

2012/7/29 宾莉金 <binlijin@gmail.com>

> hi,  all devs
>     Current when FSDataOutputStream's write and sync method are called a
> new Packet may be created and put into dataQueue, the daemon DataStreamer
> will get packet from the dataQueue and write to datanode.
>     I find every Packet will new a byte array with size is 65557(more than
> 64K). but the pktLen is just a few hundreds or a few ten hunderds . So new
> a byte array with big size but just use a very little of it. So the  utilization
> rate is very low.
>    If 1024 Packet are created one second,64M+ garbage is produced one
> second.
>    I rewrite the logic of new Packet with a PacketPool,I find the
> hregionserver's gc frequency and gc total time reduce a lot,but a single gc
> time increase a lot,so i reduce the jvm New gen size,but the same result
> get,The average RT increase and TPS reduced. I cann't see why? And wish
> some one can told me why? may be the reason is gc.
>   Although average RT increase,but more request has little rt and also
> more request has a big rt,so the  average RT increase.
> --
> *Best Regards,*
>  lijin bin

*Best Regards,*
 lijin bin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message