Mailing-List: contact dev-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hama.apache.org
Date: Wed, 31 Oct 2012 21:03:12 +0000 (UTC)
From: "Suraj Menon (JIRA)" <jira@apache.org>
To: dev@hama.apache.org
Message-ID: <1620207665.52999.1351717392906.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (HAMA-559) Add a spilling message queue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HAMA-559?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D134882=
31#comment-13488231 ]=20

Suraj Menon commented on HAMA-559:
----------------------------------

Made 3 more changes:
- Removed few ByteBuffer.remaining calls
- Relaxed the restriction that each buffer should fill completely before mo=
ving to next. (I might revert back this change)
- Added a BufferedOutputStream in between SpillingBuffer and SpillingStream=
. (This improved the efficiency by around 45%)
The performance numbers are now:=20

{noformat}

 0% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10, type=3DDIS=
K_LIST} 3809.61 ns; =CF=83=3D40.46 ns @ 10 trials
 5% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100, type=3DDI=
SK_LIST} 7113.85 ns; =CF=83=3D448.07 ns @ 10 trials
10% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000, type=3DD=
ISK_LIST} 41011.04 ns; =CF=83=3D1462.04 ns @ 10 trials
14% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000, type=3D=
DISK_LIST} 357103.20 ns; =CF=83=3D350967.71 ns @ 10 trials
19% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100000, type=
=3DDISK_LIST} 3606638.74 ns; =CF=83=3D3339040.16 ns @ 10 trials
24% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000000, type=
=3DDISK_LIST} 34951212.41 ns; =CF=83=3D1949660.29 ns @ 10 trials
29% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000000, type=
=3DDISK_LIST} 354762498.83 ns; =CF=83=3D13826670.03 ns @ 10 trials
33% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10, type=3DSPI=
LLING_BUFFER} 3914.27 ns; =CF=83=3D17.48 ns @ 3 trials
38% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100, type=3DSP=
ILLING_BUFFER} 6114.64 ns; =CF=83=3D60.98 ns @ 10 trials
43% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000, type=3DS=
PILLING_BUFFER} 31973.78 ns; =CF=83=3D143.74 ns @ 3 trials
48% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000, type=3D=
SPILLING_BUFFER} 271619.92 ns; =CF=83=3D19861.84 ns @ 10 trials
52% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100000, type=
=3DSPILLING_BUFFER} 2580930.24 ns; =CF=83=3D3416159.19 ns @ 10 trials
57% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000000, type=
=3DSPILLING_BUFFER} 25714962.79 ns; =CF=83=3D45184.42 ns @ 3 trials
62% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000000, type=
=3DSPILLING_BUFFER} 259875751.00 ns; =CF=83=3D19098186.41 ns @ 10 trials
67% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10, type=3DDIS=
K_BUFFER} 3294.61 ns; =CF=83=3D10.86 ns @ 3 trials
71% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100, type=3DDI=
SK_BUFFER} 6442.84 ns; =CF=83=3D18.10 ns @ 3 trials
76% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000, type=3DD=
ISK_BUFFER} 38160.64 ns; =CF=83=3D250.75 ns @ 3 trials
81% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000, type=3D=
DISK_BUFFER} 333411.11 ns; =CF=83=3D4376.56 ns @ 10 trials
86% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D100000, type=
=3DDISK_BUFFER} 3214465.20 ns; =CF=83=3D8146.50 ns @ 3 trials
90% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D1000000, type=
=3DDISK_BUFFER} 30817421.31 ns; =CF=83=3D21434.85 ns @ 3 trials
95% Scenario{vm=3Djava, trial=3D0, benchmark=3DSpill, size=3D10000000, type=
=3DDISK_BUFFER} 316989389.00 ns; =CF=83=3D622679.65 ns @ 3 trials

    size            type        us linear runtime
      10       DISK_LIST      3.81 =3D
      10 SPILLING_BUFFER      3.91 =3D
      10     DISK_BUFFER      3.29 =3D
     100       DISK_LIST      7.11 =3D
     100 SPILLING_BUFFER      6.11 =3D
     100     DISK_BUFFER      6.44 =3D
    1000       DISK_LIST     41.01 =3D
    1000 SPILLING_BUFFER     31.97 =3D
    1000     DISK_BUFFER     38.16 =3D
   10000       DISK_LIST    357.10 =3D
   10000 SPILLING_BUFFER    271.62 =3D
   10000     DISK_BUFFER    333.41 =3D
  100000       DISK_LIST   3606.64 =3D
  100000 SPILLING_BUFFER   2580.93 =3D
  100000     DISK_BUFFER   3214.47 =3D
 1000000       DISK_LIST  34951.21 =3D=3D
 1000000 SPILLING_BUFFER  25714.96 =3D=3D
 1000000     DISK_BUFFER  30817.42 =3D=3D
10000000       DISK_LIST 354762.50 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
10000000 SPILLING_BUFFER 259875.75 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
10000000     DISK_BUFFER 316989.39 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

vm: java
trial: 0
benchmark: Spill

{noformat}
               =20
> Add a spilling message queue
> ----------------------------
>
>                 Key: HAMA-559
>                 URL: https://issues.apache.org/jira/browse/HAMA-559
>             Project: Hama
>          Issue Type: Sub-task
>          Components: bsp core
>    Affects Versions: 0.5.0
>            Reporter: Thomas Jungblut
>            Assignee: Suraj Menon
>            Priority: Minor
>             Fix For: 0.7.0
>
>         Attachments: HAMA-559.patch-v1, spilling_buffer_cpu_usage_text_wr=
ite.png, SpillingBufferProfile-2012-10-27.snapshot, spilling_buffer_profile=
_cpu_graph_test_write.png, spilling_buffer_profile_cpugraph_writeUTF.png, s=
pillingbuffer_profile_cpu_writeUTF.png, spilling_buffer_profile_LOCK.JPG, s=
pilling_buffer_profile_timesplit_text_write.png, spilling_buffer_profile_wr=
iteUTF.png
>
>
> After HAMA-521 is done, we can add a spilling queue which just holds the =
messages in RAM that fit into the heap space. The rest can be flushed to di=
sk.
> We may call this a HybridQueue or something like that.
> The benefits should be that we don't have to flush to disk so often and g=
et faster. However we may have more GC so it is always overall faster.
> The requirements for this queue also include:
> - The message object once written to the queue (after returning from the =
write call) could be modified, but the changes should not be reflected in t=
he messages stored in the queue.
> - For now let's implement a queue that does not support concurrent readin=
g and writing. This feature is needed when we implement asynchronous commun=
ication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira