hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suraj Menon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-559) Add a spilling message queue
Date Wed, 31 Oct 2012 21:03:12 GMT

    [ https://issues.apache.org/jira/browse/HAMA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488231#comment-13488231
] 

Suraj Menon commented on HAMA-559:
----------------------------------

Made 3 more changes:
- Removed few ByteBuffer.remaining calls
- Relaxed the restriction that each buffer should fill completely before moving to next. (I
might revert back this change)
- Added a BufferedOutputStream in between SpillingBuffer and SpillingStream. (This improved
the efficiency by around 45%)
The performance numbers are now: 

{noformat}

 0% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_LIST} 3809.61 ns; σ=40.46
ns @ 10 trials
 5% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_LIST} 7113.85 ns; σ=448.07
ns @ 10 trials
10% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_LIST} 41011.04 ns; σ=1462.04
ns @ 10 trials
14% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_LIST} 357103.20 ns;
σ=350967.71 ns @ 10 trials
19% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_LIST} 3606638.74 ns;
σ=3339040.16 ns @ 10 trials
24% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_LIST} 34951212.41
ns; σ=1949660.29 ns @ 10 trials
29% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000, type=DISK_LIST} 354762498.83
ns; σ=13826670.03 ns @ 10 trials
33% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=SPILLING_BUFFER} 3914.27 ns;
σ=17.48 ns @ 3 trials
38% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=SPILLING_BUFFER} 6114.64 ns;
σ=60.98 ns @ 10 trials
43% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=SPILLING_BUFFER} 31973.78
ns; σ=143.74 ns @ 3 trials
48% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=SPILLING_BUFFER} 271619.92
ns; σ=19861.84 ns @ 10 trials
52% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=SPILLING_BUFFER} 2580930.24
ns; σ=3416159.19 ns @ 10 trials
57% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=SPILLING_BUFFER} 25714962.79
ns; σ=45184.42 ns @ 3 trials
62% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000, type=SPILLING_BUFFER} 259875751.00
ns; σ=19098186.41 ns @ 10 trials
67% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_BUFFER} 3294.61 ns; σ=10.86
ns @ 3 trials
71% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_BUFFER} 6442.84 ns; σ=18.10
ns @ 3 trials
76% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_BUFFER} 38160.64 ns;
σ=250.75 ns @ 3 trials
81% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_BUFFER} 333411.11 ns;
σ=4376.56 ns @ 10 trials
86% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_BUFFER} 3214465.20
ns; σ=8146.50 ns @ 3 trials
90% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_BUFFER} 30817421.31
ns; σ=21434.85 ns @ 3 trials
95% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000, type=DISK_BUFFER} 316989389.00
ns; σ=622679.65 ns @ 3 trials

    size            type        us linear runtime
      10       DISK_LIST      3.81 =
      10 SPILLING_BUFFER      3.91 =
      10     DISK_BUFFER      3.29 =
     100       DISK_LIST      7.11 =
     100 SPILLING_BUFFER      6.11 =
     100     DISK_BUFFER      6.44 =
    1000       DISK_LIST     41.01 =
    1000 SPILLING_BUFFER     31.97 =
    1000     DISK_BUFFER     38.16 =
   10000       DISK_LIST    357.10 =
   10000 SPILLING_BUFFER    271.62 =
   10000     DISK_BUFFER    333.41 =
  100000       DISK_LIST   3606.64 =
  100000 SPILLING_BUFFER   2580.93 =
  100000     DISK_BUFFER   3214.47 =
 1000000       DISK_LIST  34951.21 ==
 1000000 SPILLING_BUFFER  25714.96 ==
 1000000     DISK_BUFFER  30817.42 ==
10000000       DISK_LIST 354762.50 ==============================
10000000 SPILLING_BUFFER 259875.75 =====================
10000000     DISK_BUFFER 316989.39 ==========================

vm: java
trial: 0
benchmark: Spill

{noformat}
                
> Add a spilling message queue
> ----------------------------
>
>                 Key: HAMA-559
>                 URL: https://issues.apache.org/jira/browse/HAMA-559
>             Project: Hama
>          Issue Type: Sub-task
>          Components: bsp core
>    Affects Versions: 0.5.0
>            Reporter: Thomas Jungblut
>            Assignee: Suraj Menon
>            Priority: Minor
>             Fix For: 0.7.0
>
>         Attachments: HAMA-559.patch-v1, spilling_buffer_cpu_usage_text_write.png, SpillingBufferProfile-2012-10-27.snapshot,
spilling_buffer_profile_cpu_graph_test_write.png, spilling_buffer_profile_cpugraph_writeUTF.png,
spillingbuffer_profile_cpu_writeUTF.png, spilling_buffer_profile_LOCK.JPG, spilling_buffer_profile_timesplit_text_write.png,
spilling_buffer_profile_writeUTF.png
>
>
> After HAMA-521 is done, we can add a spilling queue which just holds the messages in
RAM that fit into the heap space. The rest can be flushed to disk.
> We may call this a HybridQueue or something like that.
> The benefits should be that we don't have to flush to disk so often and get faster. However
we may have more GC so it is always overall faster.
> The requirements for this queue also include:
> - The message object once written to the queue (after returning from the write call)
could be modified, but the changes should not be reflected in the messages stored in the queue.
> - For now let's implement a queue that does not support concurrent reading and writing.
This feature is needed when we implement asynchronous communication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message