hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3706) Generalize native HBase writer for additional tables
Date Wed, 17 Jun 2015 06:00:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589354#comment-14589354
] 

Joep Rottinghuis commented on YARN-3706:
----------------------------------------

Fixed code, now have successful run:
{noformat}
15/06/16 22:57:15 INFO mapreduce.Job: Counters: 23
        File System Counters
                FILE: Number of bytes read=1651635
                FILE: Number of bytes written=1927484
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=0
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=0
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Map-Reduce Framework
                Map input records=1
                Map output records=0
                Input split bytes=48
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=18
                Total committed heap usage (bytes)=325058560
        org.apache.hadoop.mapred.TimelineServicePerformanceV2$PerfCounters
                TIMELINE_SERVICE_WRITE_COUNTER=230
                TIMELINE_SERVICE_WRITE_KBS=230
                TIMELINE_SERVICE_WRITE_TIME=66
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=0
TRANSACTION RATE (per mapper): 3484.848484848485 ops/s
IO RATE (per mapper): 3484.848484848485 KB/s
TRANSACTION RATE (total): 3484.848484848485 ops/s
IO RATE (total): 3484.848484848485 KB/s
{noformat}

and an individual history file:
{noformat}
15/06/16 22:58:06 INFO mapreduce.Job: Job job_local1358267884_0001 completed successfully
15/06/16 22:58:06 INFO mapreduce.Job: Counters: 22
        File System Counters
                FILE: Number of bytes read=1651635
                FILE: Number of bytes written=1927113
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=141020
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=3
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Map-Reduce Framework
                Map input records=1
                Map output records=0
                Input split bytes=48
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=63
                Total committed heap usage (bytes)=460324864
        org.apache.hadoop.mapred.TimelineServicePerformanceV2$PerfCounters
                TIMELINE_SERVICE_WRITE_COUNTER=25
                TIMELINE_SERVICE_WRITE_TIME=145
        File Input Format Counters 
                Bytes Read=0
        File Output Format Counters 
                Bytes Written=0
TRANSACTION RATE (per mapper): 172.41379310344828 ops/s
IO RATE (per mapper): 0.0 KB/s
TRANSACTION RATE (total): 172.41379310344828 ops/s
IO RATE (total): 0.0 KB/s
{noformat}

Will make change suggested by [~zjshen], run another test run and upload a new patch.

> Generalize native HBase writer for additional tables
> ----------------------------------------------------
>
>                 Key: YARN-3706
>                 URL: https://issues.apache.org/jira/browse/YARN-3706
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>         Attachments: YARN-3706-YARN-2928.001.patch, YARN-3706-YARN-2928.010.patch, YARN-3706-YARN-2928.011.patch,
YARN-3706-YARN-2928.012.patch, YARN-3706-YARN-2928.013.patch, YARN-3706-YARN-2928.014.patch,
YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, YARN-3726-YARN-2928.004.patch,
YARN-3726-YARN-2928.005.patch, YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch,
YARN-3726-YARN-2928.008.patch, YARN-3726-YARN-2928.009.patch
>
>
> When reviewing YARN-3411 we noticed that we could change the class hierarchy a little
in order to accommodate additional tables easily.
> In order to get ready for benchmark testing we left the original layout in place, as
performance would not be impacted by the code hierarchy.
> Here is a separate jira to address the hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message