hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cosmin Lehene (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
Date Wed, 07 Jan 2015 23:19:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268484#comment-14268484
] 

Cosmin Lehene commented on HBASE-5401:
--------------------------------------

The outer loop is still there, but based on how it looks it doesn't seem it should be removed
:)

[~stack], [~ndimiduk] can you comment?

{code}
  /**
   * Per client, how many tasks will we run?  We divide number of rows by this number and
have the
   * client do the resulting count in a map task.
   */
  static int TASKS_PER_CLIENT = 10;

{code}
{code}

      for (int i = 0; i < TASKS_PER_CLIENT; i++) {
        for (int j = 0; j < opts.numClientThreads; j++) {
          TestOptions next = new TestOptions(opts);
          next.startRow = (j * perClientRows) + (i * (perClientRows/10));
          next.perClientRunRows = perClientRows / 10;
          String s = MAPPER.writeValueAsString(next);
          LOG.info("Client=" + j + ", maptask=" + i + ", input=" + s);
          int hash = h.hash(Bytes.toBytes(s));
          m.put(hash, s);
        }
      }

{code}

> PerformanceEvaluation generates 10x the number of expected mappers
> ------------------------------------------------------------------
>
>                 Key: HBASE-5401
>                 URL: https://issues.apache.org/jira/browse/HBASE-5401
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>            Reporter: Oliver Meyn
>
> With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite
10' there are 100 mappers spawned, rather than the expected 10.  The culprit appears to be
the outer loop in writeInputFile which sets up 10 splits for every "asked-for client".  I
think the fix is just to remove that outer loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message