hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13308) HiveOnSpark sumbit query very slow when hundred of beeline exectue at same time
Date Wed, 11 May 2016 17:24:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280475#comment-15280475
] 

Marcelo Vanzin commented on HIVE-13308:
---------------------------------------

There needs to be a synchronization block to access the {{server}} field; otherwise, it's
probably safe to call the {{SparkClientImpl}} constructor outside of a synchronized block,
as long as {{SparkClientFactory.stop()}} is not called before it finishes. Something like:

{code}
  public static SparkClient createClient(Map<String, String> sparkConf, HiveConf hiveConf)
      throws IOException, SparkException {
    RpcServer _server;
    synchronized (SparkClientFactory.class) {
      Preconditions.checkState(server != null, "initialize() not called.");
      _server = server;
    }
    return new SparkClientImpl(_server, sparkConf, hiveConf);
  }
{code}

Or maybe just making the {{server}} variable volatile would suffice, too (and then no synchronization
is needed in {{createClient}}).

> HiveOnSpark sumbit query very slow when hundred of beeline exectue at same time
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-13308
>                 URL: https://issues.apache.org/jira/browse/HIVE-13308
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 1.2.1
>            Reporter: wangwenli
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> backgroud: hive on spark ,  yarn cluster mode
> details:
> when using hundred of beeline submit query at the same time, we found that   yarn get
application very slow, and hiveserver is blocked at SparkClientFactory.createClient method
> after analysis, we think the synchronize on SparkClientFactory.createClient , can be
removed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message