hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
Date Mon, 26 Nov 2018 20:59:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699576#comment-16699576
] 

Hive QA commented on HIVE-20740:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 35s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 34s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 59s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m  3s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  6s{color} | {color:blue}
standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 42s{color} | {color:blue}
ql in master has 2312 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 36s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 37s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 26s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 29s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  6s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  6s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 21s{color} | {color:red}
itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608)
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 17s{color} | {color:red}
standalone-metastore/metastore-server generated 1 new + 183 unchanged - 2 fixed = 184 total
(was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 39s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m  1s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
|  |  org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration)
does not release lock on all paths  At PersistenceManagerProvider.java:on all paths  At PersistenceManagerProvider.java:[line
152] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15061/dev-support/hive-personality.sh
|
| git revision | master / 0fee288 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Remove global lock in ObjectStore.setConf method
> ------------------------------------------------
>
>                 Key: HIVE-20740
>                 URL: https://issues.apache.org/jira/browse/HIVE-20740
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>            Priority: Major
>         Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, HIVE-20740.04.patch, HIVE-20740.05.patch,
HIVE-20740.06.patch, HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, HIVE-20740.11.patch,
HIVE-20740.12.patch, HIVE-20740.13.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other clients in concurrent
workloads.
> {code}
> @Override
>   @SuppressWarnings("nls")
>   public void setConf(Configuration conf) {
>     // Although an instance of ObjectStore is accessed by one thread, there may
>     // be many threads with ObjectStore instances. So the static variables
>     // pmf and prop need to be protected with locks.
>     pmfPropLock.lock();
>     try {
>       isInitialized = false;
>       this.conf = conf;
>       this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
>       configureSSL(conf);
>       Properties propsFromConf = getDataSourceProps(conf);
>       boolean propsChanged = !propsFromConf.equals(prop);
>       if (propsChanged) {
>         if (pmf != null){
>           clearOutPmfClassLoaderCache(pmf);
>           if (!forTwoMetastoreTesting) {
>             // close the underlying connection pool to avoid leaks
>             pmf.close();
>           }
>         }
>         pmf = null;
>         prop = null;
>       }
>       assert(!isActiveTransaction());
>       shutdown();
>       // Always want to re-create pm as we don't know if it were created by the
>       // most recent instance of the pmf
>       pm = null;
>       directSql = null;
>       expressionProxy = null;
>       openTrasactionCalls = 0;
>       currentTransaction = null;
>       transactionStatus = TXN_STATUS.NO_STATE;
>       initialize(propsFromConf);
>       String partitionValidationRegex =
>           MetastoreConf.getVar(this.conf, ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
>       if (partitionValidationRegex != null && !partitionValidationRegex.isEmpty())
{
>         partitionValidationPattern = Pattern.compile(partitionValidationRegex);
>       } else {
>         partitionValidationPattern = null;
>       }
>       // Note, if metrics have not been initialized this will return null, which means
we aren't
>       // using metrics.  Thus we should always check whether this is non-null before
using.
>       MetricRegistry registry = Metrics.getRegistry();
>       if (registry != null) {
>         directSqlErrors = Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
>       }
>       this.batchSize = MetastoreConf.getIntVar(conf, ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
>       if (!isInitialized) {
>         throw new RuntimeException(
>         "Unable to create persistence manager. Check dss.log for details");
>       } else {
>         LOG.debug("Initialized ObjectStore");
>       }
>     } finally {
>       pmfPropLock.unlock();
>     }
>   }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new connection to HMS
which is trying to instantiate ObjectStore. We should either remove the lock or reduce the
scope of the lock so that it is held for a very small amount of time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message