hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11229) Mutation API: Coordinator communication with meta store should be optional
Date Fri, 10 Jul 2015 22:12:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14622972#comment-14622972
] 

Hive QA commented on HIVE-11229:
--------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744722/HIVE-11229.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9160 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4575/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4575/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4575/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12744722 - PreCommit-HIVE-TRUNK-Build

> Mutation API: Coordinator communication with meta store should be optional
> --------------------------------------------------------------------------
>
>                 Key: HIVE-11229
>                 URL: https://issues.apache.org/jira/browse/HIVE-11229
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 2.0.0
>            Reporter: Elliot West
>            Assignee: Elliot West
>              Labels: streaming_api
>         Attachments: HIVE-11229.1.patch
>
>
> [~ekoifman] raised a theoretical issue with the streaming mutation API (HIVE-10165) where
worker nodes operating in a distributed cluster might overwhelm a meta store while trying
to obtain partition locks. Although this does not happen in practice (see HIVE-11228), the
API does communicate with the meta store in this manner to obtain partition paths and create
new partitions. Therefore the issue described does in fact exist in the current implementation,
albeit in a different code path. I’d like to make such communication optional like so:
> * When the user chooses not to create partitions on demand, no meta store connection
will be created in the {{MutationCoordinators}}. Additionally, partition paths will be resolved
using {{org.apache.hadoop.hive.metastore.Warehouse.getPartitionPath(Path, LinkedHashMap<String,
String>)}} which should be suitable so long as standard Hive partition layouts are followed.
> * If the user does choose to create partitions on demand then the system will operate
as is does currently; using the meta store to both issue {{add_partition}} events and look
up partition meta data.
> * The documentation will be updated to describe these behaviours and outline alternative
approaches to collecting affected partition names and creating partitions in a less intensive
manner.
> Side note for follow up: The parameter names {{tblName}} and {{dbName}} seem to be the
wrong way around on the method {{org.apache.hadoop.hive.metastore.IMetaStoreClient.getPartition(String,
String, List<String>)}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message