hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <>
Subject [jira] [Commented] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
Date Tue, 12 Sep 2017 20:04:01 GMT


Eugene Koifman commented on HIVE-17482:

Are there any tests that prove this patch works?

The expectation on master is that all queries run in a transaction context.
(the system doesn't assert it thought I've not gotten there yet)
If you look at Driver.acquireLocks()  - it will bail if you don't have a transaction open.

Also, the caller is supposed to pass ValidTxnList to the reader in Configuration.  Where is
that happening?  the current acquireLocks does it but it sets it on Configuration but will
this be clobbered by the next fragment?

I think the semantics that we discussed earlier is that a complex query is broken down into
fragments corresponding to each scan and each is executed as a separate query in a separate
transaction.  So each fragment would start a txn, lock in the current snapshot and acquire
locks.  This would create READ_COMMITTED semantics from whole query point of view.  Is that
still the intent?  If so it's worth documenting this clearly in this bug or somewhere else.

It's not clear based on the diffs that this is what is happening.

When    _GenericUDTFGetSplits.createPlanFragment()_
calls   _CommandProcessorResponse cpr = driver.compileAndRespond(query);_
it will start a txn using TransactionManager that belongs to the SessionState in current ThreadLocal
Then it seems like the code will create a new TransactionManager  to acquire locks (which
I think will do nothing... thought it should be made to throw)

So if I'm reading this right it doesn't do the right thing

> External LLAP client: acquire locks for tables queried directly by LLAP
> -----------------------------------------------------------------------
>                 Key: HIVE-17482
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: llap
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>         Attachments: HIVE-17482.1.patch
> When using the LLAP external client with simple queries (filter/project of single table),
the appropriate locks should be taken on the table being read like they are for normal Hive
queries. This is important in the case of transactional tables being queried, since the compactor
relies on the presence of table locks to determine whether it can safely delete old versions
of compacted files without affecting currently running queries.
> This does not have to happen in the complex query case, since a query is used (with the
appropriate locking mechanisms) to create/populate the temp table holding the results to the
complex query.

This message was sent by Atlassian JIRA

View raw message