hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-11987) CompactionTxnHandler.createValidCompactTxnList() can use much less memory
Date Thu, 29 Oct 2015 02:11:27 GMT

     [ https://issues.apache.org/jira/browse/HIVE-11987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eugene Koifman updated HIVE-11987:
----------------------------------
    Description: 
This method only needs HWM and list of txn IDs in 'a' state and smallest 'o' txn id.

It's currently implemented to get the list from TxnHandler.getOpenTxnsInfo(),
which returns (txn id, state, host, user) for each txn and includes Aborted txns.

This can easily be 120 bytes or more per txn overhead (over 1 Java long) which not an issue
in general but when the system is misconfigured, the number of opened/aborted txns can get
into the millions.  This creates unnecessary memory pressure on metastore.

Should consider fixing this.
This should be easy to fix since the result of getOpenTxnsInfo() doesn't go over the wire.

Also, ValidCompactorTxnList doesn't actually need to store the 'o' txn ids, just the 'a' ones.

  was:
This method only needs HWM and list of txn IDs in 'o' state.

It's currently implemented to get the list from TxnHandler.getOpenTxnsInfo(),
which returns (txn id, state, host, user) for each txn and includes Aborted txns.

This can easily be 120 bytes or more per txn overhead (over 1 Java long) which not an issue
in general but when the system is misconfigured, the number of opened/aborted txns can get
into the millions.  This creates unnecessary memory pressure on metastore.

Should consider fixing this.


> CompactionTxnHandler.createValidCompactTxnList() can use much less memory
> -------------------------------------------------------------------------
>
>                 Key: HIVE-11987
>                 URL: https://issues.apache.org/jira/browse/HIVE-11987
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore, Transactions
>    Affects Versions: 1.1.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>            Priority: Minor
>
> This method only needs HWM and list of txn IDs in 'a' state and smallest 'o' txn id.
> It's currently implemented to get the list from TxnHandler.getOpenTxnsInfo(),
> which returns (txn id, state, host, user) for each txn and includes Aborted txns.
> This can easily be 120 bytes or more per txn overhead (over 1 Java long) which not an
issue in general but when the system is misconfigured, the number of opened/aborted txns can
get into the millions.  This creates unnecessary memory pressure on metastore.
> Should consider fixing this.
> This should be easy to fix since the result of getOpenTxnsInfo() doesn't go over the
wire.
> Also, ValidCompactorTxnList doesn't actually need to store the 'o' txn ids, just the
'a' ones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message