hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13029) NVDIMM support for LLAP Cache
Date Mon, 11 Apr 2016 22:24:25 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236103#comment-15236103
] 

Gopal V commented on HIVE-13029:
--------------------------------

bq.  The patch basically looks good to me.

I haven't tested this properly.

bq. should this be addressed, or followed up on?

The follow up for fallocate is ideally done in tje JNI layer in hadoop-common's libhadoop.so
(HADOOP-7834). 

Right now, the truncate mode allows for overcommit - so, running out of space on storage kills
the daemon with a fatal SIGBUS.

bq. The SSD cache would require the 2-tiered cache, wouldn't it?

The NVDIMM case doesn't need a 2-tiered model, because the read latencies with DAX are too
small to need the extra complexity. In fact, a 2-tier model would be slower because it would
eat up bandwidth moving data between tiers - a zero-copy paging approach directly by the OS
is much better.

The SSD mode is just left there for systems which are restricted in RAM (<64Gb), but still
want to utilize a large cache.

> NVDIMM support for LLAP Cache
> -----------------------------
>
>                 Key: HIVE-13029
>                 URL: https://issues.apache.org/jira/browse/HIVE-13029
>             Project: Hive
>          Issue Type: New Feature
>          Components: llap
>    Affects Versions: 2.1.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>            Priority: Critical
>         Attachments: HIVE-13029.1.patch
>
>
> LLAP cache has been designed so that the cache can be offloaded easily to a pmem API
without restart coherence.
> The tricky part about NVDIMMs are restart coherence, while most of the cache gains can
be obtained without keeping state across refreshes, since LLAP is not the system of record,
HDFS is.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message