phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2417) Compress memory used by row key byte[] of guideposts
Date Wed, 18 Nov 2015 02:42:11 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010075#comment-15010075
] 

James Taylor commented on PHOENIX-2417:
---------------------------------------

Thanks to [~julienledem] for pointing me to a couple of simple classes that implement a simple
prefix encoder that we could use:
- On the write side: https://github.com/Parquet/parquet-mr/blob/master/parquet-column/src/main/java/parquet/column/values/deltastrings/DeltaByteArrayWriter.java
- On the read side (materializing a single value at a time as you iterate): https://github.com/Parquet/parquet-mr/blob/fa8957d7939b59e8d391fa17000b34e865de015d/parquet-column/src/main/java/parquet/column/values/deltastrings/DeltaByteArrayReader.java#L61

We can either copy/paste or use directly (especially if there's already a Parquet dependency).

> Compress memory used by row key byte[] of guideposts
> ----------------------------------------------------
>
>                 Key: PHOENIX-2417
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2417
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>
> We've found that smaller guideposts are better in terms of minimizing any increase in
latency for point scans. However, this increases the amount of memory significantly when caching
the guideposts on the client. Guidepost are equidistant row keys in the form of raw byte[]
which are likely to have a large percentage of their leading bytes in common (as they're stored
in sorted order. We should use a simple compression technique to mitigate this. I noticed
that Apache Parquet has a run length encoding - perhaps we can use that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message