accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Felts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4708) Limit RFile block size to 2GB
Date Tue, 19 Sep 2017 15:06:00 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171853#comment-16171853
] 

Nick Felts commented on ACCUMULO-4708:
--------------------------------------

I created a couple local unit tests for the changes. They expected exceptions when the key
was too large or when a key/value pair was too large.

Because we are talking about large byte arrays, I had to edit the pom.xml to increase the
Xmx from 1G to 6G. I'm can add the tests along with the change, but it was a large increase
to the required heap space for the tests.

> Limit RFile block size to 2GB
> -----------------------------
>
>                 Key: ACCUMULO-4708
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4708
>             Project: Accumulo
>          Issue Type: Bug
>          Components: core
>            Reporter: Nick Felts
>            Assignee: Nick Felts
>              Labels: pull-request-available
>             Fix For: 1.7.4, 1.8.2, 2.0.0
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> In core/file/rfile/bcfile/BCFile.java, the block size is determined by the size of a
DataOutputStream, which returns an int. The javadoc for size() states "If the counter overflows,
it will be wrapped to Integer.MAX_VALUE."  
> This can be a problem when RFiles are saving block regions, because Accumulo has no way
of knowing how large the block actually is when the size is Integer.MAX_VALUE.  
> To fix this, a check can be put in to throw an exception if Integer.MAX_VALUE bytes (or
more) have been written.  A check can also be made when appending to the block to make sure
the key/value won't create a block that is too large.
> A check should also be made when reading in TABLE_FILE_COMPRESSED_BLOCK_SIZE and TABLE_FILE_COMPRESSED_BLOCK_SIZE_INDEX
to make sure nobody mistakenly configures larger than Integer.MAX_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message