accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Felts (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-4708) Limit RFile block size to 2GB
Date Tue, 19 Sep 2017 15:06:00 GMT


Nick Felts commented on ACCUMULO-4708:

I created a couple local unit tests for the changes. They expected exceptions when the key
was too large or when a key/value pair was too large.

Because we are talking about large byte arrays, I had to edit the pom.xml to increase the
Xmx from 1G to 6G. I'm can add the tests along with the change, but it was a large increase
to the required heap space for the tests.

> Limit RFile block size to 2GB
> -----------------------------
>                 Key: ACCUMULO-4708
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: core
>            Reporter: Nick Felts
>            Assignee: Nick Felts
>              Labels: pull-request-available
>             Fix For: 1.7.4, 1.8.2, 2.0.0
>          Time Spent: 2h
>  Remaining Estimate: 0h
> In core/file/rfile/bcfile/, the block size is determined by the size of a
DataOutputStream, which returns an int. The javadoc for size() states "If the counter overflows,
it will be wrapped to Integer.MAX_VALUE."  
> This can be a problem when RFiles are saving block regions, because Accumulo has no way
of knowing how large the block actually is when the size is Integer.MAX_VALUE.  
> To fix this, a check can be put in to throw an exception if Integer.MAX_VALUE bytes (or
more) have been written.  A check can also be made when appending to the block to make sure
the key/value won't create a block that is too large.
to make sure nobody mistakenly configures larger than Integer.MAX_VALUE.

This message was sent by Atlassian JIRA

View raw message