accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2437) Cannot create splits with MSBit set in MSByte via API
Date Fri, 07 Mar 2014 17:57:47 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924121#comment-13924121
] 

Aaron Kimball commented on ACCUMULO-2437:
-----------------------------------------

John, Thanks for the tip! The code fragment you offered will behave as I expect (creating
256 uniform splits).

As a nitpick though, the API for {{o.a.h.io.Text}} specifies that its byte stream should be
UTF-8 encoded. The constructor that you suggest will not perform any translation/validation,
so it works in this case. But it is possible that a Text object with such data in it will
experience other errors if other of its methods are used (e.g., if you call {{toString()}}
on such a text, it will return a {{RuntimeException}}). 

That having been said, reading through the source of Text, it doesn't enforce UTF-8 except
at certain boundaries like {{toString()}} or {{charAt()}}, so it is capable of holding, serializing,
and deserializing byte arrays that are not legal UTF-8... So you're probably in the clear
in practice as long as clients are mindful not to actually treat the {{Text}} instances as
true strings.

(see https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
for reference)

If that kind of API precision isn't desired, please feel free to resolve this as Invalid or
Wontfix.

> Cannot create splits with MSBit set in MSByte via API 
> ------------------------------------------------------
>
>                 Key: ACCUMULO-2437
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2437
>             Project: Accumulo
>          Issue Type: Bug
>    Affects Versions: 1.5.1
>            Reporter: Aaron Kimball
>
> I cannot create a table with 256 evenly-sliced splits using the API. I believe due to
the fact that Text can only hold valid Unicode characters, the following only generates 129
splits:
> {code}
>     TableOperations tableOps = connector.tableOperations();
>     TreeSet<Text> splits = new TreeSet<Text>();
>     for (int i = 0; i < 256; i++) { 
>       byte[] bytes = { (byte) i };
>       String theStr = new String(bytes);
>       splits.add(new Text(theStr));
>     } 
>     tableOps.addSplits(TABLE_NAME, splits);
> {code}
> Using {{getsplits}} in the shell, I see the highest split be 0x7F; while we can use byte
values 0x80 through 0xFF as leading bytes in row keys, the use of {{Text}} in the {{addSplits()}}
method makes these invalid strings to split on.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message