incubator-jena-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Allen (JIRA)" <>
Subject [jira] [Commented] (JENA-44) Support external sorting of bindings in ARQ
Date Tue, 24 May 2011 18:18:47 GMT


Stephen Allen commented on JENA-44:

Some answers with respect to JENA-45:

1) In JENA-45, bnodes are encoded with NodeFmtLib.safeBNodeLabel(String) and decoded with
a new method NodeFmtLib.decodeSafeBNodeLabel(String).  This outputs 'B' followed by the internal
blank node label encoded in hexadecimal.  This allows us to deserialize back to the proper
Jena bnode.

2) I think we need to add a way make ThresholdPolicyCount configurable externally.  Alternatively,
we can change this to actual memory usage by looking at the contents of each Binding object
as it is added.

3) Ultimately having unified memory manager (perhaps with a preset limit for each query) would
be ideal to manage operators that need it.  Looking at [1] (from the Hadoop project) gives
an indication of how this might be approached.


> Support external sorting of bindings in ARQ
> -------------------------------------------
>                 Key: JENA-44
>                 URL:
>             Project: Jena
>          Issue Type: New Feature
>          Components: ARQ
>            Reporter: Sam Tunnicliffe
>            Assignee: Paolo Castagna
>            Priority: Minor
>         Attachments: JENA-44-0.patch, JENA-44_ARQ_r8531.patch, JENA-44_ARQ_r8724.patch
> In QueryIterSort, the sorting of the contents of an Iterator<Binding> is done in
memory, using Arrays.sort. This can be problematic where the set to be sorted is large. A
possible solution could be to use an external, disk-backed algorithm. A hybrid approach may
be better, whereby we attempt the in-memory sort, but when the number of bindings encountered
goes over a certain number, resort to the disk-backed variant.

This message is automatically generated by JIRA.
For more information on JIRA, see:

View raw message