Mailing-List: contact jena-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jena-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4E09DAA2.4070508@epimorphics.com>
Date: Tue, 28 Jun 2011 14:44:02 +0100
From: Andy Seaborne <andy.seaborne@epimorphics.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.17) Gecko/20110516 Thunderbird/3.1.10
MIME-Version: 1.0
To: jena-dev@incubator.apache.org
Subject: Re: Blank nodes and MapReduce
References: <4E08C8A8.1090600@googlemail.com>
 <4E09D50E.2030307@epimorphics.com> <4E09D928.9030707@googlemail.com>
In-Reply-To: <4E09D928.9030707@googlemail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


On 28/06/11 14:37, Paolo Castagna wrote:
>
>
> Andy Seaborne wrote:
>>>    public Node create(String label) {
>>>        return Node.createAnon(new AnonId(filename + "-" + label)) ;
>>>    }
>>
>> The way I thought was to allocate a UUID per parser run (or any other
>> sufficiently large random number), xor the label into the UUID to
>> produce the bNode label.  This is a non-localised label allocation scheme.
>
> Hi Andy,
> I am not sure this would work with MapReduce as filers are split into multiple
> chunks and different machines can process splits from the same file.

Exactly - by "parser run" I mean all the separate parsing actions in one 
step of the process.  Allocate one large job random number as the base 
of bNode label generation across the whole cluster.

Per job instance, means it's different next time, important if the data 
is merged with other data.

>
> Let's say I have this file, split into two chunks:
>
>    ----------------------------
>
>    <foo:bar>  <foo:p>  _:bnode1 .      split 1
>    _:bnode1<foo:q>  "1" .
>
>    ----------------------------
>
>    _:bnode1<foo:r>  "2" .            split 2
>
>    ----------------------------
>
> I need to ensure the 'bnode1' label in split 1 and 2 refers to the same blank
> node even if the splits are parsed separately. However, the same 'bnode1' label
> from a different file must represent a different blank node. In practice, with
> MapReduce, I cannot assume that a file is parsed in a single "parser run".
>
>>
>>> Therefore, I would like to have my own
>>> LabelToNode implementation with an Allocator<String, Node>   which
>>> takes into
>>> account the filename (or an hash of it) when it creates a new blank node.
>>> But LabelToNode constructor is private.
>>>
>>> Could we make it protected?
>>
>> Now public.
>
> Thanks.
>
> Paolo
>
>>
>>>
>>> Or, alternatively, how can I construct a LabelToNode object which will
>>> be using
>>> my MapReduceAllocator?
>>
>> LabelToNode createUseLabelAsGiven()
>>
>>      Andy