Mailing-List: contact jena-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jena-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of castagna.lists@googlemail.com
 designates 209.85.212.175 as permitted sender)
Message-ID: <4F58A43D.5080709@googlemail.com>
Date: Thu, 08 Mar 2012 12:21:17 +0000
From: Paolo Castagna <castagna.lists@googlemail.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20101027)
MIME-Version: 1.0
To: jena-dev@incubator.apache.org
Subject: Re: Strategies for loading large (>500m triples) datasets
References: <339406804618164@jngomktg.net>
	<CA+LYuk8A4E+uTmqSSJ7Z45ac7OGkj3OZJ6KLdUXFmedmHHotfQ@mail.gmail.com>
	<CAORoscn9gufTdif9n+GgtjBuyXn-rMp3TiyFYYb8_vcJZE1+SA@mail.gmail.com>
	<4F4F6234.5020507@apache.org>	<4F4FAB83.9040403@googlemail.com>
	<4F54663F.5060306@googlemail.com>	<4F55C039.7000705@googlemail.com>
 <CABMc4Vb6wGDs9ta5c7h9TSfhqtd0u3cQiKomESJ4xmB76tg9yw@mail.gmail.com>
In-Reply-To: 
 <CABMc4Vb6wGDs9ta5c7h9TSfhqtd0u3cQiKomESJ4xmB76tg9yw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

FYI

Paolo Castagna wrote:
> This time, UNIX sort filled /tmp...
> I'll try specifying the --temporary-directory=DIR or, better, via $TMPDIR
> env variable (this way there is no need to change tdbloader2 script).

This time, I was able to load the Freebase data dump (converted into RDF) using tdbloader2 (which is included in TDB).

This is how I run tdbloader2 using an EC2 m1.xlarge instance (i.e. 15 GB memory):

export JVM_ARGS="-Xmx4096m -server"
export TMPDIR=/mnt/data/tmp
tdbloader2 --loc /mnt/data/freebase /mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz

Total elapsed time to load 618,465,279 triples:
~12 hours (i.e. ~10,000 triples/s overall speed)

This is the log:
Mar  7 13:11:37 ip-10-54-167-166 build:  13:11:37 -- TDB Bulk Loader Start
Mar  7 13:11:37 ip-10-54-167-166 build:  13:11:37 Data phase
Mar  7 13:11:39 ip-10-54-167-166 build: Load: /mnt/data/freebase2rdf/freebase-datadump-rdf.nt.gz -- 2012/03/07 13:11:38 UTC
Mar  7 13:11:42 ip-10-54-167-166 build: Add: 50,000 Data (Batch: 16,550 / Avg: 16,550)
Mar  7 13:11:43 ip-10-54-167-166 build: Add: 100,000 Data (Batch: 39,184 / Avg: 23,272)
[...]
Mar  7 19:13:51 ip-10-54-167-166 build: Add: 618,450,000 Data (Batch: 53,078 / Avg: 28,457)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7727]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  7 19:17:01 ip-10-54-167-166 CRON[7725]: pam_unix(cron:session): session closed for user root
Mar  7 19:24:44 ip-10-54-167-166 build: Total: 618,465,279 tuples : 22,385.15 seconds : 27,628.37 tuples/sec [2012/03/07 19:24:44 UTC]
Mar  7 19:24:45 ip-10-54-167-166 build:  19:24:44 Index phase
Mar  7 19:24:45 ip-10-54-167-166 build:  19:24:45 Index SPO
Mar  7 21:03:18 ip-10-54-167-166 build:  21:03:18 Build SPO
Mar  7 21:14:24 ip-10-54-167-166 build:  21:14:24 Index POS
Mar  7 23:38:28 ip-10-54-167-166 build:  23:38:28 Build POS
Mar  7 23:49:03 ip-10-54-167-166 build:  23:49:03 Index OSP
Mar  8 00:56:13 ip-10-54-167-166 build:  00:56:13 Build OSP
Mar  8 01:08:17 ip-10-54-167-166 build:  01:08:17 Index phase end
Mar  8 01:08:59 ip-10-54-167-166 build:  01:08:59 -- TDB Bulk Loader Finish
Mar  8 01:08:59 ip-10-54-167-166 build:  01:08:59 -- 43000 seconds

Paolo