Mailing-List: contact jena-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jena-dev@incubator.apache.org
Date: Sun, 4 Mar 2012 22:37:00 +0000 (UTC)
From: "Sarven Capadisli (Commented) (JIRA)" <jira@apache.org>
To: jena-dev@incubator.apache.org
Message-ID: 
 <701063878.20464.1330900620053.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <886968985.20162.1330888620308.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (JENA-217) tdbloader : Unsafe memory access
 operation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/JENA-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222045#comment-13222045 ] 

Sarven Capadisli commented on JENA-217:
---------------------------------------

Thanks Andy.

Just to confirm that the filesystem is ext4.

16 GB of RAM, however I'd say that the available free is from 4-6GB. The swap is 16GB.

$ java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.13) (6b20-1.9.13-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

I'll give tdbloader one more go because this time around it was going at a pretty decent speed (in comparison to my previous imports - I don't have a reason for this either). It took about 13 hours before breaking. I think there was another hour to go before finishing up.

The only issue I have with switching over to tdbloader2 or 3 is that I have to add the context URI to my N-Triples for N-Quads.

                
> tdbloader : Unsafe memory access operation
> ------------------------------------------
>
>                 Key: JENA-217
>                 URL: https://issues.apache.org/jira/browse/JENA-217
>             Project: Apache Jena
>          Issue Type: Bug
>          Components: TDB
>    Affects Versions: Fuseki 0.2.1
>         Environment: Ubuntu 10.04
>            Reporter: Sarven Capadisli
>
> First of all, I'm not sure if this is necessarily a bug or whether my system encountered some other issue. Here it is:
> I'm trying to import world-development-indicators.nt which is 68 GB into a TDB store.
> I've performed the following operation:
> java tdb.tdbloader --loc /usr/lib/fuseki/DB/DB.worldbank2 world-development-indicators --graph=http://worldbank.270a.info/graph/world-bank-indicators /data/worldbank.270a.info/indicators/en/indicator/world-bank-indicators.nt
> ..
> 360,393,432 triples loaded in 15,956.80 seconds [Rate: 22,585.57 per second]
> ..
> ** Index GSPO->GPOS: 360,202,756 slots indexed in 18,194.61 seconds [Rate: 19,797.23 per second]
> ..
> ** Index GSPO->GOSP: 360,202,756 slots indexed in 4,410.64 seconds [Rate: 81,666.85 per second]
> ..
> ** Index GSPO->POSG: 360,202,756 slots indexed in 4,354.58 seconds [Rate: 82,718.05 per second]
> ..
> ** Index GSPO->OSPG: 360,202,756 slots indexed in 4,084.50 seconds [Rate: 88,187.73 per second]
> ..
> Index GSPO->SPOG: 113,900,000 slots (Batch: 213,675 slots/s / Avg: 113,695 slots/s)
> Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
>         at com.hp.hpl.jena.tdb.lib.TupleLib.tuple(TupleLib.java:208)
>         at com.hp.hpl.jena.tdb.index.TupleIndexRecord$1.convert(TupleIndexRecord.java:195)
>         at com.hp.hpl.jena.tdb.index.TupleIndexRecord$1.convert(TupleIndexRecord.java:191)
>         at org.openjena.atlas.iterator.Iter$4.next(Iter.java:293)
>         at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.copyIndex(LoaderNodeTupleTable.java:209)
>         at com.hp.hpl.jena.tdb.store.bulkloader.BuilderSecondaryIndexesSequential.createSecondaryIndexes(BuilderSecondaryIndexesSequential.java:43)
>         at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.createSecondaryIndexes(LoaderNodeTupleTable.java:192)
>         at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.loadSecondaryIndexes(LoaderNodeTupleTable.java:91)
>         at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.loadIndexStart(LoaderNodeTupleTable.java:144)
>         at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$1.finish(BulkLoader.java:232)
>         at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadTriples$(BulkLoader.java:141)
>         at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadNamedGraph(BulkLoader.java:107)
>         at com.hp.hpl.jena.tdb.TDBLoader.loadNamedGraph$(TDBLoader.java:271)
>         at com.hp.hpl.jena.tdb.TDBLoader.loadGraph$(TDBLoader.java:246)
>         at com.hp.hpl.jena.tdb.TDBLoader.loadGraph(TDBLoader.java:177)
>         at com.hp.hpl.jena.tdb.TDBLoader.load(TDBLoader.java:112)
>         at tdb.tdbloader.loadNamedGraph(tdbloader.java:157)
>         at tdb.tdbloader.exec(tdbloader.java:142)
>         at arq.cmdline.CmdMain.mainMethod(CmdMain.java:97)
>         at arq.cmdline.CmdMain.mainRun(CmdMain.java:59)
>         at arq.cmdline.CmdMain.mainRun(CmdMain.java:46)
>         at tdb.tdbloader.main(tdbloader.java:53)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira