incubator-gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexis (JIRA)" <j...@apache.org>
Subject [jira] Created: (GORA-23) Limit result set in store reads
Date Tue, 25 Jan 2011 06:37:43 GMT
Limit result set in store reads
-------------------------------

                 Key: GORA-23
                 URL: https://issues.apache.org/jira/browse/GORA-23
             Project: Gora
          Issue Type: Bug
          Components: storage
         Environment: MySQL
            Reporter: Alexis


Once again, whatever the capacity of our system, we have a limited amount of RAM. Sooner or
later, we will eventually run out of memory.

Please refer to http://techvineyard.blogspot.com/2010/12/build-nutch-20.html#Gora for the
description of the issue:

When using MySQL as Gora backend, with the parse command, the execution hangs then crashes
because it runs out of memory, because of this query:

SELECT id,content,status,outlinks,baseUrl,typ,parseStatus,metadata,signature,markers FROM
webpage;

We are running exactly into the same issue that GORA-20. Except that we are not writing to
the store, but reading it. Currently the code loads the entire webpage table into memory.
We want to set a limit to the system call that pulls data from the database.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message