db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Korneliussen <Andreas.Kornelius...@Sun.COM>
Subject Re: [jira] Commented: (DERBY-550) BLOB : java.lang.OutOfMemoryError with network JDBC driver (org.apache.derby.jdbc.ClientDriver)
Date Wed, 12 Jul 2006 14:03:29 GMT
TomohitoNakayama wrote:
> Hello.
> 
> Regretfully I'm not sure whether I can fix this issue until 10.2 release
> because it is not clear where memory is used wastefully and
> how we can judge whether we could solve this issue.
> 

Hi.
I am not sure I could fix this for 10.2 either, therefore I suggested 
the other approach, which I would hope gives a less severe problem for 
the user to handle.

On the client side the problem occurs when 
NetStatementReply.copyEXTDTA(..) streams all the data over from the 
server and puts it into a byte array which is put into NetCursor. This 
array is later used to create LOBs. To fix it on the client side, the 
LOB implemtation classes would need to be reimplemented, so that they 
will do the streaming.

On the server side, the problem occurs in 
DRDAConnThread.readAndSetExtParam calls reader.getExtData(..). See my 
previous comments, where I suggested a solution.

It may of course be other problems also, however this is what I found 
when tracing this problem with a debugger.

> //Even if OutOfMemoryError happens, it would be not a problem in an 
> environment with very small amount of memory ....

I am not sure I understand why it would not be a problem in an 
environment with very small amount of memory. The side-effects of 
outofmemoryerror would be the same:

* they may occur anywhere in the user application
* client would hang if the DRDAConnThread stops


> //I think criterion to judge resolve of this issue is needed ...
> 
> However, I think throw Exception judging from amount of spare memory 
> would be too cunning behavor ......

I think it would be good to use a conservative approach to avoid 
consuming all heap space in the VM.

> 
> Now ... I'm in dilemma.....
> At least, I won't veto it.
> 

Well, I am not sure I will do it then, the best is to fix the real problem.

Regards

Andreas
> Best regards.
> 
> 
> Andreas Korneliussen (JIRA) wrote:
> 
>>    [ 
>> http://issues.apache.org/jira/browse/DERBY-550?page=comments#action_12420591 
>> ]
>> Andreas Korneliussen commented on DERBY-550:
>> --------------------------------------------
>>
>> Unless the streaming could be fixed for 10.2 so that we avoid 
>> OutOfMemoryError on the receiver side, I would propose the following:
>>
>> We know the size of the LOB, and can check if it can go into memory 
>> (using the Runtime class). If it cannot go into memory, we can throw 
>> an SQLException, instead of consuming all memory in the VM until we 
>> get OutOfMemoryError.
>>
>> By using this approach, we achieve the following:
>> * Side-effects on other connections in the VM: Although it is the LOB 
>> which is taking almost all the memory in the VM, the OutOfMemoryError 
>> may be thrown in another thread in the VM, causing side-effects on 
>> other connections or on the application itself.
>> * Currently, if the Network server goes out of memory when streaming 
>> data, the DRDAConnThread will stop. This causes hangs in the user 
>> applications.
>> If the streaming is fixed, there is not need to do this. Does anyone 
>> plan to fix the streaming issues for 10.2 ? If not, I will make a 
>> couple of JIRA issues to do the work of avoiding OutOfMemoryError by 
>> checking size before allocating the byte arrays.
>>
>>
>>  
>>
>>> BLOB : java.lang.OutOfMemoryError with network JDBC driver 
>>> (org.apache.derby.jdbc.ClientDriver)
>>> -----------------------------------------------------------------------------------------------

>>>
>>>
>>>         Key: DERBY-550
>>>         URL: http://issues.apache.org/jira/browse/DERBY-550
>>>     Project: Derby
>>>        Type: Bug
>>>   
>>
>>  
>>
>>>  Components: JDBC, Network Server
>>>    Versions: 10.1.1.0
>>> Environment: Any environment.
>>>    Reporter: Grégoire Dubois
>>>    Assignee: Tomohito Nakayama
>>> Attachments: BlobOutOfMem.java
>>>
>>> Using the org.apache.derby.jdbc.ClientDriver driver to access the
>>> Derby database through network, the driver is writting all the file 
>>> into memory (RAM) before sending
>>> it to the database.
>>> Writting small files (smaller than 5Mo) into the database works fine,
>>> but it is impossible to write big files (40Mo for example, or more), 
>>> without getting the
>>> exception java.lang.OutOfMemoryError.
>>> The org.apache.derby.jdbc.EmbeddedDriver doesn't have this problem.
>>> Here follows some code that creates a database, a table, and trys to 
>>> write a BLOB. 2 parameters are to be changed for the code to work for 
>>> you : DERBY_DBMS_PATH and FILE
>>> import NetNoLedge.Configuration.Configs;
>>> import org.apache.derby.drda.NetworkServerControl;
>>> import java.net.InetAddress;
>>> import java.io.*;
>>> import java.sql.*;
>>> /**
>>> *
>>> * @author  greg
>>> */
>>> public class DerbyServer_JDBC_BLOB_test {
>>>       // The unique instance of DerbyServer in the application.
>>>    private static DerbyServer_JDBC_BLOB_test derbyServer;
>>>       private NetworkServerControl server;
>>>       private static final String DERBY_JDBC_DRIVER = 
>>> "org.apache.derby.jdbc.ClientDriver";
>>>    private static final String DERBY_DATABASE_NAME = "Test";
>>>       // ###############################################################
>>>    // ############### SET HERE THE EXISTING PATH YOU WANT 
>>> ################
>>>    // ###############################################################
>>>    private static final String DERBY_DBMS_PATH =  
>>> "/home/greg/DatabaseTest";
>>>    // ###############################################################
>>>    // ###############################################################
>>>          private static int derbyPort = 9157;
>>>    private static String userName = "user";
>>>    private static String userPassword = "password";
>>>       // 
>>> ###################################################################################

>>>
>>>    // ############# DEFINE HERE THE PATH TO THE FILE YOU WANT TO 
>>> WRITE INTO THE DATABASE ###########
>>>    // ############# TRY A 100kb-3Mb FILE, AND AFTER A 40Mb OR BIGGER 
>>> FILE #########################
>>>    // 
>>> ###################################################################################

>>>
>>>    private static final File FILE = new File("/home/greg/01.jpg");
>>>    // 
>>> ###################################################################################

>>>
>>>    // 
>>> ###################################################################################

>>>
>>>       /**
>>>     * <p>Used to test the server.
>>>     */
>>>    public static void main(String args[]) {
>>>        try {
>>>            DerbyServer_JDBC_BLOB_test.launchServer();
>>>            DerbyServer_JDBC_BLOB_test server = getUniqueInstance();
>>>            server.start();
>>>            System.out.println("Server started");
>>>                   // After the server has been started, launch a 
>>> first connection to the database to
>>>            // 1) Create the database if it doesn't exist already,
>>>            // 2) Create the tables if they don't exist 
>>> already.                       
>>> Class.forName(DERBY_JDBC_DRIVER).newInstance();
>>>            Connection connection = DriverManager.getConnection 
>>> ("jdbc:derby://localhost:"+derbyPort+"/"+DERBY_DATABASE_NAME+";create=true",

>>> userName, userPassword);
>>>            System.out.println("Network JDBC connection to Derby 
>>> succeded. Database created if not created already.");
>>>                       Statement statement = 
>>> connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, 
>>> ResultSet.CONCUR_READ_ONLY);
>>>            Statement statement2;
>>>            // Create the table "file" if it doesn't already exist.
>>>            String [] tableNames={"file"};
>>>            boolean exist;
>>>            String currentTable;
>>>            ResultSet result = statement.executeQuery("SELECT 
>>> TABLENAME FROM SYS.SYSTABLES");
>>>            for (int i=0;i<tableNames.length;i++) {
>>>                exist=false;
>>>                while (result.next()){
>>>                    if 
>>> (tableNames[i].equalsIgnoreCase(result.getString(1)))
>>>                        exist=true;
>>>                }
>>>                           if (!exist) {
>>>                    statement2 = connection.createStatement();
>>>                    statement2.execute("CREATE TABLE file (" +
>>>                    "file BLOB(2G) NOT NULL)");
>>>                    connection.commit();
>>>                }
>>>                result.beforeFirst();
>>>            }
>>>            System.out.println("Table file created if not created 
>>> already");
>>>                       System.out.println("File insertion into BLOB");
>>>            FileInputStream inputStream = new FileInputStream(FILE);
>>>            PreparedStatement   preparedStatement = 
>>> connection.prepareStatement("INSERT INTO file(file) VALUES (?)");
>>>            preparedStatement.setBinaryStream(1,inputStream,(int) 
>>> FILE.length());
>>>            preparedStatement .execute();
>>>            connection.commit();
>>>            System.out.println("File inserted into BLOB");
>>>        }
>>>        catch (Exception e) {
>>>            e.printStackTrace();
>>>        }
>>>    }
>>>       /** Creates a new instance of MckoiServer
>>>     * Password is used at the database creation. It will be the 
>>> database password once created.
>>>     */       private DerbyServer_JDBC_BLOB_test() throws Exception {
>>>        System.setProperty("derby.system.home", DERBY_DBMS_PATH);
>>>               // Set the server to request an authentification.
>>>        System.setProperty("derby.authentication.provider", "BUILTIN");
>>>        System.setProperty("derby.connection.requireAuthentication", 
>>> "true");
>>>               // Create a user that can connect to Derby.
>>>        System.setProperty("derby.user."+userName, userPassword);
>>>               // Set Derby to grant full access to the created user 
>>> (to all the databases).
>>>        System.setProperty("derby.database.fullAccessUsers", userName);
>>>               //System.setProperty("derby.system.bootAll", "true");
>>>               // See if the 9157 port is already taken.
>>>        // Change it if necessary.
>>>        boolean isPortFree = false;
>>>        while ( !isPortFree ) {
>>>            try {
>>>                java.net.ServerSocket serverTest = new 
>>> java.net.ServerSocket(derbyPort);
>>>                serverTest.close();
>>>                serverTest = null;
>>>                               isPortFree = true;
>>>            }
>>>            catch (Exception e) {
>>>                System.out.println("Port already in use : "+derbyPort);
>>>                derbyPort++;
>>>                System.out.println("Try with port "+derbyPort);
>>>            }
>>>        }                   server = new 
>>> NetworkServerControl(InetAddress.getByName("localhost"),derbyPort);
>>>    }
>>>       public static void launchServer() throws Exception {
>>>        derbyServer = new DerbyServer_JDBC_BLOB_test();
>>>    }
>>>       public static DerbyServer_JDBC_BLOB_test getUniqueInstance() {
>>>        return derbyServer;
>>>    }
>>>       /**
>>>     * <p>Start the server.
>>>     */
>>>    public void start() {
>>>        try {
>>>            server.start(null);
>>>        }
>>>        catch (Exception e) {
>>>            e.printStackTrace(System.err);
>>>        }
>>>    }
>>>       /**
>>>     * <p>Stop the server.
>>>     */
>>>    public void stop() {
>>>        try {
>>>            server.shutdown();
>>>        }
>>>        catch (Exception e) {
>>>            e.printStackTrace(System.err);
>>>        }
>>>    }
>>> }
>>>   
>>
>>  
>>
> 


Mime
View raw message