db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From TomohitoNakayama <tomon...@basil.ocn.ne.jp>
Subject Re: [jira] Commented: (DERBY-550) BLOB : java.lang.OutOfMemoryError with network JDBC driver (org.apache.derby.jdbc.ClientDriver)
Date Mon, 10 Jul 2006 17:20:00 GMT
Hello.

>Would it be possible, to create a server side InputStream which requests bytes from the
client on demand, and that on the server side we could use ps.setBinaryStream(..) instead
of ps.setBytes(..) ?
>The server would then stream the bytes when doing ps.execute().  Would this work if there
are multiple parameters ?
>
It may be possible to implement the idea of "requests bytes from the 
client on demand" almostly.
//Almostly means that I think there would be some restriction in server 
side code, such as order of InputStream object to be read ....

However, I think priority is cutting down excessive expanding into 
memory in the NetworkServer code.

Reading several comments from others, I feel some kind of unhealthy 
code, which is worse than just expanding into memory at once.
// I think expanding whole image into memory is unavoidable in some 
cases, and can be decided with careful consideration.

I think it is needed to find where excessive expanding code exists ....

Best regards.


Andreas Korneliussen (JIRA) wrote:

>    [ http://issues.apache.org/jira/browse/DERBY-550?page=comments#action_12419715 ] 
>
>Andreas Korneliussen commented on DERBY-550:
>--------------------------------------------
>
>I have done some tracing of this error on the server side which confirms that this is
most likely a server side problem. 
>When the DRDAConnThread reads parameter data, it will create a byte buffer and stream
all the data of the blob into it. It happens when 
>
>DRDAConnThread.readAndSetExtParam calls reader.getExtData(..).
>
>private void readAndSetExtParam( int i, DRDAStatement stmt,
>									  int drdaType, int extLen)
>				throws DRDAProtocolException, SQLException
>		{
>			PreparedStatement ps = stmt.getPreparedStatement();
>			drdaType = (drdaType & 0x000000ff); // need unsigned value
>			boolean checkNullability = false;
>			if (sqlamLevel >= MGRLVL_7 &&
>				FdocaConstants.isNullable(drdaType))
>				checkNullability = true;
>	
>			try {	
>				byte[] paramBytes = reader.getExtData(checkNullability); <---- here the entire
LOB data gets populated in a byte array
>
>Later the paramer data is set using ps.setBytes(..).
>
>In addition, if running in debug mode, all the bytes are concatenated to the tracing string
here:
>case  DRDAConstants.DRDA_TYPE_LOBBYTES:
>					case  DRDAConstants.DRDA_TYPE_NLOBBYTES:
>						if (SanityManager.DEBUG)
>							trace("parameter value is: "+paramBytes);  <---- more memory usage
>						ps.setBytes(i+1, paramBytes);
>						break;
>
>Would it be possible, to create a server side InputStream which requests bytes from the
client on demand, and that on the server side we could use ps.setBinaryStream(..) instead
of ps.setBytes(..) ?
>The server would then stream the bytes when doing ps.execute().  Would this work if there
are multiple parameters ?
>
>  
>
>>BLOB : java.lang.OutOfMemoryError with network JDBC driver (org.apache.derby.jdbc.ClientDriver)
>>-----------------------------------------------------------------------------------------------
>>
>>         Key: DERBY-550
>>         URL: http://issues.apache.org/jira/browse/DERBY-550
>>     Project: Derby
>>        Type: Bug
>>    
>>
>
>  
>
>>  Components: JDBC
>>    Versions: 10.1.1.0
>> Environment: Any environment.
>>    Reporter: Grégoire Dubois
>>    Assignee: Tomohito Nakayama
>>    
>>
>
>  
>
>>Using the org.apache.derby.jdbc.ClientDriver driver to access the
>>Derby database through network, the driver is writting all the file into memory (RAM)
before sending
>>it to the database.
>>Writting small files (smaller than 5Mo) into the database works fine,
>>but it is impossible to write big files (40Mo for example, or more), without getting
the
>>exception java.lang.OutOfMemoryError.
>>The org.apache.derby.jdbc.EmbeddedDriver doesn't have this problem.
>>Here follows some code that creates a database, a table, and trys to write a BLOB.
2 parameters are to be changed for the code to work for you : DERBY_DBMS_PATH and FILE
>>import NetNoLedge.Configuration.Configs;
>>import org.apache.derby.drda.NetworkServerControl;
>>import java.net.InetAddress;
>>import java.io.*;
>>import java.sql.*;
>>/**
>> *
>> * @author  greg
>> */
>>public class DerbyServer_JDBC_BLOB_test {
>>    
>>    // The unique instance of DerbyServer in the application.
>>    private static DerbyServer_JDBC_BLOB_test derbyServer;
>>    
>>    private NetworkServerControl server;
>>    
>>    private static final String DERBY_JDBC_DRIVER = "org.apache.derby.jdbc.ClientDriver";
>>    private static final String DERBY_DATABASE_NAME = "Test";
>>    
>>    // ###############################################################
>>    // ############### SET HERE THE EXISTING PATH YOU WANT ################
>>    // ###############################################################
>>    private static final String DERBY_DBMS_PATH =  "/home/greg/DatabaseTest";
>>    // ###############################################################
>>    // ###############################################################
>>    
>>    
>>    private static int derbyPort = 9157;
>>    private static String userName = "user";
>>    private static String userPassword = "password";
>>    
>>    // ###################################################################################
>>    // ############# DEFINE HERE THE PATH TO THE FILE YOU WANT TO WRITE INTO THE DATABASE
###########
>>    // ############# TRY A 100kb-3Mb FILE, AND AFTER A 40Mb OR BIGGER FILE #########################
>>    // ###################################################################################
>>    private static final File FILE = new File("/home/greg/01.jpg");
>>    // ###################################################################################
>>    // ###################################################################################
>>    
>>    /**
>>     * <p>Used to test the server.
>>     */
>>    public static void main(String args[]) {
>>        try {
>>            DerbyServer_JDBC_BLOB_test.launchServer();
>>            DerbyServer_JDBC_BLOB_test server = getUniqueInstance();
>>            server.start();
>>            System.out.println("Server started");
>>        
>>            // After the server has been started, launch a first connection to the
database to
>>            // 1) Create the database if it doesn't exist already,
>>            // 2) Create the tables if they don't exist already.            
>>            Class.forName(DERBY_JDBC_DRIVER).newInstance();
>>            Connection connection = DriverManager.getConnection ("jdbc:derby://localhost:"+derbyPort+"/"+DERBY_DATABASE_NAME+";create=true",
userName, userPassword);
>>            System.out.println("Network JDBC connection to Derby succeded. Database
created if not created already.");
>>            
>>            Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
>>            Statement statement2;
>>            // Create the table "file" if it doesn't already exist.
>>            String [] tableNames={"file"};
>>            boolean exist;
>>            String currentTable;
>>            ResultSet result = statement.executeQuery("SELECT TABLENAME FROM SYS.SYSTABLES");
>>            for (int i=0;i<tableNames.length;i++) {
>>                exist=false;
>>                while (result.next()){
>>                    if (tableNames[i].equalsIgnoreCase(result.getString(1)))
>>                        exist=true;
>>                }
>>            
>>                if (!exist) {
>>                    statement2 = connection.createStatement();
>>                    statement2.execute("CREATE TABLE file (" +
>>                    "file BLOB(2G) NOT NULL)");
>>                    connection.commit();
>>                }
>>                result.beforeFirst();
>>            }
>>            System.out.println("Table file created if not created already");
>>            
>>            System.out.println("File insertion into BLOB");
>>            FileInputStream inputStream = new FileInputStream(FILE);
>>            PreparedStatement   preparedStatement = connection.prepareStatement("INSERT
INTO file(file) VALUES (?)");
>>            preparedStatement.setBinaryStream(1,inputStream,(int) FILE.length());
>>            preparedStatement .execute();
>>            connection.commit();
>>            System.out.println("File inserted into BLOB");
>>        }
>>        catch (Exception e) {
>>            e.printStackTrace();
>>        }
>>    }
>>    
>>    /** Creates a new instance of MckoiServer
>>     * Password is used at the database creation. It will be the database password
once created.
>>     */    
>>    private DerbyServer_JDBC_BLOB_test() throws Exception {
>>        System.setProperty("derby.system.home", DERBY_DBMS_PATH);
>>        
>>        // Set the server to request an authentification.
>>        System.setProperty("derby.authentication.provider", "BUILTIN");
>>        System.setProperty("derby.connection.requireAuthentication", "true");
>>        
>>        // Create a user that can connect to Derby.
>>        System.setProperty("derby.user."+userName, userPassword);
>>        
>>        // Set Derby to grant full access to the created user (to all the databases).
>>        System.setProperty("derby.database.fullAccessUsers", userName);
>>        
>>        //System.setProperty("derby.system.bootAll", "true");
>>        
>>        // See if the 9157 port is already taken.
>>        // Change it if necessary.
>>        boolean isPortFree = false;
>>        while ( !isPortFree ) {
>>            try {
>>                java.net.ServerSocket serverTest = new java.net.ServerSocket(derbyPort);
>>                serverTest.close();
>>                serverTest = null;
>>                
>>                isPortFree = true;
>>            }
>>            catch (Exception e) {
>>                System.out.println("Port already in use : "+derbyPort);
>>                derbyPort++;
>>                System.out.println("Try with port "+derbyPort);
>>            }
>>        }     
>>        
>>        server = new NetworkServerControl(InetAddress.getByName("localhost"),derbyPort);
>>    }
>>    
>>    public static void launchServer() throws Exception {
>>        derbyServer = new DerbyServer_JDBC_BLOB_test();
>>    }
>>    
>>    public static DerbyServer_JDBC_BLOB_test getUniqueInstance() {
>>        return derbyServer;
>>    }
>>    
>>    /**
>>     * <p>Start the server.
>>     */
>>    public void start() {
>>        try {
>>            server.start(null);
>>        }
>>        catch (Exception e) {
>>            e.printStackTrace(System.err);
>>        }
>>    }
>>    
>>    /**
>>     * <p>Stop the server.
>>     */
>>    public void stop() {
>>        try {
>>            server.shutdown();
>>        }
>>        catch (Exception e) {
>>            e.printStackTrace(System.err);
>>        }
>>    }
>>}
>>    
>>
>
>  
>

-- 
/*

        Tomohito Nakayama
        tomonaka@basil.ocn.ne.jp
        tomohito@rose.zero.ad.jp
        tmnk@apache.org

        Naka
        http://www5.ocn.ne.jp/~tomohito/TopPage.html

*/ 


Mime
View raw message