From derby-dev-return-119273-archive-asf-public=cust-asf.ponee.io@db.apache.org Thu Oct 11 19:02:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A927F18067E for ; Thu, 11 Oct 2018 19:02:05 +0200 (CEST) Received: (qmail 8602 invoked by uid 500); 11 Oct 2018 17:02:04 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 8512 invoked by uid 99); 11 Oct 2018 17:02:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Oct 2018 17:02:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4E2A918061D for ; Thu, 11 Oct 2018 17:02:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Asd6Y2FPnoYu for ; Thu, 11 Oct 2018 17:02:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id DDA085F356 for ; Thu, 11 Oct 2018 17:02:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1E270E2613 for ; Thu, 11 Oct 2018 17:02:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 95B0C24D4B for ; Thu, 11 Oct 2018 17:02:00 +0000 (UTC) Date: Thu, 11 Oct 2018 17:02:00 +0000 (UTC) From: "Rick Hillegas (JIRA)" To: derby-dev@db.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DERBY-7006) Investigate putting generated classes under the engine module loader MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DERBY-7006?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1664= 6758#comment-16646758 ]=20 Rick Hillegas commented on DERBY-7006: -------------------------------------- ---- h2. Summary of Experiments I have put some effort into trying to get Derby to load the generated class= es into the engine module. On core-libs-dev@openjdk.java.net, I started an = email thread titled "generated code and jigsaw modules". Two suggestions ca= me back: S1) R=C3=A9mi Forax recommended that we try loading the generated bytes as = follows: {noformat} java.lang.invoke.MethodHandles.lookup().defineClass(generatedClassBytes) {noformat} S2) Alan Bateman suggested that we study code from the java.xml module: htt= p://hg.openjdk.java.net/jdk/jdk/raw-file/tip/src/java.xml/share/classes/com= /sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.java I tried both approaches. See the attached patches: derby-7006-01-aa-remiFor= ax.diff and derby-7006-01-ac-alanBateman.diff. Both approaches solved some important problems: P1) running simple DDL and queries P2) running triggers But neither approach solved the following problem: P3) running functions which live inside jar files stored in the database After posting these results to core-libs-dev@openjdk.java.net, I received m= ore kind advice, which, nevertheless, did not fix problem P3. These are my conclusions: C1) S1 is the simpler, more straightforward solution. C2) S2 is more complicated to start out with. But I will play around with i= t more. Better solutions may occur as JPMS evolves. In the short-term, we could als= o reduce the attack surface of the exposed engine modules. See https://issu= es.apache.org/jira/browse/DERBY-7012 The following sections provide brief descriptions of the attached patches a= s well as the additional context provided by my two posts to core-libs-dev@= openjdk.java.net ---- h2. First Solution: Use MethodHandles.lookup() The main features of this solution are: MH1) A stub class in the generated package is added to the engine codeline. MH2) Generated classes are loaded into the engine module by calling support= code in java.lang.invoke.MethodHandles. The support code must be called fr= om the stub class in order to anchor the generated classes in the generated= package: {noformat} java.lang.invoke.MethodHandles.lookup().defineClass(generatedClassBytes) {noformat} MH3) An extra permission is required for the engine jar when running under = a security manager: {noformat} permission java.lang.RuntimePermission "defineClass"; {noformat} ---- h2. Second Solution: Study TemplatesImpl The main features of this solution are: TI1) A separate module is created for each generated class. TI2) The engine module exports the necessary packages to the generated modu= le. TI3) An extra permission is required for the engine jar when running under = a security manager: {noformat} permission java.lang.RuntimePermission "getProtectionDomain"; {noformat} I need to solve the following problems: TIP1) I don't know how to get the module names of jar files which are loade= d into the database. This could be solved if we divined the jar file names = when the jars are loaded into the database by SQLJ.INSTALL_JAR/SQLJ.REPLACE= _JAR and then stored the names in SYS.SYSTABLES. TIP2) I think that there is a slow memory leak when the generated class is = garbage-collected and the generated module and extra exports are orphaned. = I don't know how to remove this extra garbage. But maybe we could plug the = leak by generating only one module for each database. We would need to make= sure that all of this machinery is unloaded when the embedded driver is de= -registered. ---- h2. First Message to core-libs-dev@openjdk.java.net I am looking for advice about how to tighten up module encapsulation while = generating byte code on the fly. I ask this question on behalf of Apache De= rby, a pure-Java relational database whose original code dates back to Java= 1.2. I want to reduce Derby's attack-surface when running with a module pa= th. First a little context: A relational database is an interpreter for the SQL= language. It converts SQL queries into byte code which then runs on a virt= ual machine embedded in the interpreter. In Derby's case, the virtual machi= ne is the Java VM and the byte code is simply Java byte code. That is, a De= rby query plan is a class whose byte code is generated on the fly at run ti= me. I have converted the Apache Derby codeline into a set of jigsaw modules: ht= tps://issues.apache.org/jira/browse/DERBY-6945. Unfortunately, I had to pun= ch holes in the encapsulation of the main Derby module so that the generate= d query plans could call back into the Derby engine. That is because, by de= fault, generated query plans load into the catch-all, unnamed module. Note = that all of these generated classes live in a single package which does not= belong to any named module. 1) Is it possible to load generated code into a named module? 2) Alternatively, can someone recommend another approach for preserving mod= ule encapsulation while generating classes on the fly? I would appreciate any advice or examples which you can recommend. Thanks, -Rick ---- h2. Second Message to core-libs-dev@openjdk.java.net Thanks again to R=C3=A9mi and Alan for their advice. Unfortunately, I have = not been able to make either approach work, given another complexity of Der= by's class loading. Let me explain that additional issue. Derby lets users load jar files into the database. There they live as named= blobs of bytes. The jar files contain user-defined data types, functions, = procedures, and aggregators, which are coded in Java and can be used in SQL= statements. Derby lets users wire these jar files into a custom classpath = which drives a custom ClassLoader at query-execution time. I have not been = able to make this custom ClassLoader work with either R=C3=A9mi or Alan's a= pproach. Note that a Derby engine manages many databases and each database = can have its own custom ClassLoader. I like the simplicity of R=C3=A9mi's approach: java.lang.invoke.MethodHandles.lookup().defineClass(generatedClassBytes) This approach does indeed put the generated class where I want it: inside t= he Derby engine module. Unfortunately, the ClassLoader of the generated cla= ss is the application class loader. I can't figure out how to force the gen= erated class to use the custom ClassLoader instead. As a consequence, the = generated class cannot resolve user-defined functions which live inside jar= files in the database. Poking the customer ClassLoader into the thread's c= ontext class loader before calling MethodHandles.lookup() doesn't work. Alan's approach is a bit more complicated. It involves following the patter= n in com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl. It involve= s generating a temporary module for each generated class and then adding mo= re export directives to the engine module so that the generated module can = call back into the engine. I have to say I'm a little confused about the im= plications of slow memory leaks with this approach. I don't know what happe= ns to these generated modules and export directives when the generated clas= s is garbage-collected. More immediately, however, I am up against the same problem which plagues R= =C3=A9mi's approach: how do I get the generated module to resolve classes i= n the custom ClassLoader? More specifically, I am stuck trying to get the g= enerated module to require the user-written modules, that is, the user-writ= ten jar files. What I am missing is the ability to retrieve the module name= s of these jar files so that I can craft requires directives. The only way = I know to get a module name is to use ModuleFinder.of(Path...). Unfortunate= ly, the Path interface is an abstraction for file systems and is not a good= fit for locating a blob of bytes stored inside a database. I would appreciate any further advice about how to get over these speed bum= ps. Thanks, -Rick > Investigate putting generated classes under the engine module loader > -------------------------------------------------------------------- > > Key: DERBY-7006 > URL: https://issues.apache.org/jira/browse/DERBY-7006 > Project: Derby > Issue Type: Improvement > Components: SQL > Affects Versions: 10.15.0.0 > Reporter: Rick Hillegas > Priority: Major > Attachments: derby-7006-01-aa-remiForax.diff, derby-7006-01-ac-al= anBateman.diff > > > Right now, the generated query plans are compiled into the catch-all unna= med module. This forces us to grant reflective access to several engine pac= kages. It would be nice to encapsulate the generated classes inside the eng= ine module loader. -- This message was sent by Atlassian JIRA (v7.6.3#76005)