Return-Path: X-Original-To: apmail-asterixdb-dev-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4632818F2A for ; Sat, 13 Jun 2015 08:03:32 +0000 (UTC) Received: (qmail 77876 invoked by uid 500); 13 Jun 2015 08:03:32 -0000 Delivered-To: apmail-asterixdb-dev-archive@asterixdb.apache.org Received: (qmail 77817 invoked by uid 500); 13 Jun 2015 08:03:32 -0000 Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list dev@asterixdb.incubator.apache.org Received: (qmail 77805 invoked by uid 99); 13 Jun 2015 08:03:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Jun 2015 08:03:31 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dtabass@gmail.com designates 209.85.218.50 as permitted sender) Received: from [209.85.218.50] (HELO mail-oi0-f50.google.com) (209.85.218.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Jun 2015 08:01:14 +0000 Received: by oigz2 with SMTP id z2so33446160oig.1 for ; Sat, 13 Jun 2015 01:03:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type; bh=wUs9mxlGiOV0lDzU3g+DuCxMVwt+5BslYPrqgGNovOY=; b=Ww3ooUL82BFbDWs4drEO6I9zudxIsm3p+1AZ9SM5F4R18UA5YQBH0gJoM9pxhmCp7O xSdAQeUcANTMpPVfgmE6bpc9CXsXlno8Oj723p9OSd8Zm7EVPAHQLzsZpS2FDljcohYI aBRVuHezpMVRJ1k4h/zKSChiS1P76ir7g81ZnjrHIX44FRqdY/wj6r8Szl8yAje9+mWn Ri2su1dWBcu78c8AyQNSjVE2NJUxmwAQt1BKJCYGyPry4MmFc5QmcpEnqIapkPN5N109 vl34KZzla03hZAKFpZIye4lsdGZiJ9XgoPqC8NmCwgH8rVq45Fu+Uj7uWZ6uQ1U3++ev ftQA== X-Received: by 10.182.186.4 with SMTP id fg4mr15323346obc.7.1434182581191; Sat, 13 Jun 2015 01:03:01 -0700 (PDT) Received: from mikejcarey.local (ip72-219-187-63.oc.oc.cox.net. [72.219.187.63]) by mx.google.com with ESMTPSA id sn2sm4314752obc.11.2015.06.13.01.02.59 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Jun 2015 01:03:00 -0700 (PDT) Message-ID: <557BE3B3.6040100@gmail.com> Date: Sat, 13 Jun 2015 01:02:59 -0700 From: Mike Carey User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: dev@asterixdb.incubator.apache.org Subject: Re: Severe issue relating to indexing on nullable fields within a closed type? References: <31E8C4CE-CC21-43CF-9B1D-727BFA3EAAC6@gmail.com> <5577DAAD.2030908@gmail.com> <98BBEF2B-FB84-453E-A6A1-A0E43E0394C6@gmail.com> <00030CE9-6A25-4450-9AFC-891632498280@gmail.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------080902030904060205050904" X-Virus-Checked: Checked by ClamAV on apache.org --------------080902030904060205050904 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Agreed. The upcoming release will be major, so it is kind of our one chance to mess with the metadata in such a major way.... @ildar, I think you might be the most nimble at this particular set of issues, given your experience in this part of the code...? (This is probably a release blocker for all the feed, external data, nested indexing, and open indexing goodies....) On 6/12/15 6:43 PM, Ian Maxon wrote: > More discussion was had today about this, to summarize I think these > were the actions decided on in order of greatest to least priority: > > - Hotfix the current manifestation of the issue (done and merged) > - Change the metadata to not give optional fields in closed types an > anonymous type name (this will break current datasets). Would be > highly desirable to do this before Apache release in the next week or > so. > - Be sure that no modification occurs on MD insert, i.e. treat the > inserted object as final. > > > Thoughts? > > -Ian > > On Thu, Jun 11, 2015 at 5:28 PM, Ian Maxon wrote: >>>> The locally produced Datatype object is actually never totally valid though >>>> from the way I am seeing things. So there is no reason to cache it in the >>>> first place. >>>> >>> Indeed. But bypassing a cache is valid in this particular case only. >>> >> I think the crux of the problem though is that the MD node should >> really be treating any object it receives to insert as final. Having >> the client generate something to be inserted, then having the MD node >> mutate that upon asking it to store the information, seems like it is >> asking for trouble. It's certainly possible to do a delicate dance of >> cache invalidation, but unless there's a good reason to do that- why >> shouldn't we avoid it? >> >> - Ian >> >> >> On Thu, Jun 11, 2015 at 5:00 PM, Ildar Absalyamov >> wrote: >>> Replies inline: >>> >>>> On Jun 11, 2015, at 16:28, Ian Maxon wrote: >>>> >>>> The locally produced Datatype object is actually never totally valid though >>>> from the way I am seeing things. So there is no reason to cache it in the >>>> first place. >>>> >>> Indeed. But bypassing a cache is valid in this particular case only. >>> >>>> I talked with Mike about this today. I think the main points were as >>>> follows (correct me if I am wrong of course!): >>>> >>>> - Is it possible to move the name generation in DatatypeTupleTranslator >>>> into TypeTranslator.computeTypes? This would let the client have the fully >>>> updated object and avoid updating anything at the MD node side. >>> I was thinking about this as well. Don’t quite see why that would not be possible. >>> >>>> - The entire type name generation seems weird. Why is it that we need those >>>> names in the first place? For example take this type: >>>> >>>> >>>> create type FacebookMessageType as closed { >>>> message-id: int32, >>>> author-id: int32, >>>> in-response-to: int32?, >>>> sender-location: point?, >>>> message: string >>>> } >>>> >>>> and the Metadata entry for that type: >>>> >>>> { "DataverseName": "TinySocial", >>>> "DatatypeName": "FacebookMessageType", >>>> "Derived": { "Tag": "RECORD", >>>> "IsAnonymous": false, >>>> "EnumValues": null, >>>> "Record": { "IsOpen": false, >>>> "Fields": [ { "FieldName": >>>> "message-id", "FieldType": "int32" }, >>>> { "FieldName": >>>> "author-id", "FieldType": "int32" }, >>>> { "FieldName": >>>> "in-response-to", >>>> "FieldType": >>>> "Field_in-response-to_in_FacebookMessageType" >>>> }, >>>> { "FieldName": >>>> "sender-location", >>>> "FieldType": >>>> "Field_sender-location_in_FacebookMessageType" >>>> }, >>>> { "FieldName": >>>> "message", "FieldType": "string" } >>>> ] >>>> }, >>>> "Union": null, >>>> "UnorderedList": null, >>>> "OrderedList": null >>>> }, >>>> "Timestamp": "Thu Jun 11 11:35:33 PDT 2015" >>>> } >>>> >>>> Wouldn't it be better if we could make the Field entries for the nullable >>>> fields not dependent on this weird anonymous UNION(something,null) type? >>>> Something more like >>>> >>>> { "FieldName": "in-response-to", "FieldType":"point", "Optional":true} >>> Yeah, this whole naming thing might look weird, especially when it is applied to nullable fields, which are in essence a primitive type. >>> But for whatever reason we have chosen implementing a nullable fields as a a complex data type: union with null field. At the same time generating names for complex types is legitimate, and even required in case of anonymous nested types, e.g: >>> >>> create type FacebookMessageType as closed { >>> message-id: int32, >>> author-id: int32, >>> in-response-to: int32?, >>> sender-location: { >>> coordinate:point, >>> POI: string >>> }?, >>> message: string >>> } >>> >>>> We should definitely discuss this more on Friday >>>> >>>> - Ian >>>> >>>> On Wed, Jun 10, 2015 at 5:39 PM, Ildar Absalyamov < >>>> ildar.absalyamov@gmail.com> wrote: >>>> >>>>> Just a bit of background, the bug is not specific to RTree, but will be >>>>> triggered on any index, created on a nullable field. >>>>> In order to persist datatype with a nullable field a special internal type >>>>> with name ’Type_xxx_UnionType_yyy’ must be created on MD node side. >>>>> However the stale version of datatype (where the name of the nullable >>>>> field is not yet computed) is returned on the client, where it got cached. >>>>> >>>>> Clearly, relying on the fact that the argument supposedly passed by >>>>> reference will get updated is wrong, when things are sent via RMI (and >>>>> Ian’s patch resolves that). >>>>> However I think a better way would be to invalidate locally cached stale >>>>> copy, so when the actual index is created it will query the MD and get the >>>>> right Datatype. >>>>> That is in line with the question Niki (our visiting student from Germany) >>>>> asked me today, for which I did not had a clear answer: how do we manage >>>>> the local MD cache and invalidate it? >>>>> >>>>>> On Jun 10, 2015, at 02:43, Ian Maxon wrote: >>>>>> >>>>>> Indeed, it is weird practice at best. The real index is an RTree index on >>>>>> FacebookMessages.sender-location, which is nullable, and the code path to >>>>>> create that index just happens to trigger this. >>>>>> >>>>>> On Tue, Jun 9, 2015 at 11:35 PM, Mike Carey wrote: >>>>>> >>>>>>> This sounds like a bad practice (doing any mods on the MD node - we >>>>> should >>>>>>> view the call as passing in immutable data). >>>>>>> But I'm not sure which index is being referred to here - what index on >>>>>>> what dataset is the "actual index" below? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 6/9/15 5:11 PM, Ian Maxon wrote: >>>>>>> >>>>>>>> So I think I might have a fix for this: >>>>>>>> https://asterix-gerrit.ics.uci.edu/#/c/283/1 >>>>>>>> >>>>>>>> This bug is really insidious. What I think is happening is that in >>>>>>>> AqlTranslator, the types are computed and given to the MetadataManager >>>>> as >>>>>>>> a >>>>>>>> new Datatype, to insert into the metadata node via RMI on MetadataNode. >>>>>>>> However MetadataNode modifies the object it is given before it inserts >>>>> it >>>>>>>> into the actual index. That is where the names of the nullable types >>>>> get >>>>>>>> generated, and so the MetadataManager doesn't get to see those changes >>>>>>>> unless MetadataManager and MetadataNode are actually sharing the same >>>>>>>> object. So if it is stale, that object gets inserted into the cache, >>>>> and >>>>>>>> when the index is created, it sees the stale type information rather >>>>> than >>>>>>>> the true value that's stored in the index. >>>>>>>> >>>>>>>> My fix is just to read it again before inserting into the cache. I'm >>>>> not >>>>>>>> really sure thats the right thing to do though. And if there are any >>>>> other >>>>>>>> instances of this type (caller expecting changes on remote side to >>>>>>>> reflect), the same sort of behavior can happen. >>>>>>>> >>>>>>>> - Ian >>>>>>>> >>>>>>>> On Tue, Jun 9, 2015 at 10:12 AM, Ian Maxon wrote: >>>>>>>> >>>>>>>> My quack theory is that it is due to the RMI somehow, or that the RMI >>>>>>>>> reveals it. It isn't specific to managix. It happens as long as the CC >>>>>>>>> and >>>>>>>>> NC reside in different JVMs. >>>>>>>>> >>>>>>>>> - Ian >>>>>>>>> >>>>>>>>> On Tue, Jun 9, 2015 at 10:03 AM, Ildar Absalyamov < >>>>>>>>> ildar.absalyamov@gmail.com> wrote: >>>>>>>>> >>>>>>>>> As Ian mentioned test case already exists (which I thought was good >>>>>>>>>> enough to prevent regressions like this one), but for some reason >>>>> it’s >>>>>>>>>> not >>>>>>>>>> triggering the error, whereas if the same ddl\query is executed on a >>>>>>>>>> local >>>>>>>>>> cluster, which was prepared by managix, the error will trigger. >>>>>>>>>> >>>>>>>>>> On Jun 9, 2015, at 08:07, Mike Carey wrote: >>>>>>>>>>> Ildar, can you try to nail this one down and also add a test case if >>>>>>>>>>> you >>>>>>>>>>> can create one once this is making more sense...? >>>>>>>>>>> On Jun 8, 2015 11:58 PM, "Ildar Absalyamov" < >>>>>>>>>>> ildar.absalyamov@gmail.com >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Interesting, I guess step 1 here would be to investigate what’s >>>>>>>>>>> different >>>>>>>>>>> in test framework\AsterixHyracksIntegrationUntil setup and regular >>>>>>>>>>> cluster >>>>>>>>>>> configuration, created by managix. >>>>>>>>>>>> On Jun 8, 2015, at 23:37, Ian Maxon wrote: >>>>>>>>>>>>> I was playing around with the latest snapshot version, trying to >>>>>>>>>>>>> >>>>>>>>>>>> create a >>>>>>>>>>> fresher version of my demo docker container, and so I ran through >>>>> the >>>>>>>>>>>>> ADM/AQL 101. However I couldn't get it to work, which was really >>>>>>>>>>>>> >>>>>>>>>>>> surprising >>>>>>>>>>>> >>>>>>>>>>>>> (and distressing). The error appears after trying to create an >>>>> RTree >>>>>>>>>>>> index >>>>>>>>>>>> >>>>>>>>>>>>> on >>>>>>>>>>>>> sender-location: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> SEVERE: java.lang.NullPointerException >>>>>>>>>>>>> >>>>>>>>>>>>>> edu.uci.ics.asterix.metadata.MetadataException: >>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.metadata.MetadataNode.addIndex(MetadataNode.java:229) >>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) >>>>>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:200) >>>>>>>>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:197) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$97(TCPTransport.java:683) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1502895914.run(Unknown >>>>>>>>>>> Source) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253) >>>>>>>>>>> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) >>>>>>>>>>> at com.sun.proxy.$Proxy11.addIndex(Unknown Source) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.metadata.MetadataManager.addIndex(MetadataManager.java:417) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.aql.translator.AqlTranslator.handleCreateIndexStatement(AqlTranslator.java:920) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:261) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.api.http.servlet.APIServlet.doPost(APIServlet.java:97) >>>>>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) >>>>>>>>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546) >>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110) >>>>>>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:347) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:924) >>>>>>>>>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:781) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) >>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>>>>>>>> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:347) >>>>>>>>>>>>>> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> edu.uci.ics.hyracks.dataflow.common.data.marshalling.UTF8StringSerializerDeserializer.serialize(UTF8StringSerializerDeserializer.java:44) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.dataflow.data.nontagged.serde.AStringSerializerDeserializer.serialize(AStringSerializerDeserializer.java:47) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.dataflow.data.nontagged.serde.AStringSerializerDeserializer.serialize(AStringSerializerDeserializer.java:26) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider$1.serialize(AqlSerializerDeserializerProvider.java:208) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider$1.serialize(AqlSerializerDeserializerProvider.java:189) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.metadata.entitytupletranslators.IndexTupleTranslator.getTupleFromMetadataEntity(IndexTupleTranslator.java:260) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> edu.uci.ics.asterix.metadata.MetadataNode.addIndex(MetadataNode.java:224) >>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) >>>>>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:200) >>>>>>>>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:197) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$97(TCPTransport.java:683) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1502895914.run(Unknown >>>>>>>>>>> Source) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>>>> at >>>>>>>>>>>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>>>> ... 1 more >>>>>>>>>>>>> I started going back, trying this on older and older revisions >>>>> until >>>>>>>>>>>> it >>>>>>>>>>> stopped happening. It looks like this somehow has been lurking in >>>>> the >>>>>>>>>>>>> Open/nested indexing code for a long time. The error does not >>>>>>>>>>>>> >>>>>>>>>>>> reproduce >>>>>>>>>>> in >>>>>>>>>>>>> the testing framework, for some reason I don't understand. >>>>>>>>>>>>> >>>>>>>>>>>>> The source of the bug seems to be that the RTree index is on a >>>>> type >>>>>>>>>>>>> of >>>>>>>>>>>>> UNION(sender-location, null). The name of the type is not set >>>>> like it >>>>>>>>>>>>> should be, it is null. Then in IndexTupleTranslator, the 9th >>>>> field of >>>>>>>>>>>> the >>>>>>>>>>> new metadata record is written with the name of the type- which is >>>>>>>>>>>> null, >>>>>>>>>>> so >>>>>>>>>>>>> everything explodes. >>>>>>>>>>>>> >>>>>>>>>>>>> What is especially puzzling- is that this doesn't reproduce in >>>>>>>>>>>>> AsterixHyracksIntegrationUtil. There, the name of the type is >>>>>>>>>>>>> "Field_sender-location_in_FacebookMessageType". This type is >>>>> created >>>>>>>>>>>> too >>>>>>>>>>> in >>>>>>>>>>>>> the deployed version, however it doesn't seem to be picked up. >>>>>>>>>>>>> >>>>>>>>>>>>> I will keep digging on this, but I imagine someone more familiar >>>>> with >>>>>>>>>>>> the >>>>>>>>>>> Open/Nested indexing patch might have an idea about why this is >>>>>>>>>>>> happening. >>>>>>>>>>>> >>>>>>>>>>>>> To reproduce just try deploying the snapshot version from the >>>>> website >>>>>>>>>>>>> normally on your machine, and run through ADM/AQL 101. >>>>>>>>>>>>> >>>>>>>>>>>>> - Ian >>>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Ildar >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>> Ildar >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>> Best regards, >>>>> Ildar >>>>> >>>>> >>> Best regards, >>> Ildar >>> --------------080902030904060205050904--