drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Seddon <mr.tom.sed...@gmail.com>
Subject Re: How to load data in Drill
Date Wed, 27 Nov 2013 22:52:26 GMT
Hi,

I've done the same thing, converting one of the AMP Labs test data files
into parquet using Pig and then attempting to query via sqlline.  I'm
querying the file in HDFS, (which I have tested successfully with the
provided sample files). Here is the error:

0: jdbc:drill:schema=parquet> select * from
"/user/tom/ranking-parquet-new/part-m-00000.parquet";
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
running query.[error_id: "da7ce1f7-b0e9-4720-bf10-333bf668869d"
endpoint {
  address: "localhost"
  user_port: 31010
  bit_port: 32011
}
error_type: 0
message: "Failure while running fragment. < NullPointerException"
]
java.lang.RuntimeException: org.apache.drill.exec.rpc.RpcException: Remote
failure while running query.[error_id:
"da7ce1f7-b0e9-4720-bf10-333bf668869d"
endpoint {
  address: "localhost"
  user_port: 31010
  bit_port: 32011
}
error_type: 0
message: "Failure while running fragment. < NullPointerException"
]
at
org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:44)
at
net.hydromatic.optiq.runtime.ObjectEnumeratorCursor.next(ObjectEnumeratorCursor.java:44)
at net.hydromatic.optiq.jdbc.OptiqResultSet.next(OptiqResultSet.java:162)
at sqlline.SqlLine$BufferedRows.<init>(SqlLine.java:2499)
at sqlline.SqlLine.print(SqlLine.java:1886)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3835)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3738)
at sqlline.SqlLine.dispatch(SqlLine.java:882)
at sqlline.SqlLine.begin(SqlLine.java:717)
at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:460)
at sqlline.SqlLine.main(SqlLine.java:443)
Caused by: org.apache.drill.exec.rpc.RpcException: Remote failure while
running query.[error_id: "da7ce1f7-b0e9-4720-bf10-333bf668869d"
endpoint {
  address: "localhost"
  user_port: 31010
  bit_port: 32011
}
error_type: 0
message: "Failure while running fragment. < NullPointerException"
]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:79)
at
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:48)
at
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:33)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:142)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:127)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:724)
0: jdbc:drill:schema=parquet>


On 22 November 2013 02:12, Madhu Borkar <backmeupone@gmail.com> wrote:

> Hi Ben,
> I don't have the same sqlline script that you have mentioned. This I found
> under bin
> bin/sqlline
>
>
>
> #!/bin/bash
>
> ARGS=();
> while [ $# -gt 0 ]
> do
>   case "$1" in
>   -q) shift;
>       QUERY=$1;;
>   -e) shift;
>       QUERY=$1;;
>   -f) shift;
>       FILE=$1;;
>    *) ARGS+=($1);;
>   esac
>   shift
> done
>
> bin=`dirname "${BASH_SOURCE-$0}"`
> bin=`cd "$bin">/dev/null; pwd`
>
> . "$bin"/drill-config.sh
>
> if [ -z $JAVA_HOME ]
> then
>   JAVA=`which java`
> else
>   JAVA=`find -L $JAVA_HOME -name java | head -n 1`
> fi
>
> if [ -e $JAVA ]; then
>   echo ""
> else
>   echo "Java not found."
>   exit 1
> fi
>
> $JAVA -version 2>&1 | grep "version" | egrep -e "1.7" > /dev/null
> if [ $? -ne 0 ]; then
>   echo "Java 1.7 is required to run Apache Drill."
>   exit 1
> fi
>
> # get log directory
> if [ "$DRILL_LOG_DIR" = "" ]; then
>   export DRILL_LOG_DIR=/var/log/drill
> fi
>
>
> CP=$DRILL_HOME/jars/*:$CP
> CP=$DRILL_HOME/lib/*:$CP
> CP=$DRILL_HOME/contrib/*:$CP
> CP=$HADOOP_CLASSPATH:$CP
>
> CP=$DRILL_CONF_DIR:$CP
>
> DRILL_SHELL_JAVA_OPTS="$DRILL_SHELL_JAVA_OPTS
> -Dlog.path=$DRILL_LOG_DIR/sqlline.log"
>
> if [ -n "$QUERY" ]
> then
>   echo $QUERY | exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
> sqlline.SqlLine "${ARGS[@]}"
> elif [ -n "$FILE" ]
> then
>   exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
> sqlline.SqlLine "${ARGS[@]}" --run=$FILE
> else
>   exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
> sqlline.SqlLine "${ARGS[@]}"
> fi
> -----------------------
> Is this because I installed from binary?
> Thanks
> -Madhu
>
> On Thu, Nov 21, 2013 at 9:50 AM, Ben Becker <bbecker@maprtech.com> wrote:
>
> > Hi Madhu,
> >
> > Could you attach the complete logs during this run?  The easiest way is
> > probably to edit the sqlline shell script and remove the '
> > -Dlogback.configurationFile=...' argument like so:
> >
> > -exec java -Xmx2G
> > -Dlogback.configurationFile=./sqlparser/src/test/resources/logback.xml
> -cp
> > ./:"$(cat .classpath)" sqlline.SqlLine --verbose=true "$@"
> > *+exec java -Xmx2G -cp ./:"$(cat .classpath)" sqlline.SqlLine
> > --verbose=true "$@"*
> >
> > This will cause all log output to be directed to the console.
> >  Alternatively, you may want to log to a file or another facility by
> > editing the aforementioned logback.xml file.  One last note -- although
> > unlikely, parts of the dataset may be included in the logs, so be sure to
> > redact any sensitive information from the logs before posting.
> >
> > Thanks,
> > Ben
> >
> >
> >
> > On Tue, Nov 19, 2013 at 6:36 PM, Madhu Borkar <backmeupone@gmail.com>
> > wrote:
> >
> > > Ben,
> > > Here is log snippet using lilith..
> > > <?xml version='1.0' encoding='UTF-8'?>
> > > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "
> > > http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
> > > <html xmlns='http://www.w3.org/1999/xhtml
> > '><head><title>Title</title><link
> > > href='detailsView.css' rel='stylesheet'
> > > type='text/css'/></head><body><table><tr
> > > class='even'><th>Message</th><td><pre>Exception
during
> > > query</pre></td></tr><tr class='odd'><th>Throwable</th><td
> > > class='throwableContainer'><div class='throwable'><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>org.apache.drill.exec.rpc.RpcException</a><pre><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>Remote
> > > failure while running query.[error_id:
> > > "b9ce54dd-085c-435d-b12f-e9896c60e864"
> > > endpoint {
> > >   address: "Node-0.etouch.net"
> > >   user_port: 31010
> > >   bit_port: 32011
> > > }
> > > error_type: 0
> > > message: "Failure while running fragment. &lt; NullPointerException"
> > > ]</a></pre><hr/><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>at&nbsp;org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:79)'>at&nbsp;org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:79)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:48)'>at&nbsp;org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:48)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:33)'>at&nbsp;org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:33)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:142)'>at&nbsp;org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:142)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:127)'>at&nbsp;org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:127)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> > >
> > >
> >
> href='ste://io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)'>at&nbsp;io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)'>at&nbsp;io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)'>at&nbsp;io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)'>at&nbsp;io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)'>at&nbsp;io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)'>at&nbsp;io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)'>at&nbsp;io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)'>at&nbsp;io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)'>at&nbsp;io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)&nbsp;~[netty-common-4.0.7.Final.jar:na]</a><a
> > >
> > >
> >
> href='ste://java.lang.Thread.run(Thread.java:744)'>at&nbsp;java.lang.Thread.run(Thread.java:744)&nbsp;~[na:1.7.0_45]</a></div></td></tr><tr
> > > class='even'><th>Level</th><td>ERROR</td></tr><tr
> > >
> > >
> >
> class='odd'><th>Logger</th><td>org.apache.drill.sql.client.full.ResultEnumerator</td></tr><tr
> > > class='even'><th>Thread</th><td>main (id=1)</td></tr><tr
> > > class='odd'><th>Timestamp</th><td>2013-11-19
> > 18:26:42.023-0800</td></tr><tr
> > > class='even'><th>Call&nbsp;Location</th><td><a
> > >
> > >
> >
> href='ste://org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:43)'>at&nbsp;org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:43)</a></td></tr><tr
> > >
> > >
> >
> class='odd'><th>Logger&nbsp;Context</th><td><table><tr><th>Name</th><td>default</td></tr><tr><th>Birthtime</th><td>2013-11-19
> > >
> > >
> >
> 18:26:06.371-0800</td></tr><tr><th>Properties</th><td><table><tr><th>Key</th><th>Value</th></tr><tr><td>HOSTNAME</td><td>
> > > Node-0.etouch.net</td></tr></table></td></tr></table></td></tr><tr
> > >
> > >
> >
> class='even'><th>Source&nbsp;ID</th><td>127.0.0.1-20131119T182606709</td></tr><tr
> > >
> class='odd'><th>Local&nbsp;ID</th><td>71</td></tr></table></body></html>
> > >
> > >
> > > On Tue, Nov 19, 2013 at 9:34 AM, Ben Becker <bbecker@maprtech.com>
> > wrote:
> > >
> > > > Hi Madhu,
> > > >
> > > > Could you report which version you're using (e.g. 1.0.0-m1.rc3), and
> > > attach
> > > > the logs when this happens?  You can use Lilith to view the logs if
> you
> > > > like (see instructions at the bottom of this
> > > > page<
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/How+To+Run+Drill+with+Sample+Data#
> > > > >
> > > > ).
> > > >
> > > > Thanks,
> > > > Ben
> > > >
> > > >
> > > > On Mon, Nov 18, 2013 at 10:16 PM, Madhu Borkar <
> backmeupone@gmail.com
> > > > >wrote:
> > > >
> > > > > I have devlopment MapR cluster. I have installed Apache Drill using
> > > > > binaries. I could run queries with sample-data.
> > > > >
> > > > > I decided to load test data using pig. Here is my code
> > > > >
> > > > > ---------------------------------
> > > > >
> > > > > REGISTER file:/home/mapr/lib/parquet-pig-bundle-1.2.5.jar
> > > > >
> > > > > data = LOAD '$DATA'
> > > > >
> > > > >                 USING PigStorage(',')
> > > > >
> > > > > as (
> > > > >
> > > > >       user:    chararray,
> > > > >
> > > > >       street: chararray,
> > > > >
> > > > >       state:          chararray,
> > > > >
> > > > >       city:           chararray,
> > > > >
> > > > >       type:  chararray
> > > > >
> > > > >   );
> > > > >
> > > > >
> > > > >
> > > > > A = LIMIT data 3;
> > > > >
> > > > > dump A;
> > > > >
> > > > > fs -rmr /mapr-hive/parquet/data;
> > > > >
> > > > > STORE data INTO '/mapr-hive/parquet/data' using
> > > > > parquet.pig.ParquetStorer();
> > > > >
> > > > > ----------------------
> > > > >
> > > > > Pig script writes to output. I moved data under apache drill
> > directory.
> > > > >
> > > > >
> > > > >
> > > > > select * from "real-data/users.parquet";
> > > > >
> > > > >
> > > > >
> > > > > but when I try to query this data I get rpc error?
> > > > >
> > > > > Query failed: org.apache.drill.exec.rpc.RpcException: Remote
> failure
> > > > while
> > > > > running query.[error_id: "04ac52e4-b49e-45b8-82c5-cf480154eefd"
> > > > >
> > > > > endpoint {
> > > > >
> > > > >   address: "Node.etouch.net"
> > > > >
> > > > >   user_port: 31010
> > > > >
> > > > >   bit_port: 32011
> > > > >
> > > > > }
> > > > >
> > > > > error_type: 0
> > > > >
> > > > > message: "Failure while running fragment. < NullPointerException"
> > > > >
> > > > > ]
> > > > >
> > > > >
> > > > >
> > > > > Please, provide me help how to load data in Apache Drill.?
> > > > >
> > > > > Is there MapR configuration needed for Drill?
> > > > >
> > > > > Regards
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message