impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Tran <vtt...@cloudera.com>
Subject Re: Impala minidump dump_breakpad_symbols
Date Wed, 29 Nov 2017 13:45:07 GMT
Hi Sandish,

1) What is the OS version of the host that generated this md?
2) Was your CDH build patched?
3) What is the full output of your dump symbols command?

On Nov 29, 2017 2:13 AM, "sanysandish@gmail.com" <sanysandish@gmail.com>
wrote:

>
>
> On 2017-11-29 11:03, Vincent Tran <vttran@cloudera.com> wrote:
> > You are going to have to dump the symbols using the RPMs that match this
> > version here:
> > /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/
> > sbin-retail/impalad
> >
> >
> > Follow "Extracting from RPMs" section of this guide:
> >
> > https://cwiki.apache.org/confluence/display/IMPALA/
> Debugging+Impala+Minidumps
> >
> > On Wed, Nov 29, 2017 at 12:28 AM, sanysandish@gmail.com <
> > sanysandish@gmail.com> wrote:
> >
> > >
> > >
> > > On 2017-11-29 10:31, Vincent Tran <vttran@cloudera.com> wrote:
> > > > Hi Sandish,
> > > >
> > > > Did you get the minidump from the minicluster of this private build?
> It
> > > > looks like the hash of your private build symbols and the one from
> the md
> > > > do not match.
> > > >
> > > > You can turn the md into a core to see the original binary version
> that
> > > > generated it:
> > > >
> > > > <breakpad-home>/bin/minidump-2-core
> > > > 11850fc8-c7ec-dd15-15ba7314-1926301a.dmp >
> > > > 11850fc8-c7ec-dd15-15ba7314-1926301a.core
> > > > If you open the resulting core file in gdb, what does the last line
> say?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Nov 28, 2017 11:36 PM, "sanysandish@gmail.com" <
> sanysandish@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On 2017-11-28 21:01, Lars Volker <lv@cloudera.com> wrote:
> > > > > > The resolved minidump looks truncated, as does the output of
> minidump
> > > > > > stackwalk. This could indicate that the minidump got corrupted.
> How
> > > did
> > > > > you
> > > > > > generate the minidump?
> > > > > >
> > > > > > On Nov 28, 2017 02:20, "sanysandish@gmail.com" <
> > > sanysandish@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Thank you, Lars Volker, for quick response's
> > > > > > >
> > > > > > > The end content of /tmp/resolved.txt  file:
> > > > > > >
> > > > > > > [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt
> > > > > > >     rbp = 0x00007fd7d4fdf220   rsp = 0x00007fc10d6ad270
> > > > > > >      r8 = 0x00007fd7d4fdf220    r9 = 0x00000000ffffffff
> > > > > > >     r10 = 0x00007fc10d6ad3d0   r11 = 0x0000000000000202
> > > > > > >     r12 = 0x0000000000000006   r13 = 0x00007fc10d6ad3d0
> > > > > > >     r14 = 0xffffffffffffff92   r15 = 0x00007fc10d6ad400
> > > > > > >     rip = 0x00007fdb5e2ffa82
> > > > > > >     Found by: given as instruction pointer in context
> > > > > > >
> > > > > > > Thread 960
> > > > > > >  0  0x7fdb5e2ffa82
> > > > > > >     rax = 0xfffffffffffffdfc   rdx = 0x0000000000000001
> > > > > > >     rcx = 0xffffffffffffffff   rbx = 0x0000000010d33e20
> > > > > > >     rsi = 0x0000000000000189   rdi = 0x0000000010d33e4c
> > > > > > >     rbp = 0x00007fc0f3e7a740   rsp = 0x00007fc0f3e7a6b0
> > > > > > >      r8 = 0x0000000010d33e20    r9 = 0x00000000ffffffff
> > > > > > >     r10 = 0x00007fc0f3e7a740   r11 = 0x0000000000000202
> > > > > > >     r12 = 0x0000000000000001   r13 = 0x00007fc0f3e7a740
> > > > > > >     r14 = 0xffffffffffffff92   r15 = 0x0000000022217000
> > > > > > >     rip = 0x00007fdb5e2ffa82
> > > > > > >     Found by: given as instruction pointer in context
> > > > > > >
> > > > > > > Output of find syms/impalad:
> > > > > > > [ec2-user@jenkins tmp]$ find syms/impalad
> > > > > > > syms/impalad
> > > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0
> > > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym
> > > > > > >
> > > > > > > These are the steps I had followed:
> > > > > > > Step-1:
> > > > > > > I followed exact steps from this
> https://cwiki.apache.org/confl
> > > > > > > uence/display/IMPALA/Building+Impala and also followed
"build
> > > > > > > native-toolchain from scratch" https://cwiki.apache.org/confl
> > > > > > > uence/display/IMPALA/Building+native-toolchain+from+scratch+
> > > > > > > and+using+with+Impala
> > > > > > >
> > > > > > > Step-2:
> > > > > > > Env:
> > > > > > > export IMPALA_HOME=/home/ec2-user/impala-setup/Impala
> > > > > > > export PATH=$PATH:$IMPALA_HOME/bin
> > > > > > > export KUDU_IS_SUPPORTED=false
> > > > > > > export KUDU_CLIENT_DIR=/opt/kudu/
> > > > > > >
> > > > > > > Get breakpad symbols:
> > > > > > > dump_breakpad_symbols.py -b be/build/ --dump_syms
> > > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > > > > 7e55f1aecb60-p3/bin/dump_syms
> > > > > > > -d /tmp/syms
> > > > > > >
> > > > > > > rm /tmp/resolved.txt
> > > > > > >
> > > > > > > Run minidump_stackwalk:
> > > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > > > > > > 7e55f1aecb60-p3/bin/minidump_stackwalk
> > > /home/ec2-user/64811560-8bae-a
> > > > > 67f-22f961c1-45bf52a3.dmp
> > > > > > > /tmp/syms > /tmp/resolved.txt
> > > > > > >
> > > > > > >
> > > > > > > Please let me know If I have missed any steps??
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > Thanks you Lars Volker,
> > > > > I took few new .dmp files and ran through stackwalk.
> > > > > here is last few lines from the resolver:
> > > > >
> > > > > Loaded modules:
> > > > > 0x00400000 - 0x0283efff  impalad  ???  (main)  (WARNING: No
> symbols,
> > > > > impalad, 21AA452CEF0F8C034C7E78A639190ED40)
> > > > > 0x7fce0a33b000 - 0x7fce0a543fff  libgssapiv2.so.3.0.0  ???
> > > > > 0x7fce0a544000 - 0x7fce0a748fff  libplain.so.3.0.0  ???
> > > > > 0x7fce0a749000 - 0x7fce0a94dfff  liblogin.so.3.0.0  ???
> > > > > 0x7fce0a94e000 - 0x7fce0ad0bfff  libdb-5.3.so  ???
> > > > > 0x7fce0ad0c000 - 0x7fce0af12fff  libsasldb.so.3.0.0  ???
> > > > > 0x7fce0af13000 - 0x7fce0b118fff  libanonymous.so.3.0.0  ???
> > > > > 0x7fce0b919000 - 0x7fce0bb79fff  libpcre.so.1.2.0  ???
> > > > > 0x7fce0bb7a000 - 0x7fce0bd9efff  libselinux.so.1  ???
> > > > > 0x7fce0bda1000 - 0x7fce0bfa4fff  libkeyutils.so.1.5  ???
> > > > > 0x7fce0bfa5000 - 0x7fce0c1b2fff  libkrb5support.so.0.1  ???
> > > > > 0x7fce0c1b3000 - 0x7fce0c3b5fff  libfreebl3.so  ???
> > > > > 0x7fce0c3b6000 - 0x7fce0c5cbfff  libz.so.1.2.7  ???
> > > > > 0x7fce0c5cc000 - 0x7fce0c7fefff  libk5crypto.so.3.1  ???
> > > > > 0x7fce0c7ff000 - 0x7fce0ca02fff  libcom_err.so.2.1  ???
> > > > > 0x7fce0ca03000 - 0x7fce0cceafff  libkrb5.so.3.3  ???
> > > > > 0x7fce0cceb000 - 0x7fce0cf37fff  libgssapi_krb5.so.2.2  ???
> > > > > 0x7fce0cf38000 - 0x7fce0d140fff  libcrypt-2.17.so  ???
> > > > > 0x7fce0d16f000 - 0x7fce0d386fff  libresolv-2.17.so  ???
> > > > > 0x7fce0d389000 - 0x7fce0d746fff  libc-2.17.so  ???  (WARNING: No
> > > symbols,
> > > > > libc-2.17.so, E42D1DFF35CF1831960C5EA195CEB1360)
> > > > > 0x7fce0d74c000 - 0x7fce0d963fff  libpthread-2.17.so  ???
> (WARNING: No
> > > > > symbols, libpthread-2.17.so, 9B139160D961AA018425DBC16B7F51D40)
> > > > >
> > > > > I think I'm still not getting symbols for impala??
> > > > >
> > > > >
> > > > >
> > > >
> > > Hi Vincent Tran,
> > >
> > > Thanks for response. No the dmp file is from different CDH cluster.
> > >
> > > Core file last few lines:
> > >
> > >
> > > ??9?k   ??ol    ?I?k    ??ւ?~W???eM?x86_64/opt/
> > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/
> > > sbin-retail/impalad--flagfile=/run/cloudera-scm-agent/
> > > process/2619-impala-IMPALAD/impala-conf/impalad_flagsCDH_
> > > HCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hive-
> > > hcatalogTOMCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> > > cdh5.11.0.p0.34/lib/bigtop-tomcatXDG_SESSION_ID=c4CDH_
> > > PIG_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/lib/pigCDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.11.
> > > 0-1.cdh5.11.0.p0.34/lib/solrPARCELS_ROOT=/opt/
> cloudera/parcelsSHELL=/bin/
> > > bashCLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-
> > > connector-java.jarGLOG_logbuflevel=0CDH_HUE_PLUGINS_
> > > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/
> > > lib/hadoopCDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> > > cdh5.11.0.p0.34/lib/hiveSCM_DEFINES_SCRIPTS=/opt/cloudera/
> > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/meta/cdh_env.shCDH_
> > > AVRO_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/lib/avroCLOUDERA_ORAC
> > >  LE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java.
> > > jarCDH_VERSION=5GLOG_minloglevel=0HIVE_DEFAULT_XML=
> > > /etc/hive/conf.dist/hive-default.xmlSEARCH_HOME=/opt/
> > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/searchCMF_
> > > CONF_DIR=/etc/cloudera-scm-agentMGMT_HOME=/usr/share/
> > > cmfCGROUP_GROUP_BLKIO=IMPALA_BIN=/opt/cloudera/parcels/CDH-
> > > 5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retailPARCEL_
> > > DIRNAMES=CDH-5.11.0-1.cdh5.11.0.p0.34USER=impalaLD_LIBRARY_
> > > PATH=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/
> > > lib/impala/lib:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/lib/impala/sbin-retail:/usr/java/jdk1.8.0_131/
> > > jre/lib/amd64:/usr/java/jdk1.8.0_131/jre/lib/amd64/server:/
> > > usr/java/jdk1.8.0_131/jre/lib/amd64/server:JSVC_HOME=/opt/
> > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/bigtop-
> > > utilsCDH_SQOOP_HOME=/usr/lib/sqoopAUX_CLASSPATH=/usr/share/
> > > java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-
> > > 9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.
> > > jarHBASE_HOME=/opt/cloudera/parcels/
> > >  CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/hbaseGLOG_max_log_
> > > size=200CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> > > cdh5.11.0.p0.34/lib/impalaCDH_HTTPFS_HOME=/opt/cloudera/
> > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-httpfsCDH_
> > > MR2_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> 0.p0.34/lib/hadoop-
> > > mapreduceCM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY
> > > HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE
> > > YARN_RM_IN_STANDBY_MODECDH_HUE_HOME=/opt/cloudera/
> > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hueCDH_MR1_HOME=/
> > > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/
> > > hadoop-0.20-mapreduceCGROUP_ROOT_MEMORY=/sys/fs/cgroup/
> > > memoryGLOG_v=1PATH=/sbin:/usr/sbin:/bin:/usr/binCONF_DIR=/
> > > run/cloudera-scm-agent/process/2619-impala-IMPALADCGROUP_GROUP_MEMORY=
> > > CDH_KAFKA_HOME=/usr/lib/kafkaCDH_SENTRY_HOME=/opt/
> > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/
> > > sentryHIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/bin/../lib/hiveCDH_PARQUET_HOME=/opt/cloudera/parcels/C
> > >  DH-5.11.0-1.cdh5.11.0.p0.34/lib/parquetSUPERVISOR_GROUP_
> NAME=2619-impala-
> > > IMPALADIMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> cdh5.11.0.p0.34/lib/
> > > impalaHBASE_CONF_DIR=/run/cloudera-scm-agent/process/
> > > 2619-impala-IMPALAD/hbase-confPWD=/run/cloudera-scm-
> > > agent/process/2619-impala-IMPALADJAVA_HOME=/usr/java/
> > > jdk1.8.0_131LANG=en_US.UTF-8HADOOP_CONF_DIR=/run/
> > > cloudera-scm-agent/process/2619-impala-IMPALAD/hadoop-
> > > confCGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacctCDH_
> > > HADOOP_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/lib/hadoopCGROUP_ROOT_BLKIO=/sys/fs/cgroup/
> > > blkioCGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacctCDH_SQOOP2_
> > > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/
> > > lib/sqoop2CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> > > cdh5.11.0.p0.34/lib/sparkCDH_CRUNCH_HOME=/usr/lib/
> > > crunchCDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.
> > > cdh5.11.0.p0.34/lib/llamaJAVA_TOOL_OPTIONS=CDH_HADOOP_BIN=/
> > > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/
> > > hadoop/bin/hadoopCDH_HDFS_HOME=/opt/
> > >  cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-
> > > hdfsSUPERVISOR_ENABLED=1LIBHDFS_OPTS=-Djava.library.
> > > path=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/
> > > bin/../lib/impala/libHOME=/var/lib/impalaSHLVL=2CDH_
> > > OOZIE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.
> > > 0.p0.34/lib/oozieKEYTRUSTEE_SERVER_HOME=/usr/lib/
> > > keytrustee-serverCDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels
> > >
> >
> >
> >
> > --
> > Vincent T. Tran
> > Customer Operations Engineer
> > Cloudera, Inc.
> >
> Hi Vincent Tran,
>
> I got these RPMs  impala-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm
> impala-debuginfo-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm
>
> and generated syms
>
> dump_breakpad_symbols.py -r impala-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm
> -s impala-debuginfo-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm
> --dump_syms $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e55f1aecb60-p3/bin/dump_syms
> -d /tmp/syms
>
>
> for resolved.txt
>
> $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e
> 55f1aecb60-p3/bin/minidump_stackwalk /home/ec2-user/impalad/
> 242b57e2-a70a-695b-10ee19a1-79f2bad3.dmp /tmp/syms > /tmp/resolved.txt
>
> I'm I missing anything??
>
> Last few lines from  /tmp/resolved.txt
>
> Loaded modules:
> 0x00400000 - 0x0283efff  impalad  ???  (main)  (WARNING: No symbols,
> impalad, 21AA452CEF0F8C034C7E78A639190ED40)
> 0x7ffa54a7d000 - 0x7ffa54c85fff  libgssapiv2.so.3.0.0  ???
> 0x7ffa54c86000 - 0x7ffa54e8afff  libplain.so.3.0.0  ???
> 0x7ffa54e8b000 - 0x7ffa5508ffff  liblogin.so.3.0.0  ???
> 0x7ffa55090000 - 0x7ffa5544dfff  libdb-5.3.so  ???
> 0x7ffa5544e000 - 0x7ffa55654fff  libsasldb.so.3.0.0  ???
> 0x7ffa55655000 - 0x7ffa5585afff  libanonymous.so.3.0.0  ???
>
>
> Still same results. Do I need to get exact version of
> CDH-5.11.0-1.cdh5.11.0.p0.34 RPM'S? can you suggest me some Cloudera repo
> links where I can find exact version impala rpm's
>
> Thanks
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message