asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xikui Wang <xik...@uci.edu>
Subject Re: [jira] [Commented] (ASTERIXDB-1694) Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)
Date Fri, 21 Oct 2016 09:24:28 GMT
Hi Devs,

I'd like to put note on this problem, in case anyone hit this problem again
or have an insight of what's causing the problem.

Basically, this problem is caused by the initialization fail of Log4J in
Twitter4j(so not our problem :D). The solution is enclosed in Mingda's last
reply.
However, the cause for this problem is still unclear to us, especially the
case that it works on 4 nodes but not on 16 nodes.
Code snippet [1] shows how Twitter4J creates logger. It scans all possible
libraries and found the one that is available first. The feed adaptor
in AsterixDB will run on only one of nodes in cluster. According to Mingda,
if we shutdown the node with log problem, the new node that
adaptor running on will have the same problem.

As for a permanent solution, probably I can turn off the Twitter4j logger
in code, or hardcode the configuration to avoid this problem in the future.

If anyone have better idea, please let me know. Thanks! :)

[1]
https://github.com/yusuke/twitter4j/blob/4ebca9da71b271775624b11b5197af99a57bf175/twitter4j-core/src/internal-logging/java/twitter4j/Logger.java

On Wed, Oct 19, 2016 at 8:17 PM, mingda li <limingda1993@gmail.com> wrote:

> Dear all,
> Good news!
> For the official version of AsterixDB, the datafeed problem for Twitter can
> be solved for 16 nodes by adding a log4j.properties
> to asterix-server-0.8.9-SNAPSHOT-binary-assembly/etc and /repo. I will try
> Wail's version. And see why this can work for 4 nodes without adding the
> log4j.properties file.
>
> BTW, the log4j.properties file is as following, if you may need someday:
> # Set root logger level to DEBUG and its only appender to A1.
> log4j.rootLogger=DEBUG, A1
>
> # A1 is set to be a ConsoleAppender.
> log4j.appender.A1=org.apache.log4j.ConsoleAppender
>
> # A1 uses PatternLayout.
> log4j.appender.A1.layout=org.apache.log4j.PatternLayout
> log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>
>
>
> On Wed, Oct 19, 2016 at 8:07 PM, mingda li <limingda1993@gmail.com> wrote:
>
> > BTW, I tried to run AsterixDB's official version of Tweet feed and also
> > meet similar problem in the Node 14.
> > I follow Xikui's suggestion to add a log4j.properties
> > in asterix-server-0.8.9-SNAPSHOT-binary-assembly/etc and /repo. I
> checked
> > the log file and find it is changed to the attachment. It seems begin
> >  catch tweet but failed.
> > ​
> >  nc-red15.log
> > <https://drive.google.com/file/d/0B-3JraLWXVVGVVJ0T0VHaVpGYzA/
> view?usp=drive_web>
> > ​
> >
> >
> > On Wed, Oct 19, 2016 at 12:51 PM, mingda li <limingda1993@gmail.com>
> > wrote:
> >
> >> En, that is good suggestion.
> >> Since this is not my version of AsterixDB, we should ask Wail if he have
> >> ever set something related to twitter4j's log.
> >>
> >> @Wail, have you ever set such thing?
> >>
> >> On Tue, Oct 18, 2016 at 6:19 PM, Xikui Wang <xikuiw@uci.edu> wrote:
> >>
> >>> It looks like the log4j in Twitter4J is not correctly initialized[1].
> Did
> >>> you customize the log4j in Twitter4J configuration in your system like
> >>> this[2]? By default, it's printed to standard output.
> >>>
> >>>
> >>> [1]
> >>> http://activemq.apache.org/log4j-warn-no-appenders-could-be-
> >>> found-for-logger.html
> >>> [2] http://twitter4j.org/en/configuration.html#logger
> >>>
> >>> On Tue, Oct 18, 2016 at 5:42 PM, mingda li <limingda1993@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > When I start 16 nodes, I found the 15th node has log file different
> >>> from
> >>> > others as following.
> >>> >
> >>> > Oct 18, 2016 5:23:10 PM org.apache.hyracks.control.nc.NCDriver main
> >>> > SEVERE: Setting uncaught exception handler
> >>> org.apache.hyracks.api.lifecyc
> >>> > le.LifeCycleComponentManager@73f792cf
> >>> > Oct 18, 2016 5:23:10 PM org.apache.hyracks.control.nc.
> >>> > NodeControllerService
> >>> > start
> >>> > INFO: Starting NodeControllerService
> >>> > Oct 18, 2016 5:23:10 PM
> >>> > org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint
> >>> > start
> >>> > INFO: Starting Asterix node controller: red15
> >>> > log4j:WARN No appenders could be found for logger
> >>> > (twitter4j.TwitterStreamImpl).
> >>> > log4j:WARN Please initialize the log4j system properly.
> >>> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>> for
> >>> > more info.
> >>> > Oct 18, 2016 5:25:20 PM org.apache.hyracks.control.com
> >>> mon.work.WorkQueue$
> >>> > WorkerThread auditWaitsAndBlocks
> >>> > WARNING: Work StartTasks waited 0 times (~0ms), blocked 1 times
> (~0ms)
> >>> > Oct 18, 2016 5:25:31 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:47 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 229376 bytes
> >>> >
> >>> > The other nodes' log files are without the problem: ( I have tried
to
> >>> give
> >>> > up the node, but there will be a new node with the same problem)
> >>> > Oct 18, 2016 5:23:09 PM org.apache.hyracks.control.nc.NCDriver main
> >>> > SEVERE: Setting uncaught exception handler
> >>> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@73f792cf
> >>> > Oct 18, 2016 5:23:09 PM org.apache.hyracks.control.nc.
> >>> > NodeControllerService
> >>> > start
> >>> > INFO: Starting NodeControllerService
> >>> > Oct 18, 2016 5:23:09 PM
> >>> > org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
> >>> > INFO: Starting Asterix node controller: red6
> >>> > Oct 18, 2016 5:25:31 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:32 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 229376 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:33 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> > Oct 18, 2016 5:25:47 PM org.apache.hyracks.control.nc.Joblet close
> >>> > WARNING: Freeing leaked 131072 bytes
> >>> >
> >>> > The cluster can work well except the data feeding function on Twitter
> >>> (load
> >>> > for a while but no items loaded from Twitter). The data feeding
> >>> function of
> >>> > RSS can also work.
> >>> > The problem is wired. Does anyone have idea?
> >>> >
> >>> > Bests,
> >>> > Mingda
> >>> >
> >>> >
> >>> > On Tue, Oct 18, 2016 at 10:17 AM, mingda li <limingda1993@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Oh, sure. I just came back to school and could repeat the
> experiment
> >>> on
> >>> > > our cluster today. I will try.
> >>> > >
> >>> > > Bests,
> >>> > > Mingda
> >>> > >
> >>> > > On Tue, Oct 18, 2016 at 12:25 AM, Wail Alkowaileet <
> >>> wael.y.k@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > >> Unfortunately I couldn't reproduce the issue in our tiny cluster.
> >>> > >>
> >>> > >> @Mingda if you can post the logs it might be helpful.
> >>> > >>
> >>> > >> On Sat, Oct 15, 2016 at 12:22 AM, Yingyi Bu <buyingyi@gmail.com>
> >>> wrote:
> >>> > >>
> >>> > >> > >> Log4J in Twitter4j doesn't initialize properly.
> >>> > >> >
> >>> > >> > Can you explain that a little bit more?
> >>> > >> > What exceptions do you hit, "class not found", or sth.
else?
> >>> > >> > Thx!
> >>> > >> >
> >>> > >> > On Fri, Oct 14, 2016 at 2:20 PM, Wail Alkowaileet (JIRA)
<
> >>> > >> jira@apache.org>
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > >
> >>> > >> > >     [ https://issues.apache.org/jira/browse/ASTERIXDB-1694?
> >>> > >> > > page=com.atlassian.jira.plugin.system.issuetabpanels:comment
> >>> > >> -tabpanel&
> >>> > >> > > focusedCommentId=15576516#comment-15576516 ]
> >>> > >> > >
> >>> > >> > > Wail Alkowaileet commented on ASTERIXDB-1694:
> >>> > >> > > ---------------------------------------------
> >>> > >> > >
> >>> > >> > > After a long meeting, Mingda and I located the problem.
Log4J
> in
> >>> > >> > Twitter4j
> >>> > >> > > doesn't initialize properly.
> >>> > >> > > I don't know why that's the case. I suspect that
there's
> >>> something
> >>> > >> wrong
> >>> > >> > > in the deployment of AsterixDB.
> >>> > >> > >
> >>> > >> > > > Fail running Tweet Feed on Cluster of 16 nodes
(while
> succeed
> >>> on 4
> >>> > >> > nodes)
> >>> > >> > > > ------------------------------
> ------------------------------
> >>> > >> > > -------------
> >>> > >> > > >
> >>> > >> > > >                 Key: ASTERIXDB-1694
> >>> > >> > > >                 URL: https://issues.apache.org/
> >>> > >> > > jira/browse/ASTERIXDB-1694
> >>> > >> > > >             Project: Apache AsterixDB
> >>> > >> > > >          Issue Type: Bug
> >>> > >> > > >          Components: Feeds
> >>> > >> > > >         Environment: asterix-server-0.8.9-SNAPSHOT-
> >>> > binary-assembly
> >>> > >> run
> >>> > >> > > on cluster
> >>> > >> > > >            Reporter: Mingda Li
> >>> > >> > > >            Assignee: Xikui Wang
> >>> > >> > > >   Original Estimate: 24h
> >>> > >> > > >  Remaining Estimate: 24h
> >>> > >> > > >
> >>> > >> > > > Recently, I am trying to run the data feed
query on a
> cluster
> >>> of
> >>> > 16
> >>> > >> > > nodes. All the query run well without error. But
each time
> >>> after I
> >>> > >> > > disconnect the feed, there is none of tweet data
stored in
> >>> > dataverse.
> >>> > >> > > However, when I try to run other AQL queries, the
cluster can
> >>> work
> >>> > >> well.
> >>> > >> > > And I have ever used 4 nodes cluster to load Tweet
data
> >>> > successfully.
> >>> > >> I
> >>> > >> > > also checked the log file and find no error there.
This is
> >>> wired.
> >>> > Does
> >>> > >> > > anyone know why? Has anyone ever used the data feed
function
> on
> >>> a
> >>> > >> cluster
> >>> > >> > > of 16 nodes or more?
> >>> > >> > > > I am using a asterix-server-0.8.9-SNAPSHOT-binary-assembly
> to
> >>> > >> > configure
> >>> > >> > > cluster compiled by myself.
> >>> > >> > >
> >>> > >> > >
> >>> > >> > >
> >>> > >> > > --
> >>> > >> > > This message was sent by Atlassian JIRA
> >>> > >> > > (v6.3.4#6332)
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >>
> >>> > >> *Regards,*
> >>> > >> Wail Alkowaileet
> >>> > >>
> >>> > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message