metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: Metron Indexing Service Failing Shortly After Start
Date Tue, 27 Feb 2018 19:12:03 GMT
I'm not a python expect but I would either uninstall then install with pip
or install dependencies as needed.

On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <mcginnisd@avalonconsult.com
> wrote:

> Is there a fix available already, or how would you go about making this
> fix? I'm open to figuring it out if you have any ideas on how we should fix
> the issue. Requests was installed for me already without pip even being
> installed, likely through the MPack process or as part of a previously
> installed component. Pip reported requests had been installed through
> distutils.
>
> On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman <merrimanr@gmail.com>
> wrote:
>
>> Yep you found it.  You were unlucky enough to install a version of master
>> that had this regression.  How did you install requests?  Using pip should
>> also install the necessary dependencies.  Either way everything should
>> still work, you'll just have to manually start/stop indexing since you
>> won't ever be able to stop it.
>>
>> You are correct about HDP 2.6.
>>
>> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
>> mcginnisd@avalonconsult.com> wrote:
>>
>>> Looks like this issue was introduced in this commit:
>>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit that
>>> seems to deal with this, but it is for the Ansible deployment, so I don't
>>> believe it would affect an MPack installation:
>>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>>> c0c4c34373aec68dbd30 .
>>>
>>> I installed a newer version of requests just to see if that would fix
>>> the issue, and I'm getting a similar issue in a different package relied
>>> upon by requests now (urllib3). It seems as if something is causing sys to
>>> not exist or not point to what it should. I wonder if it has to do with the
>>> python2.6 vs python2.7 issue.
>>>
>>> Also, as I was looking through the commits, it looks like using the last
>>> official release wouldn't work for me, since we are on HDP 2.6, and changes
>>> went in since the last release to support that version of HDP.
>>>
>>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>>> mcginnisd@avalonconsult.com> wrote:
>>>
>>>> *Simon:*
>>>>
>>>> There are some errors from processing showing up in the indexing topic,
>>>> but no data is making it through without restarting Metron completely
>>>> (restarting just the indexing service doesn't seem to do it).
>>>>
>>>> I don't have easy access to the Storm UI at the moment. I have a work
>>>> around to get the logs that I'm going to look at shortly.
>>>>
>>>> I understand, but I didn't realize I had not pulled from a release
>>>> until I had already installed, and this is just a prototype cluster
>>>> anyways. I'll likely pull from an established release for the production
>>>> deployment.
>>>>
>>>> *Ryan:*
>>>>
>>>> I actually found something interesting in the ambari-agent logs on the
>>>> Metron node. See the callstack below.
>>>>
>>>> Traceback (most recent call last):
>>>>   File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>>> line 59, in run_file
>>>>     imp.load_source('__main__', script)
>>>>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>>> ckage/scripts/indexing_master.py", line 18, in <module>
>>>>     import requests
>>>>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line
>>>> 53, in <module>
>>>>     from .packages.urllib3.contrib import pyopenssl
>>>>   File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
>>>> line 61, in load_module
>>>>     if name in sys.modules:
>>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>>
>>>> Perhaps there is a missing component that needs to be installed?
>>>>
>>>> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman <merrimanr@gmail.com>
>>>> wrote:
>>>>
>>>>> If there was an error happening during the service check, it will be
>>>>> in the Ambari agent logs.  I would look there first.
>>>>>
>>>>> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
>>>>> simon@simonellistonball.com> wrote:
>>>>>
>>>>> Anything appearing on the indexing topic in kafka?
>>>>>
>>>>> Anything in the logs for the indexing topology in Storm UI?
>>>>>
>>>>> Master isn’t always the best place to start, might be worth sticking
>>>>> to a released build to kick the tyres.
>>>>>
>>>>> Simon
>>>>>
>>>>> On 27 Feb 2018, at 17:38, David McGinnis <mcginnisd@avalonconsult.com>
>>>>> wrote:
>>>>>
>>>>> All,
>>>>>
>>>>> I have a kerberized HDP 2.6.3 cluster which I have installed Metron on
>>>>> through an MPack. I am using code straight from the master branch of
the
>>>>> github as of last week.
>>>>>
>>>>> When I start Metron, all of the components correct start up. Shortly
>>>>> after, however, the Indexing Service stops working. I have not been able
to
>>>>> find any logs that seem to capture any issues, and both indexing storm
>>>>> topologies are still active and running after the failure. Those topologies
>>>>> only report a warning about one of my parsers using the default HDFS
writer
>>>>> config, which appears to be normal.
>>>>>
>>>>> Not only is this making my dashboard red when it shouldn't be, but
>>>>> also it seems as if data doesn't go through the pipeline until the entire
>>>>> Metron system is restarted. I am not confident this is related, but I
>>>>> suspect it is.
>>>>>
>>>>> Where should I go to find logs about this service, or does anyone have
>>>>> any ideas why this would be happening?
>>>>>
>>>>> --
>>>>> David McGinnis
>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc>
|
>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>> <https://twitter.com/avalonconsult>
>>>>> ------------------------------------------------------------
>>>>> -------------------------------------------------
>>>>> This message (including any attachments) contains confidential
>>>>> information
>>>>> intended for a specific individual and purpose, and is protected by
>>>>> law. If
>>>>> you are not the intended recipient, you should delete this message.
>>>>> Any
>>>>> disclosure, copying, or distribution of this message, or the taking of
>>>>> any
>>>>> action based on it, is strictly prohibited.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> David McGinnis
>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>> <https://twitter.com/avalonconsult>
>>>> ------------------------------------------------------------
>>>> -------------------------------------------------
>>>> This message (including any attachments) contains confidential
>>>> information
>>>> intended for a specific individual and purpose, and is protected by
>>>> law. If
>>>> you are not the intended recipient, you should delete this message. Any
>>>> disclosure, copying, or distribution of this message, or the taking of
>>>> any
>>>> action based on it, is strictly prohibited.
>>>>
>>>
>>>
>>>
>>> --
>>> David McGinnis
>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>> <https://twitter.com/avalonconsult>
>>> ------------------------------------------------------------
>>> -------------------------------------------------
>>> This message (including any attachments) contains confidential
>>> information
>>> intended for a specific individual and purpose, and is protected by law.
>>> If
>>> you are not the intended recipient, you should delete this message. Any
>>> disclosure, copying, or distribution of this message, or the taking of
>>> any
>>> action based on it, is strictly prohibited.
>>>
>>
>>
>
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
> ------------------------------------------------------------
> -------------------------------------------------
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>

Mime
View raw message