mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil Chmielewski <kamil....@gmail.com>
Subject Re: manage jobs log files in sandboxes
Date Fri, 06 Nov 2015 14:36:30 GMT
see
https://github.com/eirslett/mesos/commit/7eac3da95fdd39a0fb96b8d8a883e72b6b9f9943

2015-11-06 15:34 GMT+01:00 Kamil Chmielewski <kamil.chm@gmail.com>:

> Try to cantact with https://github.com/eirslett
> He can have a PoC from MesosCon Europe hackathon with Mesos writting
> containers logs to Logstash.
>
> Cheers,
> Kamil
>
>
> 2015-11-06 15:26 GMT+01:00 Paul Bell <arachweb@gmail.com>:
>
>> I've done a little reconnoitering, and the terrain looks to me as follows:
>>
>>    1. Docker maintains container log files at
>>    /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log
>>    2. Mesos maintains container STDOUT files at a
>>    slave/framework/application specific location, e.g.,
>>    /tmp/mesos/slaves/20151102-082316-370041927-5050-32381-S1/frameworks/20151102-082316-370041927-5050-32381-0000/executors/ecxprimary1.80750071-81a0-11e5-8596-82d195a34239/runs/5c767378-9599-40af-8010-a31f4c55f9dc
>>    3. The latter is mapped to the container's /mnt/mesos/sandbox
>>    4. These two files (-json.log and the STDOUT file) are different,
>>    *each* consumes disk space.
>>
>> I think that the answer to (1) is Docker's logrotate.
>>
>> As to (2), I am considering a cron job at host (not container) level that
>> drives truncate cmd (GNU coreutils) to prune these files at a certain size.
>> Obviously requires knowing the fully-qualified path under
>> /tmp/mesos/slaves, but this is readily available via "docker inspect".
>>
>> -Paul
>>
>>
>> On Fri, Nov 6, 2015 at 7:17 AM, Paul Bell <arachweb@gmail.com> wrote:
>>
>>> Hi Mauricio,
>>>
>>> Yeah....I see your point; thank you.
>>>
>>> My approach would be akin to closing the barn door after the horse got
>>> out. Both Mesos & Docker are doing their own writing of STDOUT. Docker's
>>> rotation won't address Mesos's behavior.
>>>
>>> I need to find a solution here.
>>>
>>> -Paul
>>>
>>>
>>> On Thu, Nov 5, 2015 at 10:46 PM, Mauricio Garavaglia <
>>> mauriciogaravaglia@gmail.com> wrote:
>>>
>>>> Hi Paul,
>>>>
>>>> I don't think that's going to help :(
>>>> Even if you configure a different docker log driver, Docker still send
>>>> things to stdout, which is catched by mesos and dumped in the .logs
>>>> directory in the job sandbox. For example, by default docker logs into a
>>>> json file in /var/lib/docker but mesos still writes to the sandbox.
>>>> Hi Mauricio,
>>>>
>>>> I'm grappling with the same issue.
>>>>
>>>> I'm not yet sure if it represents a viable solution, but I plan to look
>>>> at Docker's log rotation facility. It was introduced in Docker 1.8.
>>>>
>>>> If you beat me to it & it looks like a solution, please let us know!
>>>>
>>>> Thanks.
>>>>
>>>> Cordially,
>>>>
>>>> Paul
>>>>
>>>> > On Nov 5, 2015, at 9:40 PM, Mauricio Garavaglia <
>>>> mauriciogaravaglia@gmail.com> wrote:
>>>> >
>>>> > Hi guys,
>>>> >
>>>> > How can I manage the stdout/err log files generated by jobs in mesos?
>>>> for long running docker apps launched using marathon the log files can
>>>> deplete the disk of an agent, and using quotas makes the jobs to be killed
>>>> which is also not ideal. I'd like to have a way to rotate them.
>>>> >
>>>> > Is it correct to just go to the mesos agent workdir and go through
>>>> each sandbox stdout/err and rotate them? I know that could break the log
UI
>>>> but it doesn't scale very well having logs of several of GB.
>>>> >
>>>> > Thanks!
>>>>
>>>
>>>
>>
>

Mime
View raw message