chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiaqi Tan (JIRA)" <j...@apache.org>
Subject [jira] Assigned: (CHUKWA-94) SALSA state-machine extraction from Hadoop logs
Date Fri, 12 Jun 2009 14:20:07 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jiaqi Tan reassigned CHUKWA-94:
-------------------------------

    Assignee: Jiaqi Tan

> SALSA state-machine extraction from Hadoop logs
> -----------------------------------------------
>
>                 Key: CHUKWA-94
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-94
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: Data Processors
>            Reporter: Jiaqi Tan
>            Assignee: Jiaqi Tan
>         Attachments: tan.pdf
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> This is a proposed feature addition to extract state-machine views from Hadoop's logs
(TaskTracker, JobTracker, and DataNode currently supported, NameNode soon). These views are
as described in http://www.usenix.org/event/wasl08/tech/full_papers/tan/tan_html/ and will
enable analysis and diagnosis algorithms to be built on top of them.
> Building a full SALSA view involves two steps:
> 1. Incrementally parsing log entries on a per-node basis to extract states (line-by-line
reading, assuming the entire log file from a given node is available to the same process)
> 2. "Stitching" and correlating states across all logs (across nodes and across types)
to build a full state machine.
> My idea is to add SALSA as two jobs in the demux stage, with the first parsing job in
demux, and either having: 
> (a) the parsing job write its output to the permanent store with the correlating job
reading/writing from/to the permanent store, or 
> (b) the parsing job write its output back to the sinkfile and having the correlating
job reading from the sink file and writing to the permanent store.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message