apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Pujare <san...@datatorrent.com>
Subject Re: Proposing an operator for log parsing.
Date Thu, 17 Nov 2016 18:18:41 GMT
+1, I like this feature.

On 11/17/16, 3:26 AM, "Shraddha Jog" <shraddha.jog@synerzip.com> wrote:

    Dear community,
    
    We would like to add operator in malhar for parsing different types of
    logs.
    Idea of this operator is to read log data records of known formats such as
    Syslog, common log, combined log, extended log etc from the upstream in a
    DAG, parse/validate it based on the configured format and emit the
    validated POJO to the downstream.
    
    We are not targeting log formats from particular library as such but the
    default formats for common log, combined log, extended log and sys log.
    Also if user has some specific log format then that could be provided in a
    property and operator will parse the log according to the given format.
    
    Properties:
    LogFileFormat : Property to define data format for the log data record
    being read at the Input Port. It can be either from the above four default
    log formats or a json specifying fields and regular expression. More
    details can be found in the document.
    
    Ports :
    1. ParsedLog: This port shall emit the parsed/validated POJO object created
    based on the log format configured by the user.
    
    2. ErrorPort: This port shall emit the error log data record.
    
    Proposed design can be found here
    <https://docs.google.com/document/d/1RoTOUx_0chwTSahGxiIXlgACgRNfXiVv17hMezxFz74/edit?usp=sharing>
    .
    
    Thanks,
    Shraddha
    
    -- 
    This e-mail, including any attached files, may contain confidential and 
    privileged information for the sole use of the intended recipient. Any 
    review, use, distribution, or disclosure by others is strictly prohibited. 
    If you are not the intended recipient (or authorized to receive information 
    for the intended recipient), please contact the sender by reply e-mail and 
    delete all copies of this message.
    
    



Mime
View raw message