apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Proposing an operator for log parsing.
Date Thu, 17 Nov 2016 19:10:39 GMT
Good addition

Thks
Amol

On Thu, Nov 17, 2016 at 11:00 AM, Pradeep A. Dalvi <prad@apache.org> wrote:

> +1 for the feature
>
> On Thu, 17 Nov 2016 at 16:56 Shraddha Jog <shraddha.jog@synerzip.com>
> wrote:
>
> > Dear community,
> >
> > We would like to add operator in malhar for parsing different types of
> > logs.
> > Idea of this operator is to read log data records of known formats such
> as
> > Syslog, common log, combined log, extended log etc from the upstream in a
> > DAG, parse/validate it based on the configured format and emit the
> > validated POJO to the downstream.
> >
> > We are not targeting log formats from particular library as such but the
> > default formats for common log, combined log, extended log and sys log.
> > Also if user has some specific log format then that could be provided in
> a
> > property and operator will parse the log according to the given format.
> >
> > Properties:
> > LogFileFormat : Property to define data format for the log data record
> > being read at the Input Port. It can be either from the above four
> default
> > log formats or a json specifying fields and regular expression. More
> > details can be found in the document.
> >
> > Ports :
> > 1. ParsedLog: This port shall emit the parsed/validated POJO object
> created
> > based on the log format configured by the user.
> >
> > 2. ErrorPort: This port shall emit the error log data record.
> >
> > Proposed design can be found here
> > <
> > https://docs.google.com/document/d/1RoTOUx_
> 0chwTSahGxiIXlgACgRNfXiVv17hMezxFz74/edit?usp=sharing
> > >
> > .
> >
> > Thanks,
> > Shraddha
> >
> > --
> > This e-mail, including any attached files, may contain confidential and
> > privileged information for the sole use of the intended recipient. Any
> > review, use, distribution, or disclosure by others is strictly
> prohibited.
> > If you are not the intended recipient (or authorized to receive
> information
> > for the intended recipient), please contact the sender by reply e-mail
> and
> > delete all copies of this message.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message