Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: schema to just read as "byte[] array" from kafka
From: Chao Wang <chaowang@wustl.edu>
To: "Raja.Aravapalli" <Raja.Aravapalli@target.com>
Cc: "user@flink.apache.org" <user@flink.apache.org>
References: <C1BB89EC-7AC5-4320-A124-919056EE0F38@target.com>
 <1c520543-81db-2140-d617-fde6a22c73ac@wustl.edu>
Message-ID: <1a58126f-78ce-53f8-5da3-47bc3af3c97e@wustl.edu>
Date: Mon, 7 Aug 2017 12:28:35 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
In-Reply-To: <1c520543-81db-2140-d617-fde6a22c73ac@wustl.edu>
Content-Type: multipart/alternative;
 boundary="------------5A4786BEAFCE330B1E62962B"
Content-Language: en-US
Received-SPF: None (protection.outlook.com: wustl.edu does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Aug 2017 17:28:39.8183 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0201MB1797
archived-at: Mon, 07 Aug 2017 17:28:58 -0000

This is a multi-part message in MIME format.
--------------5A4786BEAFCE330B1E62962B
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

A quick update, in class MyDe:

public static class MyDe<T> extends AbstractDeserializationSchema<byte[]> {
   @Override
   public byte[] deserialize(byte[] arg0) {
     // Perform deserialization here, if needed;
     // otherwise, probably we can simply return arg0 as raw byte[]
     return arg0;
   }
}


Chao

On 08/07/2017 12:23 PM, Chao Wang wrote:
>
> Hi Raja,
>
> I just happened to work on the similar thing, and here is how to do it 
> in general, I think (In my case, I did a bit more, to deserialize a 
> tuple of <byte[],byte[]>) :
>
> FlinkKafkaConsumer010<byte[]> consumer = new 
> FlinkKafkaConsumer010<>("topic_name", new MyDe<byte[]>(), properties);
>
> and for MyDe the schema:
>
> public static class MyDe<T> extends 
> AbstractDeserializationSchema<byte[]> {
>   @Override
>   public byte[] deserialize(byte[] arg0) {
>     return new e;
>   }
> }
>
>
> Chao
>
> On 08/07/2017 10:47 AM, Raja.Aravapalli wrote:
>>
>> Hi
>>
>> I am using /SimpleStringSchema/ to deserialize a message read from 
>> kafka, but need some help to know if there is any schema available I 
>> can use rather than “SimpleStringSchema()” and instead just get 
>> “byte[]” without any deserialization happening!
>>
>> Below is code I am currently using, but instead of 
>> SimpleStringSchema() which is giving me Strings, but I want the a raw 
>> byte array Byte[]:
>>
>> FlinkKafkaConsumer08<String> myConsumer = *new 
>> *FlinkKafkaConsumer08<>(*"xxx_topic"*, *new */SimpleStringSchema(),/ 
>> properties);
>>
>> Thanks a lot.
>>
>> Regards,
>>
>> Raja.
>>
>


--------------5A4786BEAFCE330B1E62962B
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>A quick update, in class MyDe:<br>
    </p>
    public static class MyDe&lt;T&gt; extends
    AbstractDeserializationSchema&lt;byte[]&gt; {<br>
      @Override<br>
      public byte[] deserialize(byte[] arg0) {<br>
        // Perform deserialization here, if needed;<br>
        // otherwise, probably we can simply return arg0 as raw byte[]<br>
        return arg0;<br>
      }<br>
    }<br>
    <br>
    <br>
    Chao<br>
    <br>
    <div class="moz-cite-prefix">On 08/07/2017 12:23 PM, Chao Wang
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:1c520543-81db-2140-d617-fde6a22c73ac@wustl.edu">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <p>Hi Raja,</p>
      <p>I just happened to work on the similar thing, and here is how
        to do it in general, I think (In my case, I did a bit more, to
        deserialize a tuple of &lt;byte[],byte[]&gt;) :<br>
      </p>
      FlinkKafkaConsumer010&lt;byte[]&gt; consumer = new
      FlinkKafkaConsumer010&lt;&gt;("topic_name", new
      MyDe&lt;byte[]&gt;(), properties);<br>
      <br>
      and for MyDe the schema:<br>
      <br>
      public static class MyDe&lt;T&gt; extends
      AbstractDeserializationSchema&lt;byte[]&gt; {<br>
        @Override<br>
        public byte[] deserialize(byte[] arg0) {<br>
          return new e;<br>
        }<br>
      }<br>
      <br>
      <br>
      Chao<br>
      <br>
      <div class="moz-cite-prefix">On 08/07/2017 10:47 AM,
        Raja.Aravapalli wrote:<br>
      </div>
      <blockquote type="cite"
        cite="mid:C1BB89EC-7AC5-4320-A124-919056EE0F38@target.com">
        <meta name="Title" content="">
        <meta name="Keywords" content="">
        <meta name="Generator" content="Microsoft Word 15 (filtered
          medium)">
        <style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:Calibri;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:Calibri;
	color:windowtext;}
span.msoIns
	{mso-style-type:export-only;
	mso-style-name:"";
	text-decoration:underline;
	color:teal;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:Calibri;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style>
        <div class="WordSection1">
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">Hi<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">I am using
              <i>SimpleStringSchema</i> to deserialize a message read
              from kafka, but need some help to know if there is any
              schema available I can use rather than
              “SimpleStringSchema()” and instead just get “byte[]”
              without any deserialization happening!<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">Below is
              code I am currently using, but instead of
              SimpleStringSchema() which is giving me Strings, but I
              want the a raw byte array Byte[]:<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span
              style="font-size:11.0pt;color:#0432FF">FlinkKafkaConsumer08&lt;String&gt;
              myConsumer = <b>new </b>FlinkKafkaConsumer08&lt;&gt;(<b>"xxx_topic"</b>,
              <b>new </b><i><span
                  style="background:yellow;mso-highlight:yellow">SimpleStringSchema(),</span></i>
              properties);<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">Thanks a
              lot. <o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">Raja.<o:p></o:p></span></p>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>

--------------5A4786BEAFCE330B1E62962B--