Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
From: Matthias Boehm <mboehm7@googlemail.com>
Subject: Re: [Discuss] String requirements for data passed to SystemML Frames.
To: dev@systemml.incubator.apache.org
References: <OF8A05C473.728F90D9-ON88258054.0032A1D5-88258054.0032C305@notes.na.collabserv.com>
Message-ID: <d789efd8-73de-b5c2-e9ed-c8f5d88deb6d@gmail.com>
Date: Sat, 22 Oct 2016 11:45:19 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <OF8A05C473.728F90D9-ON88258054.0032A1D5-88258054.0032C305@notes.na.collabserv.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Sat, 22 Oct 2016 09:45:35 -0000

ok let me clarify a couple of things and provide an easy solution that 
resolves this issue altogether.

1) Escaping: transformencode, transformdecode, and transformapply do not 
remove quotes to provide easy to understand semantics. If users want to 
match strings with different escaping policies to the same entry it's 
the user's responsibility to handle the unquoting. The nice side effect 
is that transformencode/transformapply and transformdecode are truly 
inverse operations, at least for reversible transformations like 
recoding and dummy coding.

2) Metadata frames: The schema for meta data frames is a string column 
per original column where each transformation type has its special 
serialization format. For example, for recoding, we serialize distinct 
<token><delim><code> (one entry per row). The reason why we use the 
quote-aware splitting on parsing this meta data is a best effort to 
handle cases where delim occurs inside the quoted token. A simply 
splitting on <delim> (as done in the "fix" by PR 274) would fail in this 
situation.

3) Solution: We could, however, simply flip the serialization format to 
<code><delim><token> which allows splitting on the first occurrence of 
<delim> because <code> is guaranteed not to include <delim>. Note that 
this would loose binary backwards compatibility to existing meta data 
frames though.

Regards,
Matthias


On 10/22/2016 11:14 AM, Berthold Reinwald wrote:
> Reading SystemML frames from CSV files, and splitting strings honoring
> quotes, separators, and escaping rules follows the RFC 4180
> specification (https://tools.ietf.org/html/rfc4180#page-2). Populating
> SystemML frames from CSV files is one way, but we can also bind and
> pass Spark DataFrames with string columns to SystemML frames. Today,
> we take the Spark DataFrame strings *as is* without any checking
> whether these string values e.g. contain quotes or separator symbols,
> and whether they are escaped accordingly. Our transform capabilities
> can deal with this situation but I am a little uneasy about the fact
> that depending on where the data strings in our frames come from, they
> comply with different rules. In the case of CSV files, the fields
> comply with RFC 4180, and in the case of Spark Dataframes, the strings
> are any Java/Scala string.
>
> This may or may not be an issue but I wanted to collect some thoughts on
> this topic. Things to consider are:
>
> - reading and writing a CSV file with and without
>   transformencode/transformdecode ... should it result in the same
>   input file?
>
> - through MLContext we receive a Spark Dataframe with strings, and in
>   SystemML, we write out the CSV file, and a subsequent DML script
>   wants to read the CSV file? Would you expect the CSV file to be
>   readable by SystemML? Keep in mind that the original scala/java
>   strings may not be properly escaped.
>
> Thoughts?
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinwald@us.ibm.com
>
>