samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommy Becker <tobec...@tivo.com>
Subject Re: Issues with certain characters in config values
Date Tue, 09 Jun 2015 17:34:01 GMT
Hi Navina,
Yes, I'm seeing the issue on 0.8.0.  As I mentioned in the bug, I know the method of passing
the config is changing, but I still wanted to get the issue on record.  With regard to Util.envVarEscape(),
the assumption seems to be that the value is going through the shell at some point, but I
don't think that's true.  So the extra \s never get removed (theres no envVarUnescape()) resulting
in invalid JSON.

-Tommy

On 06/09/2015 01:15 PM, Navina Ramesh wrote:

Hi Tommy,

I wouldn't think Util.envVarEscape() is totally unnecessary. Are you
running into this issue with a version <= 0.9 ?
With 0.10, I think we no longer pass all the environment variables through
command line. We only pass the Coordinator system information via command
line.

While Util.envVarEscape() is not exactly smart in the escaping characters,
it is a vital minimum to have.

Thanks!
Navina


On 6/9/15, 5:22 AM, "Tommy Becker" <tobecker@tivo.com><mailto:tobecker@tivo.com>
wrote:



Ok, the issue with spaces appears to have been addressed by
https://issues.apache.org/jira/browse/SAMZA-530.  But quotes are still a
problem, and I'm becoming increasingly convinced that Util.envVarEscape()
method is unnecessary.

On 06/08/2015 03:02 PM, Tommy Becker wrote:
We have a stream job that uses an MVEL expression to filter messages to
process.  The expression is passed through the Config, and I noticed that
the expressions were often getting mangled and/or blowing up the job.
I've seen issues with at least 2 characters, quotes and spaces.  In
version 0.8.0 on YARN, the config gets passed through an environment
variable.  Prior to this, the JSON gets run through Util.envVarEscape()
which backslash escapes single and double quotes.  I'm trying to figure
out why this is necessary, because the backslashes never seem to get
removed and then the JSON fails to parse since \' is not a valid JSON
escape sequence.  I also can't seem to get values with spaces in them to
pass though at all (they get truncated at the first space).  I haven't
figured out what's going on there, but wondered if it could be a bug in
the YARN client.  I filed the following bug for this, can anyone shed any
light?

https://issues.apache.org/jira/browse/SAMZA-700

--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmit
hs.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
tobecker@tivo.com<mailto:tobecker@tivo.com><mailto:tobecker@tivo.com><mailto:tobecker@tivo.com><mailto:tobecker@tivo.com><mailto:tobecker@tivo.com><mail
to:tobecker@tivo.com><mailto:tobecker@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review, copying,
or distribution of this email (or any attachments) by others is
prohibited. If you are not the intended recipient, please contact the
sender immediately and permanently delete this email and any attachments.
No employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.


--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
tobecker@tivo.com<mailto:tobecker@tivo.com><mailto:tobecker@tivo.com><mailto:tobecker@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review, copying,
or distribution of this email (or any attachments) by others is
prohibited. If you are not the intended recipient, please contact the
sender immediately and permanently delete this email and any attachments.
No employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.





--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com>
tobecker@tivo.com<mailto:tobecker@tivo.com>

________________________________

This email and any attachments may contain confidential and privileged material for the sole
use of the intended recipient. Any review, copying, or distribution of this email (or any
attachments) by others is prohibited. If you are not the intended recipient, please contact
the sender immediately and permanently delete this email and any attachments. No employee
or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc.
by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message