Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
From: "Mendelson, Assaf" <Assaf.Mendelson@rsa.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Subject: structured streaming documentation does not match behavior
Thread-Topic: structured streaming documentation does not match behavior
Thread-Index: AdLl9R2pqQq5iK4OQZWhKK/UtENXTA==
Date: Thu, 15 Jun 2017 17:46:37 +0000
Message-ID: <4E2D62B698F0814BB1E367C9F243AB801216B5@MX302CL04.corp.emc.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_4E2D62B698F0814BB1E367C9F243AB801216B5MX302CL04corpemcc_"
MIME-Version: 1.0
archived-at: Thu, 15 Jun 2017 17:47:25 -0000

--_000_4E2D62B698F0814BB1E367C9F243AB801216B5MX302CL04corpemcc_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,
I have started to play around with structured streaming and it seems the do=
cumentation (structured streaming programming guide) does not match the act=
ual behavior I am seeing.
It says in the documentation that maxFilesPerTrigger (as well as latestFirs=
t) are options for the File sink. However, in fact, at least maxFilesPerTri=
gger does not seem to have any real effect. On the other hand, the streamin=
g source (readStream) which has no documentation for this option, does limi=
t the number of files.
This behavior actually makes more sense than the documentation as I expect =
the file reader to define how to read files rather than the sink (e.g. if I=
 would use a kafka sink or foreach sink, they should still get the same beh=
avior from the reading).

Thanks,
              Assaf.


--_000_4E2D62B698F0814BB1E367C9F243AB801216B5MX302CL04corpemcc_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hi,<o:p></o:p></p>
<p class=3D"MsoNormal">I have started to play around with structured stream=
ing and it seems the documentation (structured streaming programming guide)=
 does not match the actual behavior I am seeing.<o:p></o:p></p>
<p class=3D"MsoNormal">It says in the documentation that maxFilesPerTrigger=
 (as well as latestFirst) are options for the File sink. However, in fact, =
at least maxFilesPerTrigger does not seem to have any real effect. On the o=
ther hand, the streaming source (readStream)
 which has no documentation for this option, does limit the number of files=
. <o:p>
</o:p></p>
<p class=3D"MsoNormal">This behavior actually makes more sense than the doc=
umentation as I expect the file reader to define how to read files rather t=
han the sink (e.g. if I would use a kafka sink or foreach sink, they should=
 still get the same behavior from
 the reading).<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Thanks,<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp; Assaf.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>

--_000_4E2D62B698F0814BB1E367C9F243AB801216B5MX302CL04corpemcc_--