Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
Content-Type: multipart/signed;
 boundary="Apple-Mail=_E17D2E9C-6691-4BDE-8210-BAB17B4971B7";
 protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
Subject: Re: Problems upgrading Job
From: Rick Mangi <rick@chartbeat.com>
In-Reply-To: 
 <CAFvExu1Vw_tjgarDau_wVAmp3rKRTLM8Dvai99LjV1sH_emH5g@mail.gmail.com>
Date: Thu, 12 Nov 2015 13:10:54 -0500
Message-Id: <86A466C7-8C90-431E-9A5D-92E223C9174F@chartbeat.com>
References: <56DD6D62-1F49-49FD-8BF7-1965B98BDB79@chartbeat.com>
 <CAFvExu1Vw_tjgarDau_wVAmp3rKRTLM8Dvai99LjV1sH_emH5g@mail.gmail.com>
To: dev@samza.apache.org

--Apple-Mail=_E17D2E9C-6691-4BDE-8210-BAB17B4971B7
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Yi,

I pulled from master and built this morning.

Yes, that=E2=80=99s the output from JobRunner. I also tried setting a =
job.id to see if this was an issue migrating from an old task checkpoint =
topic but I got the same result.

Would you like me to open a jira ticket?

Thanks,

Rick


> On Nov 12, 2015, at 12:59 PM, Yi Pan <nickpan47@gmail.com> wrote:
>=20
> Hi, Rick,
>=20
> Did you get the fix in SAMZA-723 in your test? And could you confirm =
that
> the errors are from JobRunner log?
>=20
> -Yi
>=20
> On Thu, Nov 12, 2015 at 8:48 AM, Rick Mangi <rick@chartbeat.com> =
wrote:
>=20
>> Hi,
>>=20
>> I=E2=80=99m trying to migrate our samza jobs to 0.10.0 snapshot =
(built against the
>> latest). Everything works fine running locally (although I had to =
make some
>> changes to the local grid=E2=80=99s kafka since the checkpointing =
seems to require
>> replication_factor > 1) but when I deploy it against my production =
yarn
>> cluster I get these errors.
>>=20
>> [yarnmaster01] out: 2015-11-12 10:40:53 ZkClient [INFO] zookeeper =
state
>> changed (SyncConnected)
>> [yarnmaster01] out: 2015-11-12 10:40:53 ZkEventThread [INFO] =
Terminate
>> ZkClient event thread.
>> [yarnmaster01] out: 2015-11-12 10:40:53 ZooKeeper [INFO] Session:
>> 0x250233cdf57f2fa closed
>> [yarnmaster01] out: 2015-11-12 10:40:53 ClientCnxn [INFO] EventThread =
shut
>> down
>> [yarnmaster01] out: 2015-11-12 10:40:53 KafkaSystemAdmin [INFO]
>> Coordinator stream __samza_coordinator_metrics-reporter_1 already =
exists.
>> [yarnmaster01] out: 2015-11-12 10:40:53 JobRunner [INFO] Storing =
config in
>> coordinator stream.
>> [yarnmaster01] out: 2015-11-12 10:40:53 =
CoordinatorStreamSystemProducer
>> [INFO] Starting coordinator stream producer.
>> [yarnmaster01] out: 2015-11-12 10:40:53 KafkaSystemProducer [INFO]
>> Creating a new producer for system mykafka.
>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [INFO]
>> ProducerConfig values:
>> [yarnmaster01] out:     value.serializer =3D class
>> org.apache.kafka.common.serialization.ByteArraySerializer
>> [yarnmaster01] out:     key.serializer =3D class
>> org.apache.kafka.common.serialization.ByteArraySerializer
>> [yarnmaster01] out:     block.on.buffer.full =3D true
>> [yarnmaster01] out:     retry.backoff.ms =3D 100
>> [yarnmaster01] out:     buffer.memory =3D 33554432
>> [yarnmaster01] out:     batch.size =3D 16384
>> [yarnmaster01] out:     metrics.sample.window.ms =3D 30000
>> [yarnmaster01] out:     metadata.max.age.ms =3D 300000
>> [yarnmaster01] out:     receive.buffer.bytes =3D 32768
>> [yarnmaster01] out:     timeout.ms =3D 30000
>> [yarnmaster01] out:     max.in.flight.requests.per.connection =3D 1
>> [yarnmaster01] out:     bootstrap.servers =3D [
>> devstream01.chartbeat.net:9092]
>> [yarnmaster01] out:     metric.reporters =3D []
>> [yarnmaster01] out:     client.id =3D
>> samza_producer-metrics_reporter-1-1447342853273-4
>> [yarnmaster01] out:     compression.type =3D none
>> [yarnmaster01] out:     retries =3D 2147483647
>> [yarnmaster01] out:     max.request.size =3D 1048576
>> [yarnmaster01] out:     send.buffer.bytes =3D 131072
>> [yarnmaster01] out:     acks =3D 1
>> [yarnmaster01] out:     reconnect.backoff.ms =3D 10
>> [yarnmaster01] out:     linger.ms =3D 0
>> [yarnmaster01] out:     metrics.num.samples =3D 2
>> [yarnmaster01] out:     metadata.fetch.timeout.ms =3D 60000
>> [yarnmaster01] out:
>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [WARN] The
>> configuration batch.num.messages =3D null was supplied but isn't a =
known
>> config.
>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [WARN] The
>> configuration producer.type =3D null was supplied but isn't a known =
config.
>> [yarnmaster01] out: Exception in thread "main"
>> org.apache.samza.SamzaException:
>> org.apache.kafka.common.errors.TimeoutException: Failed to update =
metadata
>> after 60000 ms.
>> [yarnmaster01] out:     at
>> =
org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer.send(C=
oordinatorStreamSystemProducer.java:115)
>> [yarnmaster01] out:     at
>> =
org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer.writeC=
onfig(CoordinatorStreamSystemProducer.java:132)
>> [yarnmaster01] out:     at
>> org.apache.samza.job.JobRunner.run(JobRunner.scala:85)
>> [yarnmaster01] out:     at
>> org.apache.samza.job.JobRunner$.main(JobRunner.scala:43)
>> [yarnmaster01] out:     at
>> org.apache.samza.job.JobRunner.main(JobRunner.scala)
>> [yarnmaster01] out: Caused by:
>> org.apache.kafka.common.errors.TimeoutException: Failed to update =
metadata
>> after 60000 ms.
>> [yarnmaster01] out:
>>=20
>>=20
>> Warning: run() received nonzero return code 1 while executing
>> './bin/run-job.sh
>> =
-config-factory=3Dorg.apache.samza.config.factories.PropertiesConfigFactor=
y
>> --config-path=3Dfile://$PWD/conf/metrics_reporter.properties'!
>>=20
>>=20
>> This looks similar to https://issues.apache.org/jira/browse/SAMZA-560 =
but
>> I=E2=80=99m not using a StreamAppender in log4j.
>>=20
>> Any ideas? My first thought is that I might have to delete the =
existing
>> checkpoint topics but that would mean we can=E2=80=99t migrate =
completely until the
>> 10.0 release unless we want to run snapshot code in production.
>>=20
>> Thanks!
>>=20
>> Rick
>>=20
>>=20
>>=20


--Apple-Mail=_E17D2E9C-6691-4BDE-8210-BAB17B4971B7
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----

iQEcBAEBCAAGBQJWRNYuAAoJENgofZAtxzWQ1Y0IAIzfieEU/EiTBhPssE7YDh7A
gvQB/FeVETgTPnOsb5z+dBlXAELj09qAyseBqNnIyT/VMqWS7fXZbUeXdg/F5hGT
ed5UeLUC7k2SvKW9gXYU5c/n35+qXDXTa0s0lIWhgF5mAj1IXyG7L1I/exjqqTKs
wYxog3SkJpezBIUoOBlINnn9dI+hyCVjG3jErU1c66l9h6lqLzoYAy3htpxmZvNx
0rA8XCsEbIpCa6K6DmvXmtyjjGQqa8LbfJTmRiHIkaYjM/GZQAUF+/+QzMbb8YZ1
lxaxmCu6EJxEStSimuM7BBymISEijbNo/fWKnOhWeHdHagJTOxrkNXTDLagkbVk=
=8z4g
-----END PGP SIGNATURE-----

--Apple-Mail=_E17D2E9C-6691-4BDE-8210-BAB17B4971B7--