Return-Path: <user-return-99-archive-asf-public=cust-asf.ponee.io@gobblin.incubator.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 3EE15200D39
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Sat, 11 Nov 2017 15:09:54 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 3D8E7160C03; Sat, 11 Nov 2017 14:09:54 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 8D6C6160BF1
	for <archive-asf-public@cust-asf.ponee.io>; Sat, 11 Nov 2017 15:09:52 +0100 (CET)
Received: (qmail 40855 invoked by uid 500); 11 Nov 2017 14:09:51 -0000
Mailing-List: contact user-help@gobblin.incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@gobblin.incubator.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@gobblin.incubator.apache.org>
List-Post: <mailto:user@gobblin.incubator.apache.org>
List-Id: <user.gobblin.incubator.apache.org>
Reply-To: user@gobblin.incubator.apache.org
Delivered-To: mailing list user@gobblin.incubator.apache.org
Received: (qmail 40845 invoked by uid 99); 11 Nov 2017 14:09:51 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Nov 2017 14:09:51 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D9459191FD6
	for <user@gobblin.apache.org>; Sat, 11 Nov 2017 14:09:50 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 2.129
X-Spam-Level: **
X-Spam-Status: No, score=2.129 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_NONE=-0.0001,
	RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001]
	autolearn=disabled
Authentication-Results: spamd3-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=gmail.com
Received: from mx1-lw-us.apache.org ([10.40.0.8])
	by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024)
	with ESMTP id uSFWhpBioCdE for <user@gobblin.apache.org>;
	Sat, 11 Nov 2017 14:09:45 +0000 (UTC)
Received: from mail-ua0-f179.google.com (mail-ua0-f179.google.com [209.85.217.179])
	by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E83665FD72
	for <user@gobblin.incubator.apache.org>; Sat, 11 Nov 2017 14:09:44 +0000 (UTC)
Received: by mail-ua0-f179.google.com with SMTP id 21so5875467uas.13
        for <user@gobblin.incubator.apache.org>; Sat, 11 Nov 2017 06:09:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to;
        bh=WbnLbtPPd6XpkysBr4XGzobnWdAJzaZmodwngxc240s=;
        b=D+J5v9KYdKMnO3XnKB3LIOtcYO2z1WOdrMoROFsurkvuYXAYe9WtMBkgIZkEk1jnYg
         b1Cw9DnbN6lcJsPHc8WDi3W3FV4pjUEDvgwus2VswxjxZu+C34+y9qcVO5q7fVmywLYU
         i8yidtT5iyKh7/Dy7iCc8jsjjYbYJvN3fUYRMEatPvwUV3Oow/eqbz5Lr3aGnUNp98T7
         2hHcuUfDGYwnI3uCQWo+2UxQ/nbGYqKnDwfHyorRyCSZdLQ8DNw5LjrCKUlceEMc6MBe
         hhTYOiONWw/LURZtJ8rOkykQvxSY0Tzzsx++5UfQRBzb9FoAt6CXoPrIokn+YfTp97lT
         aztw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:in-reply-to:references:from:date
         :message-id:subject:to;
        bh=WbnLbtPPd6XpkysBr4XGzobnWdAJzaZmodwngxc240s=;
        b=sZYfrl+N+zrUwnKzVmahgsznvlqffbC68T2AjojCtXLlIIygJHyllwDuxe7odybLDw
         emKeT94esd+yhAsmdcaPj+9xn4JAQqTBlysWXDfet1NZ1HJj9z1hPzYtmx83H/rEFwk1
         8556y01+psl3YhbbJsFMn99QAOyOvIzoYeWNv9Z/AW7LubxmPzC8iSVJU6MxcAwTnT3Q
         5OAusAzcsvmiHdfqtoMjBwzvsVZYRhaJd3BqVN7EjwHdMIbF0631IXP6wGBbS/zKoUqU
         kovsb/O0Ilnytlru3NV+qsdEqClpqyIiW6x/7Jn78GbCntN7FVBuhKxvEnWd/oNDGhdo
         3Dzg==
X-Gm-Message-State: AJaThX7bHUuv+LZ/uKBiQ84n8veIj9d0FRuZ2O2BsXJvszT3qWuEQYua
	cHNFljBhG5sGbBpm4cnKc4OTEsHnvnP5CIB6Oz4=
X-Google-Smtp-Source: AGs4zMadw0cOSgSrKAtMu4pnL8Lbvat9E4uTOfJeDiAsaJCDc70dcuZ+OoFvrGFdp7k1tLiT19Mw0xcHQHT+V4TIkIg=
X-Received: by 10.159.55.82 with SMTP id a18mr3304427uae.130.1510409384035;
 Sat, 11 Nov 2017 06:09:44 -0800 (PST)
MIME-Version: 1.0
Received: by 10.31.93.194 with HTTP; Sat, 11 Nov 2017 06:09:43 -0800 (PST)
Received: by 10.31.93.194 with HTTP; Sat, 11 Nov 2017 06:09:43 -0800 (PST)
In-Reply-To: <CAPaCpY-5eK5tMGrCHLp4jzTRw-izV3JDEWtDczdPkCS2zEvdzA@mail.gmail.com>
References: <CAPaCpY-5eK5tMGrCHLp4jzTRw-izV3JDEWtDczdPkCS2zEvdzA@mail.gmail.com>
From: Mohan <mohandoss.tr@gmail.com>
Date: Sat, 11 Nov 2017 19:39:43 +0530
Message-ID: <CAAkdt_MHf6=KyEPRuwKpB61uW2VSaGOpgVctc=4Mi+FEJ1V1VQ@mail.gmail.com>
Subject: Re: Corrupted state file when Jobs are being run in parallel.
To: user@gobblin.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a1145d602cd0bed055db596d9"
archived-at: Sat, 11 Nov 2017 14:09:54 -0000

--001a1145d602cd0bed055db596d9
Content-Type: text/plain; charset="UTF-8"

Could you please tell me how to perform  stress test on Gobblin
And please let me know how to enable debug option in log file. Thanks

On Nov 11, 2017 7:29 PM, "Vicky Kak" <vicky.kak@gmail.com> wrote:

> Hi Guys,
>
> I have been running the stress tests and am seeing the following errors
>
> Error 1
> ************************************************************
> *********************************************************************
> 017-11-11 11:20:56 UTC INFO  [pool-11-thread-421] org.apache.hadoop.fs.FSInputChecker
> 284 - Found checksum error: b[0, 512]=53455106196f72672e617061636865
> 2e6861646f6f702e696f2e5465787425676f62626c696e2e72756e74696d
> 652e4a6f6253746174652444617461736574537461746501012a6f72672e
> 6170616368652e6861646f6f702e696f2e636f6d70726573732e44656661
> 756c74436f6465630000000044e218b9e6aad3f1aa97f2210fb5c7f0ffff
> ffff44e218b9e6aad3f1aa97f2210fb5c7f00109789c6304000002000209
> 789c630000000100010b789cebb3d50200025100f68e0ab4789ced5b7b73
> dbc611971c3b8d5ff233b6d324ad861337e9d804013e4451692643d1924c
> 51a26489962da71ece013890270238f870904479fc1592ffdb4fd1e9f4b3
> 64a6dfa3ff770f0fbe244384eca6c998d2f081bbddc5deede2f6777bcbcf
> 974da275d8266ae1a543ce90c629dbf44cb3a9e48ae93daa3663fa9b4a41
> 9195ac5c2a147373a5a9a9e9e6df4adf3c0a3ee7ff39e5ff5da7172b1beb
> ebd54663097aa6a6c52b995c9923b7333e79530e15f93954e41f8122d7fe
> 0d6f8f6fe0805bb291855d0769f8aee14b961c102da17d4625576b630b5d
> 7ae561d6954c64b7ce75d81742098639b4f036c348772835250b1dbae408
> 4f672fba1c1a2d89e85f159031870d944fe7545d4be70b46313d5f9071ba
> 24e772459445322aea331479bc2df96f1e33bf6d73eeb80b998c4d74506c
> 1f3349a356c627ca4a72467c520637fa9e
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> file:/home/Installable/gobblin-dist/working-dir/state-store/
> FlickrPageExtractorPull_137/current.jst at 0 exp: 36820587 got: 91149211
>         at org.apache.hadoop.fs.FSInputChecker.verifySums(
> FSInputChecker.java:322)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(
> FSInputChecker.java:278)
>         at org.apache.hadoop.fs.FSInputChecker.fill(
> FSInputChecker.java:213)
>         at org.apache.hadoop.fs.FSInputChecker.read1(
> FSInputChecker.java:231)
>         at org.apache.hadoop.fs.FSInputChecker.read(
> FSInputChecker.java:195)
>         at java.io.DataInputStream.readFully(DataInputStream.java:195)
>         at java.io.DataInputStream.readFully(DataInputStream.java:169)
>         at org.apache.hadoop.io.SequenceFile$Reader.init(
> SequenceFile.java:1845)
>         at org.apache.hadoop.io.SequenceFile$Reader.
> initialize(SequenceFile.java:1810)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(
> SequenceFile.java:1759)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(
> SequenceFile.java:1773)
>         at gobblin.runtime.FsDatasetStateStore.getAll(
> FsDatasetStateStore.java:119)
>         at gobblin.runtime.FsDatasetStateStore.
> getLatestDatasetStatesByUrns(FsDatasetStateStore.java:173)
>         at gobblin.runtime.JobContext.<init>(JobContext.java:136)
>         at gobblin.runtime.AbstractJobLauncher.<init>(
> AbstractJobLauncher.java:131)
>         at gobblin.runtime.local.LocalJobLauncher.<init>(
> LocalJobLauncher.java:62)
>         at gobblin.runtime.JobLauncherFactory.newJobLauncher(
> JobLauncherFactory.java:80)
>         at gobblin.runtime.JobLauncherFactory.newJobLauncher(
> JobLauncherFactory.java:59)
>         at com.bph.JobLauncherResource.search(JobLauncherResource.
> java:107)
>         at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at com.linkedin.restli.internal.server.RestLiMethodInvoker.
> doInvoke(RestLiMethodInvoker.java:186)
>         at com.linkedin.restli.internal.server.RestLiMethodInvoker.
> invoke(RestLiMethodInvoker.java:141)
>         at com.linkedin.restli.server.RestLiServer.handleResourceRequest(
> RestLiServer.java:286)
>         at com.linkedin.restli.server.RestLiServer.doHandleRequest(
> RestLiServer.java:167)
>         at com.linkedin.restli.server.BaseRestServer.handleRequest(
> BaseRestServer.java:56)
>         at com.linkedin.restli.server.DelegatingTransportDispatcher.
> handleRestRequest(DelegatingTransportDispatcher.java:56)
>         at com.linkedin.r2.filter.transport.DispatcherRequestFilter.
> onRestRequest(DispatcherRequestFilter.java:81)
>         at com.linkedin.r2.filter.FilterChainImpl$
> RestRequestFilterAdapter.onRequest(FilterChainImpl.java:328)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:55)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.compression.ServerCompressionFilter.
> onRestRequest(ServerCompressionFilter.java:126)
>         at com.linkedin.r2.filter.FilterChainImpl$
> RestRequestFilterAdapter.onRequest(FilterChainImpl.java:328)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:55)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:59)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.FilterChainImpl.onRestRequest(
> FilterChainImpl.java:103)
>         at com.linkedin.r2.filter.transport.FilterChainDispatcher.
> handleRestRequest(FilterChainDispatcher.java:74)
>         at com.linkedin.r2.transport.http.server.HttpDispatcher.
> handleRequest(HttpDispatcher.java:95)
>         at com.linkedin.r2.transport.http.server.HttpDispatcher.
> handleRequest(HttpDispatcher.java:62)
>         at com.linkedin.r2.transport.http.server.HttpNettyServer$
> Handler.messageReceived(HttpNettyServer.java:171)
>         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.
> handleUpstream(SimpleChannelUpstreamHandler.java:80)
>         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
> DefaultChannelPipeline.java:545)
>         at org.jboss.netty.channel.DefaultChannelPipeline$
> DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
>         at org.jboss.netty.handler.execution.ChannelEventRunnable.run(
> ChannelEventRunnable.java:69)
>         at org.jboss.netty.handler.execution.
> OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(
> OrderedMemoryAwareThreadPoolExecutor.java:316)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-11-11 11:20:56 UTC ERROR [pool-11-thread-421]
> com.bph.JobLauncherResource  110 -  Job Id fk_137 failed while searching
> key beryls Failed to create job launcher: org.apache.hadoop.fs.ChecksumException:
> Checksum error: file:/home/Installable/gobblin-dist/working-dir/
> state-store/FlickrPageExtractorPull_137/current.jst at 0 exp: 36820587
> got: 91149211
> 2017-11-11 11:20:56 UTC INFO  [pool-11-thread-402]
> gobblin.util.ExecutorsUtils  125 - Attempting to shutdown ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@6bce96a5[Shutting down, pool size
> = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2017-11-11 11:20:56 UTC INFO  [pool-11-thread-402]
> gobblin.util.ExecutorsUtils  144 - Successfully shutdown ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@6bce96a5[Terminated, pool size =
> 0, active threads = 0, queued tasks = 0, completed tasks = 1]
>
> ************************************************************
> *********************************************************************
>
> Error 2:
> ************************************************************
> *********************************************************************
>
> 2017-11-10 10:24:10 UTC WARN  [pool-11-thread-13] org.apache.hadoop.fs.
> ChecksumFileSystem$ChecksumFSInputChecker  154 - Problem opening checksum
> file: file:/home/Installable/gobblin-dist/working-dir/state-store/
> YoutubePageExtractorPull_138/current.jst.  Ignoring exception:
> java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>         at java.io.DataInputStream.readFully(DataInputStream.java:169)
>         at org.apache.hadoop.fs.ChecksumFileSystem$
> ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:146)
>         at org.apache.hadoop.fs.ChecksumFileSystem.open(
> ChecksumFileSystem.java:339)
>         at org.apache.hadoop.io.SequenceFile$Reader.openFile(
> SequenceFile.java:1832)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(
> SequenceFile.java:1752)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(
> SequenceFile.java:1773)
>         at gobblin.runtime.FsDatasetStateStore.getAll(
> FsDatasetStateStore.java:119)
>         at gobblin.runtime.FsDatasetStateStore.
> getLatestDatasetStatesByUrns(FsDatasetStateStore.java:173)
>         at gobblin.runtime.JobContext.<init>(JobContext.java:136)
>         at gobblin.runtime.AbstractJobLauncher.<init>(
> AbstractJobLauncher.java:131)
>         at gobblin.runtime.local.LocalJobLauncher.<init>(
> LocalJobLauncher.java:62)
>         at gobblin.runtime.JobLauncherFactory.newJobLauncher(
> JobLauncherFactory.java:80)
>         at gobblin.runtime.JobLauncherFactory.newJobLauncher(
> JobLauncherFactory.java:59)
>         at com.bph.JobLauncherResource.search(JobLauncherResource.
> java:107)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at com.linkedin.restli.internal.server.RestLiMethodInvoker.
> doInvoke(RestLiMethodInvoker.java:186)
>         at com.linkedin.restli.internal.server.RestLiMethodInvoker.
> invoke(RestLiMethodInvoker.java:141)
>         at com.linkedin.restli.server.RestLiServer.handleResourceRequest(
> RestLiServer.java:286)
>         at com.linkedin.restli.server.RestLiServer.doHandleRequest(
> RestLiServer.java:167)
>         at com.linkedin.restli.server.BaseRestServer.handleRequest(
> BaseRestServer.java:56)
>         at com.linkedin.restli.server.DelegatingTransportDispatcher.
> handleRestRequest(DelegatingTransportDispatcher.java:56)
>         at com.linkedin.r2.filter.transport.DispatcherRequestFilter.
> onRestRequest(DispatcherRequestFilter.java:81)
>         at com.linkedin.r2.filter.FilterChainImpl$
> RestRequestFilterAdapter.onRequest(FilterChainImpl.java:328)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:55)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.compression.ServerCompressionFilter.
> onRestRequest(ServerCompressionFilter.java:126)
>         at com.linkedin.r2.filter.FilterChainImpl$
> RestRequestFilterAdapter.onRequest(FilterChainImpl.java:328)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:55)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.ComposedFilter.onRequest(
> ComposedFilter.java:59)
>         at com.linkedin.r2.filter.FilterChainIterator.onRequest(
> FilterChainIterator.java:50)
>         at com.linkedin.r2.filter.FilterChainImpl.onRestRequest(
> FilterChainImpl.java:103)
>         at com.linkedin.r2.filter.transport.FilterChainDispatcher.
> handleRestRequest(FilterChainDispatcher.java:74)
>         at com.linkedin.r2.transport.http.server.HttpDispatcher.
> handleRequest(HttpDispatcher.java:95)
>         at com.linkedin.r2.transport.http.server.HttpDispatcher.
> handleRequest(HttpDispatcher.java:62)
>         at com.linkedin.r2.transport.http.server.HttpNettyServer$
> Handler.messageReceived(HttpNettyServer.java:171)
>         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.
> handleUpstream(SimpleChannelUpstreamHandler.java:80)
>         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
> DefaultChannelPipeline.java:545)
>         at org.jboss.netty.channel.DefaultChannelPipeline$
> DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
>         at org.jboss.netty.handler.execution.ChannelEventRunnable.run(
> ChannelEventRunnable.java:69)
>         at org.jboss.netty.handler.execution.
> OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(
> OrderedMemoryAwareThreadPoolExecutor.java:316)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-11-10 10:24:11 UTC ERROR [pool-11-thread-13]
> com.bph.JobLauncherResource  110 -  Job Id yt_138 failed while searching
> key ostfold Failed to create job launcher: java.io.EOFException
>
> ************************************************************
> *********************************************************************
>
> Error 3
> ************************************************************
> *********************************************************************
> 2017-11-10 13:38:49 UTC ERROR [Commit-thread-0] gobblin.runtime.SafeDatasetCommit
> 118 - Failed to persist dataset state for dataset  of job
> job_TwitterPageExtractorPull_135_1510321111647
> java.io.FileNotFoundException: Failed to rename /home/Installable/gobblin-
> dist/working-dir/state-store/TwitterPageExtractorPull_135/_tmp_/current.jst
> to /home/Installable/gobblin-dist/working-dir/state-store/
> TwitterPageExtractorPull_135/current.jst: src not found
> at gobblin.util.HadoopUtils.renamePath(HadoopUtils.java:173)
> at gobblin.util.HadoopUtils.renamePath(HadoopUtils.java:164)
> at gobblin.util.HadoopUtils.copyFile(HadoopUtils.java:333)
> at gobblin.metastore.FsStateStore.createAlias(FsStateStore.java:283)
> at gobblin.runtime.FsDatasetStateStore.persistDatasetState(
> FsDatasetStateStore.java:221)
> at gobblin.runtime.SafeDatasetCommit.persistDatasetState(
> SafeDatasetCommit.java:255)
> at gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:115)
> at gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:43)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ************************************************************
> *********************************************************************
>
> There errors are seeing during the stress tests for the same Jobs. For our
> use case we can't afford the jobs to fail due to system issue like above. I
> did some basic investigation and could find the issue could be happening to
> to non atomic operations on the state file which is of extension .jst. It
> seems we could disable the statestore, I looked at the following code in
> gobblin.runtime.JobContext::createStateStore
> ************************************************************
> *********************************************************************
> if (jobProps.containsKey(ConfigurationKeys.STATE_STORE_ENABLED) &&
>         !Boolean.parseBoolean(jobProps.getProperty(
> ConfigurationKeys.STATE_STORE_ENABLED))) {
>       return new NoopDatasetStateStore(stateStoreFs, stateStoreRootDir);
>     } else {
>       return new FsDatasetStateStore(stateStoreFs, stateStoreRootDir);
>     }
> ************************************************************
> *********************************************************************
>
> It seems that by disabling the statestore we may get over this issue, but
> for our case the source implementation is passing the information to the
> Extractor via the WorkUnitStoreState.
>
>
> We don't want the Job Retry features and hence did disable it as explained
> here
> https://gobblin.readthedocs.io/en/latest/user-guide/
> Configuration-Properties-Glossary/#retry-properties
>
> I was expecting the disabling happening by setting the follwing only
> workunit.retry.enabled=false
> we have to set this also
> task.maxretries=0
> As we don't rely on retries would it not be good to have a flag what will
> ignore the the following calls when we have have
> workunit.retry.enabled=false
>
> 1) Reading the initial value from the store
> 2) Commit the final state to the store.
>
> As mentioned about we can't disable the state store as we are generating
> some data in the Source implementation and passed to the corresponding
> Extractor implementation via State.
>
> I do anticipate of having these issues in GAAS too.
>
> I will be working to fix this issue as this is a critical issue for us.
>
> Thanks,
> Vicky
>

--001a1145d602cd0bed055db596d9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto">Could you please tell me how to perform=C2=A0 stress test=
 on Gobblin<div dir=3D"auto">And please let me know how to enable debug opt=
ion in log file. Thanks</div></div><div class=3D"gmail_extra"><br><div clas=
s=3D"gmail_quote">On Nov 11, 2017 7:29 PM, &quot;Vicky Kak&quot; &lt;<a hre=
f=3D"mailto:vicky.kak@gmail.com">vicky.kak@gmail.com</a>&gt; wrote:<br type=
=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Hi Gu=
ys,</div><div><br></div>I have been running the stress tests and am seeing =
the following errors<div><br></div><div>Error 1<br><div>*******************=
***********<wbr>******************************<wbr>************************=
******<wbr>******************************<wbr>*********</div><div><div>017-=
11-11 11:20:56 UTC INFO=C2=A0 [pool-11-thread-421] org.apache.hadoop.fs.<wb=
r>FSInputChecker=C2=A0 284 - Found checksum error: b[0, 512]=3D<wbr>5345510=
6196f72672e617061636865<wbr>2e6861646f6f702e696f2e54657874<wbr>25676f62626c=
696e2e72756e74696d<wbr>652e4a6f6253746174652444617461<wbr>73657453746174650=
1012a6f72672e<wbr>6170616368652e6861646f6f702e69<wbr>6f2e636f6d70726573732e=
44656661<wbr>756c74436f6465630000000044e218<wbr>b9e6aad3f1aa97f2210fb5c7f0f=
fff<wbr>ffff44e218b9e6aad3f1aa97f2210f<wbr>b5c7f00109789c6304000002000209<w=
br>789c630000000100010b789cebb3d5<wbr>0200025100f68e0ab4789ced5b7b73<wbr>db=
c611971c3b8d5ff233b6d324ad86<wbr>1337e9d804013e4451692643d1924c<wbr>51a2648=
9962da71ece013890270238<wbr>f870904479fc1592ffdb4fd1e9f4b3<wbr>64a6dfa3ff77=
0f0fbe244384eca6c9<wbr>98d2f081bbddc5deede2f6777bcbcf<wbr>974da275d8266ae1a=
543ce90c629db<wbr>f44cb3a9e48ae93daa3663fa9b4a41<wbr>9195ac5c2a147373a5a9a9=
e9e6df4a<wbr>df3c0a3ee7ff39e5ff5da7172b1beb<wbr>ebd54663097aa6a6c52b995c992=
3b7<wbr>333e79530e15f93954e41f8122d7fe<wbr>0d6f8f6fe0805bb291855d0769f8ae<w=
br>e14b961c102da17d4625576b630b5d<wbr>7ae561d6954c64b7ce75d817420986<wbr>39=
b4f036c348772835250b1dbae408<wbr>4f672fba1c1a2d89e85f159031870d<wbr>944fe75=
45d4be70b46313d5f9071ba<wbr>24e772459445322aea331479bc2df9<wbr>6f1e33bf6d73=
eeb80b998c4d74506c<wbr>1f3349a356c627ca4a72467c520637<wbr>fa9e</div><div>or=
g.apache.hadoop.fs.<wbr>ChecksumException: Checksum error: file:/home/Insta=
llable/<wbr>gobblin-dist/working-dir/<wbr>state-store/<wbr>FlickrPageExtrac=
torPull_137/<wbr>current.jst at 0 exp: 36820587 got: 91149211</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.fs.<wbr>FSInputChecker.verify=
Sums(<wbr>FSInputChecker.java:322)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=
 org.apache.hadoop.fs.<wbr>FSInputChecker.<wbr>readChecksumChunk(<wbr>FSInp=
utChecker.java:278)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.had=
oop.fs.<wbr>FSInputChecker.fill(<wbr>FSInputChecker.java:213)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.fs.<wbr>FSInputChecker.read1(=
<wbr>FSInputChecker.java:231)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.=
apache.hadoop.fs.<wbr>FSInputChecker.read(<wbr>FSInputChecker.java:195)</di=
v><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.DataInputStream.<wbr>readFull=
y(DataInputStream.<wbr>java:195)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at j=
ava.io.DataInputStream.<wbr>readFully(DataInputStream.<wbr>java:169)</div><=
div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at <a href=3D"http://org.apache.hadoop.io">=
org.apache.hadoop.io</a>.<wbr>SequenceFile$Reader.init(<wbr>SequenceFile.ja=
va:1845)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at <a href=3D"http://org.apa=
che.hadoop.io">org.apache.hadoop.io</a>.<wbr>SequenceFile$Reader.<wbr>initi=
alize(SequenceFile.java:<wbr>1810)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=
 <a href=3D"http://org.apache.hadoop.io">org.apache.hadoop.io</a>.<wbr>Sequ=
enceFile$Reader.&lt;init&gt;(<wbr>SequenceFile.java:1759)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at <a href=3D"http://org.apache.hadoop.io">org.apache.=
hadoop.io</a>.<wbr>SequenceFile$Reader.&lt;init&gt;(<wbr>SequenceFile.java:=
1773)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.<wbr>FsDatas=
etStateStore.getAll(<wbr>FsDatasetStateStore.java:119)</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 at gobblin.runtime.<wbr>FsDatasetStateStore.<wbr>getLates=
tDatasetStatesByUrns(<wbr>FsDatasetStateStore.java:173)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.JobContext.&lt;<wbr>init&gt;(JobCon=
text.java:136)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.<wb=
r>AbstractJobLauncher.&lt;init&gt;(<wbr>AbstractJobLauncher.java:131)</div>=
<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.local.<wbr>LocalJobLaun=
cher.&lt;init&gt;(<wbr>LocalJobLauncher.java:62)</div><div>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 at gobblin.runtime.<wbr>JobLauncherFactory.<wbr>newJobLaunche=
r(<wbr>JobLauncherFactory.java:80)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=
 gobblin.runtime.<wbr>JobLauncherFactory.<wbr>newJobLauncher(<wbr>JobLaunch=
erFactory.java:59)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.bph.JobLaun=
cherResource.<wbr>search(JobLauncherResource.<wbr>java:107)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect.<wbr>GeneratedMethodAccessor8.<wbr>=
invoke(Unknown Source)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect=
.<wbr>DelegatingMethodAccessorImpl.<wbr>invoke(<wbr>DelegatingMethodAccesso=
rImpl.<wbr>java:43)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.refl=
ect.Method.<wbr>invoke(Method.java:498)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 at com.linkedin.restli.internal.<wbr>server.RestLiMethodInvoker.<wbr>do=
Invoke(RestLiMethodInvoker.<wbr>java:186)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at com.linkedin.restli.internal.<wbr>server.RestLiMethodInvoker.<wbr=
>invoke(RestLiMethodInvoker.<wbr>java:141)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at com.linkedin.restli.server.<wbr>RestLiServer.<wbr>handleResourceR=
equest(<wbr>RestLiServer.java:286)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=
 com.linkedin.restli.server.<wbr>RestLiServer.doHandleRequest(<wbr>RestLiSe=
rver.java:167)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli=
.server.<wbr>BaseRestServer.handleRequest(<wbr>BaseRestServer.java:56)</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli.server.<wbr>Delega=
tingTransportDispatcher.<wbr>handleRestRequest(<wbr>DelegatingTransportDisp=
atcher.<wbr>java:56)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.=
r2.filter.<wbr>transport.<wbr>DispatcherRequestFilter.<wbr>onRestRequest(<w=
br>DispatcherRequestFilter.java:<wbr>81)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 at com.linkedin.r2.filter.<wbr>FilterChainImpl$<wbr>RestRequestFilterAd=
apter.<wbr>onRequest(FilterChainImpl.<wbr>java:328)</div><div>=C2=A0 =C2=A0=
 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>ComposedFilter.onRequest(<wbr=
>ComposedFilter.java:55)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linke=
din.r2.filter.<wbr>FilterChainIterator.onRequest(<wbr>FilterChainIterator.j=
ava:50)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wb=
r>compression.<wbr>ServerCompressionFilter.<wbr>onRestRequest(<wbr>ServerCo=
mpressionFilter.java:<wbr>126)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com=
.linkedin.r2.filter.<wbr>FilterChainImpl$<wbr>RestRequestFilterAdapter.<wbr=
>onRequest(FilterChainImpl.<wbr>java:328)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at com.linkedin.r2.filter.<wbr>ComposedFilter.onRequest(<wbr>Compose=
dFilter.java:55)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.f=
ilter.<wbr>FilterChainIterator.onRequest(<wbr>FilterChainIterator.java:50)<=
/div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>Compos=
edFilter.onRequest(<wbr>ComposedFilter.java:59)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>FilterChainIterator.onRequest(<wb=
r>FilterChainIterator.java:50)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com=
.linkedin.r2.filter.<wbr>FilterChainImpl.onRestRequest(<wbr>FilterChainImpl=
.java:103)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.=
<wbr>transport.<wbr>FilterChainDispatcher.<wbr>handleRestRequest(<wbr>Filte=
rChainDispatcher.java:74)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.link=
edin.r2.transport.<wbr>http.server.HttpDispatcher.<wbr>handleRequest(HttpDi=
spatcher.<wbr>java:95)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedi=
n.r2.transport.<wbr>http.server.HttpDispatcher.<wbr>handleRequest(HttpDispa=
tcher.<wbr>java:62)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r=
2.transport.<wbr>http.server.HttpNettyServer$<wbr>Handler.messageReceived(<=
wbr>HttpNettyServer.java:171)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.=
jboss.netty.channel.<wbr>SimpleChannelUpstreamHandler.<wbr>handleUpstream(<=
wbr>SimpleChannelUpstreamHandler.<wbr>java:80)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at org.jboss.netty.channel.<wbr>DefaultChannelPipeline.<wbr>send=
Upstream(<wbr>DefaultChannelPipeline.java:<wbr>545)</div><div>=C2=A0 =C2=A0=
 =C2=A0 =C2=A0 at org.jboss.netty.channel.<wbr>DefaultChannelPipeline$<wbr>=
DefaultChannelHandlerContext.<wbr>sendUpstream(<wbr>DefaultChannelPipeline.=
java:<wbr>754)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.netty.han=
dler.<wbr>execution.<wbr>ChannelEventRunnable.run(<wbr>ChannelEventRunnable=
.java:69)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.netty.handler.=
<wbr>execution.<wbr>OrderedMemoryAwareThreadPoolEx<wbr>ecutor$ChildExecutor=
.run(<wbr>OrderedMemoryAwareThreadPoolEx<wbr>ecutor.java:316)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.<wbr>ThreadPoolExecutor.ru=
nWorker(<wbr>ThreadPoolExecutor.java:1142)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at java.util.concurrent.<wbr>ThreadPoolExecutor$Worker.run(<wbr>Thre=
adPoolExecutor.java:617)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang=
.Thread.run(Thread.<wbr>java:745)</div><div>2017-11-11 11:20:56 UTC ERROR [=
pool-11-thread-421] com.bph.JobLauncherResource=C2=A0 110 -=C2=A0 Job Id fk=
_137 failed while searching key beryls Failed to create job launcher: org.a=
pache.hadoop.fs.<wbr>ChecksumException: Checksum error: file:/home/Installa=
ble/<wbr>gobblin-dist/working-dir/<wbr>state-store/<wbr>FlickrPageExtractor=
Pull_137/<wbr>current.jst at 0 exp: 36820587 got: 91149211</div><div>2017-1=
1-11 11:20:56 UTC INFO=C2=A0 [pool-11-thread-402] gobblin.util.ExecutorsUti=
ls=C2=A0 125 - Attempting to shutdown ExecutorService: java.util.concurrent=
.<wbr>ThreadPoolExecutor@6bce96a5[<wbr>Shutting down, pool size =3D 1, acti=
ve threads =3D 0, queued tasks =3D 0, completed tasks =3D 1]</div><div>2017=
-11-11 11:20:56 UTC INFO=C2=A0 [pool-11-thread-402] gobblin.util.ExecutorsU=
tils=C2=A0 144 - Successfully shutdown ExecutorService: java.util.concurren=
t.<wbr>ThreadPoolExecutor@6bce96a5[<wbr>Terminated, pool size =3D 0, active=
 threads =3D 0, queued tasks =3D 0, completed tasks =3D 1]</div></div><div>=
<div><br></div><div>******************************<wbr>********************=
**********<wbr>******************************<wbr>*************************=
*****<wbr>*********</div></div><div><br></div></div><div>Error 2:</div><div=
><div>******************************<wbr>******************************<wbr=
>******************************<wbr>******************************<wbr>****=
*****</div></div><div><br></div><div><div>2017-11-10 10:24:10 UTC WARN=C2=
=A0 [pool-11-thread-13] org.apache.hadoop.fs.<wbr>ChecksumFileSystem$<wbr>C=
hecksumFSInputChecker=C2=A0 154 - Problem opening checksum file: file:/home=
/Installable/<wbr>gobblin-dist/working-dir/<wbr>state-store/<wbr>YoutubePag=
eExtractorPull_138/<wbr>current.jst.=C2=A0 Ignoring exception:</div><div>ja=
va.io.EOFException</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.DataInp=
utStream.<wbr>readFully(DataInputStream.<wbr>java:197)</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 at java.io.DataInputStream.<wbr>readFully(DataInputStream=
.<wbr>java:169)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.=
fs.<wbr>ChecksumFileSystem$<wbr>ChecksumFSInputChecker.&lt;init&gt;(<wbr>Ch=
ecksumFileSystem.java:146)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apa=
che.hadoop.fs.<wbr>ChecksumFileSystem.open(<wbr>ChecksumFileSystem.java:339=
)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at <a href=3D"http://org.apache.had=
oop.io">org.apache.hadoop.io</a>.<wbr>SequenceFile$Reader.openFile(<wbr>Seq=
uenceFile.java:1832)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at <a href=3D"ht=
tp://org.apache.hadoop.io">org.apache.hadoop.io</a>.<wbr>SequenceFile$Reade=
r.&lt;init&gt;(<wbr>SequenceFile.java:1752)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at <a href=3D"http://org.apache.hadoop.io">org.apache.hadoop.io</a>.=
<wbr>SequenceFile$Reader.&lt;init&gt;(<wbr>SequenceFile.java:1773)</div><di=
v>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.<wbr>FsDatasetStateStore.g=
etAll(<wbr>FsDatasetStateStore.java:119)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 at gobblin.runtime.<wbr>FsDatasetStateStore.<wbr>getLatestDatasetStates=
ByUrns(<wbr>FsDatasetStateStore.java:173)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at gobblin.runtime.JobContext.&lt;<wbr>init&gt;(JobContext.java:136)=
</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.<wbr>AbstractJobL=
auncher.&lt;init&gt;(<wbr>AbstractJobLauncher.java:131)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at gobblin.runtime.local.<wbr>LocalJobLauncher.&lt;ini=
t&gt;(<wbr>LocalJobLauncher.java:62)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
at gobblin.runtime.<wbr>JobLauncherFactory.<wbr>newJobLauncher(<wbr>JobLaun=
cherFactory.java:80)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at gobblin.runti=
me.<wbr>JobLauncherFactory.<wbr>newJobLauncher(<wbr>JobLauncherFactory.java=
:59)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.bph.JobLauncherResource.<=
wbr>search(JobLauncherResource.<wbr>java:107)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at sun.reflect.<wbr>NativeMethodAccessorImpl.<wbr>invoke0(Native=
 Method)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect.<wbr>NativeMe=
thodAccessorImpl.<wbr>invoke(<wbr>NativeMethodAccessorImpl.java:<wbr>62)</d=
iv><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect.<wbr>DelegatingMethodAcc=
essorImpl.<wbr>invoke(<wbr>DelegatingMethodAccessorImpl.<wbr>java:43)</div>=
<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.reflect.Method.<wbr>invoke(Me=
thod.java:498)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli=
.internal.<wbr>server.RestLiMethodInvoker.<wbr>doInvoke(RestLiMethodInvoker=
.<wbr>java:186)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restl=
i.internal.<wbr>server.RestLiMethodInvoker.<wbr>invoke(RestLiMethodInvoker.=
<wbr>java:141)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli=
.server.<wbr>RestLiServer.<wbr>handleResourceRequest(<wbr>RestLiServer.java=
:286)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli.server.<=
wbr>RestLiServer.doHandleRequest(<wbr>RestLiServer.java:167)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.restli.server.<wbr>BaseRestServer.=
handleRequest(<wbr>BaseRestServer.java:56)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at com.linkedin.restli.server.<wbr>DelegatingTransportDispatcher.<wb=
r>handleRestRequest(<wbr>DelegatingTransportDispatcher.<wbr>java:56)</div><=
div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>transport.<w=
br>DispatcherRequestFilter.<wbr>onRestRequest(<wbr>DispatcherRequestFilter.=
java:<wbr>81)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filt=
er.<wbr>FilterChainImpl$<wbr>RestRequestFilterAdapter.<wbr>onRequest(Filter=
ChainImpl.<wbr>java:328)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linke=
din.r2.filter.<wbr>ComposedFilter.onRequest(<wbr>ComposedFilter.java:55)</d=
iv><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>FilterCh=
ainIterator.onRequest(<wbr>FilterChainIterator.java:50)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>compression.<wbr>Server=
CompressionFilter.<wbr>onRestRequest(<wbr>ServerCompressionFilter.java:<wbr=
>126)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>=
FilterChainImpl$<wbr>RestRequestFilterAdapter.<wbr>onRequest(FilterChainImp=
l.<wbr>java:328)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.f=
ilter.<wbr>ComposedFilter.onRequest(<wbr>ComposedFilter.java:55)</div><div>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>FilterChainItera=
tor.onRequest(<wbr>FilterChainIterator.java:50)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>ComposedFilter.onRequest(<wbr>Com=
posedFilter.java:59)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.=
r2.filter.<wbr>FilterChainIterator.onRequest(<wbr>FilterChainIterator.java:=
50)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>Fi=
lterChainImpl.onRestRequest(<wbr>FilterChainImpl.java:103)</div><div>=C2=A0=
 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.filter.<wbr>transport.<wbr>FilterC=
hainDispatcher.<wbr>handleRestRequest(<wbr>FilterChainDispatcher.java:74)</=
div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.transport.<wbr>http=
.server.HttpDispatcher.<wbr>handleRequest(HttpDispatcher.<wbr>java:95)</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.transport.<wbr>http.se=
rver.HttpDispatcher.<wbr>handleRequest(HttpDispatcher.<wbr>java:62)</div><d=
iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.linkedin.r2.transport.<wbr>http.serve=
r.HttpNettyServer$<wbr>Handler.messageReceived(<wbr>HttpNettyServer.java:17=
1)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.netty.channel.<wbr>Si=
mpleChannelUpstreamHandler.<wbr>handleUpstream(<wbr>SimpleChannelUpstreamHa=
ndler.<wbr>java:80)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.nett=
y.channel.<wbr>DefaultChannelPipeline.<wbr>sendUpstream(<wbr>DefaultChannel=
Pipeline.java:<wbr>545)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.=
netty.channel.<wbr>DefaultChannelPipeline$<wbr>DefaultChannelHandlerContext=
.<wbr>sendUpstream(<wbr>DefaultChannelPipeline.java:<wbr>754)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at org.jboss.netty.handler.<wbr>execution.<wbr>Cha=
nnelEventRunnable.run(<wbr>ChannelEventRunnable.java:69)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 at org.jboss.netty.handler.<wbr>execution.<wbr>Ordered=
MemoryAwareThreadPoolEx<wbr>ecutor$ChildExecutor.run(<wbr>OrderedMemoryAwar=
eThreadPoolEx<wbr>ecutor.java:316)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at=
 java.util.concurrent.<wbr>ThreadPoolExecutor.runWorker(<wbr>ThreadPoolExec=
utor.java:1142)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurre=
nt.<wbr>ThreadPoolExecutor$Worker.run(<wbr>ThreadPoolExecutor.java:617)</di=
v><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.<wbr>java=
:745)</div><div>2017-11-10 10:24:11 UTC ERROR [pool-11-thread-13] com.bph.J=
obLauncherResource=C2=A0 110 -=C2=A0 Job Id yt_138 failed while searching k=
ey ostfold Failed to create job launcher: java.io.EOFException</div></div><=
div><br></div><div><div>******************************<wbr>****************=
**************<wbr>******************************<wbr>*********************=
*********<wbr>*********</div></div><div><br></div><div>Error 3</div><div><d=
iv>******************************<wbr>******************************<wbr>**=
****************************<wbr>******************************<wbr>*******=
**</div></div><div><div>2017-11-10 13:38:49 UTC ERROR [Commit-thread-0] gob=
blin.runtime.<wbr>SafeDatasetCommit=C2=A0 118 - Failed to persist dataset s=
tate for dataset=C2=A0 of job job_TwitterPageExtractorPull_<wbr>135_1510321=
111647</div><div>java.io.FileNotFoundException: Failed to rename /home/Inst=
allable/gobblin-<wbr>dist/working-dir/state-store/<wbr>TwitterPageExtractor=
Pull_135/_<wbr>tmp_/current.jst to /home/Installable/gobblin-<wbr>dist/work=
ing-dir/state-store/<wbr>TwitterPageExtractorPull_135/<wbr>current.jst: src=
 not found</div><div><span style=3D"white-space:pre-wrap">	</span>at gobbli=
n.util.HadoopUtils.<wbr>renamePath(HadoopUtils.java:<wbr>173)</div><div><sp=
an style=3D"white-space:pre-wrap">	</span>at gobblin.util.HadoopUtils.<wbr>=
renamePath(HadoopUtils.java:<wbr>164)</div><div><span style=3D"white-space:=
pre-wrap">	</span>at gobblin.util.HadoopUtils.<wbr>copyFile(HadoopUtils.jav=
a:333)</div><div><span style=3D"white-space:pre-wrap">	</span>at gobblin.me=
tastore.<wbr>FsStateStore.createAlias(<wbr>FsStateStore.java:283)</div><div=
><span style=3D"white-space:pre-wrap">	</span>at gobblin.runtime.<wbr>FsDat=
asetStateStore.<wbr>persistDatasetState(<wbr>FsDatasetStateStore.java:221)<=
/div><div><span style=3D"white-space:pre-wrap">	</span>at gobblin.runtime.<=
wbr>SafeDatasetCommit.<wbr>persistDatasetState(<wbr>SafeDatasetCommit.java:=
255)</div><div><span style=3D"white-space:pre-wrap">	</span>at gobblin.runt=
ime.<wbr>SafeDatasetCommit.call(<wbr>SafeDatasetCommit.java:115)</div><div>=
<span style=3D"white-space:pre-wrap">	</span>at gobblin.runtime.<wbr>SafeDa=
tasetCommit.call(<wbr>SafeDatasetCommit.java:43)</div><div><span style=3D"w=
hite-space:pre-wrap">	</span>at java.util.concurrent.<wbr>FutureTask.run(Fu=
tureTask.<wbr>java:266)</div><div><span style=3D"white-space:pre-wrap">	</s=
pan>at java.util.concurrent.<wbr>Executors$RunnableAdapter.<wbr>call(Execut=
ors.java:511)</div><div><span style=3D"white-space:pre-wrap">	</span>at jav=
a.util.concurrent.<wbr>FutureTask.run(FutureTask.<wbr>java:266)</div><div><=
span style=3D"white-space:pre-wrap">	</span>at java.util.concurrent.<wbr>Th=
readPoolExecutor.runWorker(<wbr>ThreadPoolExecutor.java:1142)</div><div><sp=
an style=3D"white-space:pre-wrap">	</span>at java.util.concurrent.<wbr>Thre=
adPoolExecutor$Worker.run(<wbr>ThreadPoolExecutor.java:617)</div><div><span=
 style=3D"white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.<wbr=
>java:745)</div></div><div><div>******************************<wbr>********=
**********************<wbr>******************************<wbr>*************=
*****************<wbr>*********</div></div><div><br></div><div>There errors=
 are seeing during the stress tests for the same Jobs. For our use case we =
can&#39;t afford the jobs to fail due to system issue like above. I did som=
e basic investigation and could find the issue could be happening to to non=
 atomic operations on the state file which is of extension .jst. It seems w=
e could disable the statestore, I looked at the following code in gobblin.r=
untime.JobContext::<wbr>createStateStore</div><div>************************=
******<wbr>******************************<wbr>*****************************=
*<wbr>******************************<wbr>*********</div><div><div>if (jobPr=
ops.containsKey(<wbr>ConfigurationKeys.STATE_STORE_<wbr>ENABLED) &amp;&amp;=
</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 !Boolean.parseBoolean(<wbr>jobProps.=
getProperty(<wbr>ConfigurationKeys.STATE_STORE_<wbr>ENABLED))) {</div><div>=
=C2=A0 =C2=A0 =C2=A0 return new NoopDatasetStateStore(<wbr>stateStoreFs, st=
ateStoreRootDir);</div><div>=C2=A0 =C2=A0 } else {</div><div>=C2=A0 =C2=A0 =
=C2=A0 return new FsDatasetStateStore(<wbr>stateStoreFs, stateStoreRootDir)=
;</div><div>=C2=A0 =C2=A0 }</div></div><div><div>**************************=
****<wbr>******************************<wbr>******************************<=
wbr>******************************<wbr>*********</div><div></div></div><div=
><br></div><div>It seems that by disabling the statestore we may get over t=
his issue, but for our case the source implementation is passing the inform=
ation to the Extractor via the WorkUnitStoreState.</div><div><br></div><div=
><br></div><div>We don&#39;t want the Job Retry features and hence did disa=
ble it as explained here<br></div><div><a href=3D"https://gobblin.readthedo=
cs.io/en/latest/user-guide/Configuration-Properties-Glossary/#retry-propert=
ies" target=3D"_blank">https://gobblin.readthedocs.<wbr>io/en/latest/user-g=
uide/<wbr>Configuration-Properties-<wbr>Glossary/#retry-properties</a><br><=
/div><div><br></div><div>I was expecting the disabling happening by setting=
 the follwing only</div><h4 id=3D"m_8633475878482996135gmail-workunitretrye=
nabled" style=3D"box-sizing:border-box;margin-top:0px;font-weight:300;font-=
family:&quot;Roboto Slab&quot;,ff-tisa-web-pro,Georgia,Arial,sans-serif;fon=
t-size:16.56px;color:rgb(41,128,185);background-color:rgb(252,252,252)">wor=
kunit.retry.enabled=3Dfalse</h4><div><div>we have to set this also</div><h4=
 id=3D"m_8633475878482996135gmail-taskmaxretries" style=3D"box-sizing:borde=
r-box;margin-top:0px;font-weight:300;font-family:&quot;Roboto Slab&quot;,ff=
-tisa-web-pro,Georgia,Arial,sans-serif;font-size:16.56px;color:rgb(41,128,1=
85);background-color:rgb(252,252,252)">task.maxretries=3D0</h4></div><div>A=
s we don&#39;t rely on retries would it not be good to have a flag what wil=
l ignore the the following calls when we have have workunit.retry.enabled=
=3Dfalse</div><div><br></div><div>1) Reading the initial value from the sto=
re</div><div>2) Commit the final state to the store.</div><div><br></div><d=
iv>As mentioned about we can&#39;t disable the state store as we are genera=
ting some data in the Source implementation and passed to the corresponding=
 Extractor implementation via State.=C2=A0</div><div><br></div><div>I do an=
ticipate of having these issues in GAAS too.</div><div><br></div><div>I wil=
l be working to fix this issue as this is a critical issue for us.</div><di=
v><br></div><div>Thanks,</div><div>Vicky</div></div>
</blockquote></div></div>

--001a1145d602cd0bed055db596d9--