jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arek Kita (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
Date Fri, 01 Sep 2017 12:32:00 GMT

    [ https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150420#comment-16150420
] 

Arek Kita edited comment on OAK-6611 at 9/1/17 12:31 PM:
---------------------------------------------------------

[~tomek.rekawek], [~amitjain]: please also have a look at logs in [^oak-upgrade-with-oak-blob-cloud.fragment.log]
where you can see such async nature between oak-upgrade vs S3 datastore.

I guess that for other more complex datastores we might have similar issues but for oak-upgrade
everything has to be synchronized some how.

I remember we had some time ago a problem with segment-tar journal synchronisation (file descriptor
leaks as well) where migration hasn't been completed successfully (background processes).
Maybe [~frm] could help here as well with some pointers but this is obviously not related
this time to any segment store directly (but the problem seems to be similar as described
in OAK-5135, OAK-5326, OAK-5436 and OAK-5438).


was (Author: arkadius):
[~tomek.rekawek], [~amitjain]: please also have a look at logs in [^oak-upgrade-with-oak-blob-cloud.fragment.log]
where you can see such async nature between oak-upgrade vs S3 datastore.

I guess that for other more complex datastores we might have similar issues but for oak-upgrade
everything has to be synchronized some how.

I remember we had some time ago a problem with segment-tar journal synchronisation (file descriptor
leaks as well) where migration hasn't been completed successfully (background processes).
Maybe [~frm] could help here as well with some pointers but this is obviously not related
this time to any segment store.

> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
> -----------------------------------------------------------------------------------
>
>                 Key: OAK-6611
>                 URL: https://issues.apache.org/jira/browse/OAK-6611
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: blob-cloud, upgrade
>    Affects Versions: 1.8, 1.7.7
>            Reporter: Arek Kita
>            Assignee: Amit Jain
>            Priority: Critical
>         Attachments: oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore format/upload process
the migration ends up way quicker than S3 datastore is being migrated. This leads to a huge
number of exceptions shown due to *non synchronised* nature of *oak-upgrade* migration process
vs async S3 datastore background processes. 
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than migration
for a few cases)
> * pause it when migration is completed in a clean way (so some binaries aren't uploaded
and moved to a new datastore format) -- not sure if such mixed state is ok at all
> WDYT? 
> Please also note that this happens only when {{\-\-src-s3config \-\-src-s3datastore}}
options are specified during migration which in many cases is true (this would be the same
for the destination DataStore options). 
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is not included)
in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR  o.a.j.o.p.b.UploadStagingCache: Error adding file to backend
> java.lang.IllegalStateException: Connection pool shut down
> 	at org.apache.http.util.Asserts.check(Asserts.java:34)
> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> 	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> 	at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> 	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> 	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> 	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> 	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
> 	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
> 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
> 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
> 	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1161)
> 	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1136)
> 	at org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.write(S3Backend.java:201)
> 	at org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore$2.write(AbstractSharedCachingDataStore.java:170)
> 	at org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:341)
> 	at org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:336)
> 	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> 	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> 	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message