camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-12058) Wrong order in file idempotent store.
Date Tue, 26 Dec 2017 11:23:00 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-12058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303765#comment-16303765
] 

ASF GitHub Bot commented on CAMEL-12058:
----------------------------------------

davsclaus closed pull request #2137: CAMEL-12058 - File idempotent repository test.
URL: https://github.com/apache/camel/pull/2137
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/camel-core/src/test/java/org/apache/camel/processor/idempotent/FileIdempotentRepositoryTest.java
b/camel-core/src/test/java/org/apache/camel/processor/idempotent/FileIdempotentRepositoryTest.java
new file mode 100644
index 00000000000..7dcc0f38a87
--- /dev/null
+++ b/camel-core/src/test/java/org/apache/camel/processor/idempotent/FileIdempotentRepositoryTest.java
@@ -0,0 +1,84 @@
+package org.apache.camel.processor.idempotent;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class FileIdempotentRepositoryTest {
+
+	@Rule
+	public TemporaryFolder temporaryFolder = new TemporaryFolder();
+	
+	private FileIdempotentRepository fileIdempotentRepository = new FileIdempotentRepository();
+	private List<String> files;
+	
+	@Before
+	public void setup() throws IOException {
+		files = Arrays.asList(
+				"file1.txt.20171123",
+				"file2.txt.20171123",
+				"file1.txt.20171124",
+				"file3.txt.20171125",
+				"file2.txt.20171126",
+				"fixed.income.lamr.out.20171126",
+				"pricing.px.20171126",
+				"test.out.20171126",
+				"processing.source.lamr.out.20171126");
+		this.fileIdempotentRepository = new FileIdempotentRepository();
+	}
+	
+	@Test
+	public void testTrunkStore() throws URISyntaxException, IOException {
+		//given
+		File fileStore = temporaryFolder.newFile();
+		fileIdempotentRepository.setFileStore(fileStore);
+		fileIdempotentRepository.setCacheSize(10);
+		files.forEach(e -> fileIdempotentRepository.add(e));
+
+		//when
+		fileIdempotentRepository.trunkStore();
+
+		//then
+		Stream<String> fileContent = Files.lines(fileStore.toPath());
+		List<String> fileEntries = fileContent.collect(Collectors.toList());
+		fileContent.close();
+		//expected order
+		assertThat(fileEntries, contains(
+				"file1.txt.20171123", 
+				"file2.txt.20171123",
+				"file1.txt.20171124",
+				"file3.txt.20171125",
+				"file2.txt.20171126",
+				"fixed.income.lamr.out.20171126",
+				"pricing.px.20171126",
+				"test.out.20171126",
+				"processing.source.lamr.out.20171126"));
+
+		//current order
+/*		assertThat(fileEntries, contains(
+				"processing.source.lamr.out.20171126",
+				"test.out.20171126",
+				"fixed.income.lamr.out.20171126",
+				"pricing.px.20171126",
+				"file1.txt.20171123", 
+				"file2.txt.20171123",
+				"file1.txt.20171124",
+				"file3.txt.20171125",
+				"file2.txt.20171126"));*/
+		
+	}
+
+}


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Wrong order in file idempotent store.
> -------------------------------------
>
>                 Key: CAMEL-12058
>                 URL: https://issues.apache.org/jira/browse/CAMEL-12058
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 2.20.1, 2.20.2, 2.21.0
>            Reporter: Krzysztof Hołdanowicz
>            Assignee: Claus Ibsen
>            Priority: Minor
>              Labels: FileIdempotent
>             Fix For: 2.20.2
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I recently noticed that there is wrong entry order in file using 
> FileIdempotentRepository implementation. 
> The effect is that instead of having order like: 
> file1.txt.20171123 
> file2.txt.20171123 
> file1.txt.20171124 
> file3.txt.20171125 
> file2.txt.20171126 
> we have: 
> file1.txt.20171123 
> file1.txt.20171124 
> file2.txt.20171123 
> file2.txt.20171126 
> file3.txt.20171125 
> where date extension represents order in which particular file was consumed 
> by the idempotent file consumer. 
> As a consequence instead of initializing memory cache with newest values, 
> it is initialized (probably) based on hash function from truncStore method 
> and we consume same file more than once: 
>     protected void trunkStore() { 
>         LOG.info("Trunking idempotent filestore: {}", fileStore); 
>         FileOutputStream fos = null; 
>         try { 
>             fos = new FileOutputStream(fileStore); 
>             for (String key : cache.keySet()) { 
>                 fos.write(key.getBytes()); 
>                 fos.write(STORE_DELIMITER.getBytes()); 
>             } 
>         } catch (IOException e) { 
>             throw ObjectHelper.wrapRuntimeCamelException(e); 
>         } finally { 
>             IOHelper.close(fos, "Trunking file idempotent repository", LOG); 
>         } 
>     } 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message