Streaming Multipart for OkHttp

[vivekmuniyandi]

Needed to support streaming bulk read (used by DMSDK ExportListener), optic API, eval, resource extensions, bulk search, and large single document read with metadata. We could perhaps use a stand-alone library like nio-multipart, or the more battle-tested apache fileupload
the only concern about apache fileupload is it depends on apache commons-io which might create version conflicts if other frameworks used by our customers depend on a different version of commons-io.

[georgeajit]

As of Java 9, OpenJdk 10 and 11, we see java.lang.OutOfMemoryError: Java heap space, with multi-part parsing of large binary files (~4MB) and a large count (> 100).

Here is a snippet for QA test. The test writes 102 binary files each of size 4 MB and then reads the "uris" .

  /*
   * This test uses FileHandle to bulkload 102 binary documents,test bulk read
   * from database.
   */
  @Test
  public void test3ReadMultipleBinaryDoc() throws KeyManagementException, NoSuchAlgorithmException, Exception
  {
	  System.out.println("Inside test3ReadMultipleBinaryDoc");
	  
    try {
		String docId[] = { "Sega-4MB.jpg" };
		int count = 1;
		client = getDatabaseClient("rest-admin", "x", getConnType());
		BinaryDocumentManager docMgr = client.newBinaryDocumentManager();
		DocumentWriteSet writeset = docMgr.newWriteSet();
		File file1 = null;
		file1 = new File("src/test/java/com/marklogic/client/functionaltest/data/" + docId[0]);
		FileHandle h1 = new FileHandle(file1);
		for (int i = 0; i < ndocCount; i++) {
		  writeset.add(DIRECTORY + "binary" + i + ".jpg", h1);
		  if (count % BATCH_SIZE == 0) {
		    docMgr.write(writeset);
		    writeset = docMgr.newWriteSet();
		  }
		  count++;
		}
		if (count % BATCH_SIZE > 0) {
		  docMgr.write(writeset);
		}
		
		String uris[] = new String[ndocCount];
		for (int i = 0; i < ndocCount; i++) {
		  uris[i] = DIRECTORY + "binary" + i + ".jpg";
		}
		count = 0;
		FileHandle rh = new FileHandle();
		DocumentPage page = docMgr.read(uris);
		while (page.hasNext()) {
		  DocumentRecord rec = page.next();
		  validateRecord(rec, Format.BINARY);
		  rec.getContent(rh);
		  assertEquals("Content length :", file1.length(), rh.get().length());
		  count++;
		}
		assertEquals("document count", ndocCount, count);
		// Testing the multiple same uris will not read multiple records
		
		for (int i = 0; i < ndocCount; i++) {
		  uris[i] = DIRECTORY + "binary" + 1 + ".jpg";
		}
		count = 0;
		page = docMgr.read(uris);
		while (page.hasNext()) {
		  DocumentRecord rec = page.next();
		  validateRecord(rec, Format.BINARY);
		  count++;
		}
		System.out.println("Document count test3ReadMultipleBinaryDoc " + count);
		assertEquals("document count", 1, count);
	} catch (Exception e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	}
    finally {
    	client.release();
    }
  }

Here is the stack trace:

java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3744)
	at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at javax.mail.util.ByteArrayDataSource.<init>(ByteArrayDataSource.java:88)
	at com.marklogic.client.impl.OkHttpServices.getEntity(OkHttpServices.java:5370)
	at com.marklogic.client.impl.OkHttpServices.makeResults(OkHttpServices.java:4422)
	at com.marklogic.client.impl.OkHttpServices.getIteratedResourceImpl(OkHttpServices.java:3170)
	at com.marklogic.client.impl.OkHttpServices.getBulkDocumentsImpl(OkHttpServices.java:1045)
	at com.marklogic.client.impl.OkHttpServices.getBulkDocuments(OkHttpServices.java:930)
	at com.marklogic.client.impl.DocumentManagerImpl.read(DocumentManagerImpl.java:461)
	at com.marklogic.client.impl.DocumentManagerImpl.read(DocumentManagerImpl.java:439)
	at com.marklogic.client.impl.DocumentManagerImpl.read(DocumentManagerImpl.java:404)
	at com.marklogic.client.functionaltest.TestBulkReadSample1.test3ReadMultipleBinaryDoc(TestBulkReadSample1.java:227)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

Usage of ByteArrayDataSource in com.marklogic.client.impl.OkHttpServices.getEntity method
of OkHttpServices.java class and changes to Java Garbage collection scheme in post Java 8 releases (Java 8 does not have this issue) seem to have caused this.

For reference, this SO article seem to be explaining a similar case.

https://stackoverflow.com/questions/11290758/how-to-send-email-with-large-attach-outofmemoryerror-java-heap-space