Issue Writing Empty Document Properties

[Zipunrar]

Hi MarkLogic,

I do have an issue writing document properties with the new Java Client API Version 4.
Here are the details:

Version of MarkLogic Java Client API

4.2.0

Version of MarkLogic Server

9.0.8

Java version

Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

Input: Some code to illustrate the problem, preferably in a state that can be independently

DatabaseClient markLogic = ... // create client connection
XMLDocumentManager manager = markLogic.newXMLDocumentManager();
DocumentMetadataHandle metadata = new DocumentMetadataHandle();
// this won't work
metadata.getProperties().put("foobar", "");
StringHandle data = new StringHandle("<root>Test</root>");
// error is thrown here
manager.write("/test.xml", metadata, data);
markLogic.release();

Actual output: What did you observe? What errors did you see? Can you attach the logs? (Java logs, MarkLogic logs)

The following stacktrace is thrown:

java.lang.NullPointerException
	at com.ctc.wstx.sw.BaseStreamWriter.writeCharacters(BaseStreamWriter.java:464)
	at com.marklogic.client.io.DocumentMetadataHandle$ValueSerializer.process(DocumentMetadataHandle.java:917)
	at com.marklogic.client.impl.ValueConverter.convertFromJava(ValueConverter.java:253)
	at com.marklogic.client.impl.ValueConverter.convertFromJava(ValueConverter.java:94)
	at com.marklogic.client.io.DocumentMetadataHandle.sendPropertiesImpl(DocumentMetadataHandle.java:886)
	at com.marklogic.client.io.DocumentMetadataHandle.sendMetadataImpl(DocumentMetadataHandle.java:788)
	at com.marklogic.client.io.DocumentMetadataHandle.write(DocumentMetadataHandle.java:574)
	at com.marklogic.client.impl.StreamingOutputImpl.writeTo(StreamingOutputImpl.java:58)
	at okhttp3.MultipartBody.writeOrCountBytes(MultipartBody.java:173)
	at okhttp3.MultipartBody.writeTo(MultipartBody.java:114)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:72)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at com.burgstaller.okhttp.AuthenticationCacheInterceptor.intercept(AuthenticationCacheInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
	at okhttp3.RealCall.execute(RealCall.java:77)
	at com.marklogic.client.impl.OkHttpServices.sendRequestOnce(OkHttpServices.java:757)
	at com.marklogic.client.impl.OkHttpServices.sendRequestOnce(OkHttpServices.java:752)
	at com.marklogic.client.impl.OkHttpServices.putPostDocumentImpl(OkHttpServices.java:1609)
	at com.marklogic.client.impl.OkHttpServices.putDocument(OkHttpServices.java:1319)
	at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:920)
	at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:758)
	at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:702)
	at com.din.xscore.core.domain.marklogic.operation.MarkLogicDocumentPatcherTest.test(MarkLogicDocumentPatcherTest.java:83)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:147)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:129)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

However the root cause of this issue is a function called "StringToString" in com.marklogic.client.impl.ValueConverter at line 579 that returns a "null" value for "null" string but sadly also for "empty" strings (see line 581).

Expected output: What specifically did you expect to happen?

I would expect that I am able to write empty properties as before with Client API 3 and ML 8.
A property should accept empty strings as a valid value since that might be a real-world scenario like in my case. I want to have an empty property to express the different between "not available" and "not set". The value "null" might be worth an error, however empty string is valid in my opinion.

Thanks in advance for any help.

[ehennum]

Thanks for the clear description of the issue. It definitely sounds like something that needs to be fixed.

One workaround is to pass the properties as XML represented with any of the handles that implement XMLWriteHandle. These handles also implement DocumentMetadataWriteHandle, which is the data type of parameters that accept DocumentMetadataHandle.

That approach provides full control over the document properties but does require that the XML is valid including the declaration of the properties namespace.

[anu3990]

Hi @Zipunrar
Tried to recreate the error using the code provided by you earlier in the thread but we did not see any problems and the document was written successfully. We use the standard XMLStreamWriterImpl library that comes with java. This error might be resulting while using the StreamWriter from woodStox.
(More information here - FasterXML/woodstox#18.)

[Zipunrar]

Hi @anu3990
Thanks for the response. Sure you might be right. However I also mentioned the source of the issue within you class com.marklogic.client.impl.ValueConverter at line 579 (see directly under the long error stacktrace).

I will not say that handling "null" while writing characters is a problem of the XMLWriter but since the behavior depends on implementation of the library in use, I would prefer a single source of truth here.

So MarkLogic does define the rules for writing properties. And since this behavior changes just because I upgrade from one version of your API to another. In my opinion this should not happen.

Long story short: Giving an empty string as the value of a property should always work (independent of the underlying XMLWriter library) and writing "null" should either result in a NullPointerException or should be defensively transformed to an empty string.
The only place to change would be a single line in your code.

Thanks, and I hope that was not too cheeky.

[ehennum]

Hi, @Zipunrar .
Thanks for following up. In general, working around bugs in a lower layer in a higher layer seems like an antipractice.

In particular, while it would be puzzling for an XMLWriter implementation to accept null but not the empty string, I don't know that we can rule that out, given that we now know of an XMLWriter implementation that accepts an empty string but has decided not to accept null (which is puzzling).

Regarding the suggested change, the com.marklogic.client.impl.ValueConverter#StringToString(java.lang.String) method is reached through other call stacks besides the XMLWriter call stack, so the change would not be to a single line.

Anyway, we will try to find an alternative workaround for this puzzling decision by an XMLWriter SPI implementer.

[Zipunrar]

Hi @ehennum,

sorry but I think I don't get your point. The only thing I want is to have the same behavior that we
do had before the upgrade to MarkLogic 9. And my expectations are that if I put in a property with empty string as value, then the Java API of ML will write that empty property.

In addition I don't think that we have to struggle with the underlying XMLWriter because whatever implementation is under your API, an empty string should be excepted from any implementation.
However "null" as value not, so that could cause an exception or not depending on the implementation details.

[ehennum]

Hi, @Zipunrar , I agree that an inconsistency between a StAX implementation and the default implementation is only your problem if you are using StAX.

Like all Java libraries, our internals call the interfaces provided by core Java and depend on those interfaces working correctly at runtime.

Again, we will look for a way to fix the issue you are reporting here.

[ehennum]

Nothing better occurs to me than the brute force approach of

• Searching for all calls to XMLStreamWriter.writeCharacters() where the code isn't serializing an in-memory XML document.
• Moving the call into an if branch that makes the call only if the text argument is not null.
• Provide a comment that identifies the FasterXML bug as the reason for the check.

That way, we address any undiscovered cases that would also be affected by the FasterXML bug.

@anu3990 , does any better approach occur to you?

[anu3990]

Hi @ehennum, yes that looks like a reasonable approach.