Document failover support

[sammefford]

[vivekmuniyandi]

This is documentation on Failover in DMSDK.

Failover Support

 
In order to add failover support for DMSDK, we need two main listeners – the HostAvailabilityListener (retrying requests getting the URIs from the server) and RetryListener (retrying the QueryBatchListener after the batch is got from the server). The HostAvailabilityListener is automatically registered with the QueryBatcher and WriteBatcher when the instances are created. There are a few configurations which can be set with the HostAvailabilityListener such as minHosts (the minimum number of hosts required for the job to run), suspendTimeForHostUnavailable (the time for which the host which is unavailable should be suspended), hostUnavailableExceptions (the list of Throwable types which indicate that the host is unavailable). They come with a default setting

• minHosts set to 1
• suspendTimeForHostUnavailable set to 10 minutes
• hostUnavailableExceptions set to SocketException, SSLException, UnknownHostException

However, these default settings can be customized according to user needs by the static method available within the HostAvailabilityListener. You can customize it by the following code:

HostAvailabilityListener.getInstance(Batcher)
          .withSuspendTimeForHostUnavailable(Duration.ofMinutes(60))
   	  .withMinHosts(2)

 
This would set the minHosts to 2 and the suspendTimeForHostUnavailable to 60 minutes for the HostAvailabilityListener associated with the Batcher.
 
HostAvailabilityListener is registered automatically with both the QueryBatcher and WriteBatcher in order to facilitate retries of batches which have failed with exceptions specified in the list of hostUnavailableExceptions. If a host is detected to be down (identified by failed batches having exceptions mentioned in hostUnavailableExceptions), we check if the number of remaining hosts is greater than the minHosts in order to continue running the job. If it is less than minHosts, we stop the job.
 
If the minHosts criteria is satisfied, the existing ForestConfiguration is modified in order to take into account the host which went down. For the time mentioned in suspendTimeForHostUnavailable, the host remains suspended. We schedule an asynchronous task after the time mentioned in suspendTimeForHostUnavailable to read the ForestConfiguration again from the server. Until then, we use the modified ForestConfiguration without the host that went down.
 
The unavailable host is black listed in the ForestConfiguration and the unavailable host will be replaced with some random available host which is up and running to which we can talk to. Thus, when we get the list of forests from the modified ForestConfiguration, the list of forests wouldn’t have the unavailable host and there would be no more connection failures. This would be the configuration for time specified in suspendTimeForHostUnavailable, after which we read the ForestConfiguration from the server and update our ForestConfiguration accordingly.

Provided Listeners like ApplyTransformListener

These Listeners are registered with QueryBatcher to do some actions like applying some transform or delete documents. HostAvailabilityListener handles retries of batches that were failed and attempts to bring the batch of URIs again from the server. But there might be scenarios where the batch of URIs is got from the server but when attempting to apply the listener (do a transform or delete) the batch had failed. In order to enable retry processing the listener in that scenario, the listener must have initialized a RetryListener (a nested class of HostAvailabilityListener) and added it to its list of failure listeners. The RetryListener takes care of retrying to apply the listener to the batch which failed.
 

Create Custom Listeners that can handle Failover

In order for the listeners to handle failover, we have to have the implement the default method we have in QueryBatchListener – initializeListener(QueryBatcher) which does nothing. The custom listeners should override the default method and have the following implementation of initializeListener method in order to handle failover.

  @Override
  public void initializeListener(QueryBatcher queryBatcher) {
    HostAvailabilityListener hostAvailabilityListener = HostAvailabilityListener.getInstance(queryBatcher);
    if ( hostAvailabilityListener != null ) {
      BatchFailureListener<QueryBatch> retryListener = hostAvailabilityListener.initializeRetryListener(this);
      if( retryListener != null ) onFailure(retryListener);
    }
  }

and this method would be called for all of the QueryBatchListeners registered with the QueryBatcher when the job is started. Since initializeListener(QueryBatcher) is a default method which does nothing, calling on listeners which have the default implementation would do nothing but listeners which have overrided the default implementation to initialize the HostAvailabilityListener’s RetryListener as shown above would be called and the RetryListener would be initialized to handle failovers.

HostAvailabilityListener’s initializeRetryListener(QueryBatchListener) creates a RetryListener, if not already created, for the QueryBatchListener and returns the same. If it is already created, then it returns null. We need to add the RetryListener to the listener's failure listeners and that needs to be done only once. Hence if initializeRetryListener(QueryBatchListener) returns a RetryListener, we add it to our failure listeners list and if it returns null, we just ignore it.

[vivekmuniyandi]

Kim,

Sam had a comment on the priority of a section in the document which I had sent to you over mail. Please take that into account when you document. Many Thanks.

Vivek

[kcoleman-marklogic]

Thank, @vivekmuniyandi , I will. Is Sam's comment reflected in the version of your writeup that is shown here in this issue, or should I look for his feedback elsewhere?

I will take up this topic in earnest as soon as we're done with 9.0-3. Thank you for making the details available so promptly.

[jmakeig]

@vivekmuniyandi, this is a great write-up. Really helpful.

Is the failover behavior the default?

HostAvailabilityListener is automatically registered with the QueryBatcher and WriteBatcher when the instances are created.

That seems to imply so. Presumably then this will change the behavior of existing code (albeit generally for the better). Or am I misunderstanding.

[vivekmuniyandi]

@kcoleman-marklogic The following section in the write up is of low priority. His feedback is in the document I sent to you over mail. Sorry, I should have changed it.

There are a few configurations which can be set with the HostAvailabilityListener such as minHosts (the minimum number of hosts required for the job to run), suspendTimeForHostUnavailable (the time for which the host which is unavailable should be suspended), hostUnavailableExceptions (the list of Throwable types which indicate that the host is unavailable). They come with a default setting - minHosts set to 1, suspendTimeForHostUnavailable set to 10 minutes, hostUnavailableExceptions set to SocketException, SSLException, UnknownHostException. However, these default settings can be customized according to user needs by the static method available within the HostAvailabilityListener. You can customize it by the following code:
HostAvailabilityListener.getInstance(Batcher)
.withSuspendTimeForHostUnavailable(Duration.ofMinutes(60))
.withMinHosts(2)
This would set the minHosts to 2 and the suspendTimeForHostUnavailable to 60 minutes for the HostAvailabilityListener associated with the Batcher.

Thanks

[kcoleman-marklogic]

It's fine. I just wanted to be sure I knew where to look. Thanks for the clarification, Vivek.

[vivekmuniyandi]

Is the failover behavior the default?

@jmakeig Yes, the failover behaviour is default.

this will change the behavior of existing code

I don't understand. This was the behavior right from the start I guess and we are just documenting it now. Let me know if I have misunderstood the question. Thanks.

[kcoleman-marklogic]

@vivekmuniyandi I have a question about this section of your most excellent writeup:

Provided Listeners like ApplyTransformListener

These Listeners are registered with QueryBatcher to do some actions like applying some transform or delete documents. ... there might be scenarios where the batch of URIs is got from the server but when attempting to apply the listener (do a transform or delete) the batch had failed. In order to enable retry processing the listener in that scenar, the listener must have initialized a RetryListener ...

Do ApplyTransformListener and DeleteListener have such a retry listener attached by default, or is it something the user must do?

The ApplyTransformListener javadoc talks only about the empty response case, so that didn't clear it up for me, either.