Operation cancelled exception while executing PersistWorkflow

[Thejas007]

Describe the bug
While a step is running. Gracefully stop the host by calling workflowhost.Stop();
The last executing step will be successfully completing. but while saving the result, a Operation cancelled exception is getting thrown.

As the last completion not saved. The step is getting retried even if it was completed.

To Reproduce
While a step is running. Press Ctl+C;

Exception stack trace :

The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at MySqlConnector.Core.ConnectionPool.d__10.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 25
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.d__110.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 645
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.d__26.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 312
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.d__50.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.d__50.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.d__47.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlRelationalConnection.d__16.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.AsyncEnumerator.<InitializeReaderAsync>d__18.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.<ExecuteAsync>d__72.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.AsyncEnumerator.<MoveNextAsync>d__17.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.<SingleAsync>d__211.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.d__21`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider.d__10.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkflowCore.Services.BackgroundTasks.WorkflowConsumer.d__10.MoveNext() in C:\OpenSource\workflow-core\src\WorkflowCore\Services\BackgroundTasks\WorkflowConsumer.cs:line 54
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at WorkflowCore.Services.BackgroundTasks.WorkflowConsumer.d__10.MoveNext() in C:\OpenSource\workflow-core\src\WorkflowCore\Services\BackgroundTasks\WorkflowConsumer.cs:line 79
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkflowCore.Services.BackgroundTasks.QueueConsumer.d__18.MoveNext() in C:\OpenSource\workflow-core\src\WorkflowCore\Services\BackgroundTasks\QueueConsumer.cs:line 131

Expected behavior
Even if the operation is canceled the workflow state along with executionpointers data should be saved. So that workflow state can be resumed without executing the previously executed step.

Additional context
In WorkflowConsumer , _persistenceStore.PersistWorkflow(workflow, cancellationToken) is called with cancellation token.
So even if the step execution completed the state is not getting saved in database.

If we call PersistWorkflow without cancellationToken might fix the issue.

workflow-core/src/WorkflowCore/Services/BackgroundTasks/WorkflowConsumer.cs

Line 54 in b45525a

await _persistenceStore.PersistWorkflow(workflow, cancellationToken);

                   try
                    {
                        result = await _executor.Execute(workflow, cancellationToken);
                    }
                    finally
                    {
                        await _persistenceStore.PersistWorkflow(workflow, cancellationToken);
                        await QueueProvider.QueueWork(itemId, QueueType.Index);
                        _greylist.Remove($"wf:{itemId}");
                    }