Nodes for Conditional Execution

[blakedewey]

Is there any room in the code base for a node for use in conditional execution? I was thinking a specialized node class that ran code and returned a boolean that could then be used to terminate a branch by using the donotrun in the linear plugin or similar in the distributed plugins. Any thoughts?

[satra]

@blakedewey - i think this will require some architectural changes, but feasible. currently when a node crashes we delete the subgraph of dependent nodes. essentially the conditional branching will require similar operational logic. however, we are also now facing a scalability issue #879 that we need to resolve. if there is an architectural change required for that, we might want to role it with the necessary changes for conditional execution. do you have any thoughts on how you would build such a workflow?

[blakedewey]

I was thinking of having a separate node class that would eventually end in a boolean. If that boolean were true, we could delete the subgraph, but not add it to the error notices as is done in the case of an error, instead do some kind of skip reporting. Although if this method is not scalable, this may present a problem.

[ischwabacher]

It seems like this and #819 are the same thing— you want a node that looks at its inputs and decides either to do some work or just be a sink; that issue wants a node that looks at its inputs and decides how many copies of itself to make. I want a node that looks at its inputs and decides which of a few possible nodes to be.

After thinking about it some more, all of these cases (as well as MapNode, JoinNode and iterables) are just lifting type constructors from Input/OutputSpecs to Nodes, which is something that could conceivably be done in a general way. It might amount to from haskell import type_system though.