RF: Use consistent dot file renaming scheme

[effigies]

Only colored and hierarchical graphs are currently renamed to drop the .dot from the filename when converted to PNG or SVG.

The relevant branch is here:

nipype/nipype/pipeline/engine/workflows.py

Lines 429 to 450 in a2d054a

	if graph2use in ['hierarchical', 'colored']:
	if self.name[:1].isdigit(): # these graphs break if int
	raise ValueError('{} graph failed, workflow name cannot begin '
	'with a number'.format(graph2use))
	dotfilename = op.join(base_dir, dotfilename)
	self.write_hierarchical_dotfile(
	dotfilename=dotfilename,
	colored=graph2use == "colored",
	simple_form=simple_form)
	outfname = format_dot(dotfilename, format=format)
	else:
	graph = self._graph
	if graph2use in ['flat', 'exec']:
	graph = self._create_flat_graph()
	if graph2use == 'exec':
	graph = generate_expanded_graph(deepcopy(graph))
	outfname = export_graph(
	graph,
	base_dir,
	dotfilename=dotfilename,
	format=format,
	simple_form=simple_form)

Changes proposed in this pull request

• Factors out the renaming logic and updated command line from No longer concatenating extensions #2381 and calls from both export_graph and format_dot.

Follow-up to #2381.

Would appreciate review from @TheChymera and @djarecka.

[djarecka]

@effigies I'm trying to quickly run some of the tutorials with your branch and have this new error when run wf.write_graph("workflow_graph.dot"):

RuntimeError                              Traceback (most recent call last)
<ipython-input-9-e815aa572710> in <module>()
----> 1 wf.write_graph("workflow_graph.dot")
      2 #from IPython.display import Image
      3 
      4 #Image(filename="/Users/dorota/nipype_workshop/output/working_dir/smoothflow/workflow_graph.dot.png")

~/nipype/nipype/pipeline/engine/workflows.py in write_graph(self, dotfilename, graph2use, format, simple_form)
    436                 colored=graph2use == "colored",
    437                 simple_form=simple_form)
--> 438             outfname = format_dot(dotfilename, format=format)
    439         else:
    440             graph = self._graph

~/nipype/nipype/pipeline/engine/utils.py in format_dot(dotfilename, format)
   1311     """Dump a directed graph (Linux only; install via `brew` on OSX)"""
   1312     try:
-> 1313         formatted_dot, _ = _run_dot(dotfilename, format_ext=format)
   1314     except IOError as ioe:
   1315         if "could not be found" in str(ioe):

~/nipype/nipype/pipeline/engine/utils.py in _run_dot(dotfilename, format_ext)
   1328     cmd = 'dot -T{} -o"{}" "{}"'.format(format, formatted_dot, dotfilename)
   1329     res = CommandLine(cmd, terminal_output='allatonce',
-> 1330                       resource_monitor=False).run()
   1331     return formatted_dot, res
   1332 

~/nipype/nipype/interfaces/base/core.py in run(self, cwd, **inputs)
    513 
    514         try:
--> 515             runtime = self._run_interface(runtime)
    516             outputs = self.aggregate_outputs(runtime)
    517         except Exception as e:

~/nipype/nipype/interfaces/base/core.py in _run_interface(self, runtime, correct_return_codes)
   1020         if runtime.returncode is None or \
   1021                 runtime.returncode not in correct_return_codes:
-> 1022             self.raise_exception(runtime)
   1023 
   1024         return runtime

~/nipype/nipype/interfaces/base/core.py in raise_exception(self, runtime)
    957             ('Command:\n{cmdline}\nStandard output:\n{stdout}\n'
    958              'Standard error:\n{stderr}\nReturn code: {returncode}'
--> 959              ).format(**runtime.dictcopy()))
    960 
    961     def _get_environ(self):

RuntimeError: Command:
dot -T<built-in function format> -o"/Users/dorota/nipype_workshop/output/working_dir/smoothflow/workflow_graph.<built-in function format>" "/Users/dorota/nipype_workshop/output/working_dir/smoothflow/workflow_graph.dot"
Standard output:

Standard error:
/bin/sh: built-in: No such file or directory
Return code: 1

i can see that /Users/dorota/nipype_workshop/output/working_dir/smoothflow/workflow_graph.dot was created and nothing more

[effigies]

Ah, that was my mistake. Fixing and rebasing on master.

[effigies]

@djarecka Is there a quick way I can replicate your test locally? Just don't want to clog the circle queue if I'm going to need to make another change.

@TheChymera If you can verify that this still does what you want, that would be greatly appreciated.

[djarecka]

@effigies last night I was checking entire nipype_tutorial on CircleCi (that is set to take a current master from nipype), but today I was just testing two notebooks from basic_* notebooks that were failing (example_* takes much more time, and it looks like that the first one had the same problem). So you can try locally:

• https://github.com/miykael/nipype_tutorial/blob/master/notebooks/basic_workflow.ipynb
• https://github.com/miykael/nipype_tutorial/blob/master/notebooks/basic_graph_visualization.ipynb
• https://github.com/miykael/nipype_tutorial/blob/master/notebooks/basic_import_workflows.ipynb

[effigies]

Do you just run miykael/nipype_tutorial:latest with the nipype install directory overridden? (Sorry, I have never actually run these, and I'm not sure of the best/quickest process.)

[djarecka]

I guess you can do it, probably mine is newer djarecka/nipype_tutorial:latest (however I have to admit that forgot to update it recently).

The option that I choose today was simply open jupyter noetbook on my dev environment and try to run the notebooks (changing paths to my local copy of datasets, the same as we used during workshop, also instruction how to install can be found here ).

[djarecka]

CircleCI builds a new docker container based on Dockerfile that is generated by this script.
I should add to circle option to push the new image to DockerHub, forgot to do it...

[djarecka]

I've also built the container based on the Dockerfile from the repo and in theory can push it to DockerHub, but my connection is so slow that you will be much faster building the container on your own.

[effigies]

Okay, I've just mounted into Michael's container. I have to remove the .dot from the filenames, but I'm not getting any errors.

[djarecka]

@effigies nice! Once you merge it, I can remove all dots and ask Circle CI to run tests on all notebooks.

[effigies]

@djarecka Sounds good.

@mgxd If this passes, I propose we merge this with [skip ci] in the commit, and then make a final release commit (I assume just updating the Changelog?) for a full build. When that and @djarecka's notebook tests pass, we can release?

[mgxd]

@effigies that sounds good to me - following the checklist, we'll just have to update the __version__ and conda build first

[effigies]

@djarecka Merged. Feel free to start running tutorial tests.

@mgxd I think my job here is done. Let me know if there's anything useful I can do before release.

[djarecka]

@effigies I've started running, but all test can take 4-5h (if everything goes well). Will let you know

[effigies]

Okay. With that head time, I restarted the build for #2398, but go ahead and cancel if we start needing Circle again before it's done.

[djarecka]

@effigies @mgxd @satra - ok, the tutorial has been updated and tested. Once we have the new release, I'll create a new container and push to DockerHub.