Skip to content

Commit 632d4c5

Browse files
miykaelmiykael
authored
Merge pull request #77 from miykael/move_user_docs
Moving user documentation from main homepage to nipype_tutorial - Pull Request
2 parents 42a6e9f + 2cf5af2 commit 632d4c5

File tree

61 files changed

+4603
-851
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+4603
-851
lines changed

‎index.ipynb

Lines changed: 37 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"cells": [
33
{
44
"cell_type": "code",
5-
"execution_count": null,
5+
"execution_count": 1,
66
"metadata": {
77
"scrolled": false
88
},
@@ -63,6 +63,7 @@
6363
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_jupyter-notebook.ipynb\">Jupyter-Notebook</a>\n",
6464
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_dataset.ipynb\">BIDS & Tutorial Dataset</a>\n",
6565
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_docker.ipynb\">Docker</a>\n",
66+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_neurodocker.ipynb\">Neurodocker</a>\n",
6667
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_python.ipynb\">Python</a>\n",
6768
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_showcase.ipynb\">Nipype Showcase</a>\n",
6869
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_quickstart.ipynb\">Nipype Quickstart</a>\n",
@@ -80,28 +81,22 @@
8081
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_data_input_bids.ipynb\">Data Input with BIDS</a>\n",
8182
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_data_output.ipynb\">Data Output</a>\n",
8283
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_plugins.ipynb\">Execution Plugins</a>\n",
83-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_function_nodes.ipynb\">Function Nodes</a>\n",
84+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_function_interface.ipynb\">Function Interface</a>\n",
8485
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_iteration.ipynb\">Iteration / Iterables</a>\n",
8586
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_mapnodes.ipynb\">MapNodes</a>\n",
86-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_joinnodes.ipynb\">JoinNodes</a>\n",
87+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_joinnodes.ipynb\">JoinNode, synchronize & itersource</a>\n",
8788
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_error_and_crashes.ipynb\">Errors & Crashes</a>\n",
88-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_import_workflows.ipynb\">Import existing Workflows</a>\n",
89+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_debug.ipynb\">Debugging Nipype Workflows</a>\n",
8990
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_model_specification_fmri.ipynb\">fMRI Model Specification</a>\n",
91+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_execution_configuration.ipynb\">Execution Configuration</a>\n",
92+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_import_workflows.ipynb\">Import existing Workflows</a>\n",
9093
" </div>\n",
9194
" <p>This section will introduce you to all of the key players in Nipype. Basic concepts that you need to learn to\n",
9295
" fully understand and appreciate Nipype. Once you understand this section, you will know all that you need to know\n",
9396
" to create any kind of Nipype workflow.</p>\n",
9497
"\n",
95-
" <h2 class=\"domain-header color03\"><a class=\"domain-title\">Advanced Concepts</a></h2>\n",
98+
" <h2 class=\"domain-header color03\"><a class=\"domain-title\">Workflow Examples</a></h2>\n",
9699
" <div class=\"pure-g domain-table-container color03\">\n",
97-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_create_interfaces.ipynb\">Create Interfaces</a>\n",
98-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_interfaces_caching.ipynb\">Interfaces Caching</a>\n",
99-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_execution_configuration.ipynb\">Execution Configuration</a>\n",
100-
" </div>\n",
101-
" <p>This section is for more advanced users and Nipype developers.</p>\n",
102-
"\n",
103-
" <h2 class=\"domain-header color04\"><a class=\"domain-title\">Workflow Examples</a></h2>\n",
104-
" <div class=\"pure-g domain-table-container color04\">\n",
105100
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_preprocessing.ipynb\">Example 1: Preprocessing</a>\n",
106101
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_1stlevel.ipynb\">Example 1: 1st-level Analysis</a>\n",
107102
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_normalize.ipynb\">Example 1: Normalize Data</a>\n",
@@ -111,6 +106,17 @@
111106
" </div>\n",
112107
" <p>In this section you will find some practical examples and hands-on that show you how to use Nipype in a \"real world\" scenario.</p>\n",
113108
"\n",
109+
" <h2 class=\"domain-header color04\"><a class=\"domain-title\">Advanced Concepts</a></h2>\n",
110+
" <div class=\"pure-g domain-table-container color04\">\n",
111+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_create_interfaces.ipynb\">Create Interfaces</a>\n",
112+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_interfaces_caching.ipynb\">Interfaces Caching</a>\n",
113+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_command_line_interface.ipynb\">Nipype Command Line Interface</a>\n",
114+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_aws.ipynb\">Amazon Web Services (AWS)</a>\n",
115+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_sphinx_ext.ipynb\">Sphinx extensions</a>\n",
116+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_spmmcr.ipynb\">SPM with MATLAB Common Runtime (MCR)</a>\n",
117+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_mipav.ipynb\">Using MIPAV, JIST, and CBS Tools</a> </div>\n",
118+
" <p>This section is for more advanced users and Nipype developers.</p>\n",
119+
"\n",
114120
" <h2 class=\"domain-header color05\"><a class=\"domain-title\">Useful Resources & Links</a></h2>\n",
115121
" <div class=\"pure-g domain-table-container color05\">\n",
116122
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/resources_installation.ipynb\">Install Nipype</a>\n",
@@ -222,6 +228,7 @@
222228
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_jupyter-notebook.ipynb\">Jupyter-Notebook</a>\n",
223229
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_dataset.ipynb\">BIDS & Tutorial Dataset</a>\n",
224230
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_docker.ipynb\">Docker</a>\n",
231+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_neurodocker.ipynb\">Neurodocker</a>\n",
225232
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_python.ipynb\">Python</a>\n",
226233
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_showcase.ipynb\">Nipype Showcase</a>\n",
227234
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/introduction_quickstart.ipynb\">Nipype Quickstart</a>\n",
@@ -239,28 +246,22 @@
239246
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_data_input_bids.ipynb\">Data Input with BIDS</a>\n",
240247
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_data_output.ipynb\">Data Output</a>\n",
241248
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_plugins.ipynb\">Execution Plugins</a>\n",
242-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_function_nodes.ipynb\">Function Nodes</a>\n",
249+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_function_interface.ipynb\">Function Interface</a>\n",
243250
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_iteration.ipynb\">Iteration / Iterables</a>\n",
244251
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_mapnodes.ipynb\">MapNodes</a>\n",
245-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_joinnodes.ipynb\">JoinNodes</a>\n",
252+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_joinnodes.ipynb\">JoinNode, synchronize & itersource</a>\n",
246253
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_error_and_crashes.ipynb\">Errors & Crashes</a>\n",
247-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_import_workflows.ipynb\">Import existing Workflows</a>\n",
254+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_debug.ipynb\">Debugging Nipype Workflows</a>\n",
248255
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_model_specification_fmri.ipynb\">fMRI Model Specification</a>\n",
256+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_execution_configuration.ipynb\">Execution Configuration</a>\n",
257+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/basic_import_workflows.ipynb\">Import existing Workflows</a>\n",
249258
" </div>\n",
250259
" <p>This section will introduce you to all of the key players in Nipype. Basic concepts that you need to learn to\n",
251260
" fully understand and appreciate Nipype. Once you understand this section, you will know all that you need to know\n",
252261
" to create any kind of Nipype workflow.</p>\n",
253262
"\n",
254-
" <h2 class=\"domain-header color03\"><a class=\"domain-title\">Advanced Concepts</a></h2>\n",
263+
" <h2 class=\"domain-header color03\"><a class=\"domain-title\">Workflow Examples</a></h2>\n",
255264
" <div class=\"pure-g domain-table-container color03\">\n",
256-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_create_interfaces.ipynb\">Create Interfaces</a>\n",
257-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_interfaces_caching.ipynb\">Interfaces Caching</a>\n",
258-
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_execution_configuration.ipynb\">Execution Configuration</a>\n",
259-
" </div>\n",
260-
" <p>This section is for more advanced users and Nipype developers.</p>\n",
261-
"\n",
262-
" <h2 class=\"domain-header color04\"><a class=\"domain-title\">Workflow Examples</a></h2>\n",
263-
" <div class=\"pure-g domain-table-container color04\">\n",
264265
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_preprocessing.ipynb\">Example 1: Preprocessing</a>\n",
265266
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_1stlevel.ipynb\">Example 1: 1st-level Analysis</a>\n",
266267
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/example_normalize.ipynb\">Example 1: Normalize Data</a>\n",
@@ -270,6 +271,17 @@
270271
" </div>\n",
271272
" <p>In this section you will find some practical examples and hands-on that show you how to use Nipype in a \"real world\" scenario.</p>\n",
272273
"\n",
274+
" <h2 class=\"domain-header color04\"><a class=\"domain-title\">Advanced Concepts</a></h2>\n",
275+
" <div class=\"pure-g domain-table-container color04\">\n",
276+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_create_interfaces.ipynb\">Create Interfaces</a>\n",
277+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_interfaces_caching.ipynb\">Interfaces Caching</a>\n",
278+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_command_line_interface.ipynb\">Nipype Command Line Interface</a>\n",
279+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_aws.ipynb\">Amazon Web Services (AWS)</a>\n",
280+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_sphinx_ext.ipynb\">Sphinx extensions</a>\n",
281+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_spmmcr.ipynb\">SPM with MATLAB Common Runtime (MCR)</a>\n",
282+
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/advanced_mipav.ipynb\">Using MIPAV, JIST, and CBS Tools</a> </div>\n",
283+
" <p>This section is for more advanced users and Nipype developers.</p>\n",
284+
"\n",
273285
" <h2 class=\"domain-header color05\"><a class=\"domain-title\">Useful Resources & Links</a></h2>\n",
274286
" <div class=\"pure-g domain-table-container color05\">\n",
275287
" <a class=\"subject-link pure-u-1-4\" target=\"_blank\" href=\"notebooks/resources_installation.ipynb\">Install Nipype</a>\n",

‎notebooks/advanced_aws.ipynb

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Using Nipype with Amazon Web Services (AWS)\n",
8+
"\n",
9+
"Several groups have been successfully using Nipype on AWS. This procedure\n",
10+
"involves setting a temporary cluster using StarCluster and potentially\n",
11+
"transferring files to/from S3. The latter is supported by Nipype through\n",
12+
"`DataSink` and `S3DataGrabber`."
13+
]
14+
},
15+
{
16+
"cell_type": "markdown",
17+
"metadata": {},
18+
"source": [
19+
"## Using DataSink with S3\n",
20+
"\n",
21+
"The `DataSink` class now supports sending output data directly to an AWS S3\n",
22+
"bucket. It does this through the introduction of several input attributes to the\n",
23+
"`DataSink` interface and by parsing the `base_directory` attribute. This class\n",
24+
"uses the [boto3](https://boto3.readthedocs.org/en/latest/) and\n",
25+
"[botocore](https://botocore.readthedocs.org/en/latest/) Python packages to\n",
26+
"interact with AWS. To configure the `DataSink` to write data to S3, the user must\n",
27+
"set the ``base_directory`` property to an S3-style filepath.\n",
28+
"\n",
29+
"For example:"
30+
]
31+
},
32+
{
33+
"cell_type": "code",
34+
"execution_count": null,
35+
"metadata": {},
36+
"outputs": [],
37+
"source": [
38+
"from nipype.interfaces.io import DataSink\n",
39+
"ds = DataSink()\n",
40+
"ds.inputs.base_directory = 's3://mybucket/path/to/output/dir'"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {},
46+
"source": [
47+
"With the `\"s3://\"` prefix in the path, the `DataSink` knows that the output\n",
48+
"directory to send files is on S3 in the bucket `\"mybucket\"`. `\"path/to/output/dir\"`\n",
49+
"is the relative directory path within the bucket `\"mybucket\"` where output data\n",
50+
"will be uploaded to (***Note***: if the relative path specified contains folders that\n",
51+
"don’t exist in the bucket, the `DataSink` will create them). The `DataSink` treats\n",
52+
"the S3 base directory exactly as it would a local directory, maintaining support\n",
53+
"for containers, substitutions, subfolders, `\".\"` notation, etc. to route output\n",
54+
"data appropriately.\n",
55+
"\n",
56+
"There are four new attributes introduced with S3-compatibility: ``creds_path``,\n",
57+
"``encrypt_bucket_keys``, ``local_copy``, and ``bucket``."
58+
]
59+
},
60+
{
61+
"cell_type": "code",
62+
"execution_count": null,
63+
"metadata": {},
64+
"outputs": [],
65+
"source": [
66+
"ds.inputs.creds_path = '/home/neuro/aws_creds/credentials.csv'\n",
67+
"ds.inputs.encrypt_bucket_keys = True\n",
68+
"ds.local_copy = '/home/neuro/workflow_outputs/local_backup'"
69+
]
70+
},
71+
{
72+
"cell_type": "markdown",
73+
"metadata": {},
74+
"source": [
75+
"``creds_path`` is a file path where the user's AWS credentials file (typically\n",
76+
"a csv) is stored. This credentials file should contain the AWS access key id and\n",
77+
"secret access key and should be formatted as one of the following (these formats\n",
78+
"are how Amazon provides the credentials file by default when first downloaded).\n",
79+
"\n",
80+
"Root-account user:\n",
81+
"\n",
82+
"\tAWSAccessKeyID=ABCDEFGHIJKLMNOP\n",
83+
"\tAWSSecretKey=zyx123wvu456/ABC890+gHiJk\n",
84+
"\n",
85+
"IAM-user:\n",
86+
"\n",
87+
"\tUser Name,Access Key Id,Secret Access Key\n",
88+
"\t\"username\",ABCDEFGHIJKLMNOP,zyx123wvu456/ABC890+gHiJk\n",
89+
"\n",
90+
"The ``creds_path`` is necessary when writing files to a bucket that has\n",
91+
"restricted access (almost no buckets are publicly writable). If ``creds_path``\n",
92+
"is not specified, the DataSink will check the ``AWS_ACCESS_KEY_ID`` and\n",
93+
"``AWS_SECRET_ACCESS_KEY`` environment variables and use those values for bucket\n",
94+
"access.\n",
95+
"\n",
96+
"``encrypt_bucket_keys`` is a boolean flag that indicates whether to encrypt the\n",
97+
"output data on S3, using server-side AES-256 encryption. This is useful if the\n",
98+
"data being output is sensitive and one desires an extra layer of security on the\n",
99+
"data. By default, this is turned off.\n",
100+
"\n",
101+
"``local_copy`` is a string of the filepath where local copies of the output data\n",
102+
"are stored in addition to those sent to S3. This is useful if one wants to keep\n",
103+
"a backup version of the data stored on their local computer. By default, this is\n",
104+
"turned off.\n",
105+
"\n",
106+
"``bucket`` is a boto3 Bucket object that the user can use to overwrite the\n",
107+
"bucket specified in their ``base_directory``. This can be useful if one has to\n",
108+
"manually create a bucket instance on their own using special credentials (or\n",
109+
"using a mock server like [fakes3](https://github.com/jubos/fake-s3)). This is\n",
110+
"typically used for developers unit-testing the DataSink class. Most users do not\n",
111+
"need to use this attribute for actual workflows. This is an optional argument.\n",
112+
"\n",
113+
"Finally, the user needs only to specify the input attributes for any incoming\n",
114+
"data to the node, and the outputs will be written to their S3 bucket."
115+
]
116+
},
117+
{
118+
"cell_type": "markdown",
119+
"metadata": {},
120+
"source": [
121+
"```python\n",
122+
"workflow.connect(inputnode, 'subject_id', ds, 'container')\n",
123+
"workflow.connect(realigner, 'realigned_files', ds, 'motion')\n",
124+
"```"
125+
]
126+
},
127+
{
128+
"cell_type": "markdown",
129+
"metadata": {},
130+
"source": [
131+
"So, for example, outputs for `sub001`’s `realigned_file1.nii.gz` will be in:\n",
132+
"\n",
133+
" s3://mybucket/path/to/output/dir/sub001/motion/realigned_file1.nii.gz"
134+
]
135+
},
136+
{
137+
"cell_type": "markdown",
138+
"metadata": {},
139+
"source": [
140+
"## Using S3DataGrabber\n",
141+
"Coming soon..."
142+
]
143+
}
144+
],
145+
"metadata": {
146+
"kernelspec": {
147+
"display_name": "Python [default]",
148+
"language": "python",
149+
"name": "python3"
150+
},
151+
"language_info": {
152+
"codemirror_mode": {
153+
"name": "ipython",
154+
"version": 3
155+
},
156+
"file_extension": ".py",
157+
"mimetype": "text/x-python",
158+
"name": "python",
159+
"nbconvert_exporter": "python",
160+
"pygments_lexer": "ipython3",
161+
"version": "3.6.5"
162+
}
163+
},
164+
"nbformat": 4,
165+
"nbformat_minor": 2
166+
}

0 commit comments

Comments
 (0)