Feature Request: tutorial on loading datasets

[ronrest]

A tutorial outlining how to make use of the torch.utils.data.Dataset and torch.utils.data.DataLoader on your own data (not just the torchvision.datasets) would be good. The documentation page is quite obscure, and it is not entirely clear how these can be made use of on your own data.

Also outlining what would be good practices for when your data is:

• A numpy array
• A folder full of image files

And if pytorch has built in functions for creating queues of data, for when the data is too big to all fit in memory in one go (eg in the case of a folder full of image files).

[apaszke]

The "queue of data" is what we call a DataLoader. You're right, these docs are very needed. I'll try to write them soon. Thanks for suggestions!

[ronrest]

Great! I look forward to it. And thanks for all the great work you and your colleagues have put into creating this library and the existing documentation.

[chsasank]

In transfer learning tutorial, you can see data loading example. http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

[ShownX]

And the list of images and label

[chsasank]

New tutorial is up: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html

[JianchaoTan]

I run your code: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html
And I got this error:
File "data_loading_tutorial.py", line 393, in
print(i_batch, sample_batched['image'].size(),
TypeError: list indices must be integers, not str
when I print sample_batched, I got:
[('image', 'image', 'image', 'image'), ('landmarks', 'landmarks', 'landmarks', 'landmarks')]

[chsasank]

This is weird, because it runs fine for me. What's your pytorch version?

[JianchaoTan]

Oh, sorry, I think it is because I am using old version: '0.1.10_1'. I updated it to new version, it is ok now. Thanks!