[maskedtensor] Add safe softmax tutorial

[george-qi]

Stack from ghstack (oldest at bottom):

• [maskedtensor] Add adagrad sparse semantics tutorial #2047
• [maskedtensor] Add missing nan ops tutorial #2046
• -> [maskedtensor] Add safe softmax tutorial #2045
• [maskedtensor] Distinguish between 0 and NaN gradient #2044
• [maskedtensor] Add sparsity tutorial #2043
• [maskedtensor] Add overview tutorial #2042

[jisaacso]

I would explain a bit more here or inline what you are doing:

Let's create a Tensor with an "unsafe" column, that is a column of all -inf padding values, for which PyTorch's traditional softmax will result in nans.

Now let's create a safeter MaskedTensor from this Tensor, and show that the resultant values are masked out

[jisaacso]

I'm confused on this result. The softmax above returns two different values that are getting masked here: 0.0 and nan. Do we really want to mask out both outputs? I assume the idea is that we want to mask all input padding (aka mask all -infs). but in situations of unstable training, do 0's make a difference?

[george-qi]

I don't think the 0's make a difference in unstable training, but maybe I'm not quite understanding the question. In general for MaskedTensor, if the user has deemed a particular index as "unspecified", then it will remain unspecified or masked out in both the forward as well as the backward. And so in this case, while the underlying data that is being masked out would have been different (i.e. 0.0 vs nan), they're both unspecified and so we still want to mask them out.

For training specifically, I agree that leaving the 0's would be identical, but I think it would contradict MaskedTensor semantics and become confusing (e.g. wouldn't know when to expect masked out values vs not.)

[jisaacso]

Another high level comment similar to other tutorials -- I think this tutorial makes more sense as one component of a larger tutorial that showcases end to end training. At first glance, it's not intuitive to me why I would care to end with a MaskedTensor after softmax vs ending with a tensor. You do reference an issue on unstable training for all nan examples in a batch, but I feel like showing that first hand as part of e2e training would be more powerful.