GongXinyuu
8/22/2019 - 9:33 PM

A temporary function to avoid nan in the pytorch gumbel_softmax function.

A temporary function to avoid nan in the pytorch gumbel_softmax function.

``````import torch
import torch.nn.functional as F

def gumbel_softmax(logits, tau=1, hard=False, eps=1e-10, dim=-1):
# type: (Tensor, float, bool, float, int) -> Tensor
r"""
Samples from the `Gumbel-Softmax distribution`_ and optionally discretizes.
You can use this function to replace "F.gumbel_softmax".

Args:
logits: `[..., num_features]` unnormalized log probabilities
tau: non-negative scalar temperature
hard: if ``True``, the returned samples will be discretized as one-hot vectors,
but will be differentiated as if it is the soft sample in autograd
dim (int): A dimension along which softmax will be computed. Default: -1.

Returns:
Sampled tensor of same shape as `logits` from the Gumbel-Softmax distribution.
If ``hard=True``, the returned samples will be one-hot, otherwise they will
be probability distributions that sum to 1 across `dim`.

.. note::
This function is here for legacy reasons, may be removed from nn.Functional in the future.

.. note::
The main trick for `hard` is to do  `y_hard - y_soft.detach() + y_soft`

It achieves two things:
- makes the output value exactly one-hot
(since we add then subtract y_soft value)
(since we strip all other gradients)

Examples::
>>> logits = torch.randn(20, 32)
>>> # Sample soft categorical using reparametrization trick:
>>> F.gumbel_softmax(logits, tau=1, hard=False)
>>> # Sample hard categorical using "Straight-through" trick:
>>> F.gumbel_softmax(logits, tau=1, hard=True)

.. _Gumbel-Softmax distribution:
https://arxiv.org/abs/1611.00712
https://arxiv.org/abs/1611.01144
"""
def _gen_gumbels():
gumbels = -torch.empty_like(logits).exponential_().log()
if torch.isnan(gumbels).sum() or torch.isinf(gumbels).sum():
# to avoid zero in exp output
gumbels = _gen_gumbels()
return gumbels

gumbels = _gen_gumbels()  # ~Gumbel(0,1)
gumbels = (logits + gumbels) / tau  # ~Gumbel(logits,tau)
y_soft = gumbels.softmax(dim)

if hard:
# Straight through.
index = y_soft.max(dim, keepdim=True)[1]
y_hard = torch.zeros_like(logits).scatter_(dim, index, 1.0)
ret = y_hard - y_soft.detach() + y_soft
else:
# Reparametrization trick.
ret = y_soft
return ret``````