Introduction to Scheduled Sampling

Scheduled Sampling is a technique employed in sequence-to-sequence models to reduce the gap between training and inference procedures. Initially introduced by Bengio et al. in 2015, this method addresses the problem of "exposure bias" that typically arises in sequence generation tasks, like language modeling or machine translation.

Understanding Exposure Bias

Exposure bias occurs during the training of sequence-to-sequence models when the model is trained on the ground truth data, but during inference, it generates sequences by relying on its predictions from previous time steps. This mismatch leads to sub-optimal model performance during inference.

The Concept of Scheduled Sampling

Scheduled Sampling aims to alleviate the exposure bias by integrating generated tokens even during the training phase. Instead of always using the ground truth token as input for the next step in training, a certain percentage of previously generated tokens are used. This causes the training process to more closely mirror inference conditions, thereby potentially enhancing the model's performance.

Implementation of Scheduled Sampling in TensorFlow

In TensorFlow, Scheduled Sampling can be implemented by modifying the input feeding into the decoder during training. By using a probability schedule, model predictions can occasionally replace the ground truth labels.

Technical Example

Here's a basic example using TensorFlow to outline Scheduled Sampling:

• Helper Class: The `ScheduledEmbeddingTrainingHelper` provides capabilities for sampling embeddings during the training process. The `sampling_probability` defines the likelihood of picking the model's prediction instead of the true token.
• Scheduling Strategy: The schedule typically starts with lower probability and increases as training progresses either linearly, exponentially, or using other heuristics.
• Complex Scheduling Strategies: Researchers have explored various strategies, including adaptive schedules dependent on validation performance or adversarial approaches.
• Practical Applications: Scheduled Sampling can significantly benefit tasks like machine translation, text generation, and speech synthesis.
• Caveats: The implementation details may vary across frameworks and TensorFlow versions, particularly with the shift to TensorFlow 2.x.