Week 8 - Data Representation Patterns: Embeddings. Problem Representation Patterns: Rebalancing Patterns

Lecture recording here.

Introduction

This week we start design patterns for machine learning. We will look at one data representation pattern - the embeddings pattern. We will also look at one problem representation pattern - the rebalancing pattern. The embeddings pattern is for high-cardinality features where closeness relationships are important to preserve. It learns a data representation that maps high-cardinality data into a lower-dimensional space in such a way that the information relevant to the learning problem is preserved. The rebalancing pattern uses downsampling, upsampling, or a weighted loss function for heavily imbalanced data.

Videos

Assignment(s)

Common Categories of Machine Learning Design Patterns

Design patterns for machine learning can be broken into six categories: data representation, problem representation, patterns that modify model training, resilience, reproducibility and responsible AI. Data representation design patterns for machine learning focus on efficient and effective ways to represent and organize data for use in machine learning algorithms. Problem representation design patterns for machine learning focus on how to represent and formulate machine learning problems in a way that facilitates effective learning and modeling. Patterns that modify model training design patterns for machine learning are focused on enhancing the training process of machine learning models to improve their performance, convergence, and generalization capabilities. Resilience design patterns for machine learning are aimed at improving the robustness and fault tolerance of machine learning systems. Reproducibility design patterns for machine learning are focused on ensuring that machine learning experiments and results can be reproduced consistently. Responsible AI design patterns for machine learning focus on ensuring that machine learning systems are developed and deployed in an ethical and responsible manner. These are summarized in the image:



These are also summarized in the second half of Common Patterns.docx. The bolded patterns are the patterns we will cover in class. The number in brackets shows the popularity rank of a particular pattern. The patterns in red were covered last year but due to declining popularity will not be covered in this year's class. The patterns in green will be covered for the first time this year due to increasing popularity. Note that we cover the 16 most popular machine learning design patterns.

Machine Learning Lectures - Stanford University

For a full course on machine learning, see the playlist Stanford CS229: Machine Learning Full Course taught by Andrew Ng, Autumn 2018. Of interest to our study of machine learning design patterns is the second lecture on linear regression and gradient descent. See Stanford CS229: Machine Learning - Linear Regression and Gradient Descent.

For a shorter course on machine learning, see the playlist Machine Learning Specialization by Andrew Ng. For shorter videos on training data, see the following videos on supervised learning: #4 Machine Learning Specialization and #5 Machine Learning Specialization. See also the following videos on unsupervised learning: #6 Machine Learning Specialization and #7 Machine Learning Specialization.

The Embeddings Design Pattern

The Rationale

The rationale for the embeddings design pattern in machine learning is to represent high-dimensional categorical or discrete features in a lower-dimensional continuous vector space. Embeddings are learned representations that capture meaningful relationships and semantic information between different categories or entities present in the data.

The UML

Here is a very rough UML diagram for the embeddings pattern:

  +------------------+                +------------------+
  |  EmbeddingLayer  |<>--------------|      Model       |
  +------------------+                +------------------+
  | - inputDim: int  |                | - embeddingLayer: EmbeddingLayer
  | - embeddingDim: int               |                  |
  +------------------+                +------------------+
  | + getEmbedding() |                | + predict()      |
  +------------------+                +------------------+

1. Embedding Layer Class: This class represents the embedding layer component in a neural network model. It encapsulates the functionality of mapping categorical inputs to their corresponding embeddings. It typically contains parameters such as the input dimension, embedding dimension, and methods for forward propagation.
2. Model Class: The model class represents the overall neural network model that incorporates the embedding layer. It may include other layers such as dense layers, convolutional layers, or recurrent layers depending on the specific architecture. The model class connects the embedding layer to other components of the model.

3. Training Data Class: This class represents the training data used to train the model. It may contain the input categorical features, target variables, and other relevant data for training the embeddings. It serves as the input to the model during the training phase.
4. Inference Data Class: This class represents the data used for inference or prediction with the trained model. It may contain the categorical features for which embeddings are generated, as well as any additional input data required for prediction.

Code Example - Embeddings Data Pattern

The following is a simple example of the embeddings data pattern:
C++: Embedding.cpp.
C#: Embeddings.cs.
Java: Embeddings.java.
Python: Embeddings.py.

Common Usage

The following are some common usages of the embeddings pattern:

1. Natural Language Processing (NLP): Embeddings are widely used in NLP tasks such as text classification, sentiment analysis, named entity recognition, machine translation, and document similarity. Word embeddings, such as Word2Vec, GloVe, and fastText, capture semantic relationships between words and are used to represent textual data in a dense vector space.
2. Recommender Systems: Embeddings play a crucial role in recommender systems to capture user preferences and item characteristics. User embeddings and item embeddings are learned from historical user-item interactions and used to generate personalized recommendations. Embeddings enable the system to find similar users or items based on their embedding vectors.
3. Image and Video Processing: In computer vision tasks, embeddings are used to represent images and videos. Techniques like convolutional neural networks (CNNs) are used to learn image embeddings, which can be used for image classification, object detection, image retrieval, and more. Video embeddings can capture temporal information and are useful for tasks like action recognition and video summarization.
4. Anomaly Detection: Embeddings can be used to detect anomalies in data. By learning embeddings that capture normal patterns, deviations from the normal behavior can be identified as anomalies. This approach is commonly used in fraud detection, network intrusion detection, and outlier detection.
5. Knowledge Graphs: Embeddings are employed in knowledge graph applications to represent entities and relationships in a graph structure. Graph embeddings enable efficient similarity calculations and can be used for tasks like entity linking, link prediction, and graph-based recommendation systems.
6. Sequence Modeling: Embeddings are utilized in sequence modeling tasks, such as natural language generation, machine translation, and speech recognition. Sequence embeddings capture dependencies and context in sequential data, enabling the model to understand and generate meaningful sequences.

Code Problem - Movie Recommendations

We want to implement a system that recommends movies to a user based on a list of watched movies. We need an EmbeddingLayer class responsible for generating and retrieving embeddings. We need a Movie class to represent a movie with an ID and a title. We need a RecommenderSystem class that calls a recommendMovie function for a specific user, passing their ID and the list of movies they've already watched. The recommendMovie function takes a user ID and a list of watched movies and recommends a movie based on a users embeddings and similarity metric. The code is seen below.
Movie.h,
EmbeddingLayer.h,
RecommenderSystem.h,
MovieMain.cpp.

Code Problem - Predicting Financial Data

The following program uses historical prices as well as weights to predict a stock price for a given day. The result is a dot product of the two vectors (historical prices, weights).
VectorOperations.h, vector dot product
FinancialData.h,
StockPredictionModel.h, contains the embedded data
FinancialDataMain.cpp.

The Rebalancing Design Pattern

The Rationale

The rebalancing machine learning design pattern, also known as class rebalancing or data rebalancing, is employed in machine learning to address class imbalance issues in datasets. Class imbalance refers to a situation where the number of samples in different classes of a classification problem is significantly imbalanced, with one class having a much larger number of instances than the others.

The UML Diagram

Here is a simple UML diagram for the rebalancing design pattern:

  _______________                 ________________
  |    Dataset   |<>------------>|   Rebalancer   |
  |______________|               |________________|
  | - data       |               | - rebalance()  |
  | - labels     |               | - get_data()   |
  | - num_classes|               | - get_labels() |
  |______________|               |________________|
          ^
          |
          |
  _____________________
  |   BaseModel       |
  |___________________|
  | - train()         |
  | - predict()       |
  | - evaluate()      |
  |___________________|
         ^
         |
         |
  _____________________
  |  RebalancedModel  |
  |___________________|
  | - rebalancer      |
  | - train()         |
  | - predict()       |
  | - evaluate()      |
  |___________________|

1. The Dataset class represents the original dataset with its associated features (data) and labels (labels). It also maintains information about the number of classes in the dataset (num_classes).
2. The Rebalancer class is responsible for rebalancing the dataset. It contains methods such as rebalance() to perform the rebalancing operation, and get_data() and get_labels() to retrieve the rebalanced data and labels, respectively.
3. The BaseModel class represents the base machine learning model that can be trained, used for prediction, and evaluated. It encapsulates common functionalities like train(), predict(), and evaluate().
4. The RebalancedModel class extends the BaseModel class and introduces a rebalancer object, which is an instance of the Rebalancer class. It utilizes the rebalanced data and labels obtained from the rebalancer during the training, prediction, and evaluation processes.

Code Example - Rebalancing design pattern

The following is a simple code example of the rebalancing design pattern:
C++: Rebalancing.cpp.
C#: Rebalancing.cs.
Java: Rebalancing.java.
Python: Rebalancing.py.

Common Usage

The rebalancing design pattern is commonly used in various domains within the software industry where dealing with imbalanced datasets is a challenge. The following are some common usages of the rebalancing design pattern:

1. Fraud detection: In fraud detection systems, the number of fraudulent instances is typically significantly lower than the number of non-fraudulent instances. Rebalancing techniques can be applied to ensure that the model is trained on a balanced dataset, improving the accuracy of fraud detection.
2. Medical diagnosis: Medical datasets often suffer from class imbalance, where the number of instances belonging to certain rare medical conditions is much smaller than others. Rebalancing the dataset can help prevent the model from being biased towards the majority class and improve the accuracy of diagnosis for rare conditions.
3. Anomaly detection: Anomaly detection involves identifying rare events or outliers in a dataset. Rebalancing can be useful in scenarios where the anomalies are significantly underrepresented compared to normal instances. By rebalancing the dataset, the model can be trained to better detect and classify anomalies.
4. Credit risk assessment: When evaluating credit risk, the occurrence of default events is usually low compared to non-default events. By rebalancing the dataset, credit risk models can be trained to account for the imbalanced nature of default instances, leading to more accurate risk assessment.

Code Problem - SMOTE rebalancer (simple)

The rebalancing design pattern in machine learning involves adjusting the class distribution in the training data to handle imbalanced datasets. Here's an example in C++ that demonstrates the rebalancing design pattern using the Synthetic Minority Over-sampling Technique SMOTE to handle imbalanced data. In a real - world scenario, you would need to integrate a more sophisticated SMOTE algorithm or other methods to effectively rebalance the data before training your machine learning models.

The code is seen below:
DataSample.h,
Rebalancer.h,
RebalanceMain.cpp.

Code Problem - SMOTE algorithm

The following code contains pseudocode for the SMOTE algorithm:
Sample.h,
Smote.h,
SmoteMain.cpp.

Code Problem - SMOTE rebalancer (complex)

As above, the Rebalancer class uses the SMOTE algorithm to generate synthetic samples for the minority classes in the dataset. The rebalanced dataset is then used to train the base model (DecisionTreeModel) using the RebalancedModel class. The rebalanced model can then be used for predictions.

The code is seen below:
Dataset.h,
Rebalancer.h,
Rebalancer.cpp,
BaseModel.h,
DecisionTreeModel.h,
RebalancedModel.h,
RebalPredictor.cpp.