Skip to content

Training techniques

Techniques that wrap a training run rather than perform it: stopping it at the right moment, and verifying that the gradients driving it are correct.

early_stopping names both a module and the one function inside it, so the function is reached through the module path rather than the package root.

dlhub.training

Training Techniques

Techniques that wrap a training run rather than perform it: stopping it at the right moment, and verifying that the gradients driving it are correct.

Author

Deep Learning Reference Hub

License

MIT

dictionary_to_vector 

dictionary_to_vector(parameters: dict[str, ndarray]) -> tuple[np.ndarray, dict[str, tuple]]

Convert parameter dictionary to a single vector while preserving shape information.

Parameters:

Name Type Description Default
parameters dict[str, ndarray]

Dictionary with parameter names as keys and numpy arrays as values

 required

Returns:

Type Description
tuple[np.ndarray, dict[str, tuple]]: (theta, shapes) where:
  • theta: Single column vector containing all parameters
  • shapes: Dictionary mapping parameter names to their original shapes
Source code in src/dlhub/training/gradient_checking.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def dictionary_to_vector(
    parameters: dict[str, np.ndarray],
) -> tuple[np.ndarray, dict[str, tuple]]:
    """
    Convert parameter dictionary to a single vector while preserving shape information.

    Parameters
    ----------
    parameters : dict[str, np.ndarray]
        Dictionary with parameter names as keys and numpy arrays as values

    Returns
    -------
    tuple[np.ndarray, dict[str, tuple]]: (theta, shapes) where:
        - theta: Single column vector containing all parameters
        - shapes: Dictionary mapping parameter names to their original shapes
    """
    shapes = {}
    theta = None

    for key in sorted(parameters.keys()):  # Sort for consistent ordering
        shapes[key] = parameters[key].shape

        param_vector = np.reshape(parameters[key], (-1, 1))
        if theta is None:
            theta = param_vector
        else:
            theta = np.concatenate((theta, param_vector), axis=0)

    return theta, shapes

gradient_check 

gradient_check(parameters: dict[str, ndarray], gradients: dict[str, ndarray], X: ndarray, Y: ndarray, cost_function: Callable[[ndarray, ndarray, dict[str, ndarray]], float], epsilon: float = 1e-07) -> float

Perform gradient checking to verify analytical gradients against numerical gradients.

Parameters:

Name Type Description Default
parameters dict

Dictionary of parameters (e.g., {'W1': array, 'b1': array, ...})

 required
gradients dict

Dictionary of computed analytical gradients

 required
X ndarray

Input data

 required
Y ndarray

True labels

 required
cost_function callable

Function that computes cost given (X, Y, parameters)

 required
epsilon float

Small value for numerical differentiation

 1e-7

Returns:

Type Description
float

Relative difference between numerical and analytical gradients - < 1e-7: Excellent (gradients are likely correct) - < 1e-5: Good (gradients are probably correct) - < 1e-3: Acceptable (check implementation) - > 1e-3: Poor (likely bug in gradient computation)
Source code in src/dlhub/training/gradient_checking.py 
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def gradient_check(
    parameters: dict[str, np.ndarray],
    gradients: dict[str, np.ndarray],
    X: np.ndarray,
    Y: np.ndarray,
    cost_function: Callable[[np.ndarray, np.ndarray, dict[str, np.ndarray]], float],
    epsilon: float = 1e-7,
) -> float:
    """
    Perform gradient checking to verify analytical gradients against numerical gradients.

    Parameters
    ----------
    parameters : dict
        Dictionary of parameters (e.g., {'W1': array, 'b1': array, ...})
    gradients : dict
        Dictionary of computed analytical gradients
    X : np.ndarray
        Input data
    Y : np.ndarray
        True labels
    cost_function : callable
        Function that computes cost given (X, Y, parameters)
    epsilon : float, default=1e-7
        Small value for numerical differentiation

    Returns
    -------
    float
        Relative difference between numerical and analytical gradients
           - < 1e-7: Excellent (gradients are likely correct)
           - < 1e-5: Good (gradients are probably correct)
           - < 1e-3: Acceptable (check implementation)
           - > 1e-3: Poor (likely bug in gradient computation)
    """
    params_vector, param_shapes = dictionary_to_vector(parameters)
    grad_vector, _ = dictionary_to_vector(gradients)

    num_parameters = params_vector.shape[0]
    gradapprox = np.zeros((num_parameters, 1))

    # Each parameter is shifted in a fresh copy of the vector, so the caller's
    # `parameters` is never left holding a perturbed value.
    for i in range(num_parameters):
        theta_plus = np.copy(params_vector)
        theta_plus[i] = theta_plus[i] + epsilon
        J_plus = cost_function(X, Y, vector_to_dictionary(theta_plus, param_shapes))

        theta_minus = np.copy(params_vector)
        theta_minus[i] = theta_minus[i] - epsilon
        J_minus = cost_function(X, Y, vector_to_dictionary(theta_minus, param_shapes))

        gradapprox[i] = (J_plus - J_minus) / (2 * epsilon)  # Numerical Gradient

    # Relative Difference Computation
    numerator = np.linalg.norm(grad_vector - gradapprox)
    denominator = np.linalg.norm(grad_vector) + np.linalg.norm(gradapprox)

    if denominator == 0:
        return 0.0
    difference = numerator / denominator

    print("Gradient Check Results:")
    print(f"  Numerical gradient norm: {np.linalg.norm(gradapprox):.6f}")
    print(f"  Analytical gradient norm: {np.linalg.norm(grad_vector):.6f}")
    print(f"  Relative difference: {difference:.2e}")

    if difference < 1e-7:
        print("  ✅ Excellent! Gradients are likely correct.")
    elif difference < 1e-5:
        print("  ✅ Good! Gradients are probably correct.")
    elif difference < 1e-3:
        print("  ⚠️  Acceptable, but check your implementation.")
    else:
        print("  ❌ Poor! Likely bug in gradient computation.")

    return difference

vector_to_dictionary 

vector_to_dictionary(theta: ndarray, shapes: dict[str, tuple]) -> dict[str, np.ndarray]

Convert a parameter vector back to dictionary format using stored shapes.

Parameters:

Name Type Description Default
theta ndarray

Column vector containing all parameters

 required
shapes dict[str, tuple]

Dictionary mapping parameter names to their original shapes

 required

Returns:

Type Description
dict[str, ndarray]

Dictionary with parameter names as keys and reshaped arrays as values
Source code in src/dlhub/training/gradient_checking.py 
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
def vector_to_dictionary(
    theta: np.ndarray, shapes: dict[str, tuple]
) -> dict[str, np.ndarray]:
    """
    Convert a parameter vector back to dictionary format using stored shapes.

    Parameters
    ----------
    theta : np.ndarray
        Column vector containing all parameters
    shapes : dict[str, tuple]
        Dictionary mapping parameter names to their original shapes

    Returns
    -------
    dict[str, np.ndarray]
        Dictionary with parameter names as keys and reshaped arrays as values
    """
    parameters = {}
    start = 0

    for key in sorted(shapes.keys()):
        shape = shapes[key]
        size = np.prod(shape)  # Total number of elements

        parameters[key] = theta[start : start + size].reshape(shape)
        start += size

    return parameters

early_stopping

Early Stopping Utility

Implements a mechanism to halt model training when a monitored validation metric (such as loss or accuracy) ceases to improve after a specified number of epochs.

This prevents overfitting and saves compute by terminating training once performance plateaus.

References
  • Prechelt, L. (2012). Early Stopping — But When? In Neural Networks: Tricks of the Trade. Springer. https://link.springer.com/chapter/10.1007/978-3-642-35289-8_5
Author

Deep Learning Reference Hub

License

MIT License

Notes
  • Works by monitoring "no improvement" for patience consecutive epochs.
  • An epoch improves only if it beats the running best by more than min_delta. So best_loss is the best value that cleared that bar, which is not always the smallest value in the history: under a large min_delta, a slow drift downward never clears it and the recorded best stays where it was. This is the same rule Keras applies, and it is what makes min_delta a noise filter rather than a number that only appears in the report.

early_stopping 

early_stopping(val_losses: list[float], patience: int = 10, min_delta: float = 0.0001, verbose: bool = True) -> tuple[bool, dict]

Early stopping with detailed tracking and optional verbose output.

Parameters:

Name Type Description Default
val_losses list

Validation losses from training history

 required
patience int

Number of epochs to wait after last improvement

 10
min_delta float

Minimum decrease that counts as an improvement. A loss that falls by less than this is treated as noise, so it does not reset the patience counter.

 1e-4
verbose bool

Whether to print detailed information

 True

Returns:

Type Description
tuple[bool, dict]

whether to stop and info dict, which contains: - 'best_loss': Best validation loss seen so far - 'best_epoch': Epoch that produced it - 'epochs_since_improvement': Number of epochs since last improvement - 'current_loss': Most recent validation loss - 'improvement_needed': Loss the next epoch must beat to count - 'patience_remaining': Epochs left before stopping
Source code in src/dlhub/training/early_stopping.py 
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def early_stopping(
    val_losses: list[float],
    patience: int = 10,
    min_delta: float = 1e-4,
    verbose: bool = True,
) -> tuple[bool, dict]:
    """
    Early stopping with detailed tracking and optional verbose output.

    Parameters
    ----------
    val_losses : list
        Validation losses from training history
    patience : int, default=10
        Number of epochs to wait after last improvement
    min_delta : float, default=1e-4
        Minimum decrease that counts as an improvement. A loss that falls by less
        than this is treated as noise, so it does not reset the patience counter.
    verbose : bool, default=True
        Whether to print detailed information

    Returns
    -------
        tuple[bool, dict]
            whether to stop and info dict, which contains:
            - 'best_loss': Best validation loss seen so far
            - 'best_epoch': Epoch that produced it
            - 'epochs_since_improvement': Number of epochs since last improvement
            - 'current_loss': Most recent validation loss
            - 'improvement_needed': Loss the next epoch must beat to count
            - 'patience_remaining': Epochs left before stopping
    """
    if len(val_losses) < 2:
        return False, {"message": "Need at least 2 epochs to evaluate"}

    # Scanned rather than taken as min(val_losses), because `min_delta` decides
    # what counts as an improvement. A run that drifts down by less than
    # min_delta each epoch has a new minimum every epoch and would never stop,
    # which is the noise this parameter exists to reject.
    best_loss = val_losses[0]
    best_epoch = 0
    for epoch, loss in enumerate(val_losses[1:], start=1):
        if loss < best_loss - min_delta:
            best_loss = loss
            best_epoch = epoch

    current_epoch = len(val_losses) - 1
    epochs_since_improvement = current_epoch - best_epoch
    current_loss = val_losses[-1]
    improvement_needed = best_loss - min_delta

    info = {
        "best_loss": best_loss,
        "best_epoch": best_epoch,
        "epochs_since_improvement": epochs_since_improvement,
        "current_loss": current_loss,
        "improvement_needed": improvement_needed,
        "patience_remaining": max(0, patience - epochs_since_improvement),
    }

    should_stop = epochs_since_improvement >= patience

    if verbose:
        print("Early Stopping Check:")
        print(f"  Current Loss: {current_loss:.6f}")
        print(f"  Best Loss: {best_loss:.6f} (epoch {best_epoch})")
        print(f"  Epochs since improvement: {epochs_since_improvement}")
        print(f"  Patience remaining: {info['patience_remaining']}")
        print(f"  Should stop: {should_stop}")

    return should_stop, info

gradient_checking

Gradient Checking Utility

Provides implementation of gradient checking for neural networks: numerically approximates gradients via finite differences and compares them to analytical gradients from backpropagation.

This helps validate correctness of gradient computations and debug implementation errors.

References
  • Karpathy, A. (n.d.). Numerical Limits and Gradient Checking, in "CS231n". Stanford University.
  • Ng, A. (2017). Deep Learning Specialization: Week 3 – Gradient Checking.
Author

Deep Learning Reference Hub

License

MIT License

Notes
  • Compute numerical gradient using ε-shift method: (J(θ+ε) - J(θ-ε)) / (2ε).
  • Compare with backward-mode gradients using relative difference metric.
  • Use small ε (e.g. 1e-7), and expect relative difference < 1e-7.

gradient_check 

gradient_check(parameters: dict[str, ndarray], gradients: dict[str, ndarray], X: ndarray, Y: ndarray, cost_function: Callable[[ndarray, ndarray, dict[str, ndarray]], float], epsilon: float = 1e-07) -> float

Perform gradient checking to verify analytical gradients against numerical gradients.

Parameters:

Name Type Description Default
parameters dict

Dictionary of parameters (e.g., {'W1': array, 'b1': array, ...})

 required
gradients dict

Dictionary of computed analytical gradients

 required
X ndarray

Input data

 required
Y ndarray

True labels

 required
cost_function callable

Function that computes cost given (X, Y, parameters)

 required
epsilon float

Small value for numerical differentiation

 1e-7

Returns:

Type Description
float

Relative difference between numerical and analytical gradients - < 1e-7: Excellent (gradients are likely correct) - < 1e-5: Good (gradients are probably correct) - < 1e-3: Acceptable (check implementation) - > 1e-3: Poor (likely bug in gradient computation)
Source code in src/dlhub/training/gradient_checking.py 
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def gradient_check(
    parameters: dict[str, np.ndarray],
    gradients: dict[str, np.ndarray],
    X: np.ndarray,
    Y: np.ndarray,
    cost_function: Callable[[np.ndarray, np.ndarray, dict[str, np.ndarray]], float],
    epsilon: float = 1e-7,
) -> float:
    """
    Perform gradient checking to verify analytical gradients against numerical gradients.

    Parameters
    ----------
    parameters : dict
        Dictionary of parameters (e.g., {'W1': array, 'b1': array, ...})
    gradients : dict
        Dictionary of computed analytical gradients
    X : np.ndarray
        Input data
    Y : np.ndarray
        True labels
    cost_function : callable
        Function that computes cost given (X, Y, parameters)
    epsilon : float, default=1e-7
        Small value for numerical differentiation

    Returns
    -------
    float
        Relative difference between numerical and analytical gradients
           - < 1e-7: Excellent (gradients are likely correct)
           - < 1e-5: Good (gradients are probably correct)
           - < 1e-3: Acceptable (check implementation)
           - > 1e-3: Poor (likely bug in gradient computation)
    """
    params_vector, param_shapes = dictionary_to_vector(parameters)
    grad_vector, _ = dictionary_to_vector(gradients)

    num_parameters = params_vector.shape[0]
    gradapprox = np.zeros((num_parameters, 1))

    # Each parameter is shifted in a fresh copy of the vector, so the caller's
    # `parameters` is never left holding a perturbed value.
    for i in range(num_parameters):
        theta_plus = np.copy(params_vector)
        theta_plus[i] = theta_plus[i] + epsilon
        J_plus = cost_function(X, Y, vector_to_dictionary(theta_plus, param_shapes))

        theta_minus = np.copy(params_vector)
        theta_minus[i] = theta_minus[i] - epsilon
        J_minus = cost_function(X, Y, vector_to_dictionary(theta_minus, param_shapes))

        gradapprox[i] = (J_plus - J_minus) / (2 * epsilon)  # Numerical Gradient

    # Relative Difference Computation
    numerator = np.linalg.norm(grad_vector - gradapprox)
    denominator = np.linalg.norm(grad_vector) + np.linalg.norm(gradapprox)

    if denominator == 0:
        return 0.0
    difference = numerator / denominator

    print("Gradient Check Results:")
    print(f"  Numerical gradient norm: {np.linalg.norm(gradapprox):.6f}")
    print(f"  Analytical gradient norm: {np.linalg.norm(grad_vector):.6f}")
    print(f"  Relative difference: {difference:.2e}")

    if difference < 1e-7:
        print("  ✅ Excellent! Gradients are likely correct.")
    elif difference < 1e-5:
        print("  ✅ Good! Gradients are probably correct.")
    elif difference < 1e-3:
        print("  ⚠️  Acceptable, but check your implementation.")
    else:
        print("  ❌ Poor! Likely bug in gradient computation.")

    return difference

dictionary_to_vector 

dictionary_to_vector(parameters: dict[str, ndarray]) -> tuple[np.ndarray, dict[str, tuple]]

Convert parameter dictionary to a single vector while preserving shape information.

Parameters:

Name Type Description Default
parameters dict[str, ndarray]

Dictionary with parameter names as keys and numpy arrays as values

 required

Returns:

Type Description
tuple[np.ndarray, dict[str, tuple]]: (theta, shapes) where:
  • theta: Single column vector containing all parameters
  • shapes: Dictionary mapping parameter names to their original shapes
Source code in src/dlhub/training/gradient_checking.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def dictionary_to_vector(
    parameters: dict[str, np.ndarray],
) -> tuple[np.ndarray, dict[str, tuple]]:
    """
    Convert parameter dictionary to a single vector while preserving shape information.

    Parameters
    ----------
    parameters : dict[str, np.ndarray]
        Dictionary with parameter names as keys and numpy arrays as values

    Returns
    -------
    tuple[np.ndarray, dict[str, tuple]]: (theta, shapes) where:
        - theta: Single column vector containing all parameters
        - shapes: Dictionary mapping parameter names to their original shapes
    """
    shapes = {}
    theta = None

    for key in sorted(parameters.keys()):  # Sort for consistent ordering
        shapes[key] = parameters[key].shape

        param_vector = np.reshape(parameters[key], (-1, 1))
        if theta is None:
            theta = param_vector
        else:
            theta = np.concatenate((theta, param_vector), axis=0)

    return theta, shapes

vector_to_dictionary 

vector_to_dictionary(theta: ndarray, shapes: dict[str, tuple]) -> dict[str, np.ndarray]

Convert a parameter vector back to dictionary format using stored shapes.

Parameters:

Name Type Description Default
theta ndarray

Column vector containing all parameters

 required
shapes dict[str, tuple]

Dictionary mapping parameter names to their original shapes

 required

Returns:

Type Description
dict[str, ndarray]

Dictionary with parameter names as keys and reshaped arrays as values
Source code in src/dlhub/training/gradient_checking.py 
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
def vector_to_dictionary(
    theta: np.ndarray, shapes: dict[str, tuple]
) -> dict[str, np.ndarray]:
    """
    Convert a parameter vector back to dictionary format using stored shapes.

    Parameters
    ----------
    theta : np.ndarray
        Column vector containing all parameters
    shapes : dict[str, tuple]
        Dictionary mapping parameter names to their original shapes

    Returns
    -------
    dict[str, np.ndarray]
        Dictionary with parameter names as keys and reshaped arrays as values
    """
    parameters = {}
    start = 0

    for key in sorted(shapes.keys()):
        shape = shapes[key]
        size = np.prod(shape)  # Total number of elements

        parameters[key] = theta[start : start + size].reshape(shape)
        start += size

    return parameters