Normalization in Machine Learning — Science and Data

2023-08-30 05:14:00

Data normalization is a technique often used in machine learning to transform different variables and place them on the same common scale.

This is particularly important for models that use gradient descent or optimization-related methods, as well as models that are sensitive to the scale of input variables.

Here are two possibilities for normalization:

Normalization By Mean

Normalization by mean subtraction centers the data around zero. This technique maintains the original shape of the data distribution without changing the variance between data points. However, the data range still varies and can be large, which can be a problem for some machine learning models.

The advantage of this approach is its simplicity and preservation of the shape of the original distribution. However, it may not be the best choice if the model is sensitive to the scale of the input variables.

This is the formula:

Normalization by Standard Scaler (Average and Standard Deviation)

Normalization using mean and standard deviation (known as “z-score normalization” or StandardScaler in libraries like scikit-learn) not only centers the data, but also scales it such that it has a standard deviation of 1.

This can be particularly useful for models that are sensitive to the scales of the input variables, because it ensures that no variable has a disproportionate effect on the model due to its scale.

By subtracting the mean and dividing by the standard deviation, you are assuming that your data is approximately normal (Gaussian distribution). If your data are not normal, this approach may not be the best.

This is the formula:

Which one to use?

The choice between these two approaches depends on the context:

If the scale of the variables is important and you are using a scale-sensitive model (such as SVMs, k-NN or neural networks), then it is more appropriate to use the Standard Scaler.

If you are more interested in keeping the original shape of the distribution and only want to change the center point, then normalizing to the mean may be sufficient.

Both techniques have their own advantages and disadvantages and the choice usually depends on the specific needs of the modeling task at hand.

David Matos

Related