Why we use standard deviation , not absolute deviation

We use standard deviation in finance and variance in calculating loss functions in machine learning. Why don't we use absolute deviation instead?

In statistics, when you have large amount of sample data, you need some measures to interpret information from the data. Average/Mean is one such measure. This gives the figure, if you equally distribute the resource on each quantity, what each one will get. Mean helps when we compare two entities of different sizes. You can compare average salary of two companies, average cost of living in two cities, life expectancy of advanced and developing countries, per capita income of two countries etc.

But mean does not give any idea on how data are dispersed. Income inequality in US is a hot economic /political issue. There are statistics to show how top 1 % of population gets lion's share of income and how bottom 20% makes quite less. To capture such deviations, we need to have some measures that convey how data are distributed. ( Gini index is a common measure for income inequality in a country)

 For capturing the dispersion, we can use absolute mean deviation. Take every sample data , calculate how much distance is this located from the mean. Then, calculate average for all the distances/differences.

Let x_i denote sample data

Absolute mean deviation = Sum (abs(x_i-mean))/n

 This gives a single number which gives indication of the dispersion of all data points. Let’s consider the following set of data

 

50 50 50 50 ->  mean/avg =50 ; abs mean deviation = (|50-50| + |50-50| + |50-50| + |50-50|) /4 =0

46 54 46 54 -> mean/avg  =50 ; abs mean deviation =  (|46-50| +|54-50| +| 46-50| + |54-50|) /4 = 4

42 58 50 50 -> mean/avg  =50 ; abs mean deviation = (|42-50| + |58-50| + |50-50| +|50-50|) /4  = 4

 

The first two rows indicate the usefulness of absolute mean deviation, if you compare 2nd and 3rd rows, you can notice that even if some quantities are widely deviated from mean, absolute mean deviation is equal. This is one of the reasons why we go for standard deviation. It involves squaring of differences. What a squaring a number does? For small quantity ( <1) squaring make the number even smaller. For larger quantities (>1) squaring the number makes it larger. This is what we exactly want. We can tolerate or ignore small deviations from the mean but we want to capture even single instance of big deviation. 

 

To calculate standard deviation, in the first step variance is calculated, by squaring the differences from mean and calculating the average of squared differences.

                 V= Sum( (x_i-avg)^2) / n

Squaring , acts as a weight on each quantity and captures the wide deviation from mean. Since we square it, variance would be a huge number. To bring it back to the original unit, we take square root of it.

 

Standard deviation of 2nd row = sqrt ((4^2+4^2+4^2+4^2)/4) =4

Standard deviation of 3rd row = sqrt ((8^2+8^2+0+0)/4)=5.66

 

One more reason to use standard deviation, is once you calculate it, you can come to a conclusion on how data will be distributed around the mean. We will see this, when I write about Normal distribution. Standard deviation is used as a measure of volatility in finance. The more standard deviation ( the more is volatility) of the daily/annual returns , the more risky the asset is.