How does covariance work

PhD PhD I'd really like to know Scaling the x- and y-units by SD lead to a more interpretable summary, as discussed in the ensuing thread. Website with video Specifically this image:! Add a comment. Active Oldest Votes. Now we're ready for the explanation of covariance. Improve this answer. And very very clear. They're all red so the "covariance" is 6.

Is this right? Incidentally when I teach covariance I also use the "positive and negative rectangles" approach, but pairing each data points with the mean point. I find this makes some of the standard formulae more accessible, but on the whole I prefer your method. That would make the explanation inaccessible to the five-year-old with a box of crayons. Some of the conclusions I drew at the end would not be immediate, either. For example, it would no longer be quite so obvious that the covariance is sensitive to certain kinds of outliers.

Show 31 more comments. Peter Flom Peter Flom Also, does it apply for inverse relations too i. I tried to give a one sentence answer. Yours is much more complete. Even your "how two variables change together" is more complete, but, I think, a little harder to understand. Those are neither covariance whose unit of measure is the product of the units for the two variables nor correlation which is unitless. In this post Peter has confused the covariance with a regression coefficient of which there are two, by the way, and they usually are different.

And that returns us to the beginning: how would one convey this precise definition to the proverbial five-year-old? As always, there's a trade-off between economy of expression and accuracy: when the audience doesn't have the concepts or language needed to understand something immediately, somehow you have to weave in an explanation of that background along with your description. Doing it right requires some elaboration. Usually there's no shortcut.

Show 4 more comments. If the graph has lots more green than red, it means that y generally increases when x increases. If the graph has lots more red than green, it means that y generally decreases when x increases.

If the graph isn't dominated by one colour or the other, it means that there isn't much of a pattern to how x and y relate to each other. It's different in each of the 4 graphs as they are of different sets of data.

One confusing thing about this is that you have sort of flipped although not exactly the color scheme from whuber's answer. In trying to wrap my head around the concept, that gave some cognitive whiplash.

Did you use Python to draw it? If you did, can you kindly share your code, please? I'm doing so in case if that site shuts down or the page gets taken down when someone eons from now accesses this post ; Covariance is a measure of how much two variables change together.

Adding another one by 'CatofGrey' that helps augment the intuition: In probability theory and statistics, covariance is the measure of how much two random variables vary together as distinct from variance, which measures how much a single variable varies. Also, neither addresses the important aspect that covariance depends linearly on the scale of each variable.

Covariance uses rectangles to describe how far away an observation is from the mean on a scatter graph: If a rectangle has long sides and a high width or short sides and a short width, it provides evidence that the two variables move together.

The link is interesting. Since it has no archive on the Wayback machine it likely is new. Because it so closely parallels my three-year-old answer, right down to the choice of red for positive and blue for negative relationships, I suspect it is an unattributed derivative of the material on this site.

But if you are referring to the uniform distribution, there's nothing to calculate, because as I recall remarking in your thread at stats. Show 3 more comments. Anyone interested in improving this Kevin Wright Kevin Wright 4 4 silver badges 8 8 bronze badges. Michael R. Chernick Michael R. Chernick X and Y could have a strong quadratic relationship but have a correlation of zero.

Kingz Kingz 4 4 bronze badges. Adam Adam 10 10 silver badges 16 16 bronze badges. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. However, they would have a Covariance of 0 because their relationship is not linear. This is an important distinction to remember because it is easy to automatically apply, but not necessarily true in all cases.

We can confirm this strange result in R by creating a vector x that takes on the integers from to 10 and a vector y that squares all of the values in the x vector. However, despite being dependent, they have a Covariance of 0 so, a Covariance of 0 does not imply independence.

The reason that we cover Correlation second is that we define it in terms of Covariance i. So, Correlation is the Covariance divided by the standard deviations of the two random variables. Of course, you could solve for Covariance in terms of the Correlation; we would just have the Correlation times the product of the Standard Deviations of the two random variables. Consider the Correlation of a random variable with a constant. We know, by definition, that a constant has 0 variance again, for example, the constant 3 is always 3 , which means it also has a standard deviation of 0 standard deviation is the square root of variance.

So, if we tried to solve for the Correlation between a constant and a random variable, we would be dividing by 0 in the calculation, and we get something that is undefined. First of all, the most important part of Correlation is that it is bounded between -1 and 1. What exactly does this interval mean? The idea here is as the Correlation grows in magnitude away from 0 in either direction it means that the two variables have a stronger and stronger relationship.

Like the Covariance, the sign of the Correlation indicates the direction of the relationship: positive means that random variables move together, negative means that random variables move in different directions. The endpoints, -1 and 1, indicate that there is a perfect relationship between the two variables.

For example, the relationship between feet and inches is always that 12 inches equals 1 foot. If you plotted this relationship, it would be an exactly perfect line; the two variables relate to each other totally and completely in this fashion of course, feet only equals inches scaled to a specific factor - here, the factor is 12 - but the relationship is still a perfect mapping.

Therefore, the Correlation is 1. Now consider the other extreme on the bounds of Correlation. A Correlation of 0 means that there is no linear relationship between the two variables. We already know that if two random variables are independent, the Covariance is 0. We can see that if we plug in 0 for the Covariance to the equation for Correlation, we will get a 0 for the Correlation.

Therefore, again, independence in terms of random variables implies a Correlation of 0. However, again, the reverse is not necessarily true.

Just as a quick note, you might see artificially high Correlations. Anyways, these topics will come up in discussions with more applied tilts. Remember that with the Covariance scales arbitrarily, so we can never be sure how strong a Covariance is by looking at the magnitude. This is a nice segue into the next reason why we prefer Correlation: it is unitless, which is exactly why we can rely on the magnitude to inform us as to the strength of the relationship.

This serves to cancel units, and we are left with a value Correlation that does not change when units change! Basically, this means that the relationship between inches and feet will stay the same if we are considering the relationship between 3 inches and 3 feet multiplied by three , not just inches and feet in general. We can confirm in R again that scaling the feet and inches by a constant does not change the fact that Correlation is 1. Finally, although this is not necessarily a concept central to this book, it would be irresponsible to talk about Correlation without also talking about causation.

Causation, of course, means that changes in one variable cause a change in another variable. Unfortunately, as you have likely heard from your high school science classes, Correlation alone cannot make that strong a claim. That is, a high Correlation between two random variables indicates that the two are associated , but that their relationship is not necessarily causal in nature. Still confused? Another classic example is considering the Correlation between number of old-fashioned pirates i.

They are very strongly negatively correlated: while the number of swashbucklers has decreased and dwindled over the years, global temperatures are on the climb. The only way to prove causality is with controlled experiments, where we eliminate outside variables and isolate the effects of the two variables in question. The point and, again, this is not a primary area of concern in this predominantly theoretical book is to not jump to conclusions when we observe two random variables with high Correlations.

Usually, we apply LoTUS when we see a transformation both single and multidimensional , which we use to find the expectation of a function of a random variable or, a random variable transformed.

However, now we will be interested in not just finding the expectation of the transformation of a random variable, but the distribution of the transformation of a random variable.

This is intuitive if you consider a random variable in the official sense: a mapping of some experiment to the real line. A function of this random variable, then, is just a function of the mapping, and essentially a new mapping to the real line, or a new random variable! This formula can seem a little confusing at first, just because of all of the different notation. We can break this calculation down into two steps:.

Consider the fact that we cited earlier in the book: linear combinations of Normal random variables are still Normal random variables, and the sums of Normal random variables are still Normal random variables. Now, armed with our knowledge of transformations, we can explore this concept a bit more and provide the proof for a fact that we took for granted earlier.

As we showed in an earlier chapter, subtracting the mean from a Normal random variable and dividing by the standard deviation yields the Standard Normal again, according to the fact we cited earlier, the linear transformation is still Normal, and we can calculate the mean and variance, which come out to be 0 and 1.

We know the PDF of a Normal by now, so we write:. This is starting to look more and more recognizable. Now, we can multiply this term by the first part:. So, remember that even if transformations look complicated, you can break the calculation down into two steps: plugging into the PDF, and then taking a derivative.

So how do we decide what to use: the correlation matrix or the covariance matrix? Now let's look at some examples. We can see that all the columns are numerical and hence, we can move forward with analysis. For that, we set the scale option to false:. Here, cars. So, prcomp returns five key measures: sdev, rotation, center, scale and x. The center and scale provide the respective means and standard deviation of the variables that we used for normalization before implementing PCA.

In other words, sdev shows the square roots of the eigenvalues. The rotation matrix contains the principal component loading. This is the most important result of the function. We can represent the component loading as the correlation of a particular variable on the respective PC principal component. It can assume both positive or negative. The higher the loading value, the higher the correlation. To read this chart, look at the extreme ends top, down, left and right.

We can finish this analysis with a summary of the PCA with the covariance matrix:. All other principal components have progressively lower contribution.

For this, all we need to do is set the scale argument to true. Using the same definitions for all the measures above, we now see that the scale measure has values corresponding to each variable. We can observe the rotation matrix in a similar way along with the plot. This plot looks more informative. One significant change we see is the drop in the contribution of PC1 to the total variation.

It dropped from On the other hand, the contribution of PC2 has increased from 7 percent to 22 percent. Furthermore, the component loading values show that the relationship between the variables in the data set is way more structured and distributed. We can see another significant difference if you look at the standard deviation values in both results above.

The values from PCA while using the correlation matrix are closer to each other and more uniform as compared to the analysis using the covariance matrix. The analysis with the correlation matrix definitely uncovers better structure in the data and relationships between variables. Using the above example we can conclude that the results differ significantly when one tries to define variable relationships using covariance and correlation. This in turn, affects the importance of the variables we compute for any further analyses.

The selection of predictors and independent variables is one prominent application of such exercises. The similarities fractional differences reinforce our understanding that correlation matrix is just a scaled derivative of the covariance matrix. Any computation on these matrices should now yield the same or similar results. As expected, the results from all three relationship matrices are the same.

The charts resemble each other.