Do you understand statistical significance and how it's important to science?

1: I think that is a great summary, highlighting how the relationship between uncertainty and sample size is integral to our understanding of the data we obtain.

2: I have a good understanding of the basic statistics required up to the level I last studied (post grad enviro science). Much of the discussion by others has been around null hypothesis testing. There is a push to discard this altogether in environmental science, as was done with health sciences decades ago. The reason for this is that as David B outlined, the test statistic only tells you the probability that you would get the data you got if the null hypothesis was true. It doesn't tell you anything about whether your alternative hypothesis.

Bayesian methods which were too complex for environmental science until recently (now that computers are much more powerful) can be used instead. In such a cases you do not use a null hypothesis, rather you use measurements of the uncertainty to determine statistical significance. In health sciences they use risk ratios and odds ratios, in enviro science the methods are more complex and I can't actually explain them (I only just barely understand them enough to be able to apply them). But the reason they couldn't be done in enviro science before is that you need vary large sample sizes which can be done in epidemiology but is not possible in other aspects of the field. Now the better processing power allows us to use small amounts of observed data to develop simulations which can be used to improve the statistical power and produce the a priori probabilities we need to run a model. This is the foundation for climate predictions since there are many potential changes to variables which have never been observed and must be modelled based on simulations (eg what happens when large ice sheets melt).

I do not mind people saying that having statistically insignificant warming for 15 years does not disprove climate science theory. But to claim settled science and anyone who disagrees is a big oil stooge or is anti science is crazy. All of this just because "scientists" have written a computer program that says that warming will come back with a vengeance. You are placing scientific weight on theoretical computer climate models and not over real data by claiming "settled science". Anyone who does this are the real denialists ( for denying that you theory is falling apart), and the ones who are anti science.

1, 2) Statistical significance, pretty much like all math, is common sense. That's why everybody is good at math but not so much at thinking. (And that is in no way meant to be taken offensively to anyone.)

3) I don't know the scientific consensus on AGW, but I believe there is much disagreement in the scientific circles about the matter. We can't ignore any information or exclude anything from a possible prediction of the future climate, so saying that rising temperatures for the future are caused by human emissions of greenhouse gases and are likely to cause problems is a good idea, but inherently unproven. We can prove high temperatures kill crops and melt ice, we can prove that cars and machines emit gases which trap heat energy from leaving into space, we can prove those, but we can't prove global warming until it is over, because it is such a long term thing. So that's where your statistical significance comes into play.

That was quite a charlatan-esque answer from Mike. Let's look at it point by point why don't we?

""Based on this statement... ...you missed completely the amount of subjectivity required to assess statistical significance.""

This argument shows a complete lack of understanding of how statistics work. In analyzing time series data, longer time periods most certainly increase the confidence with which conclusions may be drawn. There is *no* subjectivity at all, what so ever other than an a priori decision of confidence level which, as Climate Realist points out, is almost always 95% confidence.

""Note also the following about... ...two different data types (or more).""

First, Type I and Type II errors have nothing to do with observational or process error. A type I error is rejecting a null hypothesis when it is actually valid whereas a type II error is accepting a null hypothesis when it is invalid. Observation error and process error are inherent and unavoidable realities of measuring and reporting latent variables (i.e. variables that cannot be directly measured - such as the global climate). Partitioning of these errors is a primary step in development of any model and knowing these words does not arm one with a weapon against statistical methods used in climate science.

""As well... ...meaningful conclusions from it.""

The first sentence is correct, a P=0.05 means that there is a 5% chance of obtaining a test statistic (F, t, Z or otherwise) as extreme or more. These tests statistics, in a nut shell, are computed as the average or summed deviation of individual observations from a generalized mean. There is *no* subjectivity in these calculations except the chosen confidence level, set a priori, that is used to determine significance.

I wish in situations like this, Mike would offer examples of statistics being used and abused by climate scientists. It is certainly hip to quote people saying how statistics can be used to say whatever you want, but no one ever seems to be able to back up such accusations with fact.

@ Mike. Quite simple really.

1. Phil Jones and significance - the data were significant at the 90% confidence level. As they set a 95% a prior confidence level he could not report a significant effect. If another year was added (which it was not) then the trend does become significant. I agree, there is subjectivity yet it is in favor of the deniers.

2. There is a big difference between filtering/homogenizing data and statistical analysis, since you don't know the difference your opinion on statistics is officially moot.

3. Please critique the statistics of the MBH 1998 paper, please! I'm dieing to know your opinion of principal component analysis (PCA) and the missteps taken along the way that, in your mind, invalidate their conclusions. For the purpose of this question, I withdraw the mootification of your statistical opinion.

4. Comparing long term records with large error terms and short confident terms will differ in confidence. The methods used to analyze such data sets are far from my understanding and therefore I will not comment on their validity. I'd also be willing to bet that you could not comment on their validity either unless there was a recent blog post on the topic.

@bob326

Point taken. However, in my opinion the subjectivity is removed as much as possible from hypothesis testing. You quote model selection and I point to AIC/BIC, you state probability distributions and I claim that each has it's defined use (one cannot supplant a Poisson with a Bernoulli because they "feel like it"). There are protocol in place to guide researchers through their decision making processes in data analysis, they don't get to chose their tests based on the result they wish and you argument seems to support this being the case. If it is I do worry for science, however I've learned that each step in the data analysis process is governed by a set protocol and stepping outside of this protocol is invalid. There are assumptions of tests that must be flexible, but this is because of the improbable nature of many assumptions.

Nevertheless, I agree with your malcontent in modern hypothesis testing but until the bulk of science moves into Bayesian inference there isn't much that can be done.

Yes, I understand the importance of "statistical significance" to science, but I just can't figure out why it is so important. Frequentist dogma, perhaps?

We must remember, and it is so often forgot, that in testing for statistical significance (SS) we are not testing the hypothesis we actually want tested. In fact we are not testing *any* hypothesis at all. Instead we are calculating the probability of seeing a statistic as large, assuming some other, often uninteresting and unimportant to the question we are looking to answer, hypothesis is true, our probability model is true, all under infinite experimental repetitions.

David B

<<There is *no* subjectivity at all, what so ever other than an a priori decision of confidence level which, as Climate Realist points out, is almost always 95% confidence.>>

I have several problems with this statement. First, while 95% is the most widely used in science, it is arbitrary. In particle physics we are subject to a higher threshold of at least 3sigma, and sometimes 5sigma or more. In social sciences it is common to use a 90% CL. In no case does it mean that our hypothesis is true if SS can be claimed, and yet that is what is typically implied.

Second, the idea that there is *no* subjectivity in calculating SS, other than choice of CL, is ludicrous. Perhaps you've spent too much time around statistical software where many assumptions are not stated explicitly. In any case, subjectivity abounds in these types of analyses. Choice of model, probability distributions, parameter estimation, test statistic, hypothesis test, etc. And the results of your test are all under the assumption that your choices are correct.

And I'm not even discussing the issues of selection bias -- choice of sample size, the extent of smoothing, the choice and reporting of test results.

But I digress. I accept AGW, but dislike the current (though shifting!) paradigm of statistical hypothesis testing.

- - - - - - - -

David,

When I stated that subjectivity exists in hypothesis testing beyond choice of CL, i didn't mean that folks go into an analysis all willy-nilly doing whatever the hell they want. There are constraints and so-called protocols in place to help guide one through the analysis, but all of them allow for some level of discretion, often coming with statistical justification, but also a disregard for shortcomings and flaws.

As to your points:

1) Neither AIC, nor BIC, nor Hannan Quinn, nor any other modified or improved IC can tell you the best model, and certainly not the true model. All they can do is tell you the *better* model in a relative sense, and under certain circumstances. In fact, they can't even tell us if our model is any good in an absolute sense, which is best characterized by the ability to forecast accurately.

2) I didn't mean to suggest that you can reasonably choose any old distribution because you feel like it. If I decide on a normal distribution for a particular dataset, I must realize that it is *at best* an approximation, even though my subsequent analysis assumes that it is not; that it is the real thing. This is wrong. And there are dozens of other probability models which one could simply justify with this or that test. Parameter estimation methods, or choice of test statistic, can be tweaked in similar fashion to produce favorable results. I'm not saying that such "tweaking" is widespread, but it's easy to fool yourself into thinking that your methods are better or more proper than some other, completely different method.

If all this seems like I'm saying that classical statistical inference is a bunch of voodoo nonsense, that's not my intent. It's still a very valuable tool for any researcher to have up his sleeve, but it is much more counterintuitive (and often unrealistic), and far less certain than many tend to believe. Something as seemingly simple and widely used as "statistical significance" illustrates this point perfectly.

1

Statistical significance isn't really pertinent IMO.

2

Yes, I generally understand why it is important and am constantly repeating my mantra that alarmists don't know what they pretend to largely becuase they assume a knowledge they don't possess. Many of the pillars of alarmism are dependant on hockey sticks that have very questionable statitistics.

3 It has been warming for about 300 years. It has been cooling for 6000, It has been cooling for 3,000,000. It has been cooling for 3 billion years. Who knows if it has warmed in the last 13 years or the last 13 days. "Temperature rising" depends on your time interval. We don't know if human emissions are primarily responsible. I haven't seen very many evaluations of the benefits and consequences of warming and increased CO2 that weren't ridiculously exaggerated. It might cause problems and that are outweighed by benefits.

Science needs some fundamental ways to evaluate their own findings once submitted; and it is important to realize, specific geological areas and the amount of numbers who are in those statistics. *Also the groups used in these studies, whether they be tests/blind or not. *or just random questioning. Once they get doctors involved, (many paid for endorsements) the public is more apt to consider them valuable. (I fall for that one myself.) It is important to keep an open mind though, because even age groups differ for the results.

SCIENCE is the EVALUATION of Beliefs.

RELIGION is a holding a set of Beliefs that compel ACTION.

Religion often wars with Science, because the compulsion to ACT on a Belief is weakened if one is still EVALUATING them.

Religion must DEFEND Beliefs. Action must DEFEND the call for Action, not Evaluate it.

It is also easy to turn Science into Religion, because what is the point of EVALUATING Beliefs unless one then ACTS on them?

The core of Warmonism is that we must ACT now.

The Science is settled, the Belief is True, only ignorant Deniers think otherwise.

And THAT is what makes AGW a Religion.

The Statistical tools of Science are not being used to EVALUATE this belief, only to DEFEND it.

Most Liberals think Religion is a BAD thing that makes people do stupid sh!t. They think this because they don't know Karl Marx was an idiot.

The problem with Religion is NOT that it compels Action, but that the Action it compels is not necessarily in the interest of the People it compels.

Hence Religion is ever a tool of Politics.

The hubris of Liberals is that they somehow believe Science is above the fray - it is after all purely the EVALUATION of Belief. If you don't share Liberal Belief, you just must be bad at EVALUATION.

This simplistic perspective leaves them blind to the intrinsically Religious nature of man. That we MUST be driven to Action by our Beliefs.

Which makes it all too easy to 'Evaluate' based on what the Actions we Desire, as opposed to what the Data suggests.

If Politics is about the control of Action

& Action is controlled by Belief

& Belief is Evaluated by Science

The probability of Science being non-political is very small.

And THAT is the statistic you don't get.

I personally see nothing to suggest that Warmons are better at Scientific Evaluation. Most the ones on Yahoo don't even know that 0.01% & 100ppm are the same number. But the point really isn't the alleged difference in math skills. I fully accept that there are many AGW supporters profoundly good at math.

The point is whether AGW is being driven by Politics or Data Evaluation.

And where is the split in Belief?

Right down party lines.

Because the ACTIONS suggested by Warmonism are SOCIALIST. That doesn't mean AGW is FALSE, it just means Socialists are highly motivated to assert that it's true. Socialists must DEFEND the Belief because it's their Religion.

Doesn't the lack of statistical warming over the last 14 years suggest further EVALUATION is needed?

How can we be on the 'settled' side of things. Is that really what the data suggests?

The Politics of control doesn't want Science, it wants Religion. Skepticism is Denial. Deniers are Climate Criminals. The pursuit of personal Power is Greed.

One does not have to defend your Agenda, if you can successfully discredit those denying the Belief that support it.

Which is what this question is.

Yes, I understand long-term trends and how they are important to science-to a certain extent. First, let me outline my limitations-I'm very good at analysis and logic; I can look at a business and indentify potential and problems with pinpoint accuracy based on the numbers, but climate science is outside of my field and statistics is not my strong point. So when I look at long term trends and the statistical probabilities that they indicate it is difficult for me to interpret how the conclusions got from point A to point B. It would be helpful for me-and I know I am not alone-to know more about the computer modeling used and the specific variables that were included. I also have some limitations when it comes to reading peer-reviewed publications, primarily in terms of vocabulary, although also some issues with knowledge of processes; for example, when I first ran across the term 'albedo' it took a while to figure out what they were talking about; I also struggle with understanding the effects of thermohaline circulation. So I often have to rely on interpretations by others, and since those reports are so frequently shaded by political or other agendas it can be hard to go back to the actual research and sort out. Finally, there is a lot of mis or disinformation that is repeated so often it can be confusing, I often gloss over the errors and outright lies initially and then go 'wait a minute...' later. Which is what I suspect the disseminators of disinformation have in mind when they publish it to begin with, except they hope people won't have the 'wait a minute' moment. For a simple and non-scientific example, in 'the Great Global Warming Swindle' there is a segment about a medical clinic that is inadequately powered by solar and how this alternative source of energy is a boondoggle because conventional power would supply plenty of electricity to operate the facility at lower cost. At first blush, this makes sense; but when you think about the time and expense to build a power grid to supply places like this clinic, the obvious solution is to add more solar panels. The 'disinformation' is obviously the clear omission of how difficult, costly and time consuming it would be to get the grid built to deliver power to remote areas like this...and how it would be funded. So you can see that the 'business' side of the equation comes easily to me in terms of the cost and benefit analysis. The clinic needs more power now, so dismissing solar when it will take years of work and billions of dollars to construct a grid when another solar panel can be installed and delivering the additional electricity in a matter of days is just stupid. Find the funding for it, buy the necessary panels, and get them installed. Second, what really constitutes a long-term trend, and is it weather or climate? For example, we see a historic period of cooling over several decades and it is ascribed to particulate emissions, or a shorter term trend that is impacted by El Nino and La Nina events. I'm uneasy with some of the conclusions reached based on time periods that are affected by variables such as these, and at this juncture don't see 30-40 years periods as long term enough to be conclusive. I understand the limitations of research and availability of data, but I am not well enough attuned to what constitutes long-term trends in terms of scientific conclusions. So my answers to your followup questions in the context I have just provided: 1. I do not have any disagreements with the general concept of how long-term trends work. 2. I understand long-term trends at least in broad terms, but am uncertain insofar as what really constitutes a 'long term' trend from which conclusions can be reached about climate. 3. I accept the consensus of science because I do not have the necessary information or expertise to reject it, with the following notations: a) I accept that global average temperatures are rising. b) I accept that human emissions of greenhouse gasses influence climate, but need more information to understand how much. c) I accept that climate change is likely to cause problems but am uncertain about the nature, extent and timing of the problems. I believe the key to the overall conclusions that may be drawn from my answer is in the three notations (a, b & c) above; these factors prevent me from supporting policies that may have geopolitical and economic impacts based solely on the conclusions and consensus to date about human emissions of greenhouse gasses.

As a general rule, scientists accept a hypothesis if it passes a 95% significance test to the effect that we can reject something called a null hypothesis. Say, if a botanist wants to determine if magnesium is essential for plant growth, he/she will compare the growth a plant which is supplemented with magnesium with the growth of a plant which is not supplemented with magnesium. But the problem with using only one pair of plants is that the difference in plant growth could be due to chance, rather than due to the magnesium. So, the botanist tests for the effect of magnesium on plant growth by testing 20 pairs of plants. If the plant treated with magnesium outgrows the plant not treated with magnesium in all 20 pairs, or in 19 out of twenty pairs, the botanist can conclude with 95% confidence that magnesium aids plant growth. If the plant treated with magnesium only outgrows the plant not treated with magnesium in 18 pairs or less, the results are not statistically significant.

Denialists love to say that Phil Jones said that there was 15 years of no significant warming and use that statement to claim that Earth is not warming. But poor statistical significance is the inevitable result of using to short of time frame to try to determine climate trends.