Quantum Computing and Statistical Significance

10/24/19

In my Nobel Prize and Statistical Significance, I showed some examples of p-values and statistical significance being used at the highest levels of science, despite criticisms that these approaches are bad for science. In this article, I will show another example of p-values and statistical significance and hypothesis testing being used for some great science, and that is in the field of quantum computing.

In basic terms, quantum computing can solve problems much faster than current computers are able to. How much faster? In Google's article Quantum Supremacy Using a Programmable Superconducting Processor, they link to the Nature article, which links to supplementary information used in the paper. In the article, they write

"We developed a new 54-qubit processor, named 'Sycamore', that is comprised of fast, high-fidelity quantum logic gates, in order to perform the benchmark testing. Our machine performed the target computation in 200 seconds, and from measurements in our experiment we determined that it would take the world's fastest supercomputer 10,000 years to produce a similar output."

On to the p-values and statistical significance. In the supplementary information, we read

• "FIG. S33. The Kolmogorov distribution function. This function is used to compute p-value from a given D_KS and number of samples N_s."
• "We see good agreements between experiment and theory. To quantify the agreements, we use the Kolmogorov Smirnov test..."
• "We then convert the distance D_KS to a p-value using the Kolmogorov distribution shown in Fig. S33. The p-value is used for rejecting the null hypothesis that the data {p_i} is consistent with the theoretical distribution."
• "We reject the null hypothesis that the experimental bitstrings are consistent with the uniform random distribution with very high confidence for this (20-qubit 14cycle) random circuit."
• FIG. S34 also shows p-values
• "The p-value for the null hypothesis of zero fidelity is generally small for every circuit, with a maximum of 0.045 for circuit number 1. We say that the null hypothesis of zero fidelity is rejected better than a 95% confidence level for each circuit."
• "On the other hand, the p-value of null hypothesis of estimated fidelity F^{^} is generally large. The p-value is between 0.18 and 0.98 for linear XEB, and between 0.33 and 0.98 for log XEB. That indicates that the empirical cumulative distribution functions ECDF(p_i) from data is quite consistent with the theoretical CDF(p_i|F^{^})."
• FIG. S36 also shows p-values
• "The fidelity distribution from 4000 bootstrap samples are shown in Fig. S37. The distribution of F^{^}₁ and F^{^}_c are each fit to a Gaussian distribution function using maximum likelihood."
• TABLE V. also shows p-values
• "The p-value for the null hypothesis of F = 0 is very small: p-value = 3 x 10^-24 from scipy, and p-value < 2.2 x 10^-16 from R. We note the more conservative value in the table. The null hypothesis of F = 0 is rejected with much higher confidence levels than individual circuits."
• "The Kolmogorov-Smirnov test on the Gaussian fit produces p-values of 0.99 and 0.41 for F^{^}₁ and F^{^}_c bootstrap distributions, respectively. It indicates that the central limit theorem is at work and the distributions are consistent with Gaussian distributions."
• "The p-value for the chi² for 11 degrees of freedom is 0.0058, indicating that it is not a very good fit. Because the correctness of the estimates of statistical uncertainties has been verified in Section VIIIG, this is attributed to systematic fluctuation in addition to degradation. It is supported by the larger variance of fidelity than the 1 sigma band in Fig. S39."
• "Therefore we form the null hypothesis that the fidelity of the quantum computer is F <= 10^-3, and the alternative hypothesis that F > 10^-3. If the alternative hypothesis is true, we can say that a classical computer can not perform the same noisy sampling task as the quantum computer. The total uncertainty on fidelity is estimated with addition in quadrature of systematic uncertainty and statistical uncertainty. The mean fidelity of 10 random circuits with 53 qubits and 20 cycles is (2.24 + 0.21) x 10^-3. The null hypothesis is therefore rejected with a significance of 6 sigma."
• uses a lot of bootstrap, simulation, and sampling theory

A critic may say "Yes, but they did this science in spite of frequentism", or "They could have used another method that they didn't use to get similar results". Both of these excuses are, oddly enough, appealing to counterfactual logic (what could have happened but didn't), the same type of logic frequentism uses with p-values that critics don't seem to care for.

In my opinion, this paper and thousands of other results show that scientists find p-values, statistical significance language and concept, sampling, etc., very useful for doing science.

Thanks for reading.