{"id":1957,"date":"2024-01-02T18:27:24","date_gmt":"2024-01-02T23:27:24","guid":{"rendered":"https:\/\/david.bookstaber.com\/Interests\/?p=1957"},"modified":"2024-01-02T18:27:24","modified_gmt":"2024-01-02T23:27:24","slug":"statistical-inference-example-testing-22lr-ammunition","status":"publish","type":"post","link":"https:\/\/david.bookstaber.com\/Interests\/2024\/01\/statistical-inference-example-testing-22lr-ammunition\/","title":{"rendered":"Statistical Inference Example: Testing .22LR Ammunition"},"content":{"rendered":"\n

Competitive shooters work hard to find the ammunition that delivers the highest precision in their guns. This isn’t always straightforward: Even among premium ammunition lines any particular barrel can show a preference for one load that produces poor results in others. Using the latest statistical techniques<\/a>, I plugged in some of the data I collected during this test of .22LR rifles<\/a>. Shown here are the recorded groups of two types of ammunition \u2013 SK Plus <\/em>and SK Match <\/em>\u2013 shot through my KIDD 10\/22.<\/p>\n\n\n\n

\"\"
Aggregated test targets shot through KIDD 10\/22 at 50 yards. The SK Plus points are labelled in the order they were shot.<\/figcaption><\/figure>\n\n\n\n

Match <\/em>is SK’s higher-end ammunition, so our expectation going into the test is that it will produce higher precision than Plus<\/em>. The purpose of the analysis here is to see how well a controlled test sustains that hypothesis.<\/p>\n\n\n\n

I like to call this the Not So Fast!<\/em> example. Remember that shooters never have enough time or ammunition, so they prefer to draw conclusions as fast as possible when running A\/B tests like this. So imagine you have first fired the 25 round group of Match <\/em>shown on the left. Running the calculations, you find its estimated sigma is 0.16″ at 50 yards with a 90% confidence interval of [0.13″, 0.19″].<\/p>\n\n\n\n

Now we want to see how SK Plus <\/em>stacks up. In the following table I show how the statistics evolve with each shot of Plus<\/em>:<\/p>\n\n\n\n

\"\"
Running statistics after each shot of SK Plus<\/em>, compared to the complete target data of SK Match<\/em><\/figcaption><\/figure>\n\n\n\n

The first few shots aren’t very good: by the third shot of Plus <\/em>we estimate it is only half as accurate as Match<\/em>: Sigma A\/B = 0.50. (That’s the middle columns in the table: “Effect Size,” which is the ratio of the estimated sigma of the two ammo types.) But three shots is a terribly small sample, which is reflected in the very wide 90% confidence interval on that estimated Effect Size: [0.33, 1.27].<\/p>\n\n\n\n

So we keep shooting, and Plus <\/em>starts to look better. After 10 shots our estimate of its dispersion (a.k.a. sigma<\/em>, which is the left three columns) has gone from over 0.3″ to barely over 0.2″. But compared to our data on Match <\/em>it’s still not looking good: by the 11th shot the 90% confidence interval on Effect Size no longer contains 1.0, which means that with 90% probability it falls short of the precision of Match<\/em>. This is also reflected in the p-value (right-most column), which collapses to single digits at this point.<\/p>\n\n\n\n

But wait! We planned to shoot 30 rounds, so let’s finish the test. By the time we’ve fired 20 rounds of Plus <\/em>we are not as confident that it is so inferior to Match<\/em>. By the end of the test our best guess is that, in this gun, Match <\/em>will produce groups only 15% tighter than Plus<\/em> (that’s 1\/0.87). And the probability of these aggregated data if there were no difference between the two (i.e., under the null hypothesis, which is the p-value) has jumped to 33%.<\/p>\n\n\n\n

What’s the point?<\/p>\n\n\n\n

    \n
  1. This example shows that small samples can suggest things that are far from the truth. We drew the worst samples of Plus <\/em>right at the start of the test.<\/li>\n\n\n\n
  2. By running a more statistically significant test, we have learned that when Match <\/em>is scarce or expensive, we probably don’t give up much performance by instead shooting Plus <\/em>through this rifle.<\/li>\n<\/ol>\n\n\n\n

    Excel file with the complete data and analysis is here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Competitive shooters work hard to find the ammunition that delivers the highest precision in their guns. This isn’t always straightforward: Even among premium ammunition lines any particular barrel can show a preference for one load that produces poor results in others. Using the latest statistical techniques, I plugged in some of the data I collected […]<\/p>\n","protected":false},"author":3,"featured_media":1958,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[133,122],"tags":[134],"class_list":["post-1957","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ballistics","category-precision","tag-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/david.bookstaber.com\/Interests\/wp-content\/uploads\/2024\/01\/SKsampleTargets.png","jetpack_shortlink":"https:\/\/wp.me\/po3d1-vz","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/posts\/1957","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/comments?post=1957"}],"version-history":[{"count":1,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/posts\/1957\/revisions"}],"predecessor-version":[{"id":1961,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/posts\/1957\/revisions\/1961"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/media\/1958"}],"wp:attachment":[{"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/media?parent=1957"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/categories?post=1957"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/david.bookstaber.com\/Interests\/wp-json\/wp\/v2\/tags?post=1957"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}