{"id":1760,"date":"2025-05-14T09:33:53","date_gmt":"2025-05-14T09:33:08","guid":{"rendered":"http:\/\/localhost:8091\/creating-preference-test-study-guide\/how-to-choose-the-best-number-of-respondents-for-your-preference-test\/"},"modified":"2025-06-30T12:53:53","modified_gmt":"2025-06-30T12:53:53","slug":"statistical-significance","status":"publish","type":"page","link":"https:\/\/www.uxtweak.com\/help\/statistical-significance\/","title":{"rendered":"How to choose the best number of respondents for your Preference Test?"},"content":{"rendered":"

Introduction \"Copy<\/a><\/h2>\n\n\n

Apart from analyzing how many respondents prefer which of your design variants in total and as a percentage, Preference Test also provides you with information about\u00a0<\/span>statistical significance of the task results.<\/b>\u00a0<\/span>How big is the chance that the given result is the effect of respondents truly preferring one design to the others?<\/p>\n\n\n\n

If you’re unfamiliar with how Preference Test calculates statistical significance, you can read more about it in our\u00a0<\/span>Preference Test Task Statistics article.<\/a>\u00a0<\/span>If you’re already familiar with the concept, continue reading to learn how to set the target number of respondents to recruit in your study to achieve results of satisfying statistical significance.<\/p>\n\n\n\n

\n \"Respondent\n
\n
\n

Respondent sees this<\/h4>\n

This is what a statistical significance looks like in UXtweak.<\/p>\n <\/div>\n

\n\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n\n\n\n

To provide enough power to the Chi-Square Test which forms the backbone of statistical significance calculation, you’ll need\u00a0<\/span>to collect a respondent sample of sufficient size. This size varies depending on the number of visual\/audio\/video designs which you’re aiming to compare.<\/b>\u00a0<\/span>Naturally, with more designs to choose from, you also need more participants if you want the test to have the same degree of reliability.<\/p>\n\n\n\n

Here are\u00a0<\/span>our recommendations for how many people to recruit to achieve the best informative results.<\/b>\u00a0<\/span>Let’s assume that we require a confidence level of at least 90% and medium effect size (meaning there will be medium differences between user preferences for individual designs – larger differences require a smaller sample size, smaller differences require a larger sample size to achieve the same confidence):<\/p>\n\n\n\n

Number of designs to compare in a single task<\/th>Recommended number of respondents for the task<\/th><\/tr><\/thead>
2<\/td>69<\/td><\/tr>
3<\/td>86<\/td><\/tr>
4<\/td>98<\/td><\/tr>
5<\/td>108<\/td><\/tr>
6<\/td>116<\/td><\/tr><\/tbody><\/table><\/figure><\/div>\n\n\n\n

We calculated these results with the help of this\u00a0<\/span>online sample size calculator.<\/a>\u00a0<\/span>You can adjust the entry parameters to calculate your own sample size in a similar manner.<\/p>\n","protected":false},"excerpt":{"rendered":"

Apart from analyzing how many respondents prefer which of your design variants in total and as a percentage, Preference Test also provides you with information about\u00a0statistical significance of the task results.\u00a0How big is the chance that the given result is the effect of respondents truly preferring one design to the others? If you’re unfamiliar with […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":27,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[24],"tags":[],"class_list":["post-1760","page","type-page","status-publish","hentry","category-preference-test"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/pages\/1760","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/comments?post=1760"}],"version-history":[{"count":5,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/pages\/1760\/revisions"}],"predecessor-version":[{"id":2660,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/pages\/1760\/revisions\/2660"}],"wp:attachment":[{"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/media?parent=1760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/categories?post=1760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.uxtweak.com\/help\/wp-json\/wp\/v2\/tags?post=1760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}