Standard Report: Tetrad Test Results

Standard Report: Tetrad Test Results

Overview

Analysis for the Tetrad data in Compusense uses a one-tailed binomial test.


Tetrad Analysis Options

When generating the Tetrad Standard Report, it is important to select the options that meet your analysis objectives. Go to Results > Reports > Create report. Click Filters and apply any necessary filters on your data before generating the report.

Select Standard report and expand the 2. Select options list to review all the options.



  1. ASTM Discrimination analysis - This analysis is a statistical non-inferiority test (which in this case is a binomial test of the lower tail) and it aligns with the ASTM standard E3009-23a. This analysis type is used in the following Discrimination tests: Triangle, Tetrad, and Duo-Trio.

    These options are selected by default. Any changes in the selections and values will be saved for the next time you run the report.

    1. Difference Test – Enter the Alpha you wish to use for this test or turn off the Difference Test analysis by removing the checkmark from the checkbox. Update the d' (d prime) by typing directly in the box. You will notice that the Pd and Pc values will automatically recalculate.

      If you update the Pd, then the d' and Pc values will automatically recalculate. Likewise, if you update the Pc value, the d' and Pd values will automatically recalculate.


    2. Equivalence Test – Enter the Alpha you wish to use for this test or turn off the Equivalence Test analysis by removing the checkmark from the checkbox. Update the d' (d prime) by typing directly in the box. You will notice that the Pd and Pc values will automatically recalculate.

      If you update the Pd, then the d' and Pc values will automatically recalculate. Likewise, if you update the Pc value, the d' and Pd values will automatically recalculate.


  2. Discrimination analysis - This analysis follows the formulas published in Sensory evaluation techniques by Meilgaard et al. This analysis has been included in Compusense since this test type was originally implemented in Compusense. These analyses options are not selected by default, but if you make any changes to the selections, they will be saved for you moving forward until you make different selections.
    1. Difference Test - This is the default analysis. Change the Alpha value if desired, or turn off the Difference Test analysis by removing the checkmark from the checkbox.


    2. Similarity Test - Analysis type used in Discrimination tests (Triangle, Tetrad, Same/Different,  3-AFC, Duo-Trio, 2-out-of-5, A-not-A, and Torgerson's) when high confidence is required that the samples are perceived the same. This analysis can be used in quality control tests, for example. Turn the Similarity Test analysis on or off when generating the report.

      Set the Alpha and Beta values according to your company standards.



Tetrad Standard Report Details

What is included in the report will depend on what you selected before generating the report, as described in the previous section of this workflow.

Report and Sample Details

The Standard report for Tetrad question data includes the following information about the report itself and about the samples:

  1. Test Name: This is the name specified in the Overview tab of the test, visible to analysts in the software, and in the reports.

  2. Number of Evaluations: The number of completed and/or in progress evaluations included in the report. Whether the 'in progress' evaluations are included in the report will depend on the filters applied before generating the report.

  3. Test Completion Date: If you set your test to Complete before generating the report, the date when you set the test to complete will be displayed here. No date will be shown if the test is not set to Complete.

  4. Samples: A table containing the information about the samples in the test that were included in the report.



ASTM Discrimination Analysis

Difference Test


  1. Above the table, the following information is displayed:
    1. The threshold values specified by the user before generating the report.

    2. Statement on how conclusions are made based on the selected alpha. The confidence % is the only value that will change in this statement depending on the Alpha value specified before the report was generated.

  2. The table displays the following details:

      1. p-value: The p-value for each tetrad test.

      2. Responses Required: The minimum number of correct responses necessary to reach the conclusion of Different.

      3. Conclusion: A conclusion is made based on whether the difference between samples is significantly larger than the difference threshold.

      4. Tetrad test number: Our example screenshot with numbered bubbles shows that a back to back Tetrad was ran; therefore there is the Tetrad 1 and Tetrad 2 row.

      5. Pooled: In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

      6. Both Tetrads: In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).


Equivalence Test




  1. Above the table, the following information is displayed:
    1. The threshold values specified by the user before generating the report.

    2. Statement on how conclusions are made based on the selected alpha. The confidence % is the only value that will change in this statement depending on the Alpha value specified before the report was generated.

  2. The table displays the following details:
    1. p-value: The p-value for each tetrad test.

    2. Responses Required: The maximum number of correct responses permitted to reach the conclusion of Equivalent.

    3. Conclusion: A conclusion is made based on whether the difference between samples is significantly larger than the difference threshold.

    4. Tetrad test number: Our example screenshot with numbered bubbles shows that a back to back Tetrad was ran; therefore there is the Tetrad 1 and Tetrad 2 row.

    5. Pooled: In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

    6. Both Tetrads: In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).


Supporting tables

Proportion Correct:


  1. N: Total number of evaluations included in the report.

  2. Correct: Count of correct responses (where the odd sample was identified correctly).

  3. Incorrect: Count of incorrect responses (where the odd sample was not identified correctly).

  4. Estimate Pc: Estimated proportion correct.

    Formula: Pc = c / n

    Where:
    c = number of correct responses.

    n = total number of evaluations.

  5. SE: Standard error of the estimate of the proportion of (momentary) discriminators.

  6. LCL: Lower confidence limit at the Alpha specified before generating the report.

  7. UCL: Upper confidence limit.

  8. Tetrad test number: Our example screenshot above, with numbered bubbles, shows that a back to back Tetrad was ran; therefore there is the Tetrad 1 and Tetrad 2 row.

  9. Pooled: In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

  10. Both Tetrads: In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).



Proportion of momentary discriminators (Pd):


  1. Estimate Pd: Estimated proportion of distinguishers.

  2. LCL: Lower confidence limit at the Alpha specified before generating the report.

  3. UCL: Upper confidence limit.

  4. Tetrad test number: Our example screenshot with numbered bubbles shows that a back to back Tetrad was ran; therefore there is the Tetrad 1 and Tetrad 2 row.

  5. Pooled: In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

  6. Both Tetrads: In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).



d prime:


  1. d' (read d prime): The measure of the extent of the overlap of panelist perception. It is the distance between the means of the two distributions measured in terms of their standard deviation.

    The larger the d’ value, the samples are more different. Where the d' value is 0 it means that the samples are indistinguishable. A d’ of 1 or less might be considered small and a d’ of 2 or more might be considered large, but it depends on various factors.

    This calculation is always done treating each response as independent, as if the test is unreplicated. For example, d' on a back-to-back test with 50 assessors who provide 2 responses, calculation is done as if there are 100 assessors who provide 1 response each.

  2. LCL: Lower confidence limit at the Alpha specified before generating the report.

  3. UCL: Upper confidence limit.

  4. Tetrad test number: Our example screenshot with numbered bubbles shows that a back to back Tetrad was ran; therefore there is the Tetrad 1 and Tetrad 2 row.

  5. Pooled: In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

  6. Both Tetrads: In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).




Discrimination Analysis

If included in the report, the tables described below are generated based on the standards for Tetrad testing that existed prior to the implementation of new ASTM standards.

The formula that you can use in Excel to reproduce the same analysis is as follows:

=1-BINOMDIST(x-1,n,0.3333333333333,TRUE)

Where:
  1. x-1 = number of correct responses minus 1

  2. n = total number of evaluations included in the analysis

Difference Test



  1. Chance. Probability of panelists correctly selecting the odd sample by chance.

  2. N. Total number of evaluations included in the report.

  3. Correct. Count of correct responses.

  4. Incorrect. Count of incorrect responses.

  5. d' (read d prime). The measure of the extent of the overlap of panelist perception. It is the distance between the means of the two distributions measured in terms of their standard deviation.

    The larger the d’ value, the samples are more different. Where the d' value is 0 it means that the samples are indistinguishable. A d’ of 1 or less might be considered small and a d’ of 2 or more might be considered large, but it depends on various factors.

    This calculation is always done treating each response as independent, as if the test is unreplicated. For example, d' on a back-to-back test with 50 assessors who provide 2 responses, calculation is done as if there are 100 assessors who provide 1 response each.

  6. p-value. The probability of observing an outcome that is more extreme than or equal to the test statistic observed assuming that the null hypothesis is true.

  7. Significant at an Alpha level. Indicator whether there is a significant difference between samples at the specified level of significance (Alpha). The default Alpha value is 0.05, but it can be changed in the results area when generating the report.

  8. Tetrad test number. Our example screenshot above, the one with numbered bubbles shows that a back to back Tetrad was ran on two samples. Below is an example of a back to back Tetrad analysis for four samples.


  9. Pooled. In the analysis where two samples (see the Samples table in our example above) were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

  10. Both Tetrads. In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

Similarity Test



  1. N. Total number of evaluations included in the report.

  2. Correct. Count of correct responses (where the odd sample was identified correctly).

  3. Responses Required. The maximum number of correct responses permitted to reach a conclusion of Similar.

  4. Conclusion. When the number of correct responses is smaller than the responses required, the samples are deemed similar. When the number of correct responses is equal to or greater than the responses required, the samples are deemed not similar.

  5. Pc(estimate). Proportion correct.

    Formula: Pc = c / n

    Where:
    c = number of correct responses.

    n = total number of evaluations.

  6. Pd(estimate). Proportion of distinguishers.

    Formula: Pd = (Pc - Pg) / (1 - Pg)

    Where:
    Pc = proportion correct.

    Pg = guessing probability.

  7. Std error (Pd). Standard error of the estimate of the proportion of (momentary) discriminators.

    Formula: (1 / (1 - Pg)) * sqrt(Pc * (1 - Pc) / n)

    Where:
    Pg = guessing probability.

    sqrt = square root.

    Pc = proportion correct.

  8. Pmin, Lower confidence limit (α=0.2).

    Formula: Pmin = Pd + Zalpha * Std error (Pd)

    Where:
    Pd = proportion of distinguishers.

    Zalpha = Alpha value. The default Alpha value is 0.10, but it can be changed before generating the report.

    Std error (Pd) =Standard error of the estimate of the proportion of (momentary) discriminators.

  9. Pmax, Upper confidence limit (β=0.01).

    Formula: Pmax = Pd + Zbeta * Std error (Pd)

    Where:
    Pd = proportion of distinguishers.

    Zbeta = Beta value. The default Beta value is 0.10, but it can be changed before generating the report.

    Std error (Pd) =Standard error of the estimate of the proportion of (momentary) discriminators.

  10. Tetrad test number. Our example with numbered bubbles shows that a back to back Tetrad was ran on two samples. Below is an example of a back to back Tetrad analysis for four samples.


  11. Pooled. In the analysis where two samples were evaluated back to back, the N is increased by multiplying the actual N by the number of tests. Each test is treated as done by different panelists even though they were seen back to back by the same panelists. Pooled analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).

  12. Both Tetrads. In a back to back Tetrad analysis (true replicate of two samples), you can see how many panelists answered both Tetrad tests correctly. The probability of panelists correctly sorting the samples for both tests by chance decreases. Both Tetrads analysis is not available in single tests and in any other variation of back to back tests (e.g. more than two tests on two samples, or more than two sample tests).


Additional Tetrad Analysis

If your test included a follow up Tetrad related Comment or a Tetrad related Choose question, your Standard report will provide you with Tetrad related Comments or Tetrad related Choose data, depending on what question type you added into your test. Let's review what each follow up question provides in the Standard report.

When Tetrad related comments are collected, the analysis will separate the panelists' comments based on their correct and incorrect Tetrad responses. In our example below, we can see that the panelist #5 correctly evaluated both Tetrads and his or her comments for both tests are in the table labeled Correct response related comments.

The Incorrect response related comments table displays the comments for all the incorrect Tetrad responses. Our example does not have any comments in it because our example test was set up with connections. The connections did not allow panelists who answered the Tetrad incorrectly to comment, however, the table still shows individual panelists and the Tetrad test # that they did not evaluate correctly.


When a Tetrad related Choose data is collected, the analysis will display the Crosstabulation and Percentage Crosstabulation tables based on panelists' correct and incorrect Tetrad responses.

  1. Count of choice selections for correct Tetrad responses.

  2. Count of choice selections for incorrect Tetrad responses.
Notes
To change the Crosstabulation format, when generating the Standard report in the Results area, expand Select options and choose what works best for your analysis needs.


Panelist Performance Report

Run a Panelist Performance report in the Panelists library to see how your panelists have performed over multiple Tetrad tests over time, allowing you to discover patterns or non-differentiators. The workflow for the report will be available soon.



Data

You can use the Data tab for another quick and easy insight into panelist performance within a single test (rather than across multiple tests as described above). The example screenshot below was taken in a test that had a back to back Tetrad scenario where two different samples were evaluated in each Tetrad (4 samples in total).



Graphs

The data from single and back to back Tetrads with two samples can also be graphed. To generate the graph:
  1. In the Results area, click the Graphs tab.

  2. In the left sliding pane, select the Tetrad question.

  3. Update any of the parameters in the right sliding pane.
Notes
Graphs cannot be generated on Tetrad related comment and CATA questions, and on back-to-back Tetrads with 4 or more samples.



    • Related Articles

    • Tetrad Test

      Overview Tetrad test is a discrimination test type that helps you determine if there is a difference between two products by asking panelists to group the four samples into 2 groups of 2. This highly recommended method is useful for a wide range of ...

    • Standard Report: Single Triangle Test Results

      Overview The Standard Report for Single Triangle test data can provide a Difference Test and a Similarity Test. The Difference Test will provide a count of Correct and Incorrect responses as well as a statement of significance. The Similarity Test ...

    • Standard Report Review

      Overview The Standard Report provides a breakdown of results by attributes (where applicable) and samples, as well as the analysis that was selected while generating the report. Question Compatibility The report is compatible with the following ...

    • Standard Report: Back to Back Triangle Test Results

      Overview The Standard Report for Back to Back Triangle test data can provide a Difference Test and a Similarity Test. The Difference Test will provide a count of Correct and Incorrect responses as well as a statement of significance. The Similarity ...

    • Standard Test

      Overview Collect quantitative data on your products using the Standard test type. Use this test type to access a variety of question types to be used in profiling, difference from control or consumer tests. Question Compatibility The Standard test ...