Overview

Penalty analysis measures the change in product liking due to the product having “too much” or “too little” of an attribute. For example, for a dark chocolate bar, you could measure the bitterness of the chocolate against the overall liking of the chocolate bar.

For consumer testing we typically recommend at least 60 results; however the Penalty analysis will run on a lower N and can still provide valuable information. 

Considerations

1. It is crucial to include a Liking and JAR question types in the test to be able to run the report.
   
   If it happens that the question types are accidentally set to a different variety of Category Scale questions, even after the data is collected, the question types can be changed to a proper type. Please do not delete results to make this change!
   
   Changing the question type is done in the Build tab, in the Category Question Options, as seen in the image below.
   
   
   
2. Penalty Analysis reports are generated as Excel Workbooks.
   
   
3. Penalty analysis is available only for tests without reps.
   
   
4. In tests with sections, the report can be generated within individual sections. The report is not available across sections in a single test nor across multiple tests.
   
   
5. Only complete sample sets will be included in the report. It is not an option to include sample sets that are not complete.

Test Setup

Generate the Report

1. The Mean Drop threshold indicates how different the mean liking of the consumers who selected JAR for a particular attribute is from the mean liking of consumers who did not select JAR for that attribute. 
   
   
2. The percentage threshold is the total percentage of consumers who selected anything other than JAR.

2. Net Penalty Colour Thresholds: If selected, these will clearly indicate which attributes are of interest in the Net Penalty Graphs. A net penalty of <0.25 is considered low impact. Potential and high impact thresholds can be selected by the analyst. 
   
   
3. The Penalty Inclusion Threshold: It allows the analyst to set a penalty analysis percentage threshold.  If the sum of the percentages for Too Much and Not Enough for a specific attribute is below the selected penalty analysis percentage, that attribute will be excluded from the Mean Drop Chart.
   
   

5.  Click 3. Select questions.

Report Details

Data

Penalty Tables

Each sample included in the analysis has its own table in this sheet (you will need to scroll down to see them all).

1. Attribute: Three rows for each attribute, one for each level on the collapsed JAR scale. In our example screenshot, the "chocolate JAR" attribute repeats three times because there are three levels (see the next item below) in the attribute.
   
   
2. Level: While typically a JAR scale would have 5 categories, in this analysis the top two and bottom two options are collapsed; therefore, only three levels are displayed and used in calculations. In our example: Too Weak, Just-about-right, and Too Strong.
   
   
3. Frequencies: The count of responses for each level within an attribute.
   
   
4. %: The frequencies converted into percentages.
   
   The formula: (Frequency/N)*100
   
   
5. Sum Liking: The sum of the liking responses within the attribute and level. In our example, the "chocolate JAR" attribute for the Too Weak level, indicates the sum liking of 25. This is determined by looking at the Data sheet. In the screenshot below, highlighted in yellow, we can see the 4 responses for the sample 1 that were less than a JAR (value lower than 3) for the "chocolate JAR" attribute. For those 4 responses we then look at the Overall Liking scores and add them up: 4 + 6 + 7 + 8 = 25.
   
   
   
6. Mean Liking: The mean liking of the sample by consumers who selected the indicated level for the given attribute.
   
   The formula: Sum Liking/Frequency
   
   
7. Mean Drop: A comparison of the mean liking of the level above or below JAR with the mean liking of the group who selected JAR. Therefore the results are displayed only in the 'too much' and 'not enough' rows; never in the JAR row.
   
   The larger the mean drop, the more that attribute/level affects the liking of the sample.
   
   
8. Label: This text will appear in the mean drop graphs.
   

Mean Drop Charts

If you have selected Penalty Inclusion when generating the report, and if the sum of the percentages for Too much and Not enough for a specific attribute is below the selected penalty inclusion percentage, the attribute will be excluded from the Mean Drop Charts.

If all attributes for a sample are below the penalty inclusion percentage threshold, that whole sample will be excluded from the Mean Drop Charts. That would mean that at the set threshold, that sample's overall liking is not affected by any of the JAR attributes included in the report.

1. x axis: This is the frequency percentage (%) of the responses for each level of a specific attribute. See the % column in the Penalty Tables sheet.
   
   
2. y axis: This is the mean drop value. See the Mean Drop column in the Penalty Tables sheet.
   
   
3. Colour (these can be edited in Excel using its built in functionality): Each sample is assigned its own colour.
   
   
4. Shapes (these can be edited in Excel using its built in functionality):  
1. Levels above JAR are represented by a triangle.
2. Levels below JAR are represented by a circle.
   
   
5. Quadrants: If you selected Mean Drop thresholds, they will be represented as two red lines on the graphs. This creates easy to read quadrants in your graph.
   
   
1. Attribute levels found in the bottom left quadrant represent attributes levels that are below both predetermined thresholds. These attribute levels are of no concern.
   
   
2. Attribute levels found in the top right quadrant represent attribute levels that are above both predetermined thresholds. This is the area of primary concern because these attribute levels indicate that they affected panelists overall liking of the sample.
   
   
3. Attribute levels found in the top left quadrant represent attribute levels that are above the mean drop threshold but below the frequency threshold. This is the area of secondary concern.
   
   
4. Attribute levels found in the bottom right quadrant represent attribute levels that are below the mean drop threshold but above the frequency threshold. This is another area of secondary concern.

Net Penalty Tables

1. Attribute: Selected attributes.
   
   
2. Category Low: The collapsed lower levels of your scale; lowest value label is shown.
   
   
3. Net Penalty: Weighted difference of the means between JAR and 'not enough' levels.
   
   The formula: Net Penalty= [Proportion indicated Not Enough] *[Mean Drop]
   
   
4. Mean Drop: The difference in the means between JAR and 'not enough' levels.
   
   
5. % Not Enough: Percentage of respondents who selected below JAR.
   
   
6. % JAR: Percentage of respondents who selected JAR.
   
   
7. % Too Much: Percentage of respondents who selected above JAR.
   
   
8. Mean Drop: The difference in the means between JAR and 'too much' levels.
   
   
9. Net Penalty: Weighted difference of the means between JAR and 'too much' levels.
   
   The formula: Net Penalty= [Proportion indicated Too Much] *[Mean Drop]
   
   
10. Category High: The collapsed higher levels of your scale; highest value label is shown.
    

Net Penalty Graphs

The Net Penalty thresholds are selected before running the report.

There is a separate graph for each sample (you will need to scroll down to see them all). Within the graph you will see the following:

1. Each attribute that meets your threshold settings.
   
   
2. The net penalty of the 'too much' and 'not enough' levels. 
   
   
3. The frequency that the JAR level was selected, displayed as a percentage. 
   

Troubleshooting Report Issues

For Penalty analysis report to be compatible and available as a reporting option in your test or section, the test or section must have at least one Category Liking question and at least one Category JAR scale in it.

The Penalty Analysis Report is Not an Option!

1. Your test or section is not a Standard test type. Discrimination tests are not compatible with the Penalty analysis report. In conclusion, there is nothing you can do to make the report compatible with such tests or sections if you already collected the data. 
   
   If it happened that you accidentally used Choose questions instead of Category questions, you might be able to export the results, create a new test using Category questions and import the results. 
   
   --or 
   
   
2. Your test or section is a Standard test type, but it is set up without any Category questions questions. This also makes the report incompatible with the test, therefore it will not be available to be selected from the list of reports.
   
   Reminder, Penalty analysis report cannot be generated if there are no Category liking and Category JAR questions in the test or section.
   
   --or
   
   
3. Your test or section is a Standard test type and it has Category questions named "Overall Liking" and "JAR", but those Category questions are not set up as Liking type and JAR type. Instead they were accidentally set up as Intensity and Purchase intent question types.
   
   Solution: Don't worry, you can still make it work! If it happens that the question types are accidentally set to a different variety of Category scale questions, even after the data is collected, the question types can be changed to a proper type. If your test is set to Complete, you will have to undo complete to be able to make the change, but please do not delete results to make this change! Merely naming a Category question as "Overall liking" or a "JAR" does not make them liking or JAR type.
   
   Changing the question type is done in the Build tab, in the Category Question Options, as seen in the image below.
   

The Penalty Analysis Report Generates, but Gives Errors When Trying to Open it!

1. Follow the steps for generating the report and focus on the step 4 where you click 2. Select options. 
   
   
2. The Penalty Inclusion setting is what is likely causing your report not to open. If the Penalty Inclusion is set to a value that excludes all the data, then there is nothing in the report, and it gives errors. For example, if you set the Penalty Inclusion to be 0.2, but your JAR data shows that for all attributes for all samples its frequency is greater than 80% then there is nothing to include in the report, therefore errors generate when trying to open the report.
   
   
   If the default Penalty Inclusion of 0.2 is your company standard, you can conclude that no attribute affected any of the samples' overall liking.
   
   If your company standard for the Penalty Inclusion is different from 0.2, specify the right value or uncheck the option and generate the report again. 

The Penalty Analysis Report Generates, but It's Missing a Sample!

1. You may have filtered out some samples before generating the report and forgot that you did that.
   
   Solution: Clear of otherwise update the Filters and generate the report again.
   
   
   
2. Similar to the previous example, this usually indicates that the threshold values set before generating the report are excluding some of the data out. 
   
   
   
   Solution: Review the threshold values before generating the report again by following these steps:
   
1. Follow the steps for generating the report and focus on the step 4 where you click 2. Select options. 
   
   
2. The Penalty Inclusion setting is what is likely causing your report to exclude some samples. If the Penalty Inclusion is set to a value that excludes some of the data, then some samples might not get included in the report. For example, if you set the Penalty Inclusion to be 0.2, but your JAR data for one or more samples is shows that for all attributes for those samples its frequency is greater than 80% then those samples will not be included in the report. In our example image we can see that the Sample 1's JAR frequency is above 80 for all attributes, therefore, the sample is not included in the report; its liking is not affected by anything at the set threshold levels.
   
   
   
   If the default Penalty Inclusion of 0.2 is your company standard, you can conclude that the overall liking of sample in question was not affected by any attribute affected.
   
   If your company standard for the Penalty Inclusion is different from 0.2, specify the right value or uncheck the option and generate the report again.