PAL
Joined: Jun 01 2005 Posts: 24

Posted: Fri Feb 03, 2006 6:01 pm Post subject: DifferenceFromControl Test 


Here's an example of the Dunnett's Statistics in your SIMS System:
Updated Summer 2022
Set up:
This document describes how to set up a DifferenceFromControl Test in SIMS software.
This test is also known as a Degree of Difference Test (DOD).
To jump right to Dunnett's T Test see #6) below.
Use this test when the project objectives include the following:
 Determine whether there is a difference between a control (reference) sample and one or more other samples.
 Estimate the degree of such difference(s).
Questionnaire example showing both an hedonic scale and a line scale.
Overview
Present to each panelist the control (reference) sample plus one or more additional samples. Panelist will rate the degree of difference between each sample and the control. Explain to panelists that some of the test samples may be identical to the control.
While it is acceptable to ask the panelists other questions about each sample, for the purpose of this discussion assume that the differencefromcontrol question is the only one on the test.
1) Questionnaire
 Create a questionnaire, Affective type, perhaps named DIFF_CTRL (reusable for future studies).
(Note: As discussed below, the degree of difference attribute can be either a Line Scale or Hedonic Scale. While it is possible to use a Descriptive questionnaire for a Line Scale attribute, setting the questionnaire type as Affective provides more flexibility to add other attribute types at a later time.)
 Create a Hedonic or Line Scale attribute which will be used to record the extent of the difference of each sample from the Control. For a Hedonic Scale the categories will look something like this:
Code: 
Choice Description Return Value
o No Difference 0
o Very Small Difference 1
o Small Difference 2
o Small/Moderate Difference 3
o Moderate Difference 4
o Moderate/Large Difference 5
o Large Difference 6
o Very Large Difference 7

For a Line Scale the Minimum Value should be 0 associated with a description of "No Difference." The Maximum Value should correspond to a description like "Very Large Difference." The scale can be either continuous or incremental. Reference Points associated with midlevel degrees of difference may be included to aid the respondents.
 Question Example: Please rate the level of difference for Sample nnn relative to the control sample.
 Set the "Attribute Seen With" option to "All Samples."
SIMS software express package file: DIFFCTRL.SIMS2000ExpressPackage.sql.ZIP
Note: This forum topic and the notes below are about presentations which include Control vs Blind Control. Control vs Control
Control vs Test Sample #1
Control vs Test Sample #2
In contrast, if your study only includes presentations of Control vs Test(s), and does not incorporate presentations for Control vs Blind Control, then we suggest the following:
 For a simple Control vs Test we recommend a simple 1 sample study, a DOD type question, and after the test, review your raw data, means, standard deviations, and other statistical methods of your choosing.
 For a simple Control vs Test1 + Control vs Test2 we recommend a simple 2 sample study, a DOD type question, and after the test, review your raw data, means, standard deviations, and other statistical methods of your choosing.
 skip all of the notes below.
Read on for:
For Control vs Blind Control , Control vs Test1 , Control vs Test2. This is a 3C3 design.
and very similar if just Control vs Blind Control , Control vs Test1. This is a 2C2 design.
and very similar Control & 3 test samples. This is a 4C4 design.
Etc., etc.
2) Experimental Design
The Experimental Design must be a Complete Block design such as 3 Samples see 3 in order to employ the Dunnett Test when the statistics are generated. It is recommended that Sample Order be randomized within each block to eliminate positional bias, which is called a "Balanced Complete Block".
The Control or Reference Sample is USUALLY designated as Sample 1 in the Experimental Design.
Control
Test Sample #1
Test Sample #2
Each Sample Set presented to a panelist will always include a sample labeled as the "Control" or "Reference." In addition the panelist will also be presented with each of the samples labeled with a threedigit Blinding Code.
The panelist will first taste the Control sample, then they will taste the 3 blind coded samples, one at a time, rating the difference from Control.
Effectively the panelist will be evaluating, in random order:
Control vs Control
Control vs Test Sample #1
Control vs Test Sample #2
In this example, the Experimental Plan should be Permutations 3 choose 3 (blocks 1 2 3 randomized).
So the panelist will compare the Control with each of the blind ordered samples. Exactly one of the ordered samples will be the same as Control, but panelist will not know which one.
3) Test Definition
 Create a new Test Definition by selecting your new Questionnaire and Experimental Design. Use whatever Panel Type option works for you.
 Print the Rotation Plan for the technician to prepare the ballots properly.
 In addition, have the technician also prepare a Control sample to include with every ballot.
4) Test Result Master / Execution
 Create a new Test Result Code, using the Test Definition above.
 Start the Test.
5) Test Administration You may need to explain to the panelists that the SIMS Sample Verification Prompt will not include the Control or Reference sample. However, each panelist must have this Control sample to complete the test properly. This explanation may be important to those panelists used to the presentation of SIMS DuoTrio tests, which explicitly include the "REF" sample in the Sample Verification Prompt.
6) After the Test is Completed  Dunnett's T Test
 Run Statistical Analysis.
 Select the "Dunnett's T Test" option
Used for Multiple Comparison Analysis of a Difference from Control test.
The Dunnett's test is similar to the Tukey test, but Dunnett's only compares your Control Sample with each of the other test samples, i.e., a Difference from Control test.
1 vs. All. Sample #1 is generally the default Control Sample for Dunnett's statistics. The Default for your Control Sample is "1 vs."
Optionally you can specify a different Control Sample for Dunnett's statistics via the list box shown, "1 vs.", "2 vs.", "3 vs." etc.
SIMS OUTPUT REPORTING NOTES: SIMS Dunnett's output.
Pvalue: The Pvalue on the main TABLE is NOT the reliable indicator of significance for Dunnetts Test.
This is because the TABLE's Pvalue shows significant difference amoung ALL the Test Samples, ANOVA.
For your Dunnett's TTests you should be looking at the individual significant differences between 1 vs 2, 1 vs 3, 1 vs 4, etc.
See your Dunnett's Pvalues in the TABLE's additional columns labeled 'PValue Dunnett 1 vs 2', 'PValue Dunnett 1 vs 3', etc.
To further interpret your Dunnett's statistical results, go to 'View Analysis Files', and click on 'Statistics Output File'.
See examples below.
Code: 
R Dunnett's Output File Example

Go to the R Section labeled ' Dunnett's Results DVn ' to determine your individual pairs significance.
Pvalues, your individual pairs Pvalues will be shown under column 'Pr(>t)'
Your Individual pairs significance levels will be shown to the right of each pvalue.
See the value for 'Minimum Significant Difference' (MSD), compare this to the individual values for 'Difference Between Means' (DBM).
Locate the MSD right near the top of the R Dunnett's Results output; See 'Minimum Significant Difference: n.nnnn'
Locate the DBM a little further down; See 'Linear Hypotheses:', the DBM is the 'Estimate' column.
If your DBM is greater than MSD (ie. DBM > MSD) then that Control vs test sample pair combination is YES significantly different.
Also see the confidence limits section, two value columns, one row for each sample comparison:
If 0 (zero) is not in the interval value range, then the pair is NOT significantly different.
Most obvious when lower number is negative and upper number is positive, 0 (zero) is in this intraval range,
then the pair is NOT significantly different.
If 0 (zero) is in the interval value range, then the pair is YES significantly different.
Most obvious when lower number & upper number are both above or below 0, 0 (zero) is NOT in this intraval range,
then the pair is YES significantly different.
R Example:
 Begin: DV1 Overall Liking (Hedonic Scale) 

 Dunnett's Results DV1 

Alpha: 0.05
Error Degrees of Freedom: 8
Error Mean Square: 1.1833
Critical Value of Dunnett's t: 2.6731
Minimum Significant Difference: 1.8391
Simultaneous Tests for General Linear Hypotheses
Multiple Comparisons of Means: Userdefined Contrasts
Linear Hypotheses:
Estimate Std. Error t value Pr(>t)
2  1 == 0 3.0000000 0.6879922 4.36051 0.0044195 **
3  1 == 0 3.6000000 0.6879922 5.23262 0.0014598 **
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Adjusted p values reported  singlestep method)
Simultaneous Confidence Intervals
Multiple Comparisons of Means: Userdefined Contrasts
Quantile = 2.6731265
95% familywise confidence level
Linear Hypotheses:
Estimate lwr upr
2  1 == 0 3.0000000 1.1609097 4.8390903
3  1 == 0 3.6000000 1.7609097 5.4390903
In this example, both sample comparisons 1 vs 3 and 1 vs 2 are significantly different.

Code: 
SAS Dunnett's Listing File Example

Go to the SAS GLM Section labeled 'Dunnett's t Tests for Xn' to determine your individual pairs significance.
See the value for 'Minimum Significant Difference' (MSD), compare this to the individual values for 'Difference Between Means' (DBM).
If your DBM is greater than MSD (ie. DBM > MSD) then that Control vs test sample pair combination is YES significantly different.
Also see the confidence limits section, two value columns, one row for each sample comparison:
If 0 (zero) is not in the interval value range, then the pair is NOT significantly different.
Most obvious when lower number is negative and upper number is positive, 0 (zero) is in this intraval range,
then the pair is NOT significantly different.
If 0 (zero) is in the interval value range, then the pair is YES significantly different.
Most obvious when lower number & upper number are both above or below 0, 0 (zero) is NOT in this intraval range,
then the pair is YES significantly different.
SAS Example:
Dependent Variable: X1 Overall Liking
The GLM Procedure
Dunnett's t Tests for X1
alpha 0.05
Error Degrees of Freedom 8
Error Mean Square 1.183333
Critical Value of Dunnett's t 2.67281
Minimum Significant Difference 1.8389
Comparisons significant at the 0.05 level are indicated by ***.
Difference
SAMPLE Between Simultaneous 95%
Comparison Means Confidence Limits
3  1 3.6000 1.7611 5.4389 ***
2  1 3.0000 1.1611 4.8389 ***
The GLM Procedure
Least Squares Means
Adjustment for Multiple Comparisons: Dunnett
H0:LSMean=
Control
SAMPLE X1 LSMEAN Pr > t
1 1.60000000
2 4.60000000 0.0044
3 5.20000000 0.0015
In this example, both sample comparisons 1 vs 3 and 1 vs 2 are significantly different.

See the Dunnett's T Test Notes in your SIMS system for more information.
Reference: Sensory Evaluation Techniques, Civille, 4th Ed. (pg. 92), 3rd Ed. (pg.86), 2nd Ed. (pg. 81)

