Sensory Computer Systems Forums

[Mary]

How can I setup a Same/Different Test in our SIMS System?

And Chi-Square Analysis.

And Fisher's Exact Test Analysis.

[John]

Here's our detailed Same/Different Test webpage, from your SIMS test set-up and thru your statistical analysis.

SAME / DIFFERENT Tests in SIMS

Updated Summer 2017

SIMS software express package of the following example: SAMEDIFF.SIMS2000ExpressPackage.sql.ZIP

The Same/Different Test is also commonly referred to as the Simple Difference Test. The Same/Different Test method is sometimes used when your two samples are unsuitable for triple or multiple test presentations, e.g. when the Triangle and Duo-trio tests can not be used.


The following describes how to set up a Same/Different Test in SIMS. Although SIMS does not have a specifically named Same/Different Test option, it is fairly easy to set up and prepare the Chi-Square Analysis.


This description assumes that you are not asking individual questions about each sample.


1) Questionnaire:

Questionnaire Example:

1. Create a questionnaire, Affective type, maybe named SAMEDIFF  (reusable for future studies)
     
2. Create a single Hedonic attribute question with two choices:
             ( )  The two samples are the SAME.  (Set Return Value = 1)
             ( )  The two samples are DIFFERENT. (Set Return Value = 0)
   - This is an important step, please double check your Return Values.
   If you use different return values, then after the test if you use our Chi-Square Analysis Excel XLS file discussed below, you'll need to adjust the all formulas in the result column G.
   
   
   
3. Set the Hedonic 'Attribute Seen With' option to 'Relative Sample Order 1.'
     
4. Set any/all other Questionnaire elements, such as Instruction boxes, same 'Seen With' as 1c. above.
   - This is an important step, please double check your selected 'Seen With' settings.
   
   
   
5. Optional Feedback to Panelist:
   - Use an Instruction Box (any page following the above Hedonic).
   - Add in the special Tag <SAME/DIFFERENT>
   - Which will transform to text SAME or DIFFERENT depending on current Panelist's Sample Set.
   - Example: Your samples were <SAME/DIFFERENT>.
   
   
   

2) Experimental Design:

    You want to present samples to the panelists in one of these four combinations:
       AA  Two Sample A (SAME)
       AB  Sample A followed by Sample B (DIFFERENT)
       BA  Sample B followed by Sample A (DIFFERENT)
       BB  Two Sample B (SAME)

1. First, create a new Experimental Plan, named SAMEDIFF  (reusable for future studies)
        Plan Type:  Affective/Descriptive
        Number of Samples(T): 2 
        Number Presented (K): 2
        Number of Blocks: 4  (each associated with each of the AB combinations above)
        Number of Reps: 1
        Save, then Go to 2nd Tab, 'Blocks'
        Edit Block  1 -> 1 1  /  2 -> 1 2  /  3 -> 2 1  /  4 -> 2 2
   If you use a Block sample pairings are not exactly 1 1, 1 2, 2 1, and 2 2, then after the test if you use our Chi-Square Analysis Excel XLS file discussed below, you'll need to adjust the Lookup Table shown in Excel columns H&I, and J&K
     
2. Create a new Experimental Design, named SAMEDIFF  (reusable for future studies)
        Design Type:  Affective
        Number of Respondents:  Usually divisible by 4 (such as 48, 60, 120, etc.)
        Number of Samples: 2
        Number of Reps: 1
        Experimental Plan:  Use SAMEDIFF, the one you created above in 2a.
        Block and Sample Presentation 'As in Plan', 'As in Plan'. ---- Do *not* select Randomization.
        Save, then Go to 2nd Tab, 'Sample Information'
        Optionally change Blinding codes and Sample Descriptions.
   
   
3. See Print Screen Example Experimental Design at the end of this Forum Post below.
   

3) Test Definition:

1. Create a new Test Definition, select your new Questionnaire and Design.
     
2. Print the Rotation Plan for technician to prepare the ballots properly.
   

4) Test Result Master / Execution:

1. Create a new Test Result Code, using the Test Definition created above.
     
2. VERY IMPORTANT: for Sample Set Selection,
              please select  'Force Panelist to Register Sample Set as Ballot Number'
     
3. Start the Test. 
     
4. Test Execution Parameters:
              Check 'Force All Panelists to Exit After Every:  1 Sample'
              Check 'Bypass Sample Set Verification Prompt' (optional)
              Check 'Bypass Sample Number Verification Prompt'
              Check 'Remove Sample Code & Count from Navigation Bar'
   

5) Test Administration:

1. Each tray should have a Ballot Number and two samples with Blinding Codes.
   Ballot Numbers will be 1 to the number of respondents (N).
   Properly match the 2 samples and Ballot numbers on each tray.
   See the Rotation Plan, Ballot Number = Sample Set.
     
2. The panelist will be asked to enter the Ballot Number.
   

6) Final Pre-Test Step:

Test your Test, as you should do with any new type of Test.

7) Data Results.

Tallying the Same/Different question results is more complicated than usual since you have to consider a) did the panelist get a Matched or Unmatched pair, and b) did the panelist respond Same or Different.

You need four numbers:
Count of Matched samples presented and panelist responded SAME
Count of UnMatched samples presented and panelist responded SAME
Count of Matched samples presented and panelist responded DIFFERENT
Count of UnMatched samples presented and panelist responded DIFFERENT


- Option 1 recommended: Run the SIMS Test Data Frequency Report, use the 3rd sort option: By Attribute - Aggregated Over All Samples. New November 2013


SIMS Test Data Frequency Report example:




- Option 2 manually: The Excel file has an older method on Excel worksheet tab #1, involving using the SIMS raw data export report to excel.

- Option 3 manually: Print out the SIMS 2000 Test Definition Rotation Plan and align with the Test Raw Data

[ForumAdmin]

After your Same/Different Test is completed:

Here's how to do the Chi-Square Analysis in Microsoft Excel. Chi-Square Analysis is also known as χ2-Analysis.

Sensory Computer Systems Excel File Available:
Same-Different_Test_Analysis_Using_Chi-Square_Using_Fishers_Exact_Test.xls (or right click and 'Save Target As' or 'Save Link As' )

Example, Chi-Square Analysis and Fisher's Exact Test Analysis:

Last edited by ForumAdmin on Thu May 18, 2023 3:39 pm; edited 9 times in total

[ForumAdmin]

Miscellaneous


Print Screen Example of Experimental Design:

**********************************************************************
**********************************************************************

Advanced

Multiple Reps (each panelist doing same/diff test more than once).

1. Use the same Ballot SAMEDIFF.
   
2. Copy Experimental Design SAMEDIFF to SAMEDIFF2, change it to 2 or more Reps. Same Experimental Plan is ok.
   
3. New Test Definition SAMEDIFF2. Review and understand the Rotation Plan.
   
4. Test Execution, same instructions as written above, plus
   
   Select [X] Allow Continuous Presentation Over Reps, panelists will signin once, do both reps in one sitting.
   Do NOT Check 'Force All Panelists to Exit After Every: 1 Sample' as stated above.
   Check 'Bypass Sample Set Verification Prompt', because only first sample set pair will be shown.
5. Final Step: Test your Test, as you should do with any new type of Test.

Advanced

Fisher's Exact Test (FET) statistical test of the equality of two independent binomial proportions.

- Sensory Computer Systems Excel File Available:
Same-Different_Test_Analysis_Using_Chi-Square_Using_Fishers_Exact_Test.xlsm (or right click and 'Save Target As' or 'Save Link As' )

- Discussed in ASTM E2139-05 Standard Test Method for Same-Different Test
- Alternate online calculators: http://www.graphpad.com/quickcalcs/contingency1.cfm and http://www.langsrud.com/fisher.htm
- More notes and formulas: http://en.wikipedia.org/wiki/Fishers_exact_test and http://faculty.vassar.edu/lowry/fisher.html
- Formula excel example: http://udel.edu/~mcdonald/statfishers.html
- Fishers Exact Test vs Chi-Square, chi-square calculates approximation P-value and it more suited to tests where the # of assessors is not sparse, ie. where all cells expected frequency values is >=5. In many cases the Chi-square is preferred because the Fisher Exact Test is difficult to calculate. Chi-Square Analysis is also known as χ2-Analysis.

References: Sensory Evaluation Techniques, Civille, 4th Ed.(pg 84), 3rd Ed.(pg 79), 2nd Ed.(pg 74).
ASTM E2139-05 Standard Test Method for Same-Different Test

Updated Summer 2017