Sensory Computer Systems Forums Index Sensory Computer Systems Forums
Welcome to the Sensory Computer Systems Online Forums!
 
 FAQFAQ   SearchSearch   RegisterRegister 
 ProfileProfile   Log inLog in   
   Sensory Computer Systems Home   

A / Not A Test

 
Post new topic   Reply to topic    Sensory Computer Systems Forums Index -> Test Design
View previous topic :: View next topic  
Author Message
Mary



Joined: Dec 21 2005
Posts: 8
Location: North America

PostPosted: Mon Feb 06, 2006 9:40 am    Post subject: A / Not A Test Reply with quote

How do I set up an A / Not A Test in SIMS 2000?

Please also show how to set up a Chi-Square Test for analysis.

Thanks, Mary
Back to top
View user's profile
ForumAdmin
Site Admin


Joined: May 20 2005
Posts: 120
Location: New Jersey

PostPosted: Wed Nov 13, 2013 2:59 pm    Post subject: A / Not A Test Reply with quote

Updated Nov 2013

The following describes how to set up an "A" - "Not A" Test in your SIMS software. Although SIMS does not have a specifically named "A" - "Not A" Test option, it is fairly easy to set up. The A - Not A Test method is used to determine whether a sensory difference exists between two products, and is sometimes used when your two samples are unsuitable for triple or multiple test presentations, e.g. when the Triangle and Duo-trio tests cannot be used. The method is often recommended as preferable to the Same/Different test when one of the two products has a position as a standard or reference product, i.e., it is the standard against which all other similar products are measured.

"A" and "Not A" refer to specific samples. In your SIMS software we will designate sample "A" as sample 1 and sample "Not A" as sample 2.

The standard test procedure first familiarizes the panelists with both a sample "A" and a sample "Not A", this occurs before the test begins, then your blind test sample(s) are presented to the panelist one at a time.

You can present just one sample to each panelist and ask them to determine if the presented sample is "A" or "Not A.". Or present multiple samples as reps to each panelist, such as Civille’s Example 6.11 which shows 10 Reps.

You may optionally also ask other attributes/questions about the test samples.


Two test designs are discussed below:

Present One Sample Per Panelist.

Present Two or More Samples Per Panelist, using multiple replications (reps). Such as Civille's Example 6.11


Other designs for presenting two or more samples exist that are more complex than the above examples, depending on how you want the A and NOT A combinations presented to each panelist, usually requiring a large n of panelists. You can experiment with SIMS 2000 multiple Rep Experimental Plans (2C1Rx) and get to any presentation required. Carefully setup an Experimental Plan and Design that meets your requirements.

And a note further down about using overblocked permutations (11, 12, 21, 22), which does not appy to A / NOT A methods.


Reference: Sensory Evaluation Techniques, Civille, 4th Ed.(pg 89), 3rd Ed.(pg 82), 2nd Ed.(pg 77).



Last edited by ForumAdmin on Fri Nov 15, 2013 9:03 am; edited 13 times in total

Back to top
View user's profile
ForumAdmin
Site Admin


Joined: May 20 2005
Posts: 120
Location: New Jersey

PostPosted: Wed Nov 13, 2013 4:52 pm    Post subject: Present One Sample Per Panelist Reply with quote


Present One Sample Per Panelist

SIMS software express package of the following example: A_NOT_A.SIMS2000ExpressPackage.sql.ZIP

Reminder: The standard test procedure first familiarizes the panelists with both a sample "A" and a sample "Not A", this occurs before the test begins, then your blind test sample is presented to the panelist.

1) Questionnaire:

    Questionnaire Example:




    1. Create a questionnaire, Affective type, named A_NOT_A (reusable for future studies)

    2a. Create a Hedonic attribute question with two choices:
      ( ) A (and set Return Value = 1)
      ( ) NOT A (and set Return Value = 2)

    2b. Set the Hedonic 'Attribute Seen With' option to 'All Samples'

    As always please test your test.

    Questionnaire Example Technician Setup:




2) Experimental Design:
    You will present either Sample A (as Sample 1) or Sample NOT A (as Sample 2) to each panelist.

  1. First, create a new Experimental Plan, named 02010002 (reusable for future studies), 2C1R1

    Creating this plan is SIMS is easy and should take under a minute, and it will always be there for future tests. The recommended name above is consistent with the standard naming system used for Experimental Plans in SIMS 2000. In this case it indicates 2 Samples, 1 Presented, and 2 Blocks.

    Plan Type: Affective/Descriptive
    Number of Samples(T): 2
    Number Presented (K): 1
    Number of Blocks: 2 (SIMS should create the blocks automatically as Block 1 -> 1 and Block 2 -> 2.)
    Number of Reps: 1


  2. Create a new Experimental Design named A_NOT_A (reusable for future studies), 2C1R1
    Design Type: Affective/Descriptive
    Number of Respondents: Usually divisible by 2
    Number of Samples: 2
    Number of Reps: 1
    Experimental Plan: Use 02010002, the one you created above.
    Block Presentation 'As in Plan', Sample Presentation 'As in Plan'
    Save, then Go to 2nd Tab, 'Sample Information'
    Optionally change Blinding codes and Sample Descriptions, Sample 1 is A, 2 is NOT A


3) Test Definition:

    3a. Create a new Test Definition, maybe named A_NOT_A (reusable for future studies), using your new Questionnaire and Design created above.
    3b. View and/or Print the Rotation Plan for technician to prepare the ballots properly.

4) Test Result Master / Execution:
    4a. Create a new Test Result Code, maybe named A_NOT_A (reusable for future studies), using the Test Definition created above.
    4b. Start the Test.

    There are no special requirements for Test Administration, since each sample is evaluated individually as to whether it is A or NOT A.


5) After the Test is completed:
    See Posting below titled After the Test is completed...


Present One Sample Per Panelist
SIMS software express package of the above example: A_NOT_A.SIMS2000ExpressPackage.sql.ZIP



Last edited by ForumAdmin on Fri Nov 15, 2013 9:20 am; edited 10 times in total

Back to top
View user's profile
ForumAdmin
Site Admin


Joined: May 20 2005
Posts: 120
Location: New Jersey

PostPosted: Wed Nov 13, 2013 6:05 pm    Post subject: Present Two or More Samples Per Panelist, multiple reps. Reply with quote


Present Two or More Samples Per Panelist, using multiple replications (reps). Such as Civille's Example 6.11

SIMS software express package of the following example: A_NOT_A_2.SIMS2000ExpressPackage.sql.ZIP

Reminder: The standard test procedure first familiarizes the panelists with both a sample "A" and a sample "Not A", this occurs before the test begins, then your blind test sample(s) are presented to the panelist one at a time.

This example assumes 2 reps for simplicity, if you need 3+ reps simply change the ballot branches as needed and the experimental design #reps.

1) Questionnaire:

    Questionnaire Example:




    1. Create a questionnaire, Affective type, maybe named A_NOT_A_2 (reusable for future studies)

    2a. Create your first Hedonic attribute question with two choices:
      ( ) A (and set Return Value = 1)
      ( ) NOT A (and set Return Value = 2)

    2b. Set the first Hedonic 'Attribute Seen With' option to 'Absolute Sample Number 1'
    2c. Set the Data Export Label to be 'A_NOT_A ABS1', where ABS1 indicates the Absolute Sample.
    ---- Panelist response is correct when panelist selects Same as A.

    3a. Create your second Hedonic attribute, easiest is to Copy the first hedonic attribute created in Step 2, and paste it directly below that attribute.

    3b. Modify the second Hedonic attribute's return values of the two hedonic choices:
      ( ) A (and set Return Value = 3)
      ( ) NOT A (and set Return Value = 4)

    3c. Set the second Hedonic 'Attribute Seen With' option to 'Absolute Sample Number 2'
    3d. Set the Data Export Label to be 'A_NOT_A ABS2', where ABS1 indicates the Absolute Sample.
    ---- Panelist response will be correct when panelist selects Not same as A.

    Why is this ballot a little more complex to setup then the previous single rep example? Answer: Because the two Hedonic attributes as described above make it much easier to compile the multiple reps data and statistics needed to evaluate the A - Not A Test.

    The first attribute created in step 2 will be shown to the panelist only when Sample A is presented (Absolute Sample Number 1) , so therefore a correct response would be evaluated as 1, and an incorrect response as 2.

    The second attribute created in step 3 will be shown to the panelist only when Sample 'Not A' is presented (Absolute Sample Number 2) , so therefore a correct response would be evaluated as 4, and an incorrect response as 3.

    This ballot setup will also work for the first scenario described above for the two most common experimental designs.

    As always please test your test.


    Questionnaire Example Technician Setup:




2) Experimental Design:
    Same setup as above for PRESENT ONE SAMPLE, except #Reps will be >=2 in your Experimental Design.
    Use the Reps option for the number of samples presented. Same Experimental Plan as above named 02010002, 2C1R1

    For example a 2 rep design where each panelist sees the same A / Not A question twice, for a block order of 1 2 or 2 1.

    One at a time, you will present either Sample A (as Sample 1) or Sample NOT A (as Sample 2) to each panelist.

  1. First, create a new Experimental Plan, named 02010002 (reusable for future studies) (and the same as above example for Present One), 2C1R1

    (The recommended Experimental Plan Name is consistent with the standard naming system used for Experimental Plans in SIMS 2000. In this case it indicates 2 Samples, 1 Presented, and 2 Blocks.)

    Plan Type: Affective/Descriptive
    Number of Samples(T): 2
    Number Presented (K): 1
    Number of Blocks: 2 (SIMS should create the blocks automatically as Block 1 -> 1 and Block 2 -> 2.)
    Number of Reps: 1 (see the multiple reps needed in next step)

  2. Create a new Experimental Design named A_NOT_A_2 (reusable for future studies, or _X for the number of Reps you need), 2C1R2, 2C1Rx
    Design Type: Affective/Descriptive
    Number of Respondents: Usually divisible by 2
    Number of Samples: 2
    Number of Reps: X (use number of Reps you need)
    Experimental Plan: Use 02010002, the one you created above.
    Block Presentation 'As in Plan', Sample Presentation 'As in Plan'
    Save, then Go to 2nd Tab, 'Sample Information'
    Optionally change Blinding codes and Sample Descriptions, Sample 1 is A, 2 is NOT A

    Civille's Example 6.11 with 10 reps would be SIMS Experimental Plan 2C1R10, 4 blocks shown in her Figure 6.15. Setup the exact presentation shown across 10 reps in a SIMS Experimental Plan. Experimental Design n=20, don't randomize Blocks or Samples.

3) Test Definition:

    3a. Create a new Test Definition, maybe named A_NOT_A_2 (reusable for future studies), using your new Questionnaire and Design created above.
    3b. NOTE: When saving the Test Definition, when asked about multiple reps blocking option, say YES, different block each rep.
    3c. View and/or Print the Rotation Plan for technician to prepare the ballots properly.

4) Test Result Master / Execution:
    4a. Create a new Test Result Code, maybe named A_NOT_A_2 (reusable for future studies), using the Test Definition created above.
    4b. NOTE: When executing test, select option for [X] Allow Continuous Presentation Over Reps.
    4c. Start the Test.

    There are no special requirements for Test Administration, since each sample is evaluated individually as to whether it is A or NOT A.

5) After the Test is completed:
    See Posting below titled After the Test is completed...


Present Two or More Samples Per Panelist, using multiple replications (reps)
SIMS software express package of the above example: A_NOT_A_2.SIMS2000ExpressPackage.sql.ZIP



Last edited by ForumAdmin on Fri Nov 15, 2013 9:40 am; edited 8 times in total

Back to top
View user's profile
ForumAdmin
Site Admin


Joined: May 20 2005
Posts: 120
Location: New Jersey

PostPosted: Wed Nov 13, 2013 8:56 pm    Post subject: Present Two or More Samples Per Panelist, using overblocked Reply with quote


Present Two or More Samples Per Panelist, using overblocked permutations (11, 12, 21, 22).

This example would deviate from the standard method for A / Not A, which states that you first familiarize the panelists with both a sample "A" and a sample "Not A", this occurs before the test begins, then your blind test sample(s) are presented to the panelist one at a time.

This example could be two seperate A / NOT A tests. First test where A is the reference sample and the second test where NOT A is the reference sample. Or restated, first test where your Control sample is the reference and the second test where your Test sample is the reference. Run the two tests, and combine the results into your chi square excel worksheet.

This example is really a SAME/DIFFERENT test, please see that forum topic here: http://www.sensorycomputersystems.com/phpbb2/viewtopic.php?t=60



Last edited by ForumAdmin on Fri Nov 15, 2013 9:25 am; edited 15 times in total

Back to top
View user's profile
ForumAdmin
Site Admin


Joined: May 20 2005
Posts: 120
Location: New Jersey

PostPosted: Wed Nov 13, 2013 9:20 pm    Post subject: After the Test is completed Reply with quote


5) After the Test is completed:

    This is a multiple Rep example.

    Quick look at the Raw Data Export, and select column option for [X] Export the choice captions instead of the data, and show Sample Descriptions.
      Column Sample Description shows the Sample Presented to the Panelist, either A or Not A
      Column A_NOT_A ABS1 shows the Panelist Responses when they where presented A (absolute sample 1)
      Column A_NOT_A ABS2 shows the Panelist Responses when they where presented A (absolute sample 2)




    Test Data Frequency Report

    5a. Run the Test Data Frequency Report on the Test. Choose the Sort Option "By Attribute Within Sample." or "By Attribute - Samples Appear in a Table Format"
    5b. Look at the hedonic attributes as created above in 1) Questionnaire.
    5c. Note the numbers in the report for the Hedonic attribute under Sample 1.
    It will look like this example (n=100):

    Code:

    Sample: Sample 1 A   (135)

        Attribute Type: Hedonic Scale   Description:  A_NOT_A ABS1

        Hedonic Labels on Questionnaire are, by Return Value in Label(n):
                Label(1) = A      (Ret value: 1)
                Label(2) = NOT A  (Ret value: 2)

                                       Cumulative  Cumulative
        Value     Frequency  Percent    Frequency   Percent
        ----------------------------------------------------------
        1            60        60.0        60        60.0
        2            40        40.0        100       100.0


    For Sample 1 A (the Sample Presented to the Panelist), the numbers in the Frequency column show how the panelists presented with Sample 1 A identified the sample. In the above example 60 panelists correctly identified the sample as A, and 40 incorrectly identified the sample as NOT A.


    Similarly, the report for the Hedonic attribute under Sample 2 will look like this:

    Code:

    Sample: Sample 2 NOT A   (246)

        Attribute Type: Hedonic Scale   Description:  A_NOT_A ABS2

        Hedonic Labels on Questionnaire are, by Return Value in Label(n):
                Label(1) = A      (Ret value: 1)
                Label(2) = NOT A  (Ret value: 2)

                                       Cumulative  Cumulative
        Value     Frequency  Percent    Frequency   Percent
         ----------------------------------------------------------
         1            35         35.0        35        35.0
         2            65         65.0        100       100.0


    For Sample 2 NOT A (the Sample Presented to the Panelist) 35 panelists incorrectly identified the sample as A, and 65 correctly identified the sample as NOT A.

    Note: Any of the Frequency Report sort options should work fine EXCEPT for the 3rd option labeled "By Attribute Aggregate Over All Samples.". Our favorite sort option here is "By Attribute - Samples Appear in a Table Format", which clearly shows which sample was presented (columns) and which samples (A or NOT A) the panelists selected.

    5e. Open this Excel spreadsheet discussed below and Go to Chi-SquareAnalysis worksheet tab.
    5f. Copy the four numbers in the Frequency column from this Test Data worksheet to the Analysis Excel spreadsheet.



Reference: Sensory Evaluation Techniques, Civille, 4th Ed.(pg 89), 3rd Ed.(pg 82), 2nd Ed.(pg 77)


A / NOT A Test:

Here's how to do the Chi-Square Analysis in Microsoft Excel.

Sensory Computer Systems Excel File Available:
A_Not A_Test_Analysis_Using_Chi-Square.xls

Example Chi-Square Analysis:


Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    Sensory Computer Systems Forums Index -> Test Design All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group