h1. Introduction What does this feature do? Enable pattern match question authors to maximise the grading reliability of the questions they create by determining how reliably a question grades answers compared to a set of human markers. AMATI simplifies the process of grading free text answers by focusing on fewer more efficient rules. It achieves this by providing: 1) statistics on the accuracy of a question and its rules 2) tools to help develop better rules Managing pmatch questions becomes easier because the author: * has fewer rules to manage * has more information to make informed decisions ** can see how accurate their question rules actually are. ** can see which rules are useful and which are redundant * can try rules and to see the updated statistics before saving the rule Background To automatically mark a free text sentence or multiple sentences the pattern match question question was developed. The Moodle pattern match question type is very powerful but equally difficult to author. Requiring carefully creating a long list of complex rules to generate a question that consistently marks student answers as well as human markers. With Pattern match the rules are generated by assuming the answers students may give. AMATI is a separate OU developed approach providing a workflow which develops grading rules based on human mark responses from to the given question. This feature adds additional functionality to the Moodle quiz pattern match authoring workflow to incorporate the AMATI methodology and support development of new grading rules using existing marked responses. The research paper Using NLP to Support Scalable Assessment of Short Free Text Responses shares the questions used to develop AMATI http://aclweb.org/anthology/W/W15/W15-0628.pdf Use cases What situations will this feature help? * Author has finished creating a question without rules and wants to upload sample responses to help create the rules. * Author has finished creating a question with rules and want to upload sample responses to test/refine the rules. * Author has previously created a question, rules and uploaded sample answers. The question has been used by students and the author wants to check/refine the rules based on additional sets of responses because the question didn’t perform as expected. h1. Functionality What can the feature actually do? Test this question When previewing a pattern match question you will now see a "Test this question" link in the top right corner. Clicking this link leads to a screen listing the sample responses attached to the question. Each response includes: * the response * the expected score * the actual score This highlights any mismatches between the score given by experienced human markers and the score given by the question. Test selected questions When responses have been uploaded to a question select the checkbox next to each response you wish to grade or click "select all". Click "Test selected responses" and each selected response will be marked by the question with the grade being recorded in the "Actual mark" column. There are 4 categories that response grading statistics fall into: 1) Matched response: the computer gives a mark a human would give 2) Missed positive: The human marked it correct, the computer marked it incorrect 3) Missed neagtive: The human marked it incorrect, the computer marked it correct 4) Ungraded: The computed mark is not available because the response has not been tested. Upload responses Responses can be added to the question in the upload response screen. To import responses create a simple CSV file with: * a header row containing mark,response * a series of rows with the human graded mark (expected mark) and response separated by a comma ** e.g. 0,jane is friends with harry Uploaded responses are associated with the specific question they were uploaded against and stored in the database for use later. Delete responses Responses can be deleted in the "Test this question" screen. select the checkbox next to each response you wish to delete or click "select all". Click "Delete" and a confirmation box will appear asking you to confirm the delete request. Click yes to delete the selected responses. h1. FAQ Are duplicates stored? To prevent duplicates only unique responses for the given question will be saved. Before an uploaded response is saved a check is made to ensure the same response is not already stored for the question. The upload feedback message reports which duplicate responses were not saved. Can an author remove test responses? Initially we have provided a standard interface allowing full editing (create, read, update, delete functionality). Later on we may restrict this functionality through capabilities within Moodle according to requirements. Does this feature change or remove users attempts? No, The test responses used are not part of the quiz or question data. Student attempts are not affected. The test responses are separate data uploaded to a separate table added specifically for this feature. They do not represent student attempts. These responses represent test data on which to verify the accurcay and reliability of a given question. There it is not possible to edit student attempts or grades held in the quiz results or the gradebook. h1. Administration Capabilities Create and read capability are clearly required by anyone using the new feature. Consensus has not been reached on which users would delete or update capability or whether this should be possible at all. The considered use case is where data is incorrect or no longer required. In this case the capability to update or remove data may be an important consideration. moodle/question View : View Question moodle/question Edit : Edit Question mod/quiz:regrade Regrade : Regrade Quiz Supported Browsers and Operating systems Use is expected to be on desktop browsers and potentially tablets. No significant user interface challenges are expected and standard moodle form elements should be sufficient to deliver the required functionality. Thus in theory all browsers that Moodle currently supports using the OU theme should be supported. Automated Testing Automated testing is crucial to validating the reliability of delivered functionality on a regular basis. In depth unit and Behat tests have been created.