Assessment Features Overview

Overview of SYLVA

SYLVA is a fully-integrated learning and assessment platform with intelligent automated grading and interactivity for student engagement.

In this video, we will cover our assessment features and examples that span across a variety of disciplines. Examples will be shown in our authoring tool where we create and distribute courseware and assessments.

Assessment Features

Full Customization: Selection Questions

Let's first start with something familiar - selection questions.

For this management example, we have the question "Select all the attributes of a transformational leader." It is a select-all question with 5 choices. We can add as many choices as desired and set up the question with one or multiple correct solutions.

We can award partial credit to students for each correct answer. Negative points is also possible.

Under Options we can see all possible variations for selection questions.

• ‍Single selection - We can setup questions with only one single correct solution or multiple.‍
• Order randomization - Order randomization is turned on by default and randomizes the order of the possible choices shown to each student.
  
• Take subset - This option allows you to use an answer pool and provide each student with only a few choices from the pool. The possible answers are randomly selected for each student. For example, we have 5 choices in our question. We can take a subset of only 4 possible choices, always inclusive of the 2 correct answers.‍
• Anchored alternative - Anchored alternative adds a choice that is always fixed as the last choice and included for every student. You can use it as a None of the Above option, or you can choose a custom answer. This is especially helpful when you don't have many alternatives because the more choices there are, the less likely random guessing will give full points.‍
• Clip final points at 0 - Lastly, when you use multiple selection and additive points, it is possible for students to receive a negative sum based on their given answers. If you include negative points for certain choices, clip final points at 0 will make sure that no student is awarded negative points for the entire question.

Let's preview the question. We see one version with a specific subset of choices and ordering.

When we preview the question again, we can see the subset and choices are randomized. Randomization is great for any type of selection question to achieve variety from a single question.

For open questions, we can use parameterization.

Preserving Academic Integrity: Parameterization

Parameters enable us to create variations of the same question by replacing question arguments such as numbers or words with a parameter. This is especially helpful to reinforce learning concepts using repetition or to prevent cheating.

In this finance example, the question states "Assume that Diamond Ltd's has a $20.35 stock price and 1000 outstanding shares. What is the market value?"

The answer is calculated by multiplying $20.35 by 1000 shares. We can make the number of shares a parameter so each student receives a question with a different number of shares.

I'll assign the number of shares as a parameter defined by x. The number of shares randomly assigned to each student will be between 1000-2000 in steps of 50. This means x will randomly be assigned as 1000, 1050, 1100 and so forth.

It is possible to use as many parameters as desired for a question.

In order for the student answer to be considered correct, the student must multiply the share price of $20.35 by the x number of shares.‍

This question is straight forward to compute, but some students might round the large number to full dollar amounts. Therefore, we account for an error tolerance based on rounding differences. We can be as strict or as flexible as we want with the error tolerance.

Let's preview the question. We see the parameter x is randomly assigned.

‍If we preview the question again, we see the parameter now changes.

For the answer, we multiply the share price $20.35 by the number of shares. I'll submit my answer and we see it is correct.

We can also test for the built-in error tolerance. I'll calculate $20.35 multiplied by the number of shares. I'll round the total. The answer is still considered correct.

In this example we covered basic numerical input. SYLVA however has the capability to grade all types of different answers.

Answer Verification: Interpreters

For free response questions, our automated grading uses interpreters to verify if a student answer is relevant.

In this finance example, the question states "Jane wants to diversify her portfolio. The current stock in her portfolio is Apple Inc. What stock should Jane pick from the timeframe January 2010 until the end of the current month?"

This question asks students to write a company as the answer. Based on the provided company, our software will first interpret the answer and then look up the financial data in the built-in knowledge base.

Under Interpreter, "Company" is selected since students will answer with a company name. We also provide the option to set the interpreter to integer, date, XLS, among others.

Correlation is a measure for similarity. For a diversified portfolio, we don't want the added stock to be highly correlated with the Apple stock. In order for the student answer to be correct, we set the condition that the correlation between Apple's current stock price and the student's given answer is less than 0.3 from January 2010. In our example, we use the threshold of 0.3 as an assumption for a diversified portfolio.

Let's test the question. Interpreters make free form questions robust because they allow students to enter only valid answers by giving them error messages when the input does not match or can't be interpreted. We set the interpreter as "Company" so if I submit another input, for example a number, I receive an error.

Let's properly test the question again. I will answer with "Starbucks". The answer is correct since it operates in a different industry.

A common source for errors in automated grading is misspellings or capitalizations. This causes issues that require manual reviewing to fix. I'll test the question again and purposely misspell "Strbucks". The software detects the misspelling and the answer is still correct.

Now let's take a company that is close to what Apple is doing. For example Google. Google has been renamed and the parent company is called Alphabet.

I'll answer with "Alphabet". The answer is incorrect since the correlation is greater than 0.3.

I'll try again and this time write "Google" even though the parent company is Alphabet. The interpreter takes into consideration this information. The answer is interpreted as Alphabet and still considered incorrect.

Now we have seen that all inputs entered in SYLVA can be interpreted for auto-grading. In addition to those listed here, SYLVA can grade everything as there are 703 interpreter types available.

Grade Everything: Interpreters (Excel)

For assessment questions, we can provide an Excel notebook for students to fill in and have the system automatically grade each notebook.

Example 1:

In this accounting spreadsheet, the students are required to fill in these five yellow cells of the equity statement based on the balance sheet provided below.

Let's look at the question setup in CREO. The question instructs students to write out the equity statement in the specific Excel cells based on the trial balance table.

The Excel file to be filled in is attached under Files.

The interpreter is set to XLSX.

The question awards a maximum of 3 points. Partial points are awarded for each of the three conditions.

Condition 1) In the first row of the spreadsheet, the students must write in "Common Stock" and the value 17500. This awards 1 point.

Condition 2) The students must write "Retained Earnings" and the value 37200. This awards 1 point.

Condition 3) The total equity must be equal to the previous 2 answers. This awards 1 point.

Let's preview the question. I click "Download attached file". After filling in the table and making the calculations, all labeled fields are completed.

In the question preview, I attach my saved file and submit the answer. I am awarded all 3 points.

Back in the Excel file, I will miscalculate only one cell. I'll test the question again. I am awarded only partial points.

Example 2:

In this second example, the students are given a spreadsheet with passenger data. The students must predict the number of passengers in the next 12 months. The student answers in the yellow cells will be graded.

Let's look at the question in CREO. The question provides students with directions to predict the number of passengers.

The Excel file is attached under Files.

The interpreter is set to XLSX.

The question awards a maximum of 1 point for calculating the correct values.

Let's preview the question. I click "Download attached file". In the spreadsheet, I'll calculate the next 12 months using the forecast function.

I upload my saved file and submit the answer. I am awarded full points.

Now we have seen that we can grade all kinds of answers automatically. Now I'll introduce interactive questions.

Interactivity Made Easy: Interactive Response

Interactive response questions allow students to interact with the question and immediately see the impact of their modifications. This allows students to focus on the intuition of the learning concept.

Example 1:

Let's first preview an example question from Economics. The question includes a dynamic supply and demand graph. The student uses the slider to increase an excise tax until the deadweight loss is greater than the consumer surplus. The final position of the slider is used as the student's answer to the question and graded accordingly.

To answer this question, the student needs the following thought process: first, where is the DWL and CS, and; second, what is the impact of the tax.

If the student is unsure about which fields on the graph are the DWL and the CS, the student can opt to show the labels. Clicking the optional button to display the labels will deduct 1 point from their answer.

Let's look at the question in CREO. The Question provides instructions for the student. The Answer type is interactive response.

In the Interactive Interface section, we setup the code for the graph.

The final step is to determine the conditions for a student’s answer to be considered correct.

Condition 1 checks if the student opted to show the DWL and CS labels or not. Displaying the labels minuses 1 point.

Condition 2 checks if the students moved the excise tax until at least 3%. This is the minimum tax for the DWL to be greater than the CS in our example. You can decide how flexible you are with student answers. Since it may be difficult for the students to position the locator on the exact location 3% on the graph, we provide a buffer to accept all answers above 3%. A correct answer awards 3 points.

Let's test the question.

I am unsure which fields are the DWL and CS so I opt to show the labels. I now see the purple section is the DWL and the orange section is the CS. I increase the excise tax using the slider until the DWL is greater than the CS. I submit my answer and receive 2 of the 3 total points.

Let's test the question again. This time I am aware the purple section in the DWL and the orange section is the CS. I increase the excise tax until the DWL is greater than the CS. I submit the answer and receive full points.

Example 2:

In this next example, students are tested on the Central Limit Theorem. The question states "Two die are rolled and their face values are added. How many times do you have to roll the die to reach the Central Limit Theorem? Adjust the slider to the correct sample size."

The student moves the slider to increase the sample size. The histogram is dynamic and recalculated based on the sample size.

Let's look at the question setup in CREO. For the Interactive Interface, we randomly generate the rolls of two 6-sided die and plot the outcomes in a histogram.

The optimal sample size in this case for the Central Limit Theorem to hold is minimum 1000. Therefore any sample size 1000 and above is considered correct.

Let's preview the question. As we increase the sample size, we see the distribution starts to form a bell curve. I'll submit my answer with the sample size at over 1000 and receive full points.

Let's preview the question again. When the sample size is very small, the distribution is uneven. The Central Limit Theorem does not hold. I'll submit the answer and the answer is incorrect.

Example 3:

For interactive response questions, we can provide a dynamic table for students to enter data.

Based on the coffee price, discount, and number of coffees purchased, the students must fill in the table for the discount received.

The setup of the question in CREO is simple.

We include the instructions in the Question. We setup a grid in the Interactive Interface.

We set conditions for each table value and award 1 point for each correct answer.

Let's test the question. Using the correct answers, I receive full points.

I'll test the question again and purposely miscalculate the total cost to the store. I receive only 3 of the 4 points possible.

Example 4:

In this last example, students are provided with a heteroskedastic graph. The students must move the data points so the graph does not reflect heteroskedasticity.

In the setup, the Question provides instructions for the students. The Interactive Interface plots a regression line with data points.

In order for the student answer to be considered correct, the spread of the squared residuals should be constant.

Let's preview the question again. I can move around each data point. The variance is recalculated based on where I place the data points.

After I finish moving the points, I submit the answer. I receive full points.

Scoring Rules: Additive vs. Holistic Consolidation

For free response or interactive response questions, we define conditions for a student answer to be considered correct. We assign points for each condition and have the possibility to award students partial points. We can take an additive or holistic grading approach.

• Additive approach: Tests each condition separately and sums the points for each condition met
• Holistic approach: Tests all conditions separately and selects the scoring rule associated with the most points

Let's preview an example question using the holistic approach. The questions asks students "Select the significant variables by using the parameter table and the residual plot. The three independent variables are TV, radio, and temperature. The dependent variable is sales per square meter."

Each tick box represents one of the variables. Using the data in the parameter table, students determine which variables are significant.

Let's view the question in CREO.

In the Question section, we see the instructions for the students. The Interactive interface includes the data table and the residual plot.

We have setup the Conditions and Scoring Rules together in a holistic approach. Each variable TV, radio, and temperature are assigned the binary variables a, b, and c.

If the student meets the first 3 conditions, 3 points are awarded. If the student understands at least the 2nd variable is insignificant, 1 point is awarded. The student receives no points in all other cases.

Let's preview the question. TV and temperature are significant variables based on the p-values. I submit my answer and am awarded the full 3 points.

I'll test the question again. I'll include the significant variable TV and not include the other 2 variables. I am awarded 1 point.

I test the question a final time. I wrongly include the insignificant variable radio. I am awarded no points.

Abstraction: Computed Inquiry

In SYLVA, each student answer is individually computed and checked against set conditions to evaluate if it is a solution. For free response questions with multiple correct solutions, our technology will compute whether or not the student answer is correct.

Example 1:

In this example, the students are asked to write an equation for the sequence {0,1,3,7,15} using the variable n.

The answer type is free response. The student must enter a formula.

There are multiple correct solutions based on how the students structure their answer. We set up the condition that the student's given answer must be equal to a sequence function in the question.

Let's preview the question. One such solution is 1/2(-2+2^n). When I submit the answer, it is correct.

Let's preview the question again. I'll submit the answer similarly to last time but add a +0. The exact input is different from last time, but the formula is evaluated each time by the software. The evaluation passes the test and I'm awarded full points.

Example 2:

In this second example, students are required to write Mathematica code less than 20 characters that evaluates to the provided list.

The grading takes a holistic approach and awards partial points based on which conditions are met. .

Let's preview the question. One solution is "Range[9]". I am awarded full points.

I'll preview the question again and test another solution "Table[i,{i,9}]". I am awarded full points.

If I simply copy and paste the provided list, I receive only partial points since I do not meet the condition of code less than 20 characters.

Our software is extremely flexible and all kind of partially correct states can be distinguished. Grading abstract things like formulas and code greatly expand the questions that can be asked and are very useful since answers cannot simply be guessed.

Manual Grading: Presentations and Projects

SYLVA's core asset is the automated grading of questions. However you can also manually grade assignments such as presentations or group projects.

For example, in one of our spring seminar courses, students were required to create a final presentation on a business modeling case. The students collaborated on the online editor, which is updated in real-time and where educators can monitor the student work.

At the end, educators manually review and grade the presentations then input the final grade in the online grade book.

Support

All of the features covered can be combined into assessment questions for improved student learning.

The Wolfram Demonstrations Project has wonderful interactive examples that can be embedded into SYLVA.

Feel free to check out our online tutorials for other example questions on our website at https://www.sylva.ac/product/tutorials‍

We offer all question setups as a service, and we are currently building a question bank so educators can reuse questions that have already been setup.

For support, please contact support@sylva.ac