Originally posted by Kwea:
give every option a 1$ chance then.
![[Big Grin]](biggrin.gif)
After adjustments, I ended up getting an 86% on hte final exam. LOL
![[Frown]](frown.gif)
You need to click on another part of the page to ensure that scrolling won't change your selection. quote:Which still means it's a poorly implemented design. If you have to scroll down to view the whole test, it's a poor design choice to utilize a widget that is scrollable. I imagine the problem wouldn't exist if they chose check boxes instead of radio buttons for answer selection.
Originally posted by fugu13:
That's just standard web browser behavior with some sorts of selection widgets in web pages. There's nothing really about the software, other than that they might consider using less changeable widgets.
quote:
Originally posted by Noemon:
Kwea, did any of your classmates run into similar trouble? You'd probably have better luck with getting a retest (and paper-based tests in the future) if a group of you petitioned for it.
quote:Having worked with many different types of medical IT environments in my day, as well as many different types of medical software, I can tell you that the likelihood of simply using a scroll wheel resulting in severe consequences is virtually nil. Any software company that creates a software suite that could result in such a mistake will very quickly find itself sued out of existence.
Originally posted by just_me:
I'm not trying to be a jerk, even though it seems like it, but...
As I understand it this is a test for nursing school? I'd just like to point out then that the medical industry uses A LOT of different software of varying sophistication, and some of it will likely be built on a web browser platform and exhibit similar behavior. Mistakes like the ones you talk about could happen - and the results can be worse than a bad grade.
quote:Yeah, see, people who make a habit of trying to piss people off regularly shouldn't ever try the whole tongue in cheek thing.
Originally posted by King of Men:
Well, I clearly am holier than thou. In particular, I poked my tongue so far into my cheek it came out on the other side. Ouch.
quote:Yeah... those people. Gotta... really... think... those people do... about making comments... about other people... when they themselves do the same thing... those... gosh darn people.
Originally posted by Boris:
quote:Yeah, see, people who make a habit of trying to piss people off regularly shouldn't ever try the whole tongue in cheek thing.
Originally posted by King of Men:
Well, I clearly am holier than thou. In particular, I poked my tongue so far into my cheek it came out on the other side. Ouch.
quote:I'm glad there are people like you at your college who are aren't willing to take crap like this lying down. Unfortunately this kind of thing is way too common, and the attitude toward it is way too relaxed.
Originally posted by Kwea:
If I don't hear from her by Monday, I go to the Dean of the whole freaking college.
quote:That made no sense whatsoever. Try again please.
Originally posted by Orincoro:
quote:Yeah... those people. Gotta... really... think... those people do... about making comments... about other people... when they themselves do the same thing... those... gosh darn people.
Originally posted by Boris:
quote:Yeah, see, people who make a habit of trying to piss people off regularly shouldn't ever try the whole tongue in cheek thing.
Originally posted by King of Men:
Well, I clearly am holier than thou. In particular, I poked my tongue so far into my cheek it came out on the other side. Ouch.
![[Smile]](smile.gif)
![[Mad]](mad.gif)
quote:I guarantee that this computer system did not replace all the scantron machines. Educators have a hard time throwing things away...they are there, somewhere.
I think this computer system replaced all the scantrons
.
quote:Check.
Originally posted by TomDavidson:
which sucks beyond all possible measure.
![[Razz]](tongue.gif)
quote:The usual way of doing multiple choice (1 right answer for full credit, no credit for anything else) is a big problem with multiple choice tests in general, and it sounds like a bigger problem for ones in the medical profession in particular.
I got a 78%, in part because I was so nervous I second-guessed myself. A lot of these nursing questions are not knowledge based questions, but "pick the BEST" type questions, where more than one answer is right.
quote:If the professor were doing this in the canonical way, you would get negative infinity points if you assigned 0 probability to the right answer. Presumably you would then fail the course.
As long as you gave it higher than 25% you got some amount of positive credit for the question. (under 25% you actually got negative credit...)
quote:Log of the probability assigned to the right answer. Add a scaling factor if you like positive numbers.
Originally posted by fugu13:
Tell me, KoM, what's the canonical way?
quote:For the method to work, there would have to be a requirement that the sum of the probabilities for all answers must = 1. To account for inadvertent mistakes in adding, your could sum the probabilities given each answer and scale the total to 1. That way giving every option a 1 would be equivalent to giving every option 0.25 (assuming 4 options).
Originally posted by Kwea:
give every option a 1$ chance then.
After adjustments, I ended up getting an 86% on hte final exam. LOL
![[Hat]](graemlins/hat.gif)
Sorry about all the stress. Hopefully, you won't have to deal with it anymore.
quote:Exactly... and he meant it. He had everyone sign something on the first day of class acknowledging that we understood the grading scheme and that a 0% on the right answer would result in immediate failure of the class.
Originally posted by King of Men:
quote:If the professor were doing this in the canonical way, you would get negative infinity points if you assigned 0 probability to the right answer. Presumably you would then fail the course.
As long as you gave it higher than 25% you got some amount of positive credit for the question. (under 25% you actually got negative credit...)
quote:There's a big cannon. They ask you to put your head inside the barrel. If you do so, you fail.
Originally posted by fugu13:
Tell me, KoM, what's the canonical way?
quote:I would auto-fail any student who gave me probabilities summing to more than one. If they sum to less than one, that's ok, it's an implicit assignment of probability to "none of the above" - although generally I would not be so evil as to make all four answers be wrong, and I'd expect students to know that.
For the method to work, there would have to be a requirement that the sum of the probabilities for all answers must = 1.
quote:Um, no? You have to decrease your probability estimate by a whole order of magnitude to change the score by 2. (Roughly.) And in any case, I said nothing about how raw scores translate to grades. Suppose an low-A student assigns 95% confidence to the right answer in 90% of cases, and 10% in the rest, then his score per question is -0.28. On a hundred questions, the A boundary is at -28. If you were slightly better than the boundary, say with a 92% chance of the 95% probability, then your expected score would be -23, and you'd need to change your estimate of no less than three questions by a whole order of magnitude - go from 95% to 9.5% - to drop down to a B. Looks quite reasonable to me.
Ah, so the canonical way is the one that creates substantial variations in score for very small differences in probability estimates. Too bad that's nonsensical for scoring tests.
quote:That's extraordinarily harsh. I've given a lot of tests and I can tell you that the probability that some students might inadvertently assign probabilities that added to over 100% in a high pressure exam is very nearly unity. Auto failing a student because they incorrectly added 4 numbers on a high pressure exam defeats the purpose of giving exams, unless your testing people on being able to add correctly under pressure. That is the sort of thing that some one would only suggest if they'd never actually been responsible for a class.
I would auto-fail any student who gave me probabilities summing to more than one. If they sum to less than one, that's ok, it's an implicit assignment of probability to "none of the above" - although generally I would not be so evil as to make all four answers be wrong, and I'd expect students to know that.
quote:It would be interesting to try to devise an optimum test taking strategy for such an exam. First off, assigning any answer a 0 probability, would be extremely risky. If you are wrong and that answer turns out to be correct, you fail. So most students would automatically give every answer some non-zero probability. The difficulty is that the scale is non-linear so the potential loss from giving an answer a very low probability is much greater than the potential gain from giving a correct answer a very high probability.
Originally posted by King of Men:
quote:Log of the probability assigned to the right answer. Add a scaling factor if you like positive numbers.
Originally posted by fugu13:
Tell me, KoM, what's the canonical way?
quote:He *did* employ it.
Originally posted by fugu13:
just_me: I was speaking specifically of the logarithmic method of scoring, which I suspect your professor did not employ.
quote:Simple math isn't an issue in any of my engineering classes either, but I can't tell you how many times I've seen students add 2 and 3 on and exam and get 6. Under the stress of an exam, students will very frequently make stupid simple errors that they undoubtedly know are incorrect. AS reasonable exam does not unduly penalize students for that kind of error.
I don't recall how he would deal with it if we ever had the total probability be >1. It never came up. As this was a graduate level course the simple math wasn't an issue (or better not have been).
quote:In this scheme, if you choose to go with option 3) rather than option 1), and you are right. It raises your score by 0.13. But if you are wrong, it drops your score by 1.6 (more than 10 times as much. Or to put it another way, answering 10 questions wrong using option 1 would be better than answering one question wrong with option 3). If you use answer 1, you would have to get 80% correct in order to do as well as using answer 2 and getting all correct.
1) 85%, 5%, 5%, 5% (reasonably sure of 1 answer)
2) 45%, 45%, 5%, 5% (reduced to 2 answers)
3) 97%, 1%, 1%, 1% (very sure of 1 answer)
![[Wink]](wink.gif)
![[Confused]](confused.gif)
quote:I must say I do not see the distinction. If you know that your estimate is likely too high, and correct accordingly, that's exactly equivalent to just lowering your estimate a bit. Bingo, just the effect I wanted.
If a student's probability estimates are likely to be incorrect, they can create a strategy that will dominate for questions with one very confident answer by putting slightly higher estimates for the perceived wrong answers.
quote:I meant, rescale the assigned probabilities to make them total 1, in accordance with your criticism that the sum wouldn't come out right.
It doesn't matter how you rescale if the ordering is screwed up due to a high variance.
quote:Why do you see this as a desirable feature? What benefit is gained by awarding such a severe penalty to some who assigns a 0 probability to the right answer? Why do you think that a student who assigned 100% to the right answer 99% of the time, and 0% 1 percent of the time deserves not only to fail but to get a lower grade than a student who assigned 1% to the correct answer 100% of the time. What possible advantage is there to it. Not only does it yield what I would consider highly nonsensical results but it is so easy for students to avoid the penalty by never assigning an zero probability even when they are certain an answer is wrong. I.E. it incourages dishonesty.
Originally posted by King of Men:
The canonical feature is failing the course if you assign a 0 probability to the right answer. The rest is details.
quote:Again, I'll caveat this by saying that I am confining my discussion to the context of the specific application (class) I encountered this scoring.
Originally posted by The Rabbit:
Not only does it yield what I would consider highly nonsensical results but it is so easy for students to avoid the penalty by never assigning an zero probability even when they are certain an answer is wrong. I.E. it incourages dishonesty.
code:void makePlots (int questions = 25) {
TH1F* ninetypc = new TH1F("ninetypc", "", 300, -35, 3);
ninetypc->SetStats(false);
ninetypc->SetMarkerColor(kBlue);
ninetypc->SetMarkerStyle(8);
ninetypc->SetMarkerSize(0.5);
TH1F* eightypc = new TH1F("eightypc", "", 300, -35, 3);
eightypc->SetStats(false);
eightypc->SetMarkerColor(kRed);
eightypc->SetMarkerStyle(8);
eightypc->SetMarkerSize(0.5);
TH1F* ninetyflat = new TH1F("ninetyflat", "", 300, -35, 3);
ninetyflat->SetStats(false);
ninetyflat->SetMarkerColor(kBlack);
ninetyflat->SetMarkerStyle(8);
ninetyflat->SetMarkerSize(0.5);
TH1F* eightyflat = new TH1F("eightyflat", "", 300, -35, 3);
eightyflat->SetStats(false);
eightyflat->SetMarkerColor(kGreen);
eightyflat->SetMarkerStyle(8);
eightyflat->SetMarkerSize(0.5);
TRandom blah;
for (int i = 0; i < 10000; ++i) {
double ninetyscore = 0;
double eightyscore = 0;
double ninetypoints = 0;
double eightypoints = 0;
for (int j = 0; j < questions; ++j) {
if (blah.Uniform() < 0.9) ninetyscore += log(0.9);
else {
ninetyscore += log(0.03);
ninetypoints--;
}
if (blah.Uniform() < 0.8) eightyscore += log(0.8);
else {
eightyscore += log(0.06);
eightypoints--;
}
}
ninetypc->Fill(ninetyscore*(25.0/questions));
eightypc->Fill(eightyscore*(25.0/questions));
ninetyflat->Fill(ninetypoints*(25.0/questions));
eightyflat->Fill(eightypoints*(25.0/questions));
}
TCanvas foo;
ninetyflat->Draw("p");
ninetypc->Draw("psame");
eightypc->Draw("psame");
eightyflat->Draw("psame");
char fname[200];
sprintf(fname, "variance%i.eps", questions);
foo.SaveAs(fname);
}
.