posted
Does anyone remember how to perform a statistical significance test? It has been a year or so since I had stats and I cannot find the formula for it online. I have some percentage data for a research project and I was curious if it was statistically significant or not (this isn't required or anything but I thought it would be nice to include it if time allows).
Posts: 306 | Registered: Jun 2003
| IP: Logged |
posted
Most likely you want to know the 95% (or 99%, if you want a much higher standard) confidence interval for your data points, to see if they might overlap.
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
posted
We need to know a lot more about the data first to figure out whether you need a T-test, F-test, chi square, ANOVA, change in model fit, etc.
Tell us whether your data is correlational or experimental, what is your sample size and whether your participants were randomly chosen from a population and randomly assigned to treatment groups. Is the data you collected nominal, ordinal, or rational? Is this test for significance post hoc (an afterthought) or had you planned this test from the beginning? What is the variance?
If you don't remember these terms, then at least describe your study in great detail.
Posts: 264 | Registered: Nov 2005
| IP: Logged |
posted
hmmm-I am starting to think that a statistical analysis is not going to work for my data.
We did some learning set formation tests on 4 different mountain lions. Basically all we have is the percent of correct responses and incorrect responses. They had a choice between 2 objects and they had to choose the correct object first(it had a piece of raw meat under it). Unfortunately, time and lack of motivation from the cougars did not really give us many trials to analyze. This is the kind of experiment that needed at least 3 months every single day...we had more like 4 days over a 4 week span Posts: 306 | Registered: Jun 2003
| IP: Logged |
posted
Your data might be eligible for a chi-square test. Let me see if I understand your experiment. You have 4 subjects. How many trials per cougar? You have one dependent variable with two values (choosing the correct object or not). You have one independent variable (whether meat is under the object or not). What else am I missing?
You want to see if the cougars chose the object with the meat significantly more often than the object without the meat.
Posts: 264 | Registered: Nov 2005
| IP: Logged |
posted
Well, that's pretty simple, then : Just calculate the probability that the cougars would choose as they did, if they had been flipping coins instead of doing whatever passes for cognition in a cougar brain. Since you have only two options this is basic binomial theorem.
Posts: 10645 | Registered: Jul 2004
| IP: Logged |
posted
Yah-I don't think this is going to work...we had high hopes for the experiment and it is a good one...but we did not produce enough trials per cougar to really analyze the data properly. The most trials we conducted were on Tecumseh and Burton (20 trials total for each) and with Mariah we failed miserably with her only getting 6 trials done (she just didn't get it).
Thanks anyways guys. I forgot all that was involved in analyzing data. I think we are just going to have to report how well the cougars caught on to the experiment concept if they did at all.
Posts: 306 | Registered: Jun 2003
| IP: Logged |
posted
how did you rule out the lions choosing based only on sense of smell? A piece of highly salient food under the correct object would pretty much define "correct" for most critters.
Anyway, typical learning studies using similar situations would have many more trials. You could fall back onto the old "graph the learning curve" way to present the data. Use an up-tick for every correct response, a flat line for every incorrect response. X-axis is trial #. Any slope other than 22.5-degrees demonstrates learning. A 45 degree slope would show perfect performance. 0 degree slope shows perfect contrarian performance.
Probably not worth trying to do statistical analysis on, but then, it is possible to do tests comparing, for example, the %correct in the first 5 trials and the last 5...something like that. At last for the two lions with 20 trials each.
You could do a test of proportions on something like that.
Or, there are some non-parametric tests that have been used in these situations too.
Personally, I'd try to graph the learning curve and leave it at that.
Good luck.
But again...was this just the lions learning to pay attention to the odor of raw meat, or did they learn something "symbolic" -- I'm assuming you were trying to do a variation of delayed matching to sample or some other classic binary choice paradigm.
posted
5 trials is nowhere near enough to compare two sets. The error would be sqrt(5) = 2.2! That's almost half the range; there is absolutely not enough information to draw any conclusions. Consider : The utter best case is that the lions get no attempts right in the first five, and all attempts right in the last five. Well, that's a two-sigma effect - which is to say, zero! Two sigmas is utterly worthless. And of course, it is highly quite unlikely (1 in 32) that the lions would get zero correct attempts in five trials anyway - indeed, so unlikely that you'd almost have to say "something weird is going on" and throw out the data.
Posts: 10645 | Registered: Jul 2004
| IP: Logged |
posted
We tried to eliminated scent based choices to find the meat by smearing the insides of both object with the meat and placing the meat under only one.
Posts: 306 | Registered: Jun 2003
| IP: Logged |
There's published literature out there with all sorts of stuff like this. It ain't pretty, but people do run the stats and come up with an answer.
Granted, you'd want replications to show it's not a fluke. They just don't have enough data.
If they were being forced to analyze it, they could look at this as an option.
Sorry you don't like it. Behavioral sciences often aren't pretty when it comes to statistical testing.
But, yes, this would definitely not pass muster in a refereed journal.
If I were looking at a student project and they at least tried to come up with some way to analyze their results, I'd give them credit for the attempt even though they lacked sufficient data.
Rackhams-- was it matching to an initial object you showed them -- then placing the meat under the matching object and allowing a choice?
I'm not sure how smearing the objects with meat would work, but it does sound like a reasonable thing to try. Again, if this were a student project, I'd give them credit for the attempt.
If this was something for a peer-reviewed journal, you'd want to demonstrate that choices were random in the absence of the match-to-sample task, just to prove the lions were choosing based on the task as you defined it, and not based on a differential in odor coming off the two choice objects.
The presence of the meat may well have been detectable even if both objects were smeared first.
Posts: 22497 | Registered: Sep 2000
| IP: Logged |
posted
The idea behind the learning set was to see if the cougars could associate that all small objects would contain the meat reward. While the general experimental design is good, it would really need to be developed frther to take into a full experimental scale. We realized we were in over our heads to get accurate results with which the scale of our experiment needed to be conducted, but by then it was too late.
I wish we had more time to be able to really conduct the experiment at its full potential because I think the results would be interesting.
Posts: 306 | Registered: Jun 2003
| IP: Logged |
If you ever get the chance to do this again, I have a few suggestions for you on how to make the study valid and fit in with a wealth of literature in other critters.
And have a more interesting study when the results are known.
Posts: 22497 | Registered: Sep 2000
| IP: Logged |
posted
Yes, I know that people advance this kind of analysis in all seriousness. That doesn't make it correct. "Behavioural science isn't pretty" is not an excuse, it's an indictment of behavioural science.
Posts: 10645 | Registered: Jul 2004
| IP: Logged |
posted
If I ever do decide to carry this experiment out to full scale, you can bet I will ask all the minds of Hatrack to help me make my study more valid. Posts: 306 | Registered: Jun 2003
| IP: Logged |
Please excuse the flippant remark regarding behavioral research. It wasn't meant in seriousness.
I do need to point out a couple of things, though:
1) I said clearly that I would accept that kind of thing from a student project, and not in peer-reviewed work.
2) I'd be far more concerned about the research methodology than I would the statistical testing.
Point #2 is important. There are lots and lots of statistical tests than can be used successfully on experiments using two-alternative choice tasks. They get at the "problem" from a variety of ways and give you a sense of what happened.
None of them are substitutes for a solid methodology that's not so easy to poke holes in. The most important thing in this situation would be to know that the animals were learning what you said they were learning -- the rate that the learned or the fact that they learned "something" is not surprising or novel, or important.
The "fact" that learning took place could indeed be shown graphically without use of statistical tests at all. Unless someone doubted the consistency of performance accuracy reaching some sufficiently high assymptotic level.
The question will remain "what did the animal learn?" Does it understand the task the same way the experimenter describes it? Or, is there some simpler explanation.
It's why two-alternative matching to sample is a better demonstration of "conceptual" learning (i.e., cognition) than is something like showing that an animal can consistently choose the smaller container -- the one under which food is hidden.
If you want to show that the animal learned to differentiate object sizes, you teach it to pick the one that matches the "sample" you just showed it (Large or small) and you randomize the trials so that there isn't some obvious pattern that could result in learning a simpler rule or concept.
And... you also have to move the receipt of food a bit in time, so that the animal's superior senses aren't fooling the experimenter into thinking a concept has been learned, when really all the thing is doing is "sniffing."
Can't even begin to tell you how many students in our lab "discovered" that gerbils can sense buried sunflower seeds, and thus don't need to "learn" anything but that we'd buried them.
The nagging simpler explanation is 99 times out of 100 (p<.0001) the real problem with behavioral experiments in animals. It's got nothing to do with expertise in statistics and everything to do with knowing the capabilities of the subjects under study, and how to craft a decent experimental design to rule out as many alternative explanations as possible.
Posts: 22497 | Registered: Sep 2000
| IP: Logged |
Printer-friendly view of this topic
![[Frown]](frown.gif)
![[Smile]](smile.gif)
