posted
For one of my classes this semester I'm doing a research project on edited English, and I need volunteers to take a survey for me. It's pretty quick and (most likely) painless, but I'd greatly appreciate any participation.
Thanks!
Edit: Link to new survey. If you already took the survey on SurveyMonkey, please retake it.
[ April 24, 2010, 02:37 PM: Message edited by: Jon Boy ]
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Jon Boy, I have a minor ethical quibble with your quiz. You need to either remove the last page where you request personal information (names and addresses) or have a process in place that deidentifies participants (i.e. makes it impossible to connect a particular individual with a specific response). You need a form for participants to check off which explains how the data will be used and gets their permission for its inclusion in your study.
When I got to the final page and you asked for my personal information (name and address,) without any explanation of how that data would be used or assurance that my privacy would be protected it made me extremely uncomfortable and I did not submit the survey.
I served on an institutional review board for human subjects research and spent a lot of time discussing what risks a person takes by participating in a survey and deciding whether studies have taken enough precautions to reduce those risks. I'm sure that makes me far more sensitive to the issues than the average guy.
This isn't intended as a criticism of you. Its not something you necessarily should have been aware of. I suspect your professors wouldn't be aware of these issues unless they've been involved with approval for human subjects research. Few people are aware that surveys and interviews require approval by a human subjects review board or have thought about the ethics of surveying people as part of an academic study. I know this is just a class project and so it isn't required to meet professional ethical standards or be reviewed. But even so, its best to be aware of the issues and follow accepted best practices.
You never know, this class project might turn up something really novel and interesting that you might want to publish or present at a professional meeting or use in a proposal for future research. If that happens, not following proper ethical standards in collecting the data could cause you some big headaches.
[ April 08, 2010, 10:14 AM: Message edited by: The Rabbit ]
Posts: 12591 | Registered: Jan 2000
| IP: Logged |
posted
Thanks for the heads-up, Rabbit. I also need to make a couple changes to remove an ordering bias, so I'll take it down for now and repost it once the changes are made.
For the record, no identifiable personal information will be used in my paper, and the paper will not be published. If it turns out to be an interesting study worth publishing, I'll probably end up revamping and redoing it.
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Here's a link to the new survey. If you already took the survey on SurveyMonkey, please take the new one if it's not too much trouble.
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Well, the results were mostly inconclusive. The three passages were an original, a lightly edited version, and a heavily edited version. Only two of the scores showed a statistically significant change, and those were for "The passage is well written" and "The author sounds professional" on the heavily edited version. The raw numbers showed a slight increase in pretty much every score from original to light to heavy, but none of the differences were significant except for those two scores.
Edited for clarity.
[ April 24, 2010, 02:20 PM: Message edited by: Jon Boy ]
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Doesn't that imply something in itself? You really shouldn't tamper with results because you don't get ones you want.
Posts: 2705 | Registered: Sep 2006
| IP: Logged |
posted
fugu: I used paired samples t-tests to compare the means of the scores for the original to those of the two edited versions.
Jon: But I'm not retesting. I mean, I might, but I don't have any immediate plans to. And if I do retest, it won't be because I didn't get "pretty" results this time.
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
No he's not. The request to re-take was the day after he originally posted (april 8), because he changed the software he was using and added a waiver. He's not re-doing anything now, just informing us of the results.
Posts: 9866 | Registered: Apr 2002
| IP: Logged |
posted
Ohh, sorry. I misunderstood. I thought that you were asking people to retest because the results were inconclusive. My apologies.
Posts: 2705 | Registered: Sep 2006
| IP: Logged |
You might be a bit confused about the timeline. First there was the first version of the survey, which had problems due to limitations of the survey software. Then he had people retake with new survey software he found he had access to. After the retaking, before which the results hadn't been looked at, was when the results were analyzed.
Of course, there's absolutely no reason to object to him running a new study after this one, either. It wasn't randomized and lacked sufficient power, so it was nothing more than some casual observational data. Even moreso, there is absolutely no problem with rerunning the study even if it were properly randomized with sufficient statistical power; the eventual model just needs to (formally or informally) take into account the multiple runs. That being useful is why we do multiple clinical trials for drugs, for instance, and don't stop at just one.
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
posted
Jon: No worries. It hadn't occurred to me to update the thread title now that the results are in.
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Jon Boy: for each variable separately? Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").
Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.
Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.
Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
quote:Originally posted by fugu13: Jon Boy: for each variable separately?
Uh, I think so. To be honest, I'm still very much a novice when it comes to stats. I used the paired samples t-test in PASW and plugged in the pairs of scores (original and light edit and original and heavy edit).
quote:Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").
Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.
Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.
Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.
Thanks for the advice. That's really helpful if I decide to redo the survey.
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
You wouldn't need to redo the survey Jon Boy. You could simply reanalyze the data from this survey.
Posts: 12591 | Registered: Jan 2000
| IP: Logged |
posted
Right. But if I decide to work something like this into my thesis, I'm going to need to get IRB approval and redo it, right?
Posts: 9945 | Registered: Sep 2002
| IP: Logged |
posted
Yeah, though you have decent odds for being IRB-exempt, depending on your school's rules (you'll still need to submit, they'll just tell you what you have planned is exempt, assuming it is).
Well, technically you probably don't even need IRB approval unless you intend to publish the results. But it would be a good idea to get it.
Let us know if/when you move forward, and we can provide some pointers on experimental design.
Posts: 15770 | Registered: Dec 2001
| IP: Logged |
Printer-friendly view of this topic
![[Smile]](smile.gif)
)
![[Razz]](tongue.gif)
