For one of my classes this semester I'm doing a research project on edited English, and I need volunteers to take a survey for me. It's pretty quick and (most likely) painless, but I'd greatly appreciate any participation.
Thanks!
Edit: Link to new survey. If you already took the survey on SurveyMonkey, please retake it.
[ April 24, 2010, 02:37 PM: Message edited by: Jon Boy ]
Posted by Lisa (Member # 8384) on :
Done.
Posted by Belle (Member # 2314) on :
Done.
Posted by AvidReader (Member # 6007) on :
Interesting, Jon Boy. I'd love to see what you find out when it's all done.
Posted by Jon Boy (Member # 4284) on :
Thanks, guys. I'll let you know what I find out in a couple weeks.
Posted by Carrie (Member # 394) on :
Done!
(And totally without pain. )
Posted by Tinros (Member # 8328) on :
Done.
Posted by Mike (Member # 55) on :
That was painful. (Just kidding (sort of).)
Posted by The Rabbit (Member # 671) on :
Jon Boy, I have a minor ethical quibble with your quiz. You need to either remove the last page where you request personal information (names and addresses) or have a process in place that deidentifies participants (i.e. makes it impossible to connect a particular individual with a specific response). You need a form for participants to check off which explains how the data will be used and gets their permission for its inclusion in your study.
When I got to the final page and you asked for my personal information (name and address,) without any explanation of how that data would be used or assurance that my privacy would be protected it made me extremely uncomfortable and I did not submit the survey.
I served on an institutional review board for human subjects research and spent a lot of time discussing what risks a person takes by participating in a survey and deciding whether studies have taken enough precautions to reduce those risks. I'm sure that makes me far more sensitive to the issues than the average guy.
This isn't intended as a criticism of you. Its not something you necessarily should have been aware of. I suspect your professors wouldn't be aware of these issues unless they've been involved with approval for human subjects research. Few people are aware that surveys and interviews require approval by a human subjects review board or have thought about the ethics of surveying people as part of an academic study. I know this is just a class project and so it isn't required to meet professional ethical standards or be reviewed. But even so, its best to be aware of the issues and follow accepted best practices.
You never know, this class project might turn up something really novel and interesting that you might want to publish or present at a professional meeting or use in a proposal for future research. If that happens, not following proper ethical standards in collecting the data could cause you some big headaches.
[ April 08, 2010, 10:14 AM: Message edited by: The Rabbit ]
Posted by Jon Boy (Member # 4284) on :
Thanks for the heads-up, Rabbit. I also need to make a couple changes to remove an ordering bias, so I'll take it down for now and repost it once the changes are made.
For the record, no identifiable personal information will be used in my paper, and the paper will not be published. If it turns out to be an interesting study worth publishing, I'll probably end up revamping and redoing it.
Posted by Jon Boy (Member # 4284) on :
Here's a link to the new survey. If you already took the survey on SurveyMonkey, please take the new one if it's not too much trouble.
Posted by Belle (Member # 2314) on :
Done. Again. Posted by jebus202 (Member # 2524) on :
Tried and failed. Does the information people are reading have to be so utterly boring?
Posted by AvidReader (Member # 6007) on :
I like house plants, so I didn't mind reading it again.
Posted by Jon Boy (Member # 4284) on :
Thanks, Belle and AvidReader (and anyone else). Again.
Sorry you have such a short attention span, Jebus. Posted by Jon Boy (Member # 4284) on :
*bump*
Is there anyone else who would like to volunteer?
Posted by Jon Boy (Member # 4284) on :
Well, the results were mostly inconclusive. The three passages were an original, a lightly edited version, and a heavily edited version. Only two of the scores showed a statistically significant change, and those were for "The passage is well written" and "The author sounds professional" on the heavily edited version. The raw numbers showed a slight increase in pretty much every score from original to light to heavy, but none of the differences were significant except for those two scores.
Edited for clarity.
[ April 24, 2010, 02:20 PM: Message edited by: Jon Boy ]
Posted by JonHecht (Member # 9712) on :
Doesn't that imply something in itself? You really shouldn't tamper with results because you don't get ones you want.
Posted by Jon Boy (Member # 4284) on :
I honestly don't know what you're saying. I haven't tampered with any results.
Posted by fugu13 (Member # 2859) on :
What test are you using for statistical significance?
Posted by JonHecht (Member # 9712) on :
What I mean to say is that you didn't get pretty results, so you are retesting.
Posted by Jon Boy (Member # 4284) on :
fugu: I used paired samples t-tests to compare the means of the scores for the original to those of the two edited versions.
Jon: But I'm not retesting. I mean, I might, but I don't have any immediate plans to. And if I do retest, it won't be because I didn't get "pretty" results this time.
Posted by dkw (Member # 3264) on :
No he's not. The request to re-take was the day after he originally posted (april 8), because he changed the software he was using and added a waiver. He's not re-doing anything now, just informing us of the results.
Posted by JonHecht (Member # 9712) on :
Ohh, sorry. I misunderstood. I thought that you were asking people to retest because the results were inconclusive. My apologies.
Posted by fugu13 (Member # 2859) on :
(edit: silly people posting before me.)
You might be a bit confused about the timeline. First there was the first version of the survey, which had problems due to limitations of the survey software. Then he had people retake with new survey software he found he had access to. After the retaking, before which the results hadn't been looked at, was when the results were analyzed.
Of course, there's absolutely no reason to object to him running a new study after this one, either. It wasn't randomized and lacked sufficient power, so it was nothing more than some casual observational data. Even moreso, there is absolutely no problem with rerunning the study even if it were properly randomized with sufficient statistical power; the eventual model just needs to (formally or informally) take into account the multiple runs. That being useful is why we do multiple clinical trials for drugs, for instance, and don't stop at just one.
Posted by Jon Boy (Member # 4284) on :
Jon: No worries. It hadn't occurred to me to update the thread title now that the results are in.
Posted by fugu13 (Member # 2859) on :
Jon Boy: for each variable separately? Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").
Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.
Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.
Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.
Posted by The Rabbit (Member # 671) on :
Those are good suggestions fugu.
Posted by Jon Boy (Member # 4284) on :
quote:Originally posted by fugu13: Jon Boy: for each variable separately?
Uh, I think so. To be honest, I'm still very much a novice when it comes to stats. I used the paired samples t-test in PASW and plugged in the pairs of scores (original and light edit and original and heavy edit).
quote:Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").
Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.
Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.
Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.
Thanks for the advice. That's really helpful if I decide to redo the survey.
Posted by The Rabbit (Member # 671) on :
You wouldn't need to redo the survey Jon Boy. You could simply reanalyze the data from this survey.
Posted by Jon Boy (Member # 4284) on :
Right. But if I decide to work something like this into my thesis, I'm going to need to get IRB approval and redo it, right?
Posted by fugu13 (Member # 2859) on :
Yeah, though you have decent odds for being IRB-exempt, depending on your school's rules (you'll still need to submit, they'll just tell you what you have planned is exempt, assuming it is).
Well, technically you probably don't even need IRB approval unless you intend to publish the results. But it would be a good idea to get it.
Let us know if/when you move forward, and we can provide some pointers on experimental design.
Posted by Jon Boy (Member # 4284) on :
![[Smile]](smile.gif)
)
![[Razz]](tongue.gif)