FacebookTwitter
Hatrack River Forum Post New Topic  Post A Reply
my profile login | register | search | faq | forum home

  next oldest topic   next newest topic
» Hatrack River Forum » Active Forums » Books, Films, Food and Culture » Help Jon Boy do his homework! Now with results!

   
Author Topic: Help Jon Boy do his homework! Now with results!
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
For one of my classes this semester I'm doing a research project on edited English, and I need volunteers to take a survey for me. It's pretty quick and (most likely) painless, but I'd greatly appreciate any participation.

Thanks!

Edit: Link to new survey. If you already took the survey on SurveyMonkey, please retake it.

[ April 24, 2010, 02:37 PM: Message edited by: Jon Boy ]

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Lisa
Member
Member # 8384

 - posted      Profile for Lisa   Email Lisa         Edit/Delete Post   Reply With Quote 
Done.
Posts: 12266 | Registered: Jul 2005  |  IP: Logged | Report this post to a Moderator
Belle
Member
Member # 2314

 - posted      Profile for Belle   Email Belle         Edit/Delete Post   Reply With Quote 
Done.
Posts: 14428 | Registered: Aug 2001  |  IP: Logged | Report this post to a Moderator
AvidReader
Member
Member # 6007

 - posted      Profile for AvidReader   Email AvidReader         Edit/Delete Post   Reply With Quote 
Interesting, Jon Boy. I'd love to see what you find out when it's all done.
Posts: 2283 | Registered: Dec 2003  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Thanks, guys. I'll let you know what I find out in a couple weeks.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Carrie
Member
Member # 394

 - posted      Profile for Carrie   Email Carrie         Edit/Delete Post   Reply With Quote 
Done!

(And totally without pain. [Smile] )

Posts: 3932 | Registered: Sep 1999  |  IP: Logged | Report this post to a Moderator
Tinros
Member
Member # 8328

 - posted      Profile for Tinros           Edit/Delete Post   Reply With Quote 
Done.
Posts: 1591 | Registered: Jul 2005  |  IP: Logged | Report this post to a Moderator
Mike
Member
Member # 55

 - posted      Profile for Mike   Email Mike         Edit/Delete Post   Reply With Quote 
That was painful. (Just kidding (sort of).)
Posts: 1810 | Registered: Jan 1999  |  IP: Logged | Report this post to a Moderator
The Rabbit
Member
Member # 671

 - posted      Profile for The Rabbit   Email The Rabbit         Edit/Delete Post   Reply With Quote 
Jon Boy, I have a minor ethical quibble with your quiz. You need to either remove the last page where you request personal information (names and addresses) or have a process in place that deidentifies participants (i.e. makes it impossible to connect a particular individual with a specific response). You need a form for participants to check off which explains how the data will be used and gets their permission for its inclusion in your study.

When I got to the final page and you asked for my personal information (name and address,) without any explanation of how that data would be used or assurance that my privacy would be protected it made me extremely uncomfortable and I did not submit the survey.

I served on an institutional review board for human subjects research and spent a lot of time discussing what risks a person takes by participating in a survey and deciding whether studies have taken enough precautions to reduce those risks. I'm sure that makes me far more sensitive to the issues than the average guy.

This isn't intended as a criticism of you. Its not something you necessarily should have been aware of. I suspect your professors wouldn't be aware of these issues unless they've been involved with approval for human subjects research. Few people are aware that surveys and interviews require approval by a human subjects review board or have thought about the ethics of surveying people as part of an academic study. I know this is just a class project and so it isn't required to meet professional ethical standards or be reviewed. But even so, its best to be aware of the issues and follow accepted best practices.

You never know, this class project might turn up something really novel and interesting that you might want to publish or present at a professional meeting or use in a proposal for future research. If that happens, not following proper ethical standards in collecting the data could cause you some big headaches.

[ April 08, 2010, 10:14 AM: Message edited by: The Rabbit ]

Posts: 12591 | Registered: Jan 2000  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Thanks for the heads-up, Rabbit. I also need to make a couple changes to remove an ordering bias, so I'll take it down for now and repost it once the changes are made.

For the record, no identifiable personal information will be used in my paper, and the paper will not be published. If it turns out to be an interesting study worth publishing, I'll probably end up revamping and redoing it.

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Here's a link to the new survey. If you already took the survey on SurveyMonkey, please take the new one if it's not too much trouble.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Belle
Member
Member # 2314

 - posted      Profile for Belle   Email Belle         Edit/Delete Post   Reply With Quote 
Done. Again. [Smile]
Posts: 14428 | Registered: Aug 2001  |  IP: Logged | Report this post to a Moderator
jebus202
Member
Member # 2524

 - posted      Profile for jebus202   Email jebus202         Edit/Delete Post   Reply With Quote 
Tried and failed. Does the information people are reading have to be so utterly boring?
Posts: 3563 | Registered: Sep 2001  |  IP: Logged | Report this post to a Moderator
AvidReader
Member
Member # 6007

 - posted      Profile for AvidReader   Email AvidReader         Edit/Delete Post   Reply With Quote 
I like house plants, so I didn't mind reading it again.
Posts: 2283 | Registered: Dec 2003  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Thanks, Belle and AvidReader (and anyone else). Again. [Smile]

Sorry you have such a short attention span, Jebus. [Razz]

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
*bump*

Is there anyone else who would like to volunteer?

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Well, the results were mostly inconclusive. The three passages were an original, a lightly edited version, and a heavily edited version. Only two of the scores showed a statistically significant change, and those were for "The passage is well written" and "The author sounds professional" on the heavily edited version. The raw numbers showed a slight increase in pretty much every score from original to light to heavy, but none of the differences were significant except for those two scores.

Edited for clarity.

[ April 24, 2010, 02:20 PM: Message edited by: Jon Boy ]

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
Itsame
Member
Member # 9712

 - posted      Profile for Itsame           Edit/Delete Post   Reply With Quote 
Doesn't that imply something in itself? You really shouldn't tamper with results because you don't get ones you want.
Posts: 2705 | Registered: Sep 2006  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
I honestly don't know what you're saying. I haven't tampered with any results.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
fugu13
Member
Member # 2859

 - posted      Profile for fugu13   Email fugu13         Edit/Delete Post   Reply With Quote 
What test are you using for statistical significance?
Posts: 15770 | Registered: Dec 2001  |  IP: Logged | Report this post to a Moderator
Itsame
Member
Member # 9712

 - posted      Profile for Itsame           Edit/Delete Post   Reply With Quote 
What I mean to say is that you didn't get pretty results, so you are retesting.
Posts: 2705 | Registered: Sep 2006  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
fugu: I used paired samples t-tests to compare the means of the scores for the original to those of the two edited versions.

Jon: But I'm not retesting. I mean, I might, but I don't have any immediate plans to. And if I do retest, it won't be because I didn't get "pretty" results this time.

Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
dkw
Member
Member # 3264

 - posted      Profile for dkw   Email dkw         Edit/Delete Post   Reply With Quote 
No he's not. The request to re-take was the day after he originally posted (april 8), because he changed the software he was using and added a waiver. He's not re-doing anything now, just informing us of the results.
Posts: 9866 | Registered: Apr 2002  |  IP: Logged | Report this post to a Moderator
Itsame
Member
Member # 9712

 - posted      Profile for Itsame           Edit/Delete Post   Reply With Quote 
Ohh, sorry. I misunderstood. I thought that you were asking people to retest because the results were inconclusive. My apologies.
Posts: 2705 | Registered: Sep 2006  |  IP: Logged | Report this post to a Moderator
fugu13
Member
Member # 2859

 - posted      Profile for fugu13   Email fugu13         Edit/Delete Post   Reply With Quote 
(edit: silly people posting before me.)

You might be a bit confused about the timeline. First there was the first version of the survey, which had problems due to limitations of the survey software. Then he had people retake with new survey software he found he had access to. After the retaking, before which the results hadn't been looked at, was when the results were analyzed.

Of course, there's absolutely no reason to object to him running a new study after this one, either. It wasn't randomized and lacked sufficient power, so it was nothing more than some casual observational data. Even moreso, there is absolutely no problem with rerunning the study even if it were properly randomized with sufficient statistical power; the eventual model just needs to (formally or informally) take into account the multiple runs. That being useful is why we do multiple clinical trials for drugs, for instance, and don't stop at just one.

Posts: 15770 | Registered: Dec 2001  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Jon: No worries. It hadn't occurred to me to update the thread title now that the results are in.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
fugu13
Member
Member # 2859

 - posted      Profile for fugu13   Email fugu13         Edit/Delete Post   Reply With Quote 
Jon Boy: for each variable separately? Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").

Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.

Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.

Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.

Posts: 15770 | Registered: Dec 2001  |  IP: Logged | Report this post to a Moderator
The Rabbit
Member
Member # 671

 - posted      Profile for The Rabbit   Email The Rabbit         Edit/Delete Post   Reply With Quote 
Those are good suggestions fugu.
Posts: 12591 | Registered: Jan 2000  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
quote:
Originally posted by fugu13:
Jon Boy: for each variable separately?

Uh, I think so. To be honest, I'm still very much a novice when it comes to stats. I used the paired samples t-test in PASW and plugged in the pairs of scores (original and light edit and original and heavy edit).

quote:
Make sure you use a multivariate test, since you're testing multiple (correlated) random variables; otherwise you won't compensate for the multiple comparisons problem. What you need is some sort of ANOVA (I think the "paired t-tests" equivalent is "within subjects ANOVA").

Also, statistical significance, especially for small models, isn't very interesting. Look for large effects, moreso, even ones that aren't statistically significant. Especially on a non-randomized, under-powered trial, where you find large effects is more likely to be interesting for later trials.

Of course, given the test you applied, the ones that seem to be statistically significant (though again, you haven't actually checked for that, since your analysis needs to compensate for the multiple comparisons problem) probably are the larger effects, so that advice is probably redundant.

Another interesting way to look at the data would be Principal Components Analysis, to see if there is underlying structure to the results. Maybe there are multiple "camps" among the survey takers, reflecting different perspectives among the survey material (as one possibility). PCA will let you search for that sort of thing.

Thanks for the advice. That's really helpful if I decide to redo the survey.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
The Rabbit
Member
Member # 671

 - posted      Profile for The Rabbit   Email The Rabbit         Edit/Delete Post   Reply With Quote 
You wouldn't need to redo the survey Jon Boy. You could simply reanalyze the data from this survey.
Posts: 12591 | Registered: Jan 2000  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Right. But if I decide to work something like this into my thesis, I'm going to need to get IRB approval and redo it, right?
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
fugu13
Member
Member # 2859

 - posted      Profile for fugu13   Email fugu13         Edit/Delete Post   Reply With Quote 
Yeah, though you have decent odds for being IRB-exempt, depending on your school's rules (you'll still need to submit, they'll just tell you what you have planned is exempt, assuming it is).

Well, technically you probably don't even need IRB approval unless you intend to publish the results. But it would be a good idea to get it.

Let us know if/when you move forward, and we can provide some pointers on experimental design.

Posts: 15770 | Registered: Dec 2001  |  IP: Logged | Report this post to a Moderator
Jon Boy
Member
Member # 4284

 - posted      Profile for Jon Boy           Edit/Delete Post   Reply With Quote 
Thanks, fugu. I really appreciate that.
Posts: 9944 | Registered: Sep 2002  |  IP: Logged | Report this post to a Moderator
   

Quick Reply
Message:

HTML is not enabled.
UBB Code™ is enabled.
UBB Code™ Images not permitted.
Instant Graemlins
   


Post New Topic  Post A Reply Close Topic   Feature Topic   Move Topic   Delete Topic next oldest topic   next newest topic
 - Printer-friendly view of this topic
Hop To:


Contact Us | Hatrack River Home Page

Copyright © 2008 Hatrack River Enterprises Inc. All rights reserved.
Reproduction in whole or in part without permission is prohibited.


Powered by Infopop Corporation
UBB.classic™ 6.7.2