Research Methodologies

A slight change in approach

2005-03-29T15:55:00.000-05:00

I'm starting to think of methodologies that will work with my research on students and self-efficacy. I looked up several sources on self-efficacy since not much has been done on GSP by itself. Self-efficacy is interesting but I'm having trouble redesigning my research question to include my thoughts on it. Right now I have several ideas, in no particular order in my head that I'm struggling to define and/or organize for my proposal.

Will students with low self-efficacy in writing place themselves in English 120?
Will students with high self-efficacy in writing place themselves in English 121?
What are the implications if the answers are yes to both questions?

How will I determine what is considered low and high self-efficacy? I think this is something I will determine by modifying what previous researchers have done to measure high and low self-efficacy. Most seem to resemble a questionaire/survey format. Once I determine who has high writing self-efficacy and who has low writing self-efficacy I can maybe interview a couple students from each group to find out more on why they enrolled in English 120 or English 121. This would give me a better idea of where they are coming from and whether we should compensate for self-efficacy in addition to placement decisions based on the GSP questionaire. Something tells me the questionaire is not enough by itself. Do we need a small form of checks and balances, or does that undermind our students' rights to place themselves in composition courses? I feel like I'm running around in circles...am I thinking too big for this sort of project?

More research...

2005-03-22T11:36:00.000-05:00

Bashford, Joanne. "Placement Decisions for First-Time-in-College Students Using the Computerized Placement Test." Office of Instructional Research, Miami-Dade Community College (1998).

This research report on the scores incoming freshmen received in computerized placement tests. These test scores are compared with the percentage of students who then passed the courses in which they placed themselves. Bashford found that when they adjusted the score ranges to more effectively match the skill levels for the courses, more students passed the courses they were enrolled in. Though just a list of tables, this information is important to consider when deciding what type of placement method is to be used for fycomp. Like other research on this issue, it seems that this method of placement is still not as accurate or useful as portfolio assessment or GSP.

Armstrong, William B. "Pre-Enrollment Placement Testing and Curricular Content: Correspondence or Misalignment?" (2001) ERIC Document Reproduction Service ED 454 895.

Armstrong looks at the relationship between placement tests scores and academic achievement. To measure academic achievement, this researcher administered placement tests before and after students took their first semesters of freshman comp. He found, by no surprise, that there was no significant change in test scores for students. It makes sense, because the tests only predict how well a student will do on that test, not necessarily the course itself. Armstrong suggests that in order for there to be a significant change in scores, curriculum needs to be more closely related to material on the placement tests. This is important to note, but do we really want to measure students' successes on placement tests? Wouldn't this be the same as giving secondary students the MEAP test and expecting that to determine how well they will do in fycomp courses? Sounds like the same thing to me.

Royer, Daniel J. and Roger Gilles. Directed self-placement: Principles and practices. Cresskill, NJ: Hampton Press 2003.

This text seems to explain the standards for self-placement standards in writing programs. Since these authors can be considered the "fathers" of directed self-placement, they manage to perfectly outline the methods for using it in any writing program. So far, I've breifly scanned through the text. However, I do think that it will offer some valuable insight on why schools who use GSP tend to stay true to what Royer and Gilles recommend. EMU has wandered a little down the beaten path with its methods, which may be good or bad; we don't know for sure yet. I will have to further consult this text to form a more clear opinion about the issue.

Armstrong, William B. "Validating Placement Tests in the Community College: The Role of Test Scores, Biographic Data, and Grade Variation." Annual Forum of the Association for Institutional Research Boston, MA 28-31 May 1995.

This study looks at the validity of placement tests and how the scores of these tests relate to student grades. The main thing that Armstrong found was that these placement tests did not predict grades significantly because there are too many outside variables that determine final grades for English courses. Some of these variables were age, gender, previous knowledge, committment to learning the material, instructor variations, etc. This study makes me wonder about how the GSP questionaires would predict how well students do in the course. Judging by this study, it seems superfluous to look at this for my own study. I think it further convinces me that I need to modify some aspects of my study to include writing self-efficacy in comparison to placement strategies.

Luna, Andrea. "A Voice in the Decision: Self-Evaluation in the Freshman English Placement Process." Reading and Writing Quarterly 19 (2003): 377-392.

At Lyndon State College writing program administrators tried placing students into freshman courses by letting them choose the course right for them AND then writing an essay defending that choice. I think this is a great idea, but one that probably works best in a small school like LSC. I'm sure it may be more difficult for something of this nature to be implemented at EMU, but still something to think about as we move toward the GSP-type procedure for fycomp. What researchers noticed were that some students would sell themselves short when deciding which course to enroll in. They may have said in their essay that they lack organization skills, but the essay itself was clearly organized. In this case, advisors compensated and placed these students in course that they thought would be better. Luna feels that this "lack of confidence" stems from the fact that students have little experience as self-evaluators of their writing. I agree. This study further confirms my questions about how well students will place themselves in the GSP method. Will our students sell themselves short as well? Maybe this is something to look at in-depth for my study. Do students' writing self-efficacy predict the course they choose to enroll in?

Borrowman, Shane. "The Trinity of Portfolio Placement: Validity, Reliability, and Curriculum Reform." Writing Program Administration 22 (1999) 7-25.

Borrowman discusses the importance of portfolio assessment for fycomp placement and how it is conducted at the University of Arizona. This program of study reitterates similar thoughts on portfolio placement that I have found in my research. What is interesting is that this program somehow has collaborated with high school teachers where both secondary and post-secondary writing instructors score the portfolios for placement into composition courses. I'm not sure if this is the best idea, unless secondary teachers are familiar with the program used at UA. I think the most important aspect of this placement procedure is that is relates to the way in which comp studies is taught at the university. This placement strategy is one of few that seem to correlate so well with what will be used in the classroom. Doing it this way is reason for making accurate placement decisions for students.

Getting there...

2005-03-20T23:17:00.000-05:00

Wachholz, Patricia B. and Carol Etheridge. "Speaking for Themselves: Writing Self-Efficacy Beliefs of High- and Low-Apprehensive Writers." ERIC Document Reproduction Service, ED 403 563.

These researchers examined the differences in writing self-efficacy in freshman students with either high or low apprehensive feelings about writing. Like many other studies on self-efficacy and writing, performance is greatly related to a student's sense of self-efficacy. They used the Daly-Miller Writing Apprehension Test with follow-up interviews of 5 high-apprehensive students and 5 low-apprehensive students. The study suggest that it is important for instructors to be aware of these students and their senses of self-efficacy in order to help all of their students achieve their writing goals. I wonder if it would be helpful to administer the Daly-Miller WAT to separate students who feel apprehensive into classrooms with peers who feel similar. The same goes for students who are not as apprehensive. The problem is that these groups are still relatively small in comparison to the vast majority of freshman students. Is the GSP questionaire indicative of apprehensive feelings of incoming freshmen? There is a connection between self-efficacy and performance, but does this guarantee that all students will make the right decision with respect to their placement in freshman composition courses? I'm not sure.

Harrington, Susanmarie et al. "The Influence of Word Processing on English Placement Test Results." Computers and Composition 17 (2000): 197-210.

This article focuses on a study that explores the impact of technology on placement testing for freshman composition. The researchers wondered if students would do better on placement tests if they were word processed, and if raters would rate tests higher if word processed. While not directly related to GSP, this study raises some interesting issues about the placement procedure and the need for making decisions based on more than impromtu essays. The researchers found that there was no difference in score between tests that were typed and tests that were hand-written. In both cases, students were placed in courses regardless of the format they completed there tests in. This study relates to GSP because it shows that placement cannot be accurately determined on impromptu essays alone. Researchers even recommend that "placement tests must match up not only with the curricula into which they place students, but also with the prior writing practices of the students who will take them" (205). Perhaps, if administered in a time-efficient way, placement would work best based on combination of both GSP and impromptu essays.

Zimmerman, Barry and Albert Bandura. "Impact of Self-Regulatory Influences on Writing Course Attainment." American Educational Research Journal 31 (1994): 845-862.

This article focuses on self-efficacy levels in freshman comp students. The researchers conducted a semi-field study where they did not control for certain variables. They administered a questionaire to determine students' efficacy levels about writing, which may be useful if I decide to create my own questionaire for my study. They found that students had high self-efficacy when it came to using sources in their papers, but low efficacy when it came to getting work done with distrations. The first result seems a little unbelievable when I compare it to my first year students, but the latter seems to fit well with what I've seen so far. Zimmerman and Bandura recommend that first year instructors evaluate self-efficacy at the beginning of each semester in order to meet their students at the appropriate level. Maybe I need to see if the GSP questionaire is a measure of self-efficacy and therefore omits the need for testing self-efficacy levels of students who have already been placed in English 120 or 121. Maybe this is how I can modify my study....

More annotations...

2005-03-20T02:47:00.000-05:00

Blakesley, David. "Directed Self-Placement in the University." Writing Program Administration 25 (2002): 9-39.

This article is one of few sources that directly relate to my research topic. However, it still seems a little general for what I was looking for. The article outlines what we should consider during the process of implementing a directed self-placement procedure for incoming freshmen. Concerns about the reactions and pressure from administrators, stakeholders, and academic advisors are presented throughout this work. Much of what Blakesley describes seems to relate to what the first-year writing committee has had to face this year. One problem that Blakesley describes is the ability to prove that this form of placement works or works better than what we have available now. Theoretically it works, but that isn't enough to convince those at the top or even those around us. He mentions looking at grade distribution and following particular remedial students as they move through college. But this will take time and funding that is not readily available. We want an answer now and a concrete answer so that we can make adjustments to this placement procedure if need be. Unfortunately, this article did not give me the answers I was hoping to find on this topic.

Willard-Traub, Margaret et al. "The Development of Large-Scale Portfolio Placement Assessment at The University of Michigan: 1992-1998." Assessing Writing 6 (1999): 41-84.

This article takes a look at the introduction of portfolio assessment for incoming freshmen at U of M. During the 6 years this placement procedure took place, writing program administrators came across great advancements and pitfalls in the system. I originally thought that high school students would have difficulty meeting the requirements for these portfolios, but according to the article 88% successfully submitted portfolios for placement. I can see how this form of placement would work at U of M because this school tends to recruit students from public schools that have strong academic programs. Eastern, on the other hand, does not attract the same type of student from the same school districts. Granted there are exceptions, but if Eastern were to use portfolio assessment for placement, I feel that students would have difficulty meeting the basic requirements. One thing I am unsure of is why U of M does not use this assessment for placement today. I can guess that time constraints, funding, and the lack of resources in other areas contributed to dropping this format of placing students in writing courses. The thing that I like about this method is that it, like GSP puts the student in more control of her academic future. Allowing for choices, whether it be through GSP or portfolio material seems to be essential in order for self-efficacy to be strong throughout the student body.

Lavelle, Ellen and Nancy Zuercher. 1999. "University Students' Beliefs about Writing and Writing Approaches." ERIC Document Reproduction Service, ED 434 541.

This study looks at the approaches students take toward writing and catagorizes them. Students were either elaborative, low self-efficacy, reflective-revisionist, spontaneous-impulsive, or procedural. Through a format of surveys and semi-formal interviews researchers explored the nature of students in freshman composition courses. This study was interesting because it gave me an idea of the sort of questions I may ask students in my research (if I do a survey, and I'm still unsure about that). These researchers asked questions like:
1) Who are you as a writer?
2) What types of writing tasks do you prefer? Why?
3) Describe your experience of writing. Does your thinking change in writing? Your interpretation of the task?
4) Are you concerned about how much time your writing task takes?
I wonder if student answers to these questions would correlate to their choices to enroll in English 120 or 121? And if asked these questions at the beginning and end of their first semester of composition, would the answers change much? This seems to be somewhat important to GSP because it emphasizes the importance of asking students to reflect on themselves as writers. The study does not talk about grades and whether students of one category of writer tend to succeed more than other students in a different category. I think I'm getting a little too off-track from my original research question here, though.

Campillo, Magda and Suzanne Pool. "Improving Writing Proficiency through Self-Efficacy Training." Annual Meeting of the American Educational Research Association, 19-23 Apr. 1999 Montreal, Quebec.

This report shows the results of a pilot program that involved teaching self-efficacy to a group of "at-risk" students who failed the writing proficiency placement exam. They found that students in this test group, when asked to retake the placement exam at the end of the summer did better than those who weren't in the test group. They didn't find a cause-effect relationship, but they are interesting results. The problem I have with this study is that students were asked to reflect on homework assignments and practice placement tests to learn how to be more accurate in their writing self-concepts. This study seems to imply that students are not necessarily better writers at the end, but better judges of their abilities when it comes to practice tests. I wonder how many students would benefit from self-efficacy training at Eastern. Do we have more "at-risk" students than other schools? If so, will incoming students not previously trained in self-efficacy inaccurately place themselves in composition courses? This study does say that these students tend to think they are doing better than they actually are. Is it possible that a majority of our students (if "at-risk") will be the same way?

Pajares, M. Frank and Margaret J. Johnson. "Confidence and Competence in Writing: The Role of Self-Efficacy, Outcome Expectancy, and Apprehension." Annual Meeting of the American Educational Research Association, 12-16 Apr. 1993 Atlanta, GA.

Pajares and Johnson study a group of undergraduate students who are on track to be elementary English instructors and how their writing self-efficacy impacts writing performance, among other things. Though they found correlations as they expected, they said that writing skills self-efficacy did not change throughout the one semester that these students were followed. They feel that because the course being taught was not a composition course per se where writing skills were focused on, it is no wonder that there were no changes in self-efficacy in this department. It would seem that self-efficacy can only be improved if instruction actively addresses the issue. If not asked to actively reflect on one's self-efficacy in writing, students have difficulty expressing their thoughts or accurately placing themselves at a level of self-efficacy that is representative of what they actually accomplish. This research further restates my concern that some students (who have not been actively instructed to think of themselves as writers) will have difficulty determining the composition course that is best for their skill level with GSP.

And the beat goes on...

2005-03-14T17:25:00.000-05:00

Matzen, Richard N. and Jeff E. Hoyt. "Basic writing placement with holistically scored essays: Research evidence." Journal of Developmental Education 28 (2004): 1, 2-4, 6, 8, 10, 12, 34.

This study focuses on using time-based essays to place students into first year composition courses. It compares the accuracy of placement between time-based and the formerly used multiple choice test format. They found that while the time-based essay format was better, it could not replace the multiple choice format and would be best used in addition to the original format. My problem with this study is that it was conducted with a small number of students (far from the size of a new freshman class). Even though the researchers mention this, they do not discuss the implications of cost or time with respect to using the time-based essay placement strategy. Sure it works with 400 or so students, but what about a number many times the size of that? Should only small universities benefit from effective placement strategies? Ideally if there was more communication between university and secondary ed. instructors/administrators the problem could be aleviated. Unfortunately, it is easier said than done.

Holmsten, Vicki. "Report from the Invisible: A Teacher-Research Project in Evaluation in a Community College Basic Writing Classroom." Annual Meeting of the National Council of Teachers of English Conference on College Composition and Communication. Atlanta, GA 24-27 Mar. 1999.

This report on a teacher-research project looks at possible ways to judge when a student is ready to move on to a regular college writing class. Though her research focuses mainly on basic/remedial college writers, the most interesting part is that her students did have a sense of their own writing skills early on. You would think that it would be difficult for remedial writers to accurately assess themselves about their level of writing because they tend to struggle with writing itself. Yet, Holmsten does say that there was a correlation between her thoughts of where her students should go next and thoughts her students had about where they should go. Not only did the students accurately determine their readiness for the next level, but a majority passed the next writing course with grades of Cs or better. The only thing I wonder is whether the fact that she mentioned her study at the beginning of the semester had an effect on her results. If students knew that she was thinking in this direction, they may have been more conditioned to be self-reflective, which led to a good sense of direction by the end of the semester. What happens if students are asked to place themselves and they have never been asked or taught to be self-reflective about their writing? I see this being a problem for students who are coming from secondary schools where it is still sometimes like Friere's "banking concept" of instruction.

More of my annotated bibliography...

2005-02-22T15:40:00.000-05:00

Royer, Daniel J. and Roger Gilles. "Basic Writing and Directed Self-Placement." Basic Writing e-Journal 2 (2000): 31 Jan. 2005 http://www.asu.edu/clas/english/composition/cbw/summer_2000_V2N2.htm

This article looks at the problematic nature of traditional placement measures, and the benefits of directed self-placement. Royer and Gilles discuss the ways they use directed self-placement at Grand Valley State University to encourage the agency of students in their writing program. The most interesting aspect of this article is that it looks like the GVSU model of directed self-placement is almost a mirror image of what we plan to do with students at EMU. I haven't asked Linda, but I think she did mention that they consulted GVSU during the process of developing the "guided self-placement" method at EMU. I like that these authors mention that the placement program needs to be proactive for students to prepare them for being proactive, not just in writing courses, but throughout the rest of the college experience.

Kizza, Immaculate. "Placement Tests: The Writer's Reactions." Annual Meeting of the Conference on College Composition and Communication, 1993, San Diego, California.

This report describes a study conducted at the University of Tennessee Chattanooga that looked at freshmen responses to writing placement tests. After taking an hour-long impromptu essay exam, students were asked to fill out a survey about their experience taking the placement test. Though the placement test seemed to "accurately" place students in the proper composition courses (meaning 98% successfully completed the course), I think the words of the students are more important. One question asked students about the clarity of the essay prompt, and a vast majority felt that they understood the statement but was unsure about what was expected of them. "Wastefulness is a necessary part of the American way of life" was the essay prompt. Students actually mentioned that "the statement was vague: Is America wasteful in time, money, environment, all of the above?," which to me seems like a valid question. The type of writing that I expect from my students at EMU is writing that addresses a more specific topic than what was given here....so why not use a topic that is reflective of what the writing program values? Maybe UT doesn't value that, or it is just this author because it is implied that students who ask this sort of quesiton "exhibited a clear lack of critical ability." To me it seems like just the opposite. It is interesting what has changed and what still remains the same at universities, since this study was conducted in 1992.

Slowly, but surely...

2005-02-15T11:50:00.000-05:00

Right now I am slowly, but surely working on my research for my annotated bibliography. I'm trying not to be overwhelmed by the information that I've found by taking one source at a time. Here are a few that I am using to get me started:

Saunders, Pearl I. 2000. Meeting the needs of entering students through appropriate placement in entry-level writing courses. ERIC Document Reproduction Service, ED 447 505.

Saunders examines the "accuracy of our entry-level English assessment practices in order to obtain pertinent information that may help to increase the academic success" (3) of students at the community college level. Her report examines the relationships between student placement scores (from ACCUPLACER-computer assisted testing software), grades, age, race and gender among freshman students at St. Louis Community College. While age, race, and gender had little to do with student grades in these entry-level courses, placement scores did. Students who were accurately placed in a course as a result of test scores had a higher retention rate than those who were inaccurately placed. According to the study, when a student was misplaced it was due to advisors, or faculty who misguided the student, not ACCUPLACER. What was interesting about this study was that they suggested that students take both enter and exit impromptu essay "exams" to ensure that there is continuity among instructors' teaching practices/grading. I wonder if these impromptu "exams" would cause anxiety for students that could get in the way of becoming better writers. If they are concerned about these upcoming "tests" throughout the semester, then they may avoid taking chances with their writing throughout the semester. I also wonder who would evaluate these "exit" exams and the amount of time it would take to look at them. Wouldn't there be discrepancies among evaluators as well?

Haswell, Richard H. 2004. Post-secondary Entry Writing Placement : A Brief Synopsis of Research. CompPile.

Haswell's essay gives an overview of how writing placement has worked in the past, and where it is headed for the future. Descriptions of tests, both essay and standardized, tend to be the trend. However, more and more institutions are changing their methods to directed self-placement. This essay gave me a good idea of where to start my research and where to begin finding some valuable sources. The works cited page of this essay is probably as much, or more helpful than the essay itself. I have already started to look for some of the original sources that Haswell points to in his essay to shape my own opinions about the writing placement techniques used in post-secondary schools.

New Ideas...but are they too narrow?

2005-02-01T13:26:00.000-05:00

So, after reading Richard Haswell's breif synopsis of Post-Secondary Entry Writing Placement I now have a better idea of where I want to go next. He has quite a few sources in his bibliographic information that I am currently trying to find, since he only mentions them briefly in his paper.

The ones I am most interested in finding are:

Matzen and Hoyt's 2004 research on holistically scored essays from the Journal of Developmental Education (unable to find on the shelf)
Royer and Gilles 2003 Directed self-placement: principles and practices
Blakesley's Directed self-placement from WPA
Willard-Traub's Portfolio assessment at U of M in Assessing Writing (possibly available through ILL)

I still want to find a way to see if students "accurately" place themselves in the appropriate course through the guided self-placement "quiz" that EMU plans to administer to incoming freshman for the Fall 2005 school year during FastTrack. The only way I can think of to measure this "accuracy" (and who knows if it would be statistically significant...I may have to rack my brain for the SPSS knowledge I retained from undergrad.) is to do a somewhat logitudinal study.

Asking students for permission to follow them throughout the placement process, and their first semester of freshman comp. at EMU. I could request to see the results of the placement "quiz" and follow the students who take the corresponding course. At the end of the course they have chosen...I will seek permission to see the final grades of these same students. If the students pass the course that they placed themselves into, then the "quiz" worked for those students. I want to use a C or higher cut-off (or whatever is considered passing for university standards).

The only problem I foresee is the issue of ethics (I would have to use students not enrolled in my course). Would it be a problem if students knew that I would be looking at their grades at the end of the semester? Would that itself skew my results? Is it necessary to have a data pool that is statistically significant? I guess I can't worry too much about that right now.

The other idea I have is to look at the success rate of students who were placed based on ACT scores prior to the implementation of the guided placement program at EMU. This would determine if there is actually a difference between the two methods. Unfortunately, this may be too big.

Update on Research

2005-01-28T09:43:00.000-05:00

I'm starting to create a better idea of what my project will look like, but most of it is in my head (not the best place to be at this point). I have been thinking a little bit more about students and directed self-placementfor fycomp...Right now I would like to see what students think of the new"questionaire/quiz" that will be used in the summer months during orientation. This new format, derived from the "Cosmo-quiz-type" layout is what EMU is planning to use to place freshmen for the Fall 2005 school year (and hopefully for years to come). I have looked at the questionaire, and have seen some useful aspects, and some problematic ones. The biggest problem I have with this format is that it assumes that students already know how to talk about writing. Many students cannot think or verbalize aspects of writing,because they have not yet been trained to think in that way. I know this because when I ask my own students about identifying "conventions" in their writing, they have no clue what I mean. This is, of course just one example but something to keep in mind.

On Reading Empirical Research in Writing...

2005-01-18T19:17:00.000-05:00

The text gives you quite a bit of info right away. I expected to be taken back by this info, but it seemed to be already known facts (sometimes hidden in archaic language). I wanted the author to "cut to the chase" without all of the background info. The background knowledge is a little bit interesting, like when she mentions the need for mechanical handbooks in the military leading to research on composition and rhetoric. Her thoughts on how to organize research with notecards scared me a little. Color-coded, word-coded, catalogued, and more seemed confusing to me. Even if I don't agree with her method of cataloguing research her point is still worth noting. Keep track of all materials and info as you go along! Unfortunately, this is something I know I am going to have difficulty with as I conduct my own research.