Assessing Research

In many ways, collecting research is the easy part. I say that because we often do NOT assess information sources very carefully. Hopefully, this lecture will provide some useful assessment methods.

Typically, when we encounter an information source we read that source and outline the information it contains. Notice that I'm suggesting that we do NOT apply any critical standards to that information. For those of you with a liberal arts background, that observation might seem anathema; after all, you have learned to read literature critically. That's certainly true; but, do you apply that same critical apparatus to the articles you read about literature? For instance, did you ever try to discover why a critic took issue with another critic? For those of you who lack even this critical perspective, I submit that you treat most information neutrally and uncritically.

Let's consider a few examples to support my description of a two part assessment process.

 Part One: Examining a "Parent" Source

First, you must be able to critically evaluate the "parent" article, the article that sends you on your assessment and research path. While this task can be accomplished ad hoc, applying a replicable method yields more useful results because they can be compared across findings. In an article on technical communication research by Pinelli and Barclay, I consider the content of each paragraph and simply list that content followed by "probes" or questions that occur to me.

Another approach is to "table" your findings. In this parent article for example:

Paragraph Number	Name the Topic	Suggest a Question	Motivate the Question
1	Studies the difference between theory and application for technical communication	What evidence will support the authors' perception that "pure" research has a positive influence on pragmatic issues faced by professional communicators?	"Pure" research influences pragmatic communication issues, and the application of such research in this manner suggests additional research questions. The relationship is symbiotic and recursive.
2	Although most professional communicators understand the importance of research, they have little training in either conducting or interpreting research.

This approach, while admittedly time-consuming, has many advantages:

• Catalogs major points,
• Provides an overview of topical persistence,
• Often, gives some indications of factual errors,
• Allows the reviewer to list major questions for additional research.

You should realize, however, that this process is merely a heuristic to help you understand the kind of mental discipline needed to assess others' research critically. At some point, you will have had enough practice and the process will become routine. For now, you should try to use a process that has the kind of structure I have suggested above. Once you have reviewed a parent article with the kind of rigor I've outlined, you can proceed to the second part of the assessment: reviewing "sibling" articles.

 Part Two: Reviewing "Sibling" Sources

Articles about scientific or medical advances often appear in newspapers and popular magazines. In the case of medical advances, we are often excited to learn about new treatments and their implications for the general public. However, often such articles merely present the implications of "ongoing" rather than "applied" research. (See the "Environmental Health Perspectives" website listed at the bottom of this page for an example).

While it is unlikely that we would seriously challenge information in such public venues, if we were truly interested in the research how would we find additional information? We would examine other sources that spoke to the same issues, including sources that were referenced in the parent article. We'll call these other sources "sibling" sources. Again, our research process needs to be both routine and replicable.

For the sake of discussion, let's assume that the parent article focuses on "global warming."

1. Make a list of all of the people cited or quoted in the parent article. Make a similar list of any secondary sources cited. These latter references should include both formal (a list of additional reading, for example) and informal ("as the Brisbane News reported on December 11, 1998").

2. Armed with these two lists, begin a search for:

• Both the general topic and any similar topics you can think of (ozone depletion, weather patterns, etc.),
• Other articles by the same author,
• Work by anyone cited or quoted in the article,
• Any other sources referenced in the article.

3. Once you find these sources, repeat the above process as often as needed based on "sibling" articles (the search may simply be limited by the relative "newness" of your topic) until you are certain you have "captured" as much detail as humanly (and sanely) possible about your topic.

 Results of Applying This Process

After completing this two part assessment process, you should have both a substantial amount of supporting information and some idea about the problems found in the parent article. This background should allow you to assess the parent article in a much more informed and critical manner.

 Turning a Critical Analysis into an Abstract

Once you understand the essential ideas in an article, you are prepared to write an Informative Abstract of that piece. There are many methods for producing such abstracts; I have pointed you to only a few here:

http://www.io.com/~hcexres/tcm1603/acchtml/abstrax.html 

I suggest that you perform your own search on "writing abstracts" to see what you can add of your own. Feel free to share any new-found resources with your colleagues in the Free Discussion Area: 7730 Issues in TPC.

 Websites of Interest

Environmental Health Perspectives

End of Page