Conducting Literature Reviews

(Instructor's Note: Much of the following discussion relies on two texts:

Fink, Arlene. Conducting Research Literature Reviews: From Paper to the Internet. Thousand Oaks, CA: SAGE Publications, 1998.

Girden, Ellen R. Evaluating Research Articles: From Start to Finish. Thousand Oaks, CA: SAGE Publications, 1996.)

Conducting a literature review provides a researcher with an overview and evaluation of available resources on a specific topic. This assignment prepares you to:

• Select a research problem and state it as a series of searchable questions
• Identify likely subject headings and keywords to structure a search
• Establish search criteria to identify appropriate paper and electronic sources
• Create an evaluation methodology for the "best" sources and records that ensures the validity of both quantitative and descriptive studies, as well as meta-analyses
• Prepare a structured analytical approach for abstracts and use it to analyze and report results

 Select and State a Research Question

Although stating a research question may seem relatively straightforward, it is often the least well-defined aspect of a research program. While many people understand the difference between over- and under-generalization, they still create research questions that violate these principles. An additional difficulty occurs when a researcher revises the question during the research process. First and foremost, a researcher must have a consistent and well-defined focus. Let's consider a simple research question:

Why do readers refuse to read texts?

Although this question seems like a resolvable issue, consider what has been left out:

• What kinds of readers are we talking about? People read for all sorts of reasons. Someone reading a novel for pleasure may stop reading because they find the text too scary, objectionable, boring, or poorly written; all of those reactions may be idiosyncratic and unresolvable. Other readers may be using a text to support completing a task, such as servicing an aerial navigation beacon; in this instance, rejecting a text has more serious consequences. Or, one may be interested in why school children refuse to read.
• Why are they supposed to be reading? Reading for pleasure, reading to gather task-supportive information, reading to acquire conceptual knowledge, and the like are all very different activities.
• What "kinds" of texts? The idea of textuality spans a considerable range of theory. Texts can be thought of as "genre" that arise to support the needs of various combinations of audience and purpose. For instance, a tutorial text supports a specific audience and purpose; so too do genre such as reference manuals, comic books, novels, newspapers, and annual reports.
• What do the terms "refuse" and "read" mean in this question? Reading may seem like a well-defined process; however, it can require readers to consider and process more than textual material. Many texts have non-textual elements: illustrations (of all kinds), data displays, and document conventions (headings and emphasis techniques, for instance). The reading process is further complicated by the use of "dynamic" or moving elements in electronic information; these must also be "read." Refusing, or rejecting, an information source is fraught with similar definitional difficulties. Festinger, for example, provides an interesting perspective on an entire research field called "source credibility."
• Where does the problem really lie? The question assumes that something is wrong with the way potential readers view the text. Isn't it as likely that some problem in the text causes this reader behavior?

So, the research question could be posited in a more specific way:

What features of the Non-Directional Beacon Field Service Manual make it difficult for service personnel to complete their tasks?

Sounds good, but this question introduces a new set of difficulties.

• Is it likely that this specific manual has been researched? Even the most generous guess would be NO (unless this company happens to conduct a rigorous internal research program). Perhaps the manual can be defined as a more typical genre: tutorial, reference, procedural, etc.
• How can we characterize the features of this manual? This question is perhaps the most difficult to answer. For example, the difficulty may simply be that the text is "poorly written." But, how would we know that, a priori, and how would we characterize poor writing? It might also be, and probably is, the case that a combination of text features, rather than a single feature, is responsible for reader behaviors. In this instance, it might be the case that locators--table of contents, index, running headers, etc.--may simply be flawed; or, it might be that page numbers are missing or incorrect. For the researcher, it is enough to recognize that interactions among a text's features are many and varied.
• In a similar vein, how can we characterize the audience that the question identifies? "Field service personnel" may be trained technicians who know how to service this equipment, but they may not be engineers. Thus, a manual that advises a technician to perform an adjustment based on a calculation may be addressing the wrong potential audience.
• What are "tasks"? Still sounds like an easy one, but the researcher needs to be specific. Are these simple "monkey" tasks that rely on "muscle-memory"; that is, are they tasks that are done frequently in exactly the same manner? Or, are these unique tasks that require diagnosis and decision-making? Obviously, such tasks are VERY different in character.

Given this further problem-setting analysis, a third rendition of the research question might be:

What characteristics of procedural, or reference, manuals make it difficult for service personnel to complete their routine tasks?

At this point, the researcher could offer further clarification by defining manual characteristics, creating a profile of service personnel, and defining routine tasks.

 Identify Subject Headings and Keywords

Once you have a stable and well-defined research question (and any supporting corollaries, definitions, or sub-questions), you can begin to identify subject headings and keywords. Generally, you should rely on subject headings, which tend to be broader in scope, if you are uncertain about the inclusive character of your research question. You can also use these subject headings to check completed research based on keywords to be sure you have not missed important sources.

If we use our research question stated above, subject headings such as reading behaviors, reading research, and the like might provide some information.

In contrast, keywords have been compiled from titles, abstracts, and subject headings. Before we consider what keywords might be found in a database, let's try to identify potential keywords in our research question:

What characteristics of procedural, or reference, manuals make it difficult for service personnel to complete their routine tasks?

Potential keywords: procedures, reference, referential, manuals, texts, documentation, service, tasks; procedural manuals, reference manuals, reading behavior, audience analysis, task definition.

You may also want to add such search terms as specific authors (known authorities, for example), titles, title words, journals, and even dates to help limit (or expand) your search possibilities.

Since search techniques vary considerably among possible sources, a model search will not be suggested here. Instead, you should create your own model search and try searching various sources. It is likely that you will find more information during this search than you expected. For example, it is likely that a database search will return some initial, and early, indication if there is the potential for narrowing (or expanding) the search terms. This model search will also help you to discern the database's behavior in terms of "grammar" and methods used to retrieve records. Be sure to read all help information to broaden your own awareness of the search potential of any one database.

 Establish Search Criteria

Although you have now identified potential keywords, you need to decide which keywords seem most important for your research needs, which are likely to yield any useful records, and which data sources will provide those records. All of these decisions are difficult, but they do become easier as you exercise your research abilities over time.

Begin this process by identifying the major sources in your field (or the field most closely related to your research question). You ought to review Lecture 1 and the material for week 10 for this course. Of course, these lists change over time and you need to check even their accuracy. But, they are a good starting point.

Keep in mind that you should begin EVERY research task by discovering what you (and what others) ALREADY know about a topic. That is, identify ANY existing bibliographic collections first. For example, two published bibliographies were used to provide background as this course was developed:

Alred, Gerald J. The St. Martin's Bibliography of Business and Technical Communication.

Hull, Debra L. Business and Technical Communication: A Bibliography, 1975-1985.

These two collections bring research in the field up to about 1996 (though only a few citations beyond 1990 appear in the Alred collection). Despite this "declared" coverage, even a cursory examination of Alred's text reveals some interesting deficiencies. For instance, the World Wide Web, for better or worse, isn't even mentioned. Well, when did that technology come into everyday use? (An interesting research question in itself, wouldn't you say?)

But I would still maintain that the KNOWN should always be your starting point. At least, it allows you to decide if your topic is even represented in any database. Based on your ability to collect these known resources, you can then establish how much searching you will have to do to assure yourself that you have, indeed, captured all of the relevant information about your research question.

After collecting these known resources, you should begin your search with the databases that index the major journals in your field. No easy answer here! The best way to learn where journals in your field are indexed is to read them. Often, the journal's front matter will tell you what databases index the journal. Review the major journals in your field, often! I've given you a start in the Week 13 topic (see course calendar) by listing the journals that typically appear in major bibliographies in the field or in Technical Communication's "Recent and Relevant" section. You, once again, need to assess the appropriateness of these resources for your own needs.

 Create an Evaluation Process

I'm advocating a two-part evaluation process, consisting of two passes or "screens" of a group of studies. The first screen examines the studies from a practical and feasible perspective. Practically, the screen aims to decide if each study is usable, authoritative, and concentrated on the topic. It also provides the first opportunity to sift through a large group of studies to exclude some records from further review. Typical feasibility decisions at this stage are based on comprehensiveness, English or foreign language, publication credibility, research methods, research setting, and participant characteristics. Of course, not all of these elements need be present for a study to be valuable; some studies rely on subjects and quasi- or experimental design, while others are primarily theoretical.

The second screen identifies the best studies by critically examining the quality of the methodology. Quality is, at best, a "fuzzy" term; but, it can be defined with enough specificity for our needs. From one perspective, quality can be a measure of the "scientific" accuracy of a study. That is, the more rigorous and quantitative studies might be rated as the highest quality. Unfortunately, this perspective would rate a sound theoretical discussion as low in quality. For now, I am going to offer a set of quality characteristics that support exactly this approach; but, I encourage you to consider the materials I offer elsewhere in class lecture notes on our website.

• Does the study provide sufficient information about methodology? Whether quantitative or qualitative, you should be able to tell how the researcher intends to treat and/or develop the topic.
• Is the design internally and externally valid? "Internally valid" studies are free from errors and biases caused by changes in participants over time, the selection process, historical events, attrition, or even changes in instrumentation. "External validity" refers to the consequences of the ways in which participants are selected and/or assigned to research conditions.
• Are the supporting data sources reliable or authoritative? Since you have been assessing journals in your field and identifying authorities, you should be developing an understanding of the hierarchy of quality generally recognized in your profession.
• Are the analytic methods appropriate for the topic and the study's conclusions? I'd review the Lecture for week 11.
• Are the results meaningful in statistical, practical, or theoretic terms? Since many studies in our profession are NOT quantitative, you should also be willing to consider the importance of practical, qualitative, and theoretical studies. Although decidedly more difficult to assess, they should not be rejected.
• Are the results presented in a coherent manner? If the logic of the results does seem to follow from the evidence, there's no problem. But, you need to be critical. For statistical studies, the potential for arguing other interpretations of the results is restricted, though not completely; for other studies the "results" may not be as obvious. For example, if the work presents a theory, the only valid conclusion is a "test" of that theory. Absent that information, the theory survives solely on the reputation and authority of the author, bolstered by the plausibility of the author's supporting argument.
• Does the study recognize both its strengths and its weaknesses? A point often neglected. While we are likely to review competing research or other viewpoints in a literature review early in an article, we may not take a similar approach in discussing results or conclusions. Often, pointing out weaknesses leads others to respond with good solutions for those weaknesses. At any rate, the idea is to foster research AND to be honest about our own.

End of Page