Another View of Research

Although the traditional major division for research has been along the quantitative-qualitative axis, other category schemes can be used. Here are some of the more obvious examples (adapted from Lauer, Janice M. and J. William Asher. Composition Research: Empirical Designs. NY: OUP, 1988.)

 Case Studies

Often the first empirical research done on a topic seeks to identify the important variables. Basically, this descriptive effort tries to decide which elements, in the incredible welter of interactions, deserve closer examination. Typically, case studies proceed qualitatively by focusing on individuals, small groups, or larger closed environments.

Research designs for case studies include subject selection, developing hypotheses or background theories, collecting data, developing salient variables, seeking agreement among observers or coders, offering results (usually rich narrative accounts of events).

A good checklist for performing such studies should include:

1. Offering the underlying theories that support the study,
2. Describing the subjects and the basis for their selection,
3. Summarizing data collection methods,
4. Defining the kinds of data collected,
5. Identifying the variables found and defining them adequately,
6. Detailing how variables were interrelated and tabulated,
7. Explaining the coding agreement level and how it was calculated,
8. Summarizing the form(s)--summaries, figures, narrative, etc.-- used to report the results,
9. Cataloging problems in the study and suggesting remedies.

 Ethnographic Studies

An outgrowth of social science disciplines that regard direct cultural observation as necessary, this methodology generally implants an observer in a condition where that person watches events over time. In examining human communication behavior, this process involves choosing the environment, selecting observers, and establishing an observation period.

To develop resilient data observers typically record three kinds of notes: observational (ongoing activities), methodological (tactics), theoretical (speculations about applicable theory). These data are analyzed by a group of coders and placed into categories that form the basis for variables. Finally, these variables are "tested" against schema to try to account for observed behaviors.

Ethnography, as a qualitative research technique, offers some interesting advantages to researchers of human communication behavior. It does, for instance, provide a rich account of the complexity of communication events. But, it also has its share of weaknesses. For example, ethnographic studies tend to produce an enormous amount of "data," and that data can be difficult to categorize. Second, the order in which coders receive the data may influence their perceptions; first in, first out. Other difficulties include: judgment confidence, information availability, positive and negative instances, internal consistency, dubious reliability, missing information, population size, and sampling techniques. As you can see, this can be a very extensive list.

Checklist for ethnographic studies

1. What are the supporting theories?
2. Who are the subjects and environments and why them?
3. What data was collected and how?
4. What variables were identified and defined?
5. What role did the observer play?
6. How are the variables interrelated?
7. How were the results reported? Detail?
8. Can we draw any useful conclusions?
9. What kinds of problems does the study have and how can they be remedied?

 Samplings and Surveys

These two techniques are somewhat like the earlier types but they base their findings on a smaller group selected from a larger, and similar, population.

Sample selection requires the researcher to select a segment (n) of a larger population (N) to study. Two elements control the n figure. First, researchers must select a feasible number of units that will allow them to collect and analyze data adequately. This feasibility is generally expressed as Confidence Limits. For example, if a researcher finds that in a study involving 30 (n) subjects a variable occurs 60% of the time, in the entire population (N) that same variable can be said to occur 60% plus or minus 18% of the time based on the table below:

Sample Size and Confidence Interval Limits

While it initially appears that confidence levels are typically quite small for small populations, there are a variety of correct factors that can be applied to confidence limits. For example, if the N population is relatively small, less than 100, then a correction factor can be calculated. Basically, that correction looks like:

For example, if a researcher finds that in a study involving 30 (n) subjects selected from a potential population of 100 (N) subjects a variable occurs 60% of the time, in the entire population (N) that same variable can be said to occur 60% plus or minus 18% of the time.

However, since the n represents a relatively large percentage of the total population N, the 18% is corrected by .84 and now the researcher can say: In the entire population (N) that same variable can be said to occur 60% plus or minus 16.5% of the time. Other kinds of corrections can also be applied.

Once a sample size has been established, the method for data collection--questionnaires, paper collection, interviews, test results--must be determined.

Surveys and questionnaires generally contain two types of questions: objectively scored and open-ended responses. The former can be useful because they can be scored easily; the latter requires more coding. One aspect of surveys that often confounds a study is the fact that response rate, or even non-responses to questions, reduces the confidence level targeted in the study plan. The obvious response to this difficulty is to use more subjects than you intended to achieve your confidence level goal.

Data collected in a survey may be of three types: nominal, interval, or rank order. Nominal data is simply things we can count: comma splices, sentence length, occurrence of "vocal" punctuation (ahem), etc. Interval data comes from scores , ratings, and grades. Rank order data places items in a hierarchical order.

At this point we can begin to consider the role of statistics in data analysis. But I would suggest to you that it is important to begin to keep some of the definitions I have offered, and will offer, in mind. They are pretty basic stuff.

Four additional terms are important here. Mean refers to the average of the score on whatever measure is being applied. The next three terms measure individual differences or indications of degree of variability among subjects tested using a particular variable. Range represents the highest score minus the lowest score on a variable in a sample. Standard deviation is about one-fifth to one-sixth of the range. Variance measures the amount of individual differences among scores and is the square of the standard deviation.

All of these try to determine data precision and all depend on the amount of variability of each characteristic in the studied population. For example, nominal data reported in percentages derives its standard deviation from the mean.

Checklist for Sampling Survey studies

1. What population (N) is the studied based on? Why?
2. What was the sample?
3. What sampling method was used? Why?
4. What were the confidence limits? Can they be accepted?
5. What theory lies behind the survey and its questions?
6. What kind of data were collected (nominal, interval, rank order)? How? Why?
7. What were the variables?
8. What percentage of returns did the study produce?
9. Can any conclusions be drawn?

 Quantitative Descriptive Studies

These studies isolate important variables, define them, quantify them, and/or interrelate them. Based on these activities these studies report the results of statistical analysis. However, these studies should be regarded as descriptive, not experimental, research because they do not utilize either control groups or treatments.

This research also requires a relatively large number of subjects. For example, a general rule of thumb for number of subjects is ten times the number of variables. Thus, four variables would require a minimum of forty subjects. In addition, subjects should be appropriate for the kind of variables tested. If you are testing writing ability, the subjects should be drawn from a pool of writers (or writing students). Sometimes it is useful to select a random sample based on specific subject characteristics. You might want to have a random combination of education, job function, gender, etc. Drawing subjects from a larger representative pool allows researchers to claim that their subject pool (n) is representative of the larger population (N).

Collecting data can be done in any manner so long as the method remains identical for all participants.

Selecting variables represents perhaps the most time-consuming and difficult aspect of testing. Variables must be selected based on some theoretic assumptions or from those identified by other research methods (ethnographic, etc.). Two types of variables are selected. Independent variables usually refer to those differences that participants exhibit prior to testing (age, education, gender, etc.); dependent variables are those introduced by the researcher for study (a communication behavior). In a reading study, for instance, the independent variables might be degree completed, professional field, and gender; the independent variables might be ability to decode graphics, understanding of international icons, and reading miscues.

All of these research efforts should be controlled by hypotheses based on some theoretical assumptions. To make the most of the power of offering hypotheses, you should always be willing to entertain both your hypothesis and the "null," or competing, hypothesis. For example, if you hypothesize in the study in the above paragraph that: Engineers will decode graphics better than managers; then you should also entertain the possibility that managers will decode graphics better than engineers.

Once data has been collected you can analyze that data (for two variables) based on several decisions. Which condition did you test:

1. Relationships among two variables in a single subject group?
2. Differences among two or more subject groups on a single variable at a time?

To study the condition you collected data for, use the following table as a guide.

Statistical Analysis for Two Variables (from Lauer)

Some useful definitions to keep in mind.

Standard Deviation measures individual differences or dispersion of scores for a variable; it indicates the amount of individual difference in a variable. For example, in our graphic study (above) we might find that gender exhibits a large standard deviation. That may tell us that either there is considerable difference or that we need to consider our sample size. Typically, large samples have six standard deviations in a set of scores; smaller samples about five (See Lauer, p.91).

Chi Square shows the possible relationship between two or more subject groups and one nominal variable. In our demonstration study, for instance, we might want to examine the relationship between the nominal variable male/female (gender) and the nominal variable "decoding graphics behavior." (Lauer, p. 92).

Correlations assess the relationship between pairs of interval data.

Analysis of Variance and the F-test are used when one variable is nominal (sex, age), the other interval (time, score).

T-Test assesses the relationship of a dichotomous nominal (graphic type--scientific versus popular) and an interval variable (time to decode).

In each of the above instances, researchers try to relate two variables. However, more complex studies may require you to analyze the relationships among a number of variables. These studies are supported by Multivariate Analyses. For example, in our demonstration study, we might want to know if gender and professional field have an interrelationship with success in decoding a graphic. The most typical analysis method for resolving such a problem would be some form of factor analysis.

Regardless of the statistical method we use, we do want some assurance that we are interpreting the results of our calculations accurately. Typically this is accomplished by assessing significance and amount of variance. Significance tries to respond to the possibility that the reported relationship could have happened based on chance alone. To assess significance the calculation is compared to tabled values of chance distribution. If the value is higher than that in the table, the result is reported as significant, or unlikely to have occurred by chance. This significance is reported as a p value (probability level) based on either 1 or 5 percent levels. At five percent, this indicates that the chances would have been 1 in 20 cases or 1 in 100 cases; at one percent, chance becomes even less likely.

Checklist for Quantitative Descriptive studies

1. What hypotheses did the study pose?
2. What importance did the lit review give to the study>
3. What are the variables? Their type? Source?
4. Who were the subjects? What was the ratio of subjects to variables?
5. How was the data collected?
6. What kinds of statistical analysis were performed? Were they appropriate?
7. What did the researchers conclude? What kinds of interactions supported those conclusions?
8. Were any relationships significant? At what levels?

 Predication and Classification

Based on the strength of variables these studies try to determine the relationship between several variables and a single criterion. Prediction research forecasts an interval variable; classification, a nominal variable. These approaches are employed to predict future behavior. For example, assigning students to a specific educational treatment (college versus trade school) might be based on such a study.

 Measurement

Using existing standard instruments (Nelson Denny Form B), tests (GRE), attitude scale (personality testing), observation schemes, and questionnaire items (teaching evaluations) to compare current situations to known results.

 True Experiments

Controlled experiments.

 Quasi-Experiments

Similar to true experiments but the subject pool cannot be randomized.

 Meta-Analysis

Mathematically assesses strength and validity of studies done in a specific research venue.

End of Page