Qualitative Research
Mutiara Budi Azhar
Faculty of Medicine Sriwijaya University
- Understanding Non-numerical Data -

Two approaches to Research
Quantitative
• emphasis on the countable--numbers and measures
• statistical methods of data analysis
• traditionally, when people have spoken of "research" they have often meant quantitative research
Qualitative
• emphasis on words, pictures, descriptions or narratives
• emphasis on understanding how humans make sense of the setting they are in through symbols, rituals, social roles, social structures, etc.
Qualitative Research
• Quantitative research attempts to understand "how things really are" in a social setting and believes it is possible do this through empirical methods
• Qualitative research attempts to understand human experiences from the perspective of those who experience them
Quantitative vs. Qualitative
Use Quantitative
• when we know a problem well enough that we know exactly what to measure to answer our question
• if we believe the human experience we are studying is one that can be reduced to numbers and measures
Use Qualitative
• when knowledge is sketchy or we have little theoretical understanding of a phenomenon
• if we believe what we are studying is best understood through the subjective experiences of people involved
Is studying human beings just like studying anything else?
• Weber: when we study human beings, we have access to a kind of information we do not have in other circumstances-- we can "put ourselves into their shoes"
• He called this subjective understanding "verstehen"
Characteristics of Qualitative Research
• Admittedly subjective
• seeks to understand, rather than explain
• reliance on inductive logic--reasoning from concrete experience to abstract theory
• Seeks to generate hypotheses, not to test them
• data are processed as received
• the researcher is the data collection instrument
Some Types of Qualitative Research
• Ethnographic Studies
• Cross-cultural Research
• Case Studies
• Oral Histories
Ethnographic Studies
• Ethnographic studies immerse the researcher within a social group, usually for a prolonged period of time--"subjective soaking"
• The purpose is to help the researcher understand beliefs, values, attitudes, roles, social structures and norms within particular social settings
Perspectives in Ethnographic Studies
o Emic--the Insider’s perspective
the perspective of an individual socialized to daily life in a culture and participating fully in it
o Etic--the outsider’s perspective
the traditional ethnographic approach, that of an outsider trying to ‘translate’ an unfamiliar culture into terms she or he can understand
Cross-cultural Research
• Strategies to resist cultural encapsulation
• Cultural immersion experiences
• Obtaining information directly from cultural minorities about values and normative behavior rooted in the culture
• Collaboration with key informants from the culture on aspects of study design prior to beginning study
Case Studies
• A case study is a type of exploratory research designed to advance knowledge of a phenomenon
• Case studies are appropriate when:
o little is known about the area being studied
o the area studies involves illegal behavior or behavior that is not socially sanctioned
o It is impossible to draw a representative sample of participants
Strengths
• possible to achieve insights unavailable using quantitative methods
• interesting and gratifying
• often on cutting edge of knowledge-building
Weaknesses
• limited capacity to generate definitive knowledge
• small sample not necessarily representative
• ability to generalize to other cases extremely limited
Oral Histories
• Relies on firsthand accounts of individuals who were in a situation
• Often used to capture memories of elders
• Can be single case study or a group of narratives can be collected
Approaches to Understanding Data in Qualitative Research
• Grounded theory
building theory from data by proposing a tentative theory based on part of the data collected, then testing that against additional data
• Feminist Research
the position people have in the social structure influences their experiences and their ways of understanding
Grounded Theory
• In grounded theory, raw data is "coded", that is, classified according to what it seems to mean
• In the process of building theory, these coding schemes become more elaborate and more focused
Characteristics of Feminist Research
• Argues that traditional methods of inquiry often are biased
• supports an egalitarian relationship between researcher and participants
• often focused on action research

Common Qualitative Methods
In this subsection we describe the most common qualitative methods employed in project evaluations. These include observations, in-depth interviews, and focus groups. We also cover briefly some other less frequently used qualitative techniques. Advantages and disadvantages are summarized.
Observations
Observational techniques are methods by which an individual or individuals gather firsthand data on programs, processes, or behaviors being studied. They provide evaluators with an opportunity to collect data on a wide range of behaviors, to capture a great variety of interactions, and to openly explore the evaluation topic. By directly observing operations and activities, the evaluator can develop a holistic perspective, i.e., an understanding of the context within which the project operates. This may be especially important where it is not the event that is of interest, but rather how that event may fit into, or be impacted by, a sequence of events. Observational approaches also allow the evaluator to learn about things the participants or staff may be unaware of or that they are unwilling or unable to discuss in an interview or focus group.
When to use observations.
Observations can be useful during both the formative and summative phases of evaluation. For example, during the formative phase, observations can be useful in determining whether or not the project is being delivered and operated as planned. In the hypothetical project, observations could be used to describe the faculty development sessions, examining the extent to which participants understand the concepts, ask the right questions, and are engaged in appropriate interactions. Such formative observations could also provide valuable insights into the teaching styles of the presenters and how they are covering the material.
Observations during the summative phase of evaluation can be used to determine whether or not the project is successful. The technique would be especially useful in directly examining teaching methods employed by the faculty in their own classes after program participation. Exhibits 3 and 4 display the advantages and disadvantages of observations as a data collection tool and some common types of data that are readily collected by observation.
Readers familiar with survey techniques may justifiably point out that surveys can address these same questions and do so in a less costly fashion. Critics of surveys find them suspect because of their reliance on self-report, which may not provide an accurate picture of what is happening because of the tendency, intentional or not, to try to give the "right answer." Surveys also cannot tap into the contextual element. Proponents of surveys counter that properly constructed surveys with built in checks and balances can overcome these problems and provide highly credible data. This frequently debated issue is best decided on a case-by-case basis.
Recording Observational Data
Observations are carried out using a carefully developed set of steps and instruments. The observer is more than just an onlooker, but rather comes to the scene with a set of target concepts, definitions, and criteria for describing events. While in some studies observers may simply record and describe, in the majority of evaluations, their descriptions are, or eventually will be, judged against a continuum of expectations.



Exhibit 1
Advantages and disadvantages of observations
Advantages
Provide direct information about behavior of individuals and groups
Permit evaluator to enter into and understand situation/context
Provide good opportunities for identifying unanticipated outcomes
Exist in natural, unstructured, and flexible setting
Disadvantages
Expensive and time consuming
Need well-qualified, highly trained observers; may need to be content experts
May affect behavior of participants
Selective perception of observer may distort data
Investigator has little control over situation
Behavior or set of behaviors observed may be atypical

Observations usually are guided by a structured protocol. The protocol can take a variety of forms, ranging from the request for a narrative describing events seen to a checklist or a rating scale of specific behaviors/activities that address the evaluation question of interest. The use of a protocol helps assure that all observers are gathering the pertinent information and, with appropriate training, applying the same criteria in the evaluation. For example, if, as described earlier, an observational approach is selected to gather data on the faculty training sessions, the instrument developed would explicitly guide the observer to examine the kinds of activities in which participants were interacting, the role(s) of the trainers and the participants, the types of materials provided and used, the opportunity for hands-on interaction, etc
Exhibit 2
Types of information for which observations are a good source
The setting - The physical environment within which the project takes place.
The human, social environment - The ways in which all actors (staff, participants, others) interact and behave toward each other.
Project implementation activities - What goes on in the life of the project? What do various actors (staff, participants, others) actually do? How are resources allocated?
The native language of the program - Different organizations and agencies have their own language or jargon to describe the problems they deal with in their work; capturing the precise language of all participants is an important way to record how staff and participants understand their experiences.
Nonverbal communication - Nonverbal cues about what is happening in the project: on the way all participants dress, express opinions, physically space themselves during discussions, and arrange themselves in their physical setting.
Notable non-occurrences - Determining what is not occurring although the expectation is that it should be occurring as planned by the project team, or noting the absence of some particular activity/factor that is noteworthy and would serve as added information.
The protocol goes beyond a recording of events, i.e., use of identified materials, and provides an overall context for the data. The protocol should prompt the observer to
• Describe the setting of program delivery, i.e., where the observation took place and what the physical setting was like;
• Identify the people who participated in those activities, i.e., characteristics of those who were present;
• Describe the content of the intervention, i.e., actual activities and messages that were delivered;
• Document the interactions between implementation staff and project participants;
• Describe and assess the quality of the delivery of the intervention; and
• Be alert to unanticipated events that might require refocusing one or more evaluation questions.
Field notes are frequently used to provide more in-depth background or to help the observer remember salient events if a form is not completed at the time of observation. Field notes contain the description of what has been observed. The descriptions must be factual, accurate, and thorough without being judgmental and cluttered by trivia. The date and time of the observation should be recorded, and everything that the observer believes to be worth noting should be included. No information should be trusted to future recall.
The use of technological tools, such as battery-operated tape recorder or dictaphone, laptop computer, camera, and video camera, can make the collection of field notes more efficient and the notes themselves more comprehensive. Informed consent must be obtained from participants before any observational data are gathered.
The Role of the Observer
There are various methods for gathering observational data, depending on the nature of a given project. The most fundamental distinction between various observational strategies concerns the extent to which the observer will be a participant in the setting being studied. The extent of participation is a continuum that varies from complete involvement in the setting as a full participant to complete separation from the setting as an outside observer or spectator. The participant observer is fully engaged in experiencing the project setting while at the same time trying to understand that setting through personal experience, observations, and interactions and discussions with other participants. The outside observer stands apart from the setting, attempts to be non-intrusive, and assumes the role of a "fly-on-the-wall." The extent to which full participation is possible and desirable will depend on the nature of the project and its participants, the political and social context, the nature of the evaluation questions being asked, and the resources available. "The ideal is to negotiate and adopt that degree of participation that will yield the most meaningful data about the program given the characteristics of the participants, the nature of staff-participant interactions, and the sociopolitical context of the program" (Patton, 1990).
In some cases it may be beneficial to have two people observing at the same time. This can increase the quality of the data by providing a larger volume of data and by decreasing the influence of observer bias. However, in addition to the added cost, the presence of two observers may create an environment threatening to those being observed and cause them to change their behavior. Studies using observation typically employ intensive training experiences to make sure that the observer or observers know what to look for and can, to the extent possible, operate in an unbiased manner. In long or complicated studies, it is useful to check on an observer’s performance periodically to make sure that accuracy is being maintained. The issue of training is a critical one and may make the difference between a defensible study and what can be challenged as "one person’s perspective."
A special issue with regard to observations relates to the amount of observation needed. While in participant observation this may be a moot point (except with regard to data recording), when an outside observer is used, the question of "how much" becomes very important. While most people agree that one observation (a single hour of a training session or one class period of instruction) is not enough, there is no hard and fast rule regarding how many samples need to be drawn. General tips to consider are to avoid atypical situations, carry out observations more than one time, and (where possible and relevant) spread the observations out over time.
Participant observation is often difficult to incorporate in evaluations; therefore, the use of outside observers is far more common. In the hypothetical project, observations might be scheduled for all training sessions and for a sample of classrooms, including some where faculty members who participated in training were teaching and some staffed by teachers who had not participated in the training.
Issues of privacy and access.
Observational techniques are perhaps the most privacy-threatening data collection technique for staff and, to a lesser extent, participants. Staff fear that the data may be included in their performance evaluations and may have effects on their careers. Participants may also feel uncomfortable assuming that they are being judged. Evaluators need to assure everyone that evaluations of performance are not the purpose of the effort, and that no such reports will result from the observations. Additionally, because most educational settings are subject to a constant flow of observers from various organizations, there is often great reluctance to grant access to additional observers. Much effort may be needed to assure project staff and participants that they will not be adversely affected by the evaluators’ work and to negotiate observer access to specific sites.
Interviews
Interviews provide very different data from observations: they allow the evaluation team to capture the perspectives of project participants, staff, and others associated with the project. In the hypothetical example, interviews with project staff can provide information on the early stages of the implementation and problems encountered. The use of interviews as a data collection method begins with the assumption that the participants’ perspectives are meaningful, knowable, and able to be made explicit, and that their perspectives affect the success of the project. An interview, rather than a paper and pencil survey, is selected when interpersonal contact is important and when opportunities for follow-up of interesting comments are desired.
Two types of interviews are used in evaluation research: structured interviews, in which a carefully worded questionnaire is administered; and in-depth interviews, in which the interviewer does not follow a rigid form. In the former, the emphasis is on obtaining answers to carefully phrased questions. Interviewers are trained to deviate only minimally from the question wording to ensure uniformity of interview administration. In the latter, however, the interviewers seek to encourage free and open responses, and there may be a tradeoff between comprehensive coverage of topics and in-depth exploration of a more limited set of questions. In-depth interviews also encourage capturing of respondents’ perceptions in their own words, a very desirable strategy in qualitative data collection. This allows the evaluator to present the meaningfulness of the experience from the respondent’s perspective. In-depth interviews are conducted with individuals or with a small group of individuals.
In-depth interviews.
An in-depth interview is a dialogue between a skilled interviewer and an interviewee. Its goal is to elicit rich, detailed material that can be used in analysis (Lofland and Lofland, 1995). Such interviews are best conducted face to face, although in some situations telephone interviewing can be successful.
In-depth interviews are characterized by extensive probing and open-ended questions. Typically, the project evaluator prepares an interview guide that includes a list of questions or issues that are to be explored and suggested probes for following up on key topics. The guide helps the interviewer pace the interview and make interviewing more systematic and comprehensive. Lofland and Lofland (1995) provide guidelines for preparing interview guides, doing the interview with the guide, and writing up the interview
The dynamics of interviewing are similar to a guided conversation. The interviewer becomes an attentive listener who shapes the process into a familiar and comfortable form of social engagement - a conversation - and the quality of the information obtained is largely dependent on the interviewer’s skills and personality (Patton, 1990). In contrast to a good conversation, however, an in-depth interview is not intended to be a two-way form of communication and sharing. The key to being a good interviewer is being a good listener and questioner. Tempting as it may be, it is not the role of the interviewer to put forth his or her opinions, perceptions, or feelings. Interviewers should be trained individuals who are sensitive, empathetic, and able to establish a non-threatening environment in which participants feel comfortable. They should be selected during a process that weighs personal characteristics that will make them acceptable to the individuals being interviewed; clearly, age, sex, profession, race/ethnicity, and appearance may be key characteristics. Thorough training, including familiarization with the project and its goals, is important. Poor interviewing skills, poor phrasing of questions, or inadequate knowledge of the subject’s culture or frame of reference may result in a collection that obtains little useful data.
When to use in-depth interviews.
In-depth interviews can be used at any stage of the evaluation process. They are especially useful in answering questions such as those suggested by Patton (1990):
• What does the program look and feel like to the participants? To other stakeholders?
• What are the experiences of program participants?
• What do stakeholders know about the project?
• What thoughts do stakeholders knowledgeable about the program have concerning program operations, processes, and outcomes?
• What are participants’ and stakeholders’ expectations?
• What features of the project are most salient to the participants?
• What changes do participants perceive in themselves as a result of their involvement in the project?
Specific circumstances for which in-depth interviews are particularly appropriate include
• complex subject matter;
• detailed information sought;
• busy, high-status respondents; and
• highly sensitive subject matter.
In the hypothetical project, in-depth interviews of the project director, staff, department chairs, branch campus deans, and non-participant faculty would be useful. These interviews can address both formative and summative questions and be used in conjunction with other data collection methods. The advantages and disadvantages of in-depth interviews are outlined in Exhibit 3.
When in-depth interviews are being considered as a data collection technique, it is important to keep several potential pitfalls or problems in mind.
• There may be substantial variation in the interview setting. Interviews generally take place in a wide range of settings. This limits the interviewer’s control over the environment. The interviewer may have to contend with disruptions and other problems that may inhibit the acquisition of information and limit the comparability of interviews.
• There may be a large gap between the respondent’s knowledge and that of the interviewer. Interviews are often conducted with knowledgeable respondents, yet administered by less knowledgeable interviewers or by interviewers not completely familiar with the pertinent social, political, or cultural context. Therefore, some of the responses may not be correctly understood or reported. The solution may be not only to employ highly trained and knowledgeable staff, but also to use interviewers with special skills for specific types of respondents (for example, same status interviewers for high-level administrators or community leaders). It may also be most expedient for the project director or senior evaluation staff to conduct such interviews, if this can be done without introducing or appearing to introduce bias.

Exhibit 3
Advantages and disadvantages of in-depth interviews
Advantages
• Usually yield richest data, details, new insights
• Permit face-to-face contact with respondents
• Provide opportunity to explore topics in depth
• Afford ability to experience the affective as well as cognitive aspects of responses
• Allow interviewer to explain or help clarify questions, increasing the likelihood of useful responses
• Allow interviewer to be flexible in administering interview to particular individuals or circumstances
Disadvantages
• Expensive and time-consuming
• Need well-qualified, highly trained interviewers
• Interviewee may distort information through recall error, selective perceptions, desire to please interviewer
• Flexibility can result in inconsistencies across interviews
• Volume of information too large; may be difficult to transcribe and reduce data

Exhibit 4 outlines other considerations in conducting interviews. These considerations are also important in conducting focus groups, the next technique that we will consider.


Exhibit 4.
Considerations in conducting in-depth interviews and focus groups
Factors to consider in determining the setting for interviews (both individual and group) include the following:
• Select a setting that provides privacy for participants.
• Select a location where there are no distractions and it is easy to hear respondents speak.
• Select a comfortable location.
• Select a non-threatening environment.
• Select a location that is easily accessible for respondents.
• Select a facility equipped for audio or video recording.
• Stop telephone or visitor interruptions to respondents interviewed in their office or homes.
• Provide seating arrangements that encourage involvement and interaction.
Recording interview data.
Interview data can be recorded on tape (with the permission of the participants) and/or summarized in notes. As with observations, detailed recording is a necessary component of interviews since it forms the basis for analyzing the data. All methods, but especially the second and third, require carefully crafted interview guides with ample space available for recording the interviewee’s responses. Three procedures for recording the data are presented below.
In the first approach, the interviewer (or in some cases the transcriber) listens to the tapes and writes a verbatim account of everything that was said. Transcription of the raw data includes word-for-word quotations of the participant’s responses as well as the interviewer’s descriptions of participant’s characteristics, enthusiasm, body language, and overall mood during the interview. Notes from the interview can be used to identify speakers or to recall comments that are garbled or unclear on the tape. This approach is recommended when the necessary financial and human resources are available, when the transcriptions can be produced in a reasonable amount of time, when the focus of the interview is to make detailed comparisons, or when respondents’ own words and phrasing are needed. The major advantages of this transcription method are its completeness and the opportunity it affords for the interviewer to remain attentive and focused during the interview. The major disadvantages are the amount of time and resources needed to produce complete transcriptions and the inhibitory impact tape recording has on some respondents. If this technique is selected, it is essential that the participants have been informed that their answers are being recorded, that they are assured confidentiality, and that their permission has been obtained.
A second possible procedure for recording interviews draws less on the word-by-word record and more on the notes taken by the interviewer or assigned note-taker. This method is called "note expansion." As soon as possible after the interview, the interviewer listens to the tape to clarify certain issues and to confirm that all the main points have been included in the notes. This approach is recommended when resources are scarce, when the results must be produced in a short period of time, and when the purpose of the interview is to get rapid feedback from members of the target population. The note expansion approach saves time and retains all the essential points of the discussion. In addition to the drawbacks pointed out above, a disadvantage is that the interviewer may be more selective or biased in what he or she writes.
In the third approach, the interviewer uses no tape recording, but instead takes detailed notes during the interview and draws on memory to expand and clarify the notes immediately after the interview. This approach is useful if time is short, the results are needed quickly, and the evaluation questions are simple. Where more complex questions are involved, effective note-taking can be achieved, but only after much practice. Further, the interviewer must frequently talk and write at the same time, a skill that is hard for some to achieve.
Focus Groups
Focus groups combine elements of both interviewing and participant observation. The focus group session is, indeed, an interview (Patton, 1990) not a discussion group, problem-solving session, or decision-making group. At the same time, focus groups capitalize on group dynamics. The hallmark of focus groups is the explicit use of the group interaction to generate data and insights that would be unlikely to emerge without the interaction found in a group. The technique inherently allows observation of group dynamics, discussion, and firsthand insights into the respondents’ behaviors, attitudes, language, etc.
Focus groups are a gathering of 8 to 12 people who share some characteristics relevant to the evaluation. Originally used as a market research tool to investigate the appeal of various products, the focus group technique has been adopted by other fields, such as education, as a tool for data gathering on a given topic. Focus groups conducted by experts take place in a focus group facility that includes recording apparatus (audio and/or visual) and an attached room with a one-way mirror for observation. There is an official recorder who may or may not be in the room. Participants are paid for attendance and provided with refreshments. As the focus group technique has been adopted by fields outside of marketing, some of these features, such as payment or refreshment, have been eliminated.
When to use focus groups.
When conducting evaluations, focus groups are useful in answering the same type of questions as in-depth interviews, except in a social context. Specific applications of the focus group method in evaluations include
• identifying and defining problems in project implementation;
• identifying project strengths, weaknesses, and recommendations;
• assisting with interpretation of quantitative findings;
• obtaining perceptions of project outcomes and impacts; and
• generating new ideas.
In the hypothetical project, focus groups could be conducted with project participants to collect perceptions of project implementation and operation (e.g., Were the workshops staffed appropriately? Were the presentations suitable for all participants?), as well as progress toward objectives during the formative phase of evaluation (Did participants exchange information by e-mail and other means?). Focus groups could also be used to collect data on project outcomes and impact during the summative phase of evaluation (e.g., Were changes made in the curriculum? Did students taught by participants appear to become more interested in class work? What barriers did the participants face in applying what they had been taught?).
Although focus groups and in-depth interviews share many characteristics, they should not be used interchangeably. Factors to consider when choosing between focus groups and in-depth interviews are included in Exhibit 7.
Developing a Focus Group
An important aspect of conducting focus groups is the topic guide. The topic guide, a list of topics or question areas, serves as a summary statement of the issues and objectives to be covered by the focus group. The topic guide also serves as a road map and as a memory aid for the focus group leader, called a "moderator." The topic guide also provides the initial outline for the report of findings.
Focus group participants are typically asked to reflect on the questions asked by the moderator. Participants are permitted to hear each other’s responses and to make additional comments beyond their own original responses as they hear what other people have to say. It is not necessary for the group to reach any kind of consensus, nor it is necessary for people to disagree. The moderator must keep the discussion flowing and make sure that one or two persons do not dominate the discussion. As a rule, the focus group session should not last longer than 1 1/2 to 2 hours. When very specific information is required, the session may be as short as 40 minutes. The objective is to get high-quality data in a social context where people can consider their own views in the context of the views of others, and where new ideas and perspectives can be introduced.
The participants are usually a relatively homogeneous group of people. Answering the question, "Which respondent variables represent relevant similarities among the target population?" requires some thoughtful consideration when planning the evaluation. Respondents’ social class, level of expertise, age, cultural background, and sex should always be considered. There is a sharp division among focus group moderators regarding the effectiveness of mixing sexes within a group, although most moderators agree that it is acceptable to mix the sexes when the discussion topic is not related to or affected by sex stereotypes.
Determining how many groups are needed requires balancing cost and information needs. A focus group can be fairly expensive, costing $10,000 to $20,000 depending on the type of physical facilities needed, the effort it takes to recruit participants, and the complexity of the reports required. A good rule of thumb is to conduct at least two groups for every variable considered to be relevant to the outcome (sex, age, educational level, etc.). However, even when several groups are sampled, conclusions typically are limited to the specific individuals participating in the focus group. Unless the study population is extremely small, it is not possible to generalize from focus group data.
Recording focus group data.
The procedures for recording a focus group session are basically the same as those used for in-depth interviews. However, the focus group approach lends itself to more creative and efficient procedures. If the evaluation team does use a focus group room with a one-way mirror, a colleague can take notes and record observations. An advantage of this approach is that the extra individual is not in the view of participants and, therefore, not interfering with the group process. If a one-way mirror is not a possibility, the moderator may have a colleague present in the room to take notes and to record observations. A major advantage of these approaches is that the recorder focuses on observing and taking notes, while the moderator concentrates on asking questions, facilitating the group interaction, following up on ideas, and making smooth transitions from issue to issue. Furthermore, like observations, focus groups can be videotaped. These approaches allow for confirmation of what was seen and heard. Whatever the approach to gathering detailed data, informed consent is necessary and confidentiality should be assured.

Exhibit 5.
Which to use: Focus groups or indepth interviews?
Factors to consider Use focus groups when... Use in-depth interview when...
Group interaction interaction of respondents may stimulate a richer response or new and valuable thought. group interaction is likely to be limited or nonproductive.
Group/peer pressure group/peer pressure will be valuable in challenging the thinking of respondents and illuminating conflicting opinions. group/peer pressure would inhibit responses and cloud the meaning of results. Color Color Color Color
Sensitivity of subject matter subject matter is not so sensitive that respondents will temper responses or withhold information. subject matter is so sensitive that respondents would be unwilling to talk openly in a group.
Depth of individual responses the topic is such that most respondents can say all that is relevant or all that they know in less than 10 minutes. the topic is such that a greater depth of response per individual is desirable, as with complex subject matter and very knowledgeable respondents.
Data collector fatigue it is desirable to have one individual conduct the data collection; a few groups will not create fatigue or boredom for one person. it is possible to use numerous individuals on the project; one interviewer would become fatigued or bored conducting all interviews.
Extent of issues to be covered the volume of issues to cover is not extensive. a greater volume of issues must be covered.
Continuity of information a single subject area is being examined in depth and strings of behaviors are less relevant. it is necessary to understand how attitudes and behaviors link together on an individual basis.
Experimentation with interview guide enough is known to establish a meaningful topic guide. it may be necessary to develop the interview guide by altering it after each of the initial interviews.
Observation by stakeholders it is desirable for stakeholders to hear what participants have to say. stakeholders do not need to hear firsthand the opinions of participants.
Logistics geographically an acceptable number of target respondents can be assembled in one location. respondents are dispersed or not easily assembled for other reasons.
Cost and training quick turnaround is critical, and funds are limited. quick turnaround is not critical, and budget will permit higher cost.
Availability of qualified staff focus group facilitators need to be able to control and manage groups interviewers need to be supportive and skilled listeners.
Having highlighted the similarities between interviews and focus groups, it is important to also point out one critical difference. In focus groups, group dynamics are especially important. The notes, and resultant report, should include comments on group interaction and dynamics as they inform the questions under study.
Other Qualitative Methods
This section outlines less common but, nonetheless, potentially useful qualitative methods for project evaluation. These methods include document studies, key informants, alternative (authentic) assessment, and case studies.
Document Studies
Existing records often provide insights into a setting and/or group of people that cannot be observed or noted in another way. This information can be found in document form. Lincoln and Guba (1985) defined a document as "any written or recorded material" not prepared for the purposes of the evaluation or at the request of the inquirer. Documents can be divided into two major categories: public records, and personal documents (Guba and Lincoln, 1981).
Public records are materials created and kept for the purpose of "attesting to an event or providing an accounting" (Lincoln and Guba, 1985). Public records can be collected from outside (external) or within (internal) the setting in which the evaluation is taking place. Examples of external records are census and vital statistics reports, county office records, newspaper archives, and local business records that can assist an evaluator in gathering information about the larger community and relevant trends. Such materials can be helpful in better understanding the project participants and making comparisons between groups/communities.
For the evaluation of educational innovations, internal records include documents such as student transcripts and records, historical accounts, institutional mission statements, annual reports, budgets, grade and standardized test reports, minutes of meetings, internal memoranda, policy manuals, institutional histories, college/university catalogs, faculty and student handbooks, official correspondence, demographic material, mass media reports and presentations, and descriptions of program development and evaluation. They are particularly useful in describing institutional characteristics, such as backgrounds and academic performance of students, and in identifying institutional strengths and weaknesses. They can help the evaluator understand the institution’s resources, values, processes, priorities, and concerns. Furthermore, they provide a record or history not subject to recall bias.
Personal documents are first-person accounts of events and experiences. These "documents of life" include diaries, portfolios, photographs, artwork, schedules, scrapbooks, poetry, letters to the paper, etc. Personal documents can help the evaluator understand how the participant sees the world and what she or he wants to communicate to an audience. And unlike other sources of qualitative data, collecting data from documents is relatively invisible to, and requires minimal cooperation from, persons within the setting being studied (Fetterman, 1989).
The usefulness of existing sources varies depending on whether they are accessible and accurate. In the hypothetical project, documents can provide the evaluator with useful information about the culture of the institution and participants involved in the project, which in turn can assist in the development of evaluation questions. Information from documents also can be used to generate interview questions or to identify events to be observed. Furthermore, existing records can be useful for making comparisons (e.g., comparing project participants to project applicants, project proposal to implementation records, or documentation of institutional policies and program descriptions prior to and following implementation of project interventions and activities).
The advantages and disadvantages of document studies are outlined in Exhibit 6.
Key Informant
A key informant is a person (or group of persons) who has unique skills or professional background related to the issue/intervention being evaluated, is knowledgeable about the project participants, or has access to other information of interest to the evaluator. A key informant can also be someone who has a way of communicating that represents or captures the essence of what the participants say and do. Key informants can help the evaluation team better understand the issue being evaluated, as well as the project participants, their backgrounds, behaviors, and attitudes, and any language or ethnic considerations. They can offer expertise beyond the evaluation team. They are also very useful for assisting with the evaluation of curricula and other educational materials. Key informants can be surveyed or interviewed individually or through focus groups.
In the hypothetical project, key informants (i.e., expert faculty on main campus, deans, and department chairs) can assist with (1) developing evaluation questions, and (2) answering formative and summative evaluation questions.
The use of advisory committees is another way of gathering information from key informants. Advisory groups are called together for a variety of purposes:
• To represent the ideas and attitudes of a community, group, or organization;
• To promote legitimacy for project;
• To advise and recommend; or
• To carry out a specific task.
Members of such a group may be specifically selected or invited to participate because of their unique skills or professional background; they may volunteer; they may be nominated or elected; or they may come together through a combination of these processes.
The advantages and disadvantages of using key informants are outlined in Exhibit 7.

Exhibit 6.
Advantages and disadvantages of document studies
Advantages
• Available locally
• Inexpensive
• Grounded in setting and language in which they occur
• Useful for determining value, interest, positions, political climate, public attitudes, historical trends or sequences
• Provide opportunity for study of trends over time
• Unobtrusive
Disadvantages
• May be incomplete
• May be inaccurate; questionable authenticity
• Locating suitable documents may pose challenges
• Analysis may be time consuming
• Access may be difficult



Exhibit 7.
Advantages and disadvantages of using key informants
Advantages
• Information concerning causes, reasons, and/or best approaches from an "insider" point of view
• Advice/feedback increases credibility of study
• Pipeline to pivotal groups
• May have side benefit to solidify relationships between evaluators, client

References
http://www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/CHAP_3.HTM
http://www.fhsu.edu/~lnsmith/FallSemester/Soc362-MethodsOfResearch/lecture_notes/L-WhatIsQualitativeResearch.htm
http://www.wilderdom.com/research/QualitativeVersusQuantitativeResearch.html
http://www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/CHAP_3.HTM
Bernard, H. R. (1988) Research Methods in Cultural Anthropology. SAGE Publication, Inc. California.
Patton, M. Q. (1987) How to Use Qualitative Methods in Evaluation. SAGE Publication, Inc, Beverly Hills.






Qualitative versus Quantitative Research:
Key Points in a Classic Debate
Features of Qualitative & Quantitative Research
Qualitative Quantitative
"All research ultimately has
a qualitative grounding"
- Donald Campbell "There's no such thing as qualitative data.
Everything is either 1 or 0"
- Fred Kerlinger
The aim of qualitative analysis is a complete, detailed description. In quantitative research we classify features, count them, and construct statistical models in an attempt to explain what is observed.
Recommended during earlier phases of research projects. Recommended during latter phases of research projects.
Researcher may only know roughly in advance what he/she is looking for. Researcher knows clearly in advance what he/she is looking for.
The design emerges as the study unfolds. All aspects of the study are carefully designed before data is collected.
Researcher is the data gathering instrument. Researcher uses tools, such as questionnaires or equipment to collect numerical data.
Data is in the form of words, pictures or objects. Data is in the form of numbers and statistics.
Qualitative data is more 'rich', time consuming, and less able to be generalized. Quantitative data is more efficient, able to test hypotheses, but may miss contextual detail.
Researcher tends to become subjectively immersed in the subject matter. Researcher tends to remain objectively separated from the subject matter.
(quotes are from Miles & Huberman (1994, p. 40). Qualitative Data Analysis)
Main Points
• Qualitative research involves analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact).
• Quantitative research involves analysis of numerical data.
• The strengths and weaknesses of qualitative and quantitative research are a perennial, hot debate, especially in the social sciences. The issues invoke classic 'paradigm war'.
• The personality / thinking style of the researcher and/or the culture of the organization is under-recognized as a key factor in preferred choice of methods.
• Overly focusing on the debate of "qualitative versus quantitative" frames the methods in opposition. It is important to focus also on how the techniques can be integrated, such as in mixed methods research. More good can come of social science researchers developing skills in both realms than debating which method is superior.


FOCUS GROUP DISCUSSI0N
Mutiara Budi Azhar
Fakultas Kedokteran Universitas Sriwijaya Palembang
Mei 2002


PENDAHULUAN

• Focus group discussion (FGD) sering juga disebut group in-depth interview atau focus group interview.
• Teknik ini mula-mula digunakan dalam pemasaran produk industri untuk memahami kecenderungan perilaku konsumen.
• Kemudian teknik ini digunakan juga oleh psikiater dan psikolog dalam group therapy, dengan asumsi individu-individu yang mempunyai masalah yang sama akan lebih mudah saling mengutarakan pendapatnya dan saling berbagi rasa.
• Dalam teknik yang termasuk salah satu teknik pengumpulan data dalam riset kualitatif ini, biasanya digunakan wawancara semiterstruktur (membutuhkan interview guide); setiap anggota kelompok didorong untuk mengungkapkan perasaan, sikap, dan persepsinya mengenai topik yang dibahas.
• Dengan teknik ini dapat diperoleh informasi dalam jumlah besar dalam waktu yang relatif singkat.
• Diskusi dipandu oleh seorang moderator; moderator harus mampu menjalin hubungan yang akrab dengan responden sehingga responden dapat secara jujur dan terbuka mengungkapkan sikap, emosi dan perasaannya berdasarkan pengalaman subjektifnya.

PEMILIHAN RESPONDEN DAN KELOMPOK

• Partisipan sebaiknya homogen dalam hal latar belakang status sosial dan intelektualitasnya. Bila tidak, proses diskusi antar-partisipan tidak akan produktif karena tiap anggota bisa saja mengemukakan sesuatu yang tidak dimengerti atau asing bagi peserta lainnya. Kehomogenan dalam hal jenis kelamin dan umur juga perlu diperhatikan.
• Partisipan harus punya pengalaman tentang topik yang dibahas; bila yang dibahas adalah masalah pelayanan rawat inap RSMH Palembang, maka partisipan sebaiknya pernah dirawat di rumah sakit itu.
• Individu yang sudah pernah berpartisipasi dalam FGD untuk topik yang sama (dalam 6-12 bulan terakhir), sebaiknya jangan dipilih menjadi partisipan.




JUMLAH PESERTA DAN JUMLAH SESI

• Jumlah peserta diskusi yang dianggap ideal adalah 6-12; bila peserta kurang dari 6, biasanya ada hambatan dari peserta untuk mengekspresikan diri; bila peserta lebih dari 12, biasanya partisipasi individual menjadi terbatas.
• Rekrutmen sebaiknya 20% diatas jumlah partisipan yang diinginkan karena pada waktu yang telah dijanjikan bisa saja ada calon peserta yang tidak hadir; Bila diinginkan untuk melibatkan 8 orang, dianjurkan untuk merekrut 10 orang.
• Jumlah sesi FGD sangat tergantung kepada segmen populasi yang diinginkan serta biaya dan waktu yang tersedia.


MODERATOR

• Seorang moderator harus mempunyai keterampilan melakukan wawancara dan mempunyai pengetahuan dan dasar teori yang cukup tentang topik yang dibahas; dengan bekal tersebut, ia dapat menjaring informasi yang inginkan sebanyak dan seteliti mungkin dengan bias yang minimal..
• Ia harus mampu memahami dan mengelola perilaku kelompok sehingga diskusi menjadi hidup dan dinamis, dalam arti terjadi interaksi interpersonal dari anggota kelompok. Jadi moderator bukan melakukan wawancara mendalam secara individual kepada peserta diskusi.
• Ia harus ramah serta mampu menciptakan keakraban dan iklim yang rileks, memiliki empati yang tinggi, cukup fleksibel, namun cukup tegas dalam memandu diskusi.
• Mampu dengan elegan menghindari dominasi peserta tertentu dalam diskusi.
• Tahu kapan melakukan probing dan kapan menutup mulut, kapan menggunakan bahasa verbal dan kapan menggunakan bahasa nonverbal.
• Sebaiknya moderator berjenis kelamin sama dengan peserta diskusi walaupun dalam topik tertentu, jenis kelamin moderator yang berbeda dengan jenis kelamin partisipan dapat juga lebih menguntungkan.
• Mengingat besarnya peran moderator, sebaiknya yang akan menjadi moderator harus dididik dan dilatih secara khusus, plus pengalaman trial and error.
• Alat bantu seperti tape recorder diperlukan, meskipun mencatat adalah keharusan.
LINGKUNGAN FISIK

• Sedapat mungkin jangan di rumah, kantor ataupun tempat-tempat yang membuat peserta segan untuk berbicara bebas.
• Lebih baik peserta dan moderator sama-sama duduk di kursi yang disusun melingkar satu lapis; peserta jangan duduk mengelilingi meja seperti dalam suatu konperensi.
• Bila peneliti meminta orang lain menjadi moderator, idealnya diskusi dilakukan di dalam ruang dengan fasilitas one way mirror. Peneliti mengamati dari sisi lain tanpa peserta merasa diobservasi.
• Bila diskusi cukup panjang, baiknya diselingi dengan istirahat; umumnya diskusi diselesaikan dalam 60-90 menit.

KEUNTUNGAN TEKNIK FGD

• Memungkinkan terjadinya sinergisme: dari diskusi dapat muncul informasi, ide, dan pandangan yang lebih luas.
• Memungkinkan terjadinya snowballing: komentar satu peserta dapat memacu reaksi berantai dari peserta lainnya.
• Memungkinkan terjadinya stimulation: pengalaman dalam kelompok merupakan suatu hal yang menyenangkan sehingga mendorong partisipan untuk lebih aktif.
• Memungkinkan munculnya rasa aman (security): di dalam kelompok, individu merasa lebih berani dan bebas berbicara dibanding sendirian.
• Memungkinkan munculnya spontanitas: jawaban tidak lagi sepenuhnya jawaban individu, namun merupakan hasil proses diskusi.