ATLAS.ti networks as a mechanism for establishing trustworthiness

On Tuesday morning I attended a special topics webinar on ATLAS.ti network views conducted by Susanne Friese.

The network view tool in ATLAS.ti enables the researcher to create "interactive mind maps," something I have already written a bit about in a previous post.

Basically, the hermeneutic unit (the ATLAS project file) is the total network. All objects added to or created within ATLAS (documents, artifacts, memos, quotations, codes) instantly become nodes in the network when they are linked to something else (for instance, a code to a quotation, a memo to a code, and so on), and these links may then be visualized.

I sat in on the session in hopes of learning more about this ATLAS function, which I have come to appreciate as a scaffold for both analysis and writing (harder and harder to separate these two processes, I am learning).

As I listened to Susanne Friese and her "vondervul" German accent, I also made a few connections between network views and this week's readings in EP659, which focus on trustworthiness and issues of quality in qualitative inquiry (Anfara, Brown & Mangione, 2002; Tracy, 2010).

According to Anfara et al., "...[A] key part of qualitative research is how we account for ourselves, how we reveal that world of secrets" (p. 29). Neither Anfara and his colleagues nor Tracy specifically mention CAQDAS as a tool for ensuring quality, but both articles discuss transparency of data management, coding, and analysis as a mechanism for strengthening research quality.

Tracy, for example, provides a more than adequate rationale for use of digital tools in her discussion of "rich rigor," one of the eight major criteria in her reconceptualization of excellent qualitative research. She writes,

Rigorous data analysis may be achieved through providing the reader with an explanation about the process by which the raw data are transformed and organized into the research report. Despite the data-analysis approach, rigorous analysis is marked by transparency regarding the process of sorting, choosing, and organizing the data. (p. 841)

And Anfara et al.’s example of “code mapping” (p. 32) is quite simply a picture of one researcher’s analytic process that could easily be depicted using ATLAS.ti’s network view. Although, with charts and tables, one must be critically conscious of not merely showing a hierarchy of codes, which Friese adamantly warns against. (See "Analytic capabilities of network views" below.) The researcher builds a network view in ATLAS based on his or her interpretation of relationships across codes and categories, thus making the network view tool indispensable for conducting constant comparative analyses, building "audit trails," and "documenting the procedures used to generate categories" (Anfara et al., p. 33).

Creswell (2013) is more direct in his endorsement of CAQDAS. In the chapter on validation strategies in the newest edition of his Qualitative Inquiry and Research Design, he mentions the use of computer programs to assist in recording and analyzing data as one of several ways for enhancing the stability and dependability of findings, or, what is known in positivistic terms as "reliability."

Networks are compelling visual diagrams that add transparency to the researcher's process, and I am now considering how I might incorporate them into the write-up of my own research findings or my dissertation appendices.

Here are a few other notes I took from Tuesday’s webinar:

Basic concepts of network views

• Strong and weak links are denoted by solid (strong) and dotted (weak) lines
• Weak links between nodes are unnamed. They exist between memos and quotations, codes and quotations, memos and memos, and between families and their members.
• Named links express relationships between two codes or two quotations
• Named links may be "directed/transitive" ------> and "non-directed/symmetric" <-------> 
• Background colors in network views coordinate to code colors (if you use color)
• Hyperlinks occur on the data level, between two codes or between two quotes

How to link objects

• You can drag and drop from anywhere, bringing objects into the network view manager. You can also import nodes into the network (any object from the HU). For instance, you can select a code and import all its quotations.
• Take time to play with the display options under the Display drop-down menu
• In the relations editor, you can create your own relations with colored lines of different point sizes (See “How to create relations” below.)
• Comments can be added to relations and are denoted with a tilde just as with other comments in other areas of ATLAS
• NEVER DELETE AN OBJECT from the network view, use the "remove from network view" option instead
• Creating links between data opens up different kinds of relations, as opposed to linking codes, which is more conceptual
• You can save a network as a graphic file (png, gif, jpg) and insert into a PowerPoint or MSWord doc

How to create relations

• Open Relations Editor and expand the window until you see the Edit tab
• Decide on the relation you want and crate a unique identifier, the first three letters and then the actual memo text (e.g. REA is the identifier for "is reason for")

Working with hyperlinks

Example of "star" links

• Hyperlinks are "stars" or "chains" of links
• Quotes may be linked within and across documents

Analytic capabilities of network views

• Codes are just topics and areas of interest within the data, they describe
• Networks take it to the conceptual level, so networks are for linking across categories and depicting relationships between data, not for building code hierarchies. DO NOT USE NETWORK VIEW FUNCTION TO REPRESENT YOUR CODE STRUCTURE. IT'S TWO DIFFERENT LEVELS OF ANALYSIS.
• Use network view after coding when you start to see relationships.
• ATLAS.ti version 7 allows you to filter codes that you bring into the view. You can hit Reset Filter to bring back all codes.
• Code families are just grouping mechanisms for codes. Families are filters: create family, then turn on filter.

References

Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 31(7), 28–38.

 Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed., Kindle version.). Los Angeles: SAGE Publications, Inc.

 Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837–851.

ATLAS.ti networks as a mechanism for establishing trustworthiness

Reflecting on Watt (2007)

Watt's 2007 piece, "On Becoming a Qualitative Researcher," is a first-person narrative of a novice researcher who "put reflexivity to the test by keeping a research journal" (pp. 82-83).

Unlike positivist studies, which are assessed by familiar standards of validity and reliability, the rigor of qualitative inquiry is determined more subjectively, beginning with the researcher herself through a process of reflexivity. In the absence of precise designs and formulas to guide their practice, qualitative researchers, Watt explains, must be willing to self-scrutinize. 

Reflexivity, as defined by Glesne (2006), "involves critical reflection on how researcher, research participants, setting, and phenomenon of interest interact and influence each other" (p. 6). Reflexivity statements "provide a kind of map to the decisions you make so that the reader can better understand (and question) the interpretations you make" (p. 127).

As a student, Watt understood reflexivity on a theoretical level but wasn't sure what it looked like in practice.

This is the third time in as many years that I have read Watt's article, and with each new reading, I gain new insights. Not surprisingly, this time around, my insights were colored by the fact that I am now immersed in my own dissertation project.

What did surprise me was just how intimately connected the practice of reflexivity is to improving trustworthiness in research, a topic I will try to address at the end of this post.

A refined sense of the reflexive process
First, putting aside textbook definitions, my understanding of reflexivity has grown more refined, especially as it relates to the act of writing and developing identity.  I was really struck by the influence of a (somewhat) regular writing regime on Watt's own personal development as much as her research process. This is the kind of exploratory and personal journal writing that Laurel Richardson advocates in "Writing: A Method of Inquiry," which Watt refers to several times.

In her research journal, Watt summarized her readings from the literature, reflected on events in the field, recorded participant data, and made notes about her methodology. "Through using writing as a method of inquiry I was able to make links between how I carried out my study, reflective journal entries, and the literature on qualitative methodology" (p. 98).  In other words, she developed the deliberative stance as advocated by Piantanida and Garman (2009) and came to recognize the "centrality of writing as a way of coming to know" (Ch. 9, “Experiential Text as a Content for Theorizing,” para 12).

More importantly, through writing, the deliberative stance becomes internalized: "Reflective writing allowed me to meaningfully construct my own sense of what it means to become a qualitative researcher" (Watt, p. 83).

On becoming an expert
Writing is one way to negotiate moments of conflict and disequilibrium. Like discussion and other forms of peer mediation, reflective writing creates a feedback loop -- a conversation with oneself -- for improving practice and developing expertise.

Yet, as Watt points out, it is during moments of crisis and tension that novice researchers typically slow down or shut down their writing entirely.

For instance, when faced with the task of data analysis, Watt was at a "complete loss" (p. 95), and so she made lots of charts and diagrams. She explains, "I was so focused on the need to do something with the data that I did not consider journaling as a means to think things through, on both a personal and a research level. That was a mistake. In retrospect, this was perhaps the time I needed it most" (p. 96).

The mistake is common in my field, education. Like novice qualitative researchers, new teachers experience a profound sense of disequilibrium. Overwhelmed and isolated, they put reflection on the back burner.

But reflexivity is the antidote to repeating mistakes, and building upon a foundation of subjective experience is a hallmark of expertise. Watt cites Eisner's (1991) concept of educational "connoisseurship," in which seasoned practitioners leverage their subjectivities in productive ways. Piantanida and Garman also cite Eisner's work in their Chapter 5 discussion of deliberation as a means of attaining phronesis, "a valuing of wisdom that can guide action within the complexities of unfolding experience."

Whether it's "critical subjectivity," "connoiseurship," or "phronesis," these ideas are very useful to me because they are nuanced versions of what I would simply call "developing expertise," and teachers developing expertise about technology is the topic of my dissertation. For purposes of my study, I have found the CHAT concept known as "expansive learning" (Engestrom) to be helpful, but I like how Eisner's term goes directly at the teacher-learner, teacher-researcher experience. I may need to borrow or develop a more precise term other than "expertise" (something else to reflect upon later).

At any rate, I have noted before in this blog the parallel developmental trajectories of novice qualitative researchers and teachers-as-learners insofar as stance, disposition, and expertise are concerned. And after re-reading Watt, I have a renewed sense of how reflexivity mediates this process.

The Multiple Realities perspective
I have also referred previously in this blog to the Multiple Realities perspective, a central notion in my theoretical frameworks. I was excited to make a connection between Watt's article and Labbo and Reinking's (1999) seminal theoretical piece, "Negotiating the Multiple Realities of Technology in Literacy Research and Instruction."

Although Labbo and Reinking do not specifically use the term "reflexivity," they put forth the Multiple Realities perspective as a way for New Literacies researchers to monitor and leverage subjectivities in a way that strengthens research-to-practice connections:

For example, when a question related to instructional  practice begins with the phrase "What  does the research say about...?" we  believe it should be followed by an explicit consideration of which reality or set of realities is being considered. Doing so means that the answer will inherently be more complex than citing a string of studies and drawing conclusions from them. It also suggests that it may be important to identify the realities to which a potential answer does not apply or why a question is not a relevant or particularly good one within certain realities. (p. 488)

The Multiple Realities perspective, then, is a framework for guiding reflexive thinking within New Literacies research projects. I had never thought about it in this way until now! Maybe I need to be more intentional about referencing it in the methods section of my Chapter 3.

Implications for use of digital tools
Finally, a note about digital technologies and the role they serve in supporting reflexive practice: Watt's descriptions of her struggles to corral and make sense of the data, the codes, and the categories highlighted the value (for me) of digital tools generally, and CAQDAS in particular. This last insight will only become more compelling with the passage of time (Watt's article is now six years old) and my own increasing awareness of how to use digital tools for qualitative inquiry.

Trustworthiness
The common thread that connects each of the above ideas is how they promote trustworthiness in qualitative research. The discipline of regular, reflexive writing -- "chronicling one's thinking" -- helps the researcher mediate the meaning of her experiences and continually develop her expertise.

As Watt explains, "Revisiting my study has strengthened my confidence in my ability to negotiate the  complex process of qualitative inquiry, and I now see myself as a researcher. The multiple layers of reflection drawn upon in writing and revising this paper have made me more cognizant of how far I have come, and have taken me further along the path to becoming a qualitative researcher" (p. 98).

Reflexivity develops an internal authority of self-knowledge (identity) and self-efficacy, alternate "measures," if you will, of excellence and trustworthiness in research.  Reflexivity develops an authentically authoritative voice that delivers the research findings and interpretations in a way that resonates with readers.

References

Glesne, C. (2005). Becoming qualitative researchers: An introduction (3rd ed.). Boston: Allyn & Bacon.

 Labbo, L. D., & Reinking, D. (1999). Negotiating the multiple realities of technology in literacy research and instruction. Reading Research Quarterly, 34(4), 478–492. doi:10.1598/RRQ.34.4.5

Piantanida, M., & Garman, N. B. (2009). The qualitative dissertation: A guide for students and faculty (2nd ed., Kindle version.). Thousand Oaks, CA: Corwin.

Watt, D. (2007). On becoming a qualitative researcher: The value of reflexivity. The Qualitative Report, 12(1), 82–101.

Reflecting on Watt (2007)

New transcription routines

I have been preparing transcripts from virtual class recordings and interactive/co-generative interviews for my dissertation project.  As I have done in the past with other projects, I am creating verbatim transcripts in a standardized format to “aid the handling, comparison, and sharing of language data” (Lapadat & Lindsay, 1999, p. 70). 

My primary tool for transcription, is InqScribe software, which has many features that I have come to rely on, especially the control for playback speed and the embedded timecodes that enable me to jump to specific locations in my audio files. However, one aspect I do not like about InqScribe is its output. I must copy and paste the raw transcript into a Word document, where I bold, italicize, and perform other formatting functions as needed.

My journey down the transcription road (more like grueling death march) happened to coincide with my recent venture into the case study literature.  For this reason, I closely attended to what the case study methodologists had to say about interviewing and transcribing, and I wondered if my readings would influence my process of data collection and handling.  In fact, they have, and I have added a few "new routines" to my process. 

Gisting

Somewhere along the way, I heard about this practice of capturing the "gist" of researcher-participant exchanges  -- maybe in ethnography? I can't remember. 

Nevertheless, to gist an interview or conversation, the researcher writes down everything that he or she can recall as soon as possible following the encounter. 

Gisting is essentially a good habit of all fieldwork. We were taught in intro to qualitative research to immediately review notes and reflect in writing after the fact. Sadly, it is not a habit I ever developed. Typically, when I am in the field, I am running hither and yon, usually because I have overscheduled the day and am late to the next engagement. As qualitative researchers, this may be our most debilitating mistake in the interview process. I realize this now.

In the short passage on interviewing techniques in The Art of Case Study Research, Stake says that the single most important thing the case researcher can do is "insist on ample time and space immediately following the interview to prepare the facsimile and interpretive commentary" (p. 66). In fact, Stake argues persuasively against any transcription, saying the "facsimile" is all anyone really wants to see. Participants are likely to be put off by the length of a typical transcript and "the inelegance of their own sentences" (p. 66). 

Briefly, Stake's interview tips are:

1. rather than tape record or write furious notes, listen and make a few notes
2. ask for clarification
3. after the interview (within a few hours) reconstruct the account in your own words
4. submit the facsimile to the participant for accuracy and stylistic improvement

I think I have the first two steps down pat, but I am not prepared to give up recording. I am a fairly decent listener and always take a few notes, and my notes work like markers for later navigation through the recording. On the downside of that, I have never mastered the art of discreetly noting the passage of time or approximate minutes transpired during an interview. (It seems rude to glance at my cell phone or clock during the interview. I don't want to wrongly cue the speaker that we are out of time.)

Insofar as reconstructing the account immediately after the interview (Step 3), this is something I would like to do better. What I have been doing based on Stake's advice is to deliberately stop after every 10 or so minutes of transcription and write and reflect on key ideas, impressions, and insights that accumulated in my brain. This has become my version of gisting the interview, and I'm very pleased with the results! When I finally finish transcribing -- and, unfortunately, this does drag out the transcription process even more so than ever -- I have a nice narrative summary of the interview with my interpretive commentary woven throughout. I insert timestamps as critical reference markers, should I or a participant want to go into the original transcript to read the exact words. 

Maintaining trustworthiness

I am starting to include the gist of the interview with the verbatim transcript when I submit it to the interviewee as part of my new member checking routine. So far, I have received one reaction from an interviewee, and it was wholly positive.  Stake says to expect this. Member checking is a necessary step, even when participants don't respond. If the researcher is anxious his or her facts or interpretations are off-base, it is up to the researcher to probe more deeply.

Member checks are one of several strategies identified by Yamagata-Lynch (2010) for maintaining trustworthiness in activity theoretical studies (the method of my dissertation). Other strategies include prolonged engagement, persistent observation, and triangulation. I feel relatively confident about upholding these standards during my study, but I have never purposefully applied member checks before. This is new territory for me.

While Yin does not use the term "member check," he says a major procedure in doing a case study is inviting participants and informants to view drafts of the report. Yin fixates on "corroboration" and "validity" as the main reason for member checks. He says participants may disagree with interpretation but should validate facts of the case. The case study is not complete until the researcher resolves these disagreements. 

I would argue the purpose is to check interpretations for "coherence," per Piantanida and Garman (2009). Yin allows for this possibility, too. In cases where "validity" or "objective truth" may not be the point, the process of member checking is helpful for identifying different participant perspectives, which will be portrayed in the final case study report. And, Yin says, the researcher ultimately has the "discretionary option" and does not have to accommodate participants' reinterpretations (Yin, Ch. 6, "Reviewing the Draft Case Study: A Validating Procedure," para 3).

As with gisting, member checking adds time. Also, Yin advises, participant-reviewers may use the opportunity to open a fresh line of dialogue about the study. The researcher must plan ahead and budget time accordingly.

I have only started to do member checks, but to expedite the process, I am finding the Evernote notetaking app to be especially useful.

It was not my plan initially to use Evernote so deliberately in my data collection process. In January, during my very first interview with a teacher-learner, my digital audio recorder was making me nervous, indicating it was low on both memory and battery life. I didn't have a back-up recording device -- or so I thought! I remembered I could use the built--in recording feature of Evernote. I opened my iPad, opened Evernote, created a note using the participant's pseudonym as a title, and clicked on the Record Audio button in the top menu. With tablet devices becoming almost ever-present in daily life, it was easy to place the iPad unobtrusively on the table between me and the interviewee while we talked. It may even have been less distracting than the microrecorder.

Later, when I prepared to send the transcript with gisting to the participant, it occurred to me that I could use the Share tool in Evernote and the original audio file would embed in an email. So, I copied and pasted the gist and the transcript from Word into the Evernote note with audio and sent it. 

I am hopeful that the member checking routine will enrich my study. According to Yin, "Often, the opportunity to review the draft also produces further evidence, as the informants and participants may remember new materials that they had forgotten during the initial data collection period" (Yin, Ch. 6, "Reviewing the Draft Case Study: A Validating Procedure," para 2).

In the event this does not happen (and it may not), I still wonder if this could not serve as the basic premise of eventual follow-up interviews, asking participants to react to the short gist of their first interview (as opposed to, "What did you think of the transcript?")? 

Group update

Last week our group workshopped Brian's Chapter 3. I learned a lot from reading Brian's work and got some ideas about interviewing as well as a lead on another possible resource, ‪Learning From Strangers: The Art and Method of Qualitative Interview Studies by Weiss. The group also selected a reading for the week of  Feb. 24, Edwards (1998). We plan to discuss the reading and workshop some of Nalani's early coding efforts from a pilot study she conducted.

References

Lapadat, J. C., & Lindsay, A. C. (1999). Transcription in research and practice: From standardization of technique to interpretive positionings. Qualitative Inquiry, 5(1), 64 –86. doi:10.1177/107780049900500104

Piantanida, M., & Garman, N. B. (2009). The qualitative dissertation: A guide for students and faculty (2nd ed., Kindle version.). Thousand Oaks, CA: Corwin.

Stake, R. E. (1995). The art of case study research. Thousand Oaks, CA: SAGE Publications.

Yamagata-Lynch, L. C. (2010). Activity systems analysis methods: Understanding complex learning environments. New York: Springer.

Yin, R. K. (2008). Case study research: Design and methods (4th ed., Kindle version.). Los Angeles: SAGE.

New transcription routines