welcome everyone to the third of four webinars on interprofessional
collaborative practice program evaluation all of which are based on the
National centers practical guide number five incorporating IPCP teamwork
assessment into program evaluation welcome back to all of you who joined us
for our first two webinars and we are equally delighted to welcome those of
you who are dialing in for the first time so as many of you already know the
National Center is a jointly accredited provider of interprofessional continuing
education credits for the healthcare team this statement describes the nature
of this presentation and the one hour of continuing education credit available to
those who selected that option at the point of registration if you have
questions about continuing education please let us know in the chat box and
we will answer them at the close of this webinar this slide states that our
presenter Barret Michael Locke does not have a conflict of interest in this
presentation my name is Connie Schmitz and as many of
you already know I am a consultant working with the National Center and the
lead author of practical guide 5 it's my really great pleasure to introduce you
today to a wonderful co-author of this guide and today's presenter Michael
Barrett McCulloch Barrett is an associate professor in the Department of
Sociology and the associate dean of interprofessional education in the
College of Health Sciences at the University of Delaware in this role he
has spent the better part of last year designing the interprofessional and
interdisciplinary curriculum thread for health pre professionals at the
University he also serves as the director of the center for
interprofessional development education and research additionally he holds an
adjunct research professors position with the Sydney Medical College at
Thomas Jefferson University his research primarily focuses on the socialization
and professionalization processes within the Health Professions
he is passionate about qualitative methods for ite CP evaluation and we are
delighted to have him built on our previous webinars by discussing the when
why and how of applying these methods to the evaluation design data collection
and analysis I know that I will personally learn from something from him
today and I hope you do as well and with that Barrett I will pass the ball to you
all right excellent Thank You Connie and also thank you for
it to Angela and Stephanie from the National Center for all their time and
effort and support with this I also want to thank the National Center for its
support with putting together a guide number five and creating these
opportunities for us to dive a little deeper into these key aspects so welcome
to everyone and Connie is right I am very passionate about qualitative
methods and I think that's going to come out in a lot of these slides and some of
the things I talked about hopefully I don't move too quickly but we are in a
time limit and there's a lot of things to talk about so once again this series
of webinars is an opportunity for us to dive a little bit deeper in the key
aspects of the guide number five unfortunately we didn't have enough
space to really go too far in depth about qualitative methods or any
specific real you know methodological input in approaches so this provides
that opportunity here here's a breakdown of the contents of guide number five and
this is probably a slide you've seen before we're going to go down and talk
about parts mostly two and three today so hopefully we'll get through that
here's the learning objectives for the day of course given our time limit we'll
simply cover some of the basics and fundamentals but I do want all of you to
consider me a kind of a personal resource I know that may seem odd but I
know there aren't a ton of people out there doing qualitative methods and
qualitative data analysis within IPE and I PCP evaluation and assessment so
please consider me a resource from here on out there's a Q&A session at the end
and there's also the upcoming short course at the next summit and I'll be
there as well but also always available via email or dance
or any questions about qualitative methods in general or personal project
reviews that type of stuff so I teach qualitative methods at the undergraduate
but also the graduate level and at the professional level I give a number of
seminars to clinicians and providers but at the graduate level we cover classes
that are you know the basic level and then another whole semester course on
advanced qualitative methods so trying to fit this in with an hour is going to
be a bit of a heavy lift but I think we're going to be able to get through
some some key points here's the agenda for today we're going to talk about some
evaluation plan components review some basic qualitative methods that are
commonly used in IPE and IPC P evaluation and assessment I'm going to
provide two specific examples and provide the reasoning behind utilizing
qualitative methods for those and then we're going to discuss the real
fundamentals of qualitative data analysis that's going to be a pretty
unique section we're also going to do a little bit of a deeper dive into
grounded theory which I know some of you may be interested in a spoiler alert not
everything is grounded theory so we'll be talking about that as well you might
have seen this slide before if you participate in the other webinars these
are the components for the evaluation plan when thinking about qualitative
methods specifically we have to take many aspects of the context and the
design into consideration before we even begin because these factors will impact
not only if we utilize qualitative methods to evaluate and assess but which
methods specifically how will utilize them who will lead in the evaluation and
assessment what type of data we'll get and how we'll be able to analyze that
data like with any evaluation and assessment protocol or project your
Guiding evaluation questions need to be refined and specific from the get-go but
also you have to be fairly flexible in order to move around and I think this
the 15 questions and decisions that you probably have seen other side you
probably seen before you know it really helps to point that out
each of these questions are important when considering quality to methods for
your evaluation assessment and they should be mapped out beforehand but also
you have to remain flexible one of the beauties of qualitative methods is that
you're going open up a lot of cans of worms and
that's important because when you're doing that when you're scraping below
the surface and you're really digging in you're exploring things that you may not
have known was going on or that was happening or perspectives of attitudes
that you weren't aware of and in doing so that will impact your evaluation
guiding questions but also questions related to context design and logistics
so you have to be flexible with qualitative methods I think this is
probably one of the best resources you can utilize and again you're going to
mark it up and you're going to be flexible with it but the evaluation
planning tool is an extremely useful mechanism to utilize in order did I set
your potential qualitative approach the tool allows the team to examine what it
is you know quote on is what it is they're actually interested in exploring
where that is is or where they think it is and the best way to extract that is
that data so all of these things are very important and again sometimes
usually utilizing qualitative methods you don't really know what you're
looking for until you get in there there we go so some of you may have seen this
cartoon before it's always utilized when people are talking about qualitative
methods when why do we utilize qualitative methods well when were you
when they're utilizing with IP and IP CP evaluation/assessment
it's because we often know that participants whether those participants
be students providers patients patients families faculty they all have a story
right so when we're evaluating programs and teamwork we all know that people
have a story they have a perspective they have experiences they have
attitudes their beliefs views understandings and a lot of times scales
just can't get at that stuff no matter how good the scale is how about it is
how reliable it is how often it's been utilized it really doesn't get at
everything there are elements of programs that are essential to an
effective evaluation assessment program that must be observed or experienced to
be understood and we really need that on the ground in the trenches perspective
and oftentimes qualitative methods are the best approach to getting at those
those line item questions even like in scales you know can't really detect
qualitative methods help us dig deeper too
get underneath the surface to explore the broader context one of the things
that happens a lot with evaluation/assessment is no matter how
good our protocol is we tend to forget that these programs plan these
initiatives and the intervention don't exist in the vacuum there's a larger
context and culture that surrounds them and we also need to understand that and
how they may be impacting what we're trying to study and what we're trying to
evaluate we can help us identify and examine issues that we hadn't thought
about it helps us foster connections and honestly it qualitative methods provide
voice really a lot of times that we kind of just forget that participants are
more than data points and so we need to understand the voices in the perspective
of those engaged in the program and in the initiatives in interventions that
we're trying to explore and evaluate there's a humanist and a humanism to
qualitative method that because we are you know face to face in with
qualitative methods often with interviews or observations or focus
groups but it can make us feel a little bit uncomfortable and messy sometimes
people think that qualitative method is a little too touchy-feely and that may
lend to a lack of validity which is not true at all but neglecting that
humanists in the humanism of it especially with evaluation that can be
kind of cut and dry sometimes really doesn't allow us to get that good data
that we need qualitative is qualitative approaches allow us to see how the
pieces fit together and also help us tackle big ambiguous topics like success
an impact because honestly what may be success to you may not be success to
whoever is engaged in the program and so we need to understand what that is what
are we actually trying to find out if a program we're evaluating a program if
it's working or not what is working mean for all the participants so here are the
fundamentals of certain people are saying to move a little closer to
microphone I will do that I apologize there we go hopefully that's better the
fundamentals of certain qualitative methods and so I've highlighted the ones
that are utilized most frequently in evaluation assessment those are
interviews focus groups and observations interviews are you know the respondent
is asked a series of questions where they
to the project or the protocol there are differences between structured
semi-structured and unstructured structured interviews you have an
interview guide which is your list of questions right and you follow those to
the letter and you ask all of your participants the exact same questions a
semi-structured you still have your interview guide but you allow
participants and respondents to go off a little bit and you allow to explore what
they may be saying you may add some questions here or there you may add some
refine questions all right that may be going on unstructured you're going in
there you're just gonna let yours gonna bring up a topic and you're gonna have a
discussion right in that sense you can also conduct interviews over the phone
you can do them in person in a new way that people have been
utilizing more and more often and maybe some of you are used to this is online
like through Skype or FaceTime or things like that
I prefer in-person because I like to read body language and facial effect
which i think is important also you can judge pauses a little bit better when
you're doing face to face or even you know online focus groups are often very
popular those are where groups of individuals are led to read three
questions and scenarios by a researcher or by the PI of the evaluation protocol
they can also be structured semi-structured or unstructured you have
to be very careful about who you're involving in your focus groups in terms
of power and authority dynamics especially during evaluation and
assessment so that's something to really plan out beforehand so you wouldn't want
to have a student and perhaps an attending
you know in the same same like same same group also in terms of different
professions that can be a challenge as well due to nested hierarchy amongst the
health occupations so those are things to think about observations the
researcher observes individuals or groups or just groups of individuals in
their natural kind of setting whatever you're trying to observe whether it be
an intervention that's on the ground or interactions could also be observing a
setting and just seeing what the happenings are there it could be a full
participant in terms of participant observation you could be engaged in the
actual program as a faculty member or as a learner and be doing the observations
but could also be completely removed from
the program and observing from an outsider perspective being a complete
observer content analysis there at the bottom is also pretty commonly used
that's where you're analyzing text instead of gathering information from
people and by text I mean something very broadly you could be using emails or
books or pamphlets that describe particular courses but also music lyrics
of videos type of pictures those types of things all of that is considered
content journals are fairly popular lately in terms of having students keep
personal journals or having patients keep personal journals during
experiences in terms of evaluation it's a little tough to analyze it takes a
little practice with those Auto ethnography is extremely challenging and
I really only recommend that for those who are very well versed in kind of
ethnography in general but then also qualitative methods overall because that
can involve interviews observations focus groups as well as you're being a
complete participant observer so that's a little bit more challenging the key
with all of these methods however is something that see right Mills used to
say which was sociology allows you to make the familiar strange right and so
these methods allow you to do that right when you're digging beneath the surface
you're actually questing questioning the familiar questioning the normative
things that we take for granted you're going to explore those types of things
and that's what qualitative methods allow us to do what you need to ask
yourself before embarking on this though is when is it a good time to utilize
interviews observations and focus groups and that's where that evaluation plan
comes in really well so you knowing the fundamentals of those methods you can
say oh this might be an attempt there's a good opportunity to utilize these
methods and why those over surveys right if surveys are quick and dirty they're
usually cheap with and you can get a whole smattering of a sample with
qualitative methods your sample is going to be pretty small usually because it's
time-consuming its effort consuming and it's also cost it cost a ton to do these
things sometimes and I mean that honestly in terms of transcription
services and whether or not you compensate subjects organizing time for
people when they have time for work and you have to compensate that maybe even
your own time as a researcher or perhaps even a
clinician or a faculty member you had to buy out your own time to engage in these
kind of heavy methods and that's the truth so they can be pretty expensive so
a lot of people don't do them for those reasons but also not feeling very
comfortable with them qualitative methods require a vulnerability and an
intimacy that can feel awkward if you're not used to them because you have to be
open to the data and open to your participants I've also put down at the
bottom before I change the slide the tools that we often utilize and one of
the things I want to make sure that I emphasize is that you yourself when
you're doing qualitative methods are a research tool you are a filter you know
whether you are they're asking questions in terms of a focus groups or interview
or conducting observations yourself but also you yourself are a filter in terms
of the data you are getting not just through interpretation but from the
other people or persons that are interacting with you whether they're
answering questions you and all your trappings are a filter in terms of what
happened and that is important to understand sampling is also a big issue
we're doing with qualitative methods you know who do you sample why is it
important how do you get to them feasibility issue again so these things
are important to think about and that evaluation plan that grid that I showed
earlier that's really important to utilize this is probably my favorite
slide of all time I love this slide so I've never actually put this on a slide
but I asked this question all the time I get asked to do a lot of projects with
provide with clinicians healthcare professionals and those that are
involved in faculty at medical schools nursing schools and they say you know
they bring me on to do the qualitative aspects of their evaluation and
assessment and it's great because the biggest question is are you experienced
have you had an opportunity to be formally trained in qualitative methods
and I mean that have you taken a series of classes in qualitative data gathering
and in qualitative data analysis and if you haven't it's a really good idea to
utilize your resources utilize those that have it's not you wouldn't ask
someone without any training and quantitative methods or statistical
analysis to run a factor analysis for you it's like you wouldn't ask someone
to go run marathon that hadn't been you know
running distance for a little while at least it's just not gonna work and it
might get hurt right and you might really get terrible data so my advice to
everyone is really do a self check before you embark on this and say do I
have the training to be capable of doing this type of of these methods in this
type of valuation assessment and if not try to find someone who does whether you
assess your kind of social science department that might be around campus
or your education departments or actually go out and get the training
yourself it's a good question to ask one more thing before we kind of dig just a
little bit deeper I put this right in the start always show your work with
qualitative methods keep meticulous notes in every step that you take and I
mean that from the moment you start to utilize qualitative methods you say I'm
going to do interviews okay well why all right so you start mapping that out you
map out your sampling frame you map out how you're going to do that that helps
with an IRB protocol but then also this will help when you start writing it up
for any kind of publication or white paper or whatever your evaluation and
assessment protocol is going to turn into right you've got to keep notes on
and all decisions and all steps from the very start
you want to write meticulous and specifically detailed notes on all of it
it's very important especially related to data analysis and interpretation all
right so I'm going to start with some examples from my own work and I picked
these examples because these in my opinion are actually really reflective
of the fundamentals of why we utilize qualitative methods so I've provided a
site at the bottom there and I'm just going to run through this relatively
quickly and provide some background so this particular project was an
evaluation of on a specific program at Thomas Jefferson the health mentors
program and we were interested in kind of exploring specific things that we
didn't think were present in previous attempts to evaluate similar programs so
students perceptions of IPE programs in their attitudes of abilities of other
health professions and in of IP in general the findings were mixed in the
literature right some programs we're having a positive impact somewhere not
and most of them were utilizing scales and we thought that kind of lacked the
in-depth understanding of really what we were trying to explore
of the qualitative studies that looked at those issues there was limited
sampling and by that I mean they were sampling post enrollment only or the
impact of one single event or one single program or a non required program they
gathered the data at only one time so there was a lack or at least a fairly
good lack of longitudinal evidence and there was also a lack of exploring that
broader picture again this perception that programs just exist in a vacuum
well what is it about the program but also what is it around the program right
that was happening within the institution within the students lives
that could be impacting these attitudes and perceptions right if we don't
explore that we might be missing key factors so the health mentors program
for those of you who not aware they have done the revamping recently of this
program so this was when we did the project this is what the program looked
like it's a two year longitudinal IPE program required for first the first two
years of training for med nursing pharmacy ot PT and couple and Family
Therapy there are student groups and they're mixed by profession they're led
by a patient from the community with one or more chronic conditions and some of
your institutions they have a similar program like this the students work the
student groups work with a community member help navigate health care system
life with the illness various facets of day-to-day they learn more about each
other in informal informal ways and the program was doing quite well according
to their previous research the previous evaluation and assessment had shown that
was mostly survey based it had shown there was a positive impact from this
program on attitudes and perceptions but we they were interested in exploring a
little bit of why that was so now what was it about the perceptions and
attitudes that the students had what was the bigger picture that was going on the
larger context that could be impacting how Wow why were they turning positive
and what was the students experience with the program but also with into
professionalism as it existed at Thomas Jefferson at that point I like to
utilize the iceberg metaphor in terms of understanding why we utilize qualitative
methods surveys can really help you understand if attitudes or perceptions
change the if write that can give you oh yeah this scale tells us it did change
but quality methods help us go below the surface
right we can see what are the factors in and around the program that are
impacting these possible changes and how do they work and that was really the key
thing right that's where the why and the how our so for our evaluation assessment
protocol and why qualitative methods were essential we wanted to get at that
how and why and and the what actually as well so to address of our objectives and
what we felt were gaps in the previous literature we went a little bit more in
depth with our evaluation assessment we utilize the I apologize for all the
words on this slide I know it's a big no-no with PowerPoint but it allows me
to get to my point we utilize the following sampling methodological
approach so we had 20 students randomly selected from six different health
disciplines one of the questions you could ask yourself and if we have time
later on I can answer them is why why only 20 students right there is far more
students that were involved in this program why only 20 students and how was
the sampling from each of the health disciplines decided now I've given you
the citation for that particular article that describes that but I can go more in
depth later on we did in depth semi-structured interviews at the ends
of year 1 of the program and year two of the program 39 of the 40 interviews were
conducted over the phone and one interview was conducted in person and I
can tell you right now immediately I was disappointed that I had to do them over
the phone and not in person because I would have loved to seen the students
perspectives also you might have noticed that maybe we're missing a group that we
should have interviewed maybe we should have been interviewing the community
member as well to get their perspective maybe we should have interviewed the
faculty when you're engaging in qualitative message you have to think
about that sampling frame go back to your evaluation plan and say okay what
are we really looking for here what it why is that important so the students
were asked I put those in the box they're these types of topics these
types of questions like I said it was semi structured so we allow the students
to talk more about things we wanted to know what it was about the program yes
but also what could it be around and about the program as well so I'm going
to talk a little bit about the the analysis of the data a little bit later
on and I'm going to move through the example to kind of provide an
understanding of where why the quality the methods are utilized
in these instances but of course like I said I'll answer some questions at the
end as well so moving to the next example this is a more IP CP approach
barriers to and facilitators of interprofessional interdepartmental
interventions unearthing department culture and I provide the site there as
well this was in collaboration with our affiliated healthcare healthcare
delivery institution and it was a it was extremely interesting project so once
again the evaluation assessment of a particular IP GP programs had mostly
been focused on one clinical setting be the approach to exploring interventions
and their impact or success I thought was in our team thought was limited in
terms of what it was sampling who was sampling and the lack of longitudinal
approaches and also a lack of context and culture because we know again these
programs don't exist in the vacuum what else could be going on that's impacting
the success or improvement of these initiatives the intervention was
designed to enhance transporting care from patients from the IDI to the Mick
you so we were interested in the transition itself but also of the
perspectives of the intervention of those in it those on the ground the
clinicians and providers that were involved in this process so to explore
the intricacies of the interprofessional interdepartmental teams as personnel
from the IDI and Mick you began working together we engaged in a kind of a
longitudinal approach and what we wanted to do was really explore this specific
intervention and the perspectives attitudes and context of that
intervention so what had done what they had done at this institution was
redesign how they transport a patient's from the 'i deeds of NICU right so they
created this Mickey Weller team or mat in a comprised of a nurse and a
physician assistant from the Mick you responsible for the quick response to
the notification sent from the IDI now this is a little unique and why I
provided this slide is that this wasn't how it works before before when patient
when they IDI wanted a transport they would call to make you and they would
say hey you know come down here check it out and some of you probably notice a
lot better than I do being a sociology I'm not involved in this all the time
but what ends up happening with this particular dis intervention was that now
the IDI had the authority to activate this Mickey will alert team whenever it
wanted whenever it thought it was it was necessary and the mcubed team had to
come down before they would come down and think man maybe maybe not and they
could leave here they had to come down and the Matt response the IDI they
receives a bedside face-to-face handoff from the IDI team and then they assume
that make you alert him assume total responsibility for the patient's care
under the supervision of the Mickey attending so this they would even stay
until Abed was ready the Mickey team would stay in the IDI until the bed was
ready up in the Mickey so this changed a lot of different
things and some of you again who are clinicians and practitioners are
probably like oh yeah I could see what's going on I had no idea what this really
meant I was just interested in understanding the context the
perspective the attitudes of the program itself but what ended up happening and
I'll talk about a little bit of later was really we we unearthed cultural
differences that I had no idea about just by asking questions about
perceptions we unearthed like department cultures that had such an impact on this
initiative that it almost slowed the whole thing it almost brought it to a
stop at one point and that was really important we hadn't utilized qualitative
methods we wouldn't have understood how bad it was getting for the people on the
ground keep on wanting to use a certain button to change these slides I
apologize okay so understand we sought to understand the barriers and
facilitators to this transportation interesting the actors perceptions how
did this happen does it work what could things that could be happening or
hurting and we're using utilizing the alert team as a kind of conduit to
understand that so again we could utilize surveys and believe me the
clinicians on this team the providers on this team were very interested the
number one thing that came out of this was that it did improve the transport
and patient outcomes and they had there's a whole other paper about that
that came out so they were interested in time of transport and patient outcomes
those things could be measured by quantitative outcomes by survey methods
by exploring time and all those things but what about the context the
environment the department culture the people involved how did that impact
though that time and those patient outcomes and the whole process of it we
needed qualitative methods to engage in that so we engaged in a series of focus
groups to explore those issues and we we looked at the attitudes of providers
regarding the initiative and what was working what was working well what
wasn't now this is what's important when you're consulting focus groups so notice
that we conducted the focus groups with the folks from the IDI separately then
the focus groups with the McCue staff and I'm sure maybe all immediately
understand like oh that was a good idea well I thought from my perspective again
they're working together and they may not enjoy working together and we're not
going to get at that if we have a focus group that it combines all of them
together right so if you're trying to assess an IPE program and you're looking
for perceptions attitudes and experiences from students that are from
nursing PT ot and you want to know how they enjoy working together well you're
not going to want to throw them all on the batch because they may have some
things to say about some people that they don't want others to hear right so
we wanted their unique perspective so we arranged them separately and we did it
at two different times one was one month after the implementation of the
intervention and we conducted three separate focus groups and then we did
another one at t2 which was six months after t1 as it's very similar questions
followed up on questions or things that it was on earth during t1 and we again
we did three separate focus groups there to each of the focus groups lasted about
60 minutes and here's another key aspect to think about it was led by two or more
of the study team members I was always leading the focus groups but I always
had a clinician and a provider with me to help translate and what I mean is in
terms of words that were being used about the process that I didn't
understand that would affect the questions I would ask and so that was
important also they had a in ground knowledge of the people that were there
they knew the personality so they knew if someone was going to start to
commandeer the focus group they knew also how to kind of navigate that that
sensitive way of pushing the question off to people who may not talk because
of the particular occupation because of relationships those sets of things are
important again utilizing your resources is very
important when you're doing focus groups and knowing who you're inviting it's
very important as well all right like I said I'm going to talk a little bit
about data analysis today and I want to make sure that we have enough time for
that hopefully I didn't move too quickly in terms of the why and the when
utilizing qualitative methods and those two examples and again I'm looking at
the time and I think we're going to have a good amount of time for for Q&A so
again I can go more in-depth after that but I do want to provide just a nice
teaser into data analysis which some of you may not be familiar with or maybe
some of you are but I think it's very important and this will be a bit of a
heavy lift hence the the icon there and I apologize
up front if I move too fast I'm going to cover some of the fundamentals but I
want to provide you with enough detail so that you feel comfortable embarking
into these waters on your own I'm also providing a number of resources through
citations on the slides that I consider to be essential reading for doing
qualitative data analysis excellent citations when you're writing up
analysis sections or methodology sections in your own white papers or
program reports evaluation reports or even publishable papers so feel free to
utilize those as well and again I'm also a good resource to utilize so moving
forward when we're talking about analysis we're talking about coding
often these terms are utilized interchangeably so analysis for
quantitative qualitative methods is often referred to as coding but what is
it well codes are labels that assigns symbolic meaning to the descriptive or
inferential information compiled during a study that's I'll break it down I
promise they are attached to data chunks of varying size and can be
straightforward and quite descriptive labels or maybe even more complex from
metaphor esque I tend to utilize the metaphor esque approach but let me back
up what I mean by data chunks and some of you who work with qualitative or
qualitative data are probably aware of what I'm talking about but for example
let's say you have a transcript of an interview so you've conducted an
interview you may be you audio recorded you've gone ahead and had
someone transcribe that audio recording for you whether it's a professional
service that you go outside for or maybe it's someone you have working for you
whatever you do it yourself and you've transcribed that audio recording
verbatim of the interview so now you're going through that interview you're
looking at all that data and you're looking at you're saying I'm interested
in looking at conflict that's something I'm interested in looking at a potential
conflict within this intervention so you'll go through and you'll look at the
data and you'll try to identify the chunks of that data maybe it's a couple
pair maybe it's one paragraph maybe it's two lines maybe it's one line maybe it's
two pages where respondent has talked about what you think is conflict so
you've interpreted a series of data as relating to the idea that you have of
conflict in a sense what you have done there you Circle it you write conflict
in the margins and you have now coded a chunk regardless of what size of that
data as reflective of concept of conflict excuse me so you can say like
these can be are a super assigning symbolic meaning to the descriptive or
inferential because you inferred information compiled during that study
you've attached it to a data chunk and you've been straightforward to some
extent with that conflict but you have to really define what that code is and
we'll go about that in a second with a code you are actually capturing what
we'll call the essence of that chunk of data and it's a form of interpretation
every aspect every stage of analysis is interpretation in quality with
qualitative data so when you're coding you can actually move forward and have
that that consistent analysis coding is a form of data reduction and it's
actually a form of simplification it can be very basic unfortunately what's
happened is we see a lot of that in the in publications lately where people are
using very basic approaches to qualitative data analysis but it should
be more sophisticated so think about this way if you saw a paper on that was
utilizing quantitative method and the entire methods and results
section maybe just they just gave you mean median and mode you probably
consider that to be not a good paper so think about the same way in terms of
data analysis when we're just coding at the surface level and you're only giving
us that that's not going to be a very well interpreted paper that's not going
to be a really sophisticated paper and we'll go over some things of how we can
get make it more sophisticated one key to remember the coding is analysis but
it's just one aspect of analysis it's not V I'll talk more about that what I
mean by that in a second you want to be able to build the relationships and
uncover themes its the linkages between the codes between the data that's the
key that's the next step so coding I said is analysis if it's only one aspect
the other almost more important aspect is when you start to combine those
linkages when you start to see how the data connects when you're uncovering
those themes and patterns and categories codings codes are organizing principles
that are not set in stone they're not static so I talked about being
vulnerable being intimate being fluid these things are extremely important not
just when you're conducting the methods but also when you're conducting the data
analysis we move forward here so here's a good what I think a good list of key
phrases and terms that we utilize we're talking about coding or analysis right
so there's deductive coding and deductive coding is where you're
utilizing a list of concepts or topics that are coming from a predetermined
framework so maybe you have a series of guiding evaluation questions that have
key concepts or terms that you're interested in exploring so you're
interested in exploring professional identity formation right and you might
already know that or you might know that through previous research professional
identity formation is a key outcome for the program you're evaluating so you
want to go ahead you're going to use that as a deductive code right you're
going ahead ahead of time with that and maybe some other things like
interprofessional identity maybe solidarity teamwork these types of big
broad codes that you're going to go into your data already have these codes in
mind that's deductive right now an inductive
approach also known as open coding to some extent because you're open to the
data you're allowing the data to speak to you write the codes emerge
progressively through data collection right so also when you're interviewing
or doing your focus groups or doing your observations things are coming to light
as you're collecting that data but also when you're reviewing that data when
you're reading it right when you begin your analysis you said so this is
starting to come up I'm starting to see issues related to humanism or empathy or
psychological safety you know key things that we might be interesting that we
didn't really know we're going to be there oh this is starting to come
together right that's more inductive coding you're going to consistently
revise your codes and then I mean that you're not you're not going to stick
with your codes codes will decay and what I mean by that is you might go
through a couple of transcripts a couple of data points and you'll be like oh
this is a great codes fantastic Wow I'm really getting some good stuff and then
all of a sudden transcripts 10 through 25 have no evidence of that at all and
it the code itself has decayed right they also refine so you might start off
with something like professional identity formation but then you'll
refine it to have a series of other underlying either sub codes or just more
definition to it what you mean by professional identity formation or what
it relates to and you have to adapt to the data as the analyzer you have to
adapt that I cannot stress enough how important it is to not force your codes
you have to be open you have to be vulnerable to this and I know I'm using
these touchy-feely words that may not feel comfortable but I tell you it's the
best way to get used to this all right you have to be open you can't force
things a coding scheme is the conceptual web of how codes relate to one another
but also they're fuller meanings also I would say meh mowing and jotting
is one of the most important things you can possibly do with coding when I
receive my transcripts you know I used to code back in the day with colored
pencils I would use a specific color or a code and then I would go through my
transcripts and I would oh here's here's this point you know here's empathy and
I'd highlight that and I'd write apathy in the side and I'd have a
codebook that would be color-coordinated that became very time-intensive
so now I've gotten to a point where I just use pen I circle chunks of data I
write the code on the side and then I write notes as well and be like well
this might connect to such-and-such but I also keep a legal pad to my right
or my left and I start to make notes about what I'm seeing in the data all
right so this is a really important about mem owing and jotting it's
extremely porting kind of keeps a running tally it helps you stay detailed
but also keeps a running tally of what you're doing and a codebook of course
keep one make a list of your codes and what they mean it's very important so
what is coding and so here a couple of citations that I I'm really strongly
advocate for am I use these in my classes all the time I use them
personally in terms of citations but also reference materials referring back
to certain processes according to miles in vermin and saldaña there are cycles
of coding they break them down into two cycles first cycle which is working with
chunks of data and then second cycle which is more refined I like to call it
kind of break and rocks and then refining diamonds the issue to think
about they do a really nice job of mapping this out in terms of its first
cycle is breaking rocks second cycle is really getting more refined when you're
going through data analysis you're actually going through a series of first
cycles and a series of second cycles it's really important to remember that
it's not just oh I'm done with first Lego let's move on it's not like that
they do a way of identifying and categorizing them well but it's really
an ongoing process that you have to continually work with now this next
slide and this is where I said it was a bit of a heavy lift it can be a bit
overwhelming and to be fair a part of me wanted it to be this is again utilizing
those citations this is just what Miles uberman and
Saldana the key people in the field in terms of writing the book on data
analysis for qualitative methods actually lists as just the types of
types of strategies of first cycle coding you can see all the different
times that we go through right and sometimes you utilize some of these
strategies you know interchangeably with others you might switch from process to
value of course you're going to utilize
evaluation coding but to be honest in a little ticket trick of the trade here is
it's very rarely even in sociological literature even in the highest end of
qualitative journals you will not always see people be so specific about the
strategy they utilized in first or second coding now some do don't get me
wrong some do and I think that's fantastic and I think it's very helpful
to be so specific but because you switched so often especially if you're
doing inductive coding you may be engaging in something and then switch to
another type of method like in vivo coding which you're really coding in
terms of people's voices in the tone and things like that you may quickly switch
to emotion coding and you may not be like I did this for 20 minutes than I
did this that's not how it works but I did want to show you all the different
types and synergies for just the first cycle of coding according to the key
people in the field so again it helps you break those rocks these are all
methods utilized breaking rocks I do want to bring up a couple real quick
though not all of them by any means I don't have time for that
but I do want to bring up a couple so causation coding right under procedural
which is the type look at the why particular outcomes come about right so
this is very useful in terms of evaluation and assessment your mapping
pathways between your making connections between your codes as well and it's also
excellent for grounded theory which I will talk about in a little bit right to
foster those connections there's magnitude coding which is under
grammatical so this is kind of what I would call my arch-nemesis in the
clinical role when I when I do a lot of evaluation assessment for healthcare
delivery professionals I think this is more of a supplemental work but
oftentimes it gets published and honestly sometimes when I put forth a
manuscript to a key clinical journal and enhance qualitative methods they demand
that I provide them a frequency count of particular words or phrases so I might
say you know this clearly has weight this team emerged I would give them all
the analysis of how I identified it but then they come back and say well how
often did it did it people talk about it how often did it come up explicitly in
the conversation now from my perspective and I'm sure you've heard others say
this just because something is said or happened the number times and
make sure it's important or quote-unquote significant or actually
impactful to the program right but you can do a frequency analysis of specific
words if you are interested now some of you may be interested in a deductive
approach that utilizes that and that's perfectly well and good there's no
problem with that right and that's fine and I'll actually
show you something I did with with this magnitude coding but it is kind of more
of a frequency analysis of sorts sub coding is something that you're
consistently going to do you have a major code and then in the process of
refining you'll see other things kind of come out of that I'll just get more
specific it'll be you know empathy related to patience empathy related to
colleagues that kind of stuff those are sub coding so you have two different sub
codes there on the code of empathy there's also simultaneous coding where
the overlap occurrence of two or more codes on a data chunk that's going to
happen all the time and that's where you start developing those connections and
those patterns because you say oh wait this relates to this right so it takes a
kind of a trained eye on that one but people can do it get your hands dirty
with the data and you can find that second cycle according to these
colleagues you're finding the patterns your reoccurring categories you're
looking for your pattern codes the things that are coming out that are
connecting it's the connective tissue is that you're coding out here you're
looking for the threads that tie those bits together I have to stress to
everyone don't go here for some people go there for the pad let me find
dependence develop your codes first your fundamental list of codes and then make
your connections don't rush through it it happens quickly and it can happen
actually as you're doing first cycle so you just got to be attentive and you got
to be vulnerable to it but don't stay locks you know don't get locked in it
don't say this pattern is happening and then try to force things into that
pattern stay open stay loose it's very important this is where theoretical
constructs will come in and the relationships to the codes themselves
some quick notes in terms of understand the process of coding there's more than
one way to code my students always want to know you know and people I work with
what's the one right way there's really not
you saw all the different types of first cycle coding and there's number of
different ways to go about doing that whether you read them first read them at
all when you're working with colleagues there's a number of what
to do it but you have to start somewhere always memo and jot pre-code activities
do you read them before you start coding to get familiar with them
what could should be coded that really depends on your guiding evaluation
questions in your evaluation plan the garbage in garbage out just like if
you're working out if you don't do a great work out if you don't put forth
some effort you're not going to get a great outcome it's the same thing with
data analysis year if you don't put forth good methods and rigorous methods
and try to get that data and get a good sample you're gonna get junk d
contextualizing and recontextualizing remember you are a you are a filter you
are interpreting constantly and this is something I like to call the Goldilocks
dilemma this is where your codes might be too general or they might be too
specific right both of those are problematic so always be open to
refining and letting decaying codes just die out like don't force him I did say I
would provide some examples from my own work in terms of data analysis I'm not
going to run through this it again a lot of words but what I've done in these
next two slides is provide kind of a blueprint an outline if you will some
key points stuff that I just took from our and now our methods and analysis
section from those paper so one of the things to look at is the things I bolted
the multi-step inductive and deductive coding things I talked about identifying
patterns how we did that we utilize inductive codes making comparisons right
and then one of the things I do want to reference here is that we actually what
we ended up doing was once I developed the codes and went through first and
second cycle a ton I threw those away after I had my codebook came back to a
clean set of transcripts and actually went ahead and used my codes my patterns
as codes and recoated all of those transcripts again so again did it in
another cycle of getting more in-depth so you can feel free to use these
resources and again use me as a resource as well but again I'm near his for the
the mat program we did open coding utilize those factor factors as codes as
well we did a frequency analysis to identify how many times factors were
actually raised and then we looked at the degree of specific especially excuse
me to those categories add to the
intervention so how close were they to intervention and how close were they to
the out sending context of the institution or those within it right we
also practice in decoder reliability which is very important when you have
others coding you want to make sure you have some similarities in terms of how
you're coding as well now real quick is I am running at a time
and people are letting me know so what I want to go right here is I'm going to go
real quick through grounded theory just some key fundamental points and then
open it up to questions so the take-home message here and I provided a list of
citations here for you so feel free to utilize these as resources and
understanding grant theory not everything is a grounded theory or a
grounded Theory approach or a constructivist grounded Theory approach
if people have told me that they feel like they need to utilize that in their
papers because it adds sophistication because qualitative methods has a
negative stigma so they feel like they say oh I did grounded theory that
they're adding some sophistication that's not the case right one of the key
things remember and I've taken this direct quote from one of the resources
there the final product of a grounded Theory study or even a grounded
evaluation or assessment protocol or ground 3 nestled with an assessment
evaluation or assessment is an integrated and comprehensive grounded
theory that explains a process or scheme associated with a phenomena you have
when you're saying you're utilizing grounded theory you have to generate
theory it can be an add-on to a fairly popular theory within the literature as
it is that's fine but you have to generate some kind of theoretical
concept process or construct right that is the key and a lot of times I'd say
you know I don't haven't done a frequency analysis I'd say it's probably
close to 85% of the papers that I review that say they're doing grounded theory
are not doing grounded theory especially an evaluation and assessment and in the
interprofessional realm particularly so in terms of these coding processes for
grounded theory I said I'm going to move a little bit quickly
there's initial coding and then there's focus coding just like that first cycle
and second cycle and again you'll have these slides so you'll be able to use
these as a resource as well but I do want to show you some
key factors open coating is utilized frequently within grounded theory actual
coding categories are more refined just like in the general second cycle and
then selective coating where you have a core category where you're starting to
identify those Q via reticle constructs and you're piecing the theory together
into a story and relating it to other categories granda Theory coding
generates the bones of your analysis theoretical integration will assemble
these bones into a working skeleton oftentimes when people are doing ground
in theory they just leave the bones there they forget to bring it all
together with axial coding and then theoretical coding in discussion in
terms of the more classic folks in Grenaa theory like Strauss and Corbin
and Glasser this axial coding and theoretical coding are the key elements
this is where Sharma's might start to kind of differentiate a little bit Shana
is a perspective on grounded theory and there are differing perspectives on
grounded theory and all have their own merits of course she breaks away a
little bit in terms of axial and theoretical coding all right and so I
mean I'm not trying to put words in her mouth but that's where it kind of gets a
little bit more for the people and a little less more theoretically rounded
right so it's very important to understand the key elements with
grounded theory and I think all of them have the merits and they have their
similarities but there's also a number of differences as well and again I can
go more into Decimus later one of the last things I want to leave you about is
is the questions about - in vivo or not - in vivo right this is not an
endorsement by any means but here's a list of a qualitative software so to
analyze qualitative data I use in vivo or match QD a I have a colleague right
across the hall who utilizes at least Ti and hyper research different people
utilize different things the one thing I will stress it helps to break rocks I
will give it that but it does kind of eliminate that intimacy with your data
when you're not doing it by hand but these are very useful when you're
utilizing a large amount of data so some takeaway points as I wrap up qualitative
methods are useful and valuable in various evaluation assessment designs
right the how why what and when you're going to pick the right methods to the
job and you need to utilize your guiding evaluations questions
for that know before you dig understand what you're getting yourself into
utilize that evaluation plan analysis manners engage in detail not everything
is grounded theory and of course use your resources use your colleague use
your resources use the settings you have so my advertisements for the
end-of-course Doug Archibald coming up on may 22nd with utilizing quantitative
methods to collect and analyze your data make sure you register for that and of
course save the date for the next summit I'll be there as well the whole team
will be there actually if I remember correctly and that's from July 29 to
August 1st but we'll have a short course for the evaluation for program
evaluation and guide number five in which we are actually Doug and I are
given a little bit more space to talk more in depth about methods but we'll
also go much more in depth about the program number five as well so it's a
great supplement to these webinars and with that I know I went fast and I
apologize towards you but I did want to leave some time for questions so do
anybody have any questions
thanks so much Barrett and yes we've been monitoring a couple of the
questions coming in the first one we got asks you to talk a little bit more about
how to collect qualitative data online and what some challenges are advice you
might give especially during the IRB part which I am assuming is the
recruitment yeah so ISO IRB in terms of designing the actual protocol for your
whole research design and human research assuming subjects research yet so online
it really depends on what you're trying to do if you're going to do online in
terms of sending out emails to solicit responses and it still is that your
sample that's and that's that's you can do that IRB allows that you just have to
send in your new your solicitation email to with your protocol to say what it's
asking them to do I often include a consent form in there as well so the
students know what they're getting themselves into or the participants know
what they're getting themselves into and if you're looking at
there's an online program that you want to evaluate that's an interesting ball
of wax but you can utilize that in the same way that you would utilize an
in-person program or intervention that you wanted to evaluate but in terms of
using IRB and if you're trying to get online recruitment that's really
utilizing a strong solicitation email and I have examples of that I'm more
than willing to share and also providing ajust enough information for people to
understand what you're doing and they know what they're signing up for if I
didn't answer that question I apologize but let me know any experience a second
question asks in a focus group type of qualitative study in multiple
institutions do you usually have the same person read the groups in all
institutions or if you have a different person in each institution as long as
that leader has experience oh it's a good question so so we qualitative
method it's hard to control for a lot of things one of the things if it's
possible if it's possible setting up the design to where the person leading the
focus group is the same person for each of the groups regardless of the location
that would be wonderful but again feasibility is what's going to
steer that ship if that's not feasible then you have to have an agreement
between the people leading the focus groups understanding what's going on
maybe even doing a practice focus group beforehand to make sure both people
understand how their the other is going to go about leading the group so you're
not taking the group's in different directions or even meandering in one way
that you wouldn't do for another so you know best-case scenario have one person
lead it and then worst case scenario just makes sure they're both on the same
page in terms of moving forward or the all three of them are because if you're
working with the de team and you have three or four it may just be easier to
have each of them do their own just make sure everyone's on the same page
great thanks Brett I've got one question and then maybe we can wrap up after that
I thought it was great how you mentioned using the literature as you identified
the need for qualitative Memphis for your two examples and then even how you
focus your methods I'm just wondering if there are seminal
review articles in the IPE CP literature that you might recommend people read in
order to understand what have we learned from qualitative research in the IPE CP
field to date you left you leave me with a stumper alone you close down I was
doing so well and then you just draw this this one that I thought was going
to be a slow pitch but ends up being a fast ball yeah right right down the
middle and you can email people later but if you guess I'm going to take that
route there are there are a number of I put it this way so I review a ton of
articles that are evaluation and assessment with qualitative methods and
I just from that alone as well as my own research in citing certain people there
are a ton of excellent approaches and some very complicated and sophisticated
qualitative methods that I would love to compile if I if I have the time and I
definitely will before the short course that's really what we what we've learned
the results are you know effect like a meta-analysis of qualitative studies I
will get if any of the audience members has one in mind that'd be great at the
time nothing's coming to mind but it doesn't mean it doesn't exist so I will
dig in the trenches and see if I can come up with one or two but there are
just extremely great examples that you can utilize as frameworks for your own
work I think that's one of the best ways to do that good good no you hit the ball
out of the park okay all right well and also I do want to add one more thing cuz
you thought about literature and when I was talking about Grenaa theory I move
kind of quickly and I do want to add that constant comparison is also a
technique utilized in grounded theory in which you're comparing not only data to
data but also data to literature consistently from the start of your data
gathering and also those are methods you know the open coding actual coding
theoretical coding constant comparison those types of things that are specific
to grounded theory but they're not just for grounded theory you can utilize
those on your own in terms of any kind of inductive or deductive approach
they're not just for grounded theory so don't be afraid to utilize them in your
own opportunities as well thank you so much ferret one thing
before we log off here as I am going to put an evaluation link in the chat box
if you all could click on that link and take a couple minutes to fill out that
evaluation we would really appreciate it and that concludes our session today
thanks so much for joining us and we look forward to meeting you and hearing
from some of you in the future
For more infomation >> Onion Soup Recipe - Duration: 5:20. 
For more infomation >> Suzuki Baleno - Duration: 0:52. 
For more infomation >> Opel Mokka X - Duration: 0:59. 
For more infomation >> Jurassic World Alive Hack 2019 - Cheats for FREE unlimited Cash and Coins (GPS) - Duration: 5:28. 
For more infomation >> VW Tiguan - Duration: 1:23. 
For more infomation >> Skoda Kodiaq - Duration: 1:13. 
For more infomation >> Skoda Octavia - Duration: 1:05. 
For more infomation >> Skoda Octavia - Duration: 1:09. 
For more infomation >> Skoda Octavia - Duration: 1:12. 
For more infomation >> Fiat 500 - Duration: 0:59. 
For more infomation >> Actualité TV - Golden Globes : les stars porteront des rubans symboliques pour la lutte contre le h - Duration: 2:07. 
For more infomation >> « J'ai peur pour ce couple » : Brigitte Macron malmenée, mais elle ne se laisse pas faire ! - Duration: 2:33. 
For more infomation >> Circle K Texas: GUESS THE SCORE at the Rodeo | SA Live | KSAT 12 - Duration: 4:15. 
No comments:
Post a Comment