Developmental evaluators, where are you?


#1

I’m Carolyn, I work as an independent evaluator in Vancouver, BC, Canada. My practice is increasingly moving into primarily developmental evaluation work, and I’m really hopeful that this discussion board can become a place to have deep conversations with other developmental evaluators as we figure this niche out. I’m in a constant state of improv and invention myself. Books are a great resource, but a thriving community of practice is even better!


#2

Hi Carolyn, I’m Rachael, in Springfield, MA. The vast majority of my work is developmental evaluation. Thanks for thinking of starting a topic area on it!


#3

Hey @c_camman and @R8chLawrence

You are both in-the-know evaluators. There are many, and probably many who will show up on this board, who have no clue what a developmental evaluator is or what they do. People who would probably ask, “what’s an MQP?” with a straight face.

Given that kind audience. Can you give a forum briefer on what you think it means to be a developmental evaluator?


#4

Thanks, @clysy, that’s a great point. Even among developmental evaluators, I think it’s worth bringing some clarity to our different interpretations and understandings, which I’m sure we have!

So I would love to know how other people doing developmental evaluation work make sense of what we do. In the interests of generating dialogue and building community on this thread, I can put some questions out there for people to answer (and which I will come back and answer myself):

  1. How do you define or describe developmental evaluation?
  2. What’s a burning question you have about DE?
  3. Where do you look for inspiration/wisdom/guidance about DE?

#5

Okay, I’ll give it a shot @c_camman

  1. Developmental evaluation was spawned by the idea that summative evaluation and formative evaluation are inadequate to meet the needs of many programs. What do you do if you don’t have some concretely defined program, activity, or intervention. Not ready for the traditional kind of pass/fail summative evaluation. Your needs and activities are too broad even for the tweaking style of a formative evaluation.

  2. Is developmental evaluation defined enough to not be a tautology? It can’t just be everything Summative/Formative is not.

  3. Simple answer, Michael Quinn Patton. Honest answer, Google…


#6

Here are my answers!

  1. Right now I describe my work to people as “learning as we go”, with developmental evaluation as a process that helps us learn and adapt in real time as we both develop better understandings of the complex contexts we’re intervening in as well as change that context by interacting with it. So the nature of the evaluand is changing while (and in part because of) being evaluated, and DE supports this.
  2. How much of a coherent practice is it really? I’ve had to do a lot of “making it up as I go” (which is kind of inherent to it, and also a feature of there not being a strong digital gathering space for all us practitioners flung out across the globe), so how many different individual “strands” of DE are there out there, evolving in relative isolation?
  3. Working with designers, particularly people with formal design training who also work in social change settings, so they bring the rigour of their design thinking to the complexity of change work. To me that’s the space where evaluation and design come together in a powerful way, where the veil between them is thinnest, if you will, and DE doesn’t feel like an “alternative” approach but the most natural, obvious approach that drives and is driven by the design process.

#7

Here’s my two cents, or perhaps four in the current economy:

  1. How do you define or describe developmental evaluation?

Developmental evaluation is evaluation done with the purpose of informing and contributing to the development of a program or product. (Differentiated from evaluation of one’s personal development, or evaluation associated with difference in human development). Evaluators who work in this way provide data-informed feedback to programs that are often more in “prototype” than well-defined. Rather than having the goal of better adhering to an existing model, it’s to help inform the development of a model where either none exists or the model is too vague to actually be a model.

  1. What’s a burning question you have about DE?

Honestly, is this really a paradigm in and of itself, or is it more a series of competencies and preferences about work environment? There’s a lot of overlap with other evaluation practices in terms of methods and use – it’s the status of the program that is really different.

  1. Where do you look for inspiration/wisdom/guidance about DE?

My mentor/advisor/boss (Sharon Rallis) for one. Others I’d include besides the obvious MQP are Cheryl-Anne Poth and Cheryl Baldwin. I think dealing with complexity theories is pretty important.

More later…but a start.


#8

Whether DE is a coherent practice is such an interesting question. DE was purposely developed to be fluid and non-prescriptive; I suppose it has to be with its focus on adaptation, change, and environments that are complex, dynamic and changing. As Patton writes, the 8 DE Principles must be applied, but in ways that are responsive to the context of each unique project.

So there are potentially endless options for implementation, which I think relates to what you’re saying with ‘different individual strands of DE’. Where I get hung up now is how I should feel about this. On the one hand, there’s the flexibility that complex situations require - great. But on the other hand, it makes it more difficult to assess DE approaches that others apply (an issue for things like research on DE and applying DE practices across projects). I’m not exactly sure how to address this issue (hooray for raising more questions than answers!).


#9

My hope for this site is that we can find ways to encourage discussion @evan_poncelet
And in my mind, there is no better way to encourage discussion than asking more questions.
Answers are overrated.


#10

Agreed, questions are way more interesting than answers!

I’m already fascinated by the diversity of ways that we’re approaching and describing DE. I know the practice is one that many people have trouble “putting their finger on”, myself included sometimes. I think I went with the word “coherence” specifically because it gets at the way something can be vast and complex and yet still have some kind of logical continuity and thread that holds it together. Cities are large, multifaceted, and ever-changing, but we still generally have a sense of a city having coherence–an identity, boundaries, perceived commonalities within it that distinguish it from other cities or surrounding rural areas (albeit sometimes with some fuzzy liminal spaces).

I hope others weigh in as well!


#11

That’s a useful analogy, I like it! This discussion gets at @R8chLawrence’s burning question too, I think. DE overlaps with other eval theories and practices, like a city borrowing ideas from another (city policies, planning and development, etc.), but it’s a unique theory as a whole. Sort of a ‘whole being greater than the sum of its parts’ situation.

I also think it’s ironic that DE is kind of a trans-theory in the field evaluation, which itself is a trans-discipline :grin:


#12

I’m going to take a shot. Developmental evaluation undergirds the innovation activities of an organization by forging a partnership between summative and formative assessment. DE is the yin and yang of evaluation that sees harmony in the tumult and fluidity of social change.


#13

Great conversation. I’m finding that being principles-focused rather than model-based is a core distinction of DE. Increasingly for me, principles-focused developmental evaluation constitutes the niche. What grounds DE, since it is not model-based (and not aimed at improving or judging a model), is adherence to principles. Principles-focused DE is for principles-driven innovators changing the world through adherence to fundamental principles. Developmental evaluators, then, must also be principles-focused, adhering to the principles of DE.
https://journals.sagepub.com/doi/abs/10.1177/1098214015626295?journalCode=ajec


#14

Hey @MQP do you have any resources that go through the principles that don’t require someone to pay for access? I can access the journal with my AEA membership, but that’s not necessarily going to be true for the target audience here.

I found this on AEA365 which covers the basics.
https://aea365.org/blog/developmental-eval-week-michael-quinn-patton-on-developmental-evaluation-principles/

But I have to be honest. This graphic just makes me dizzy.


#15

Not sure if this is kosher so apologies for linking it if it isn’t, but I am aware of a PDF excerpt from the DE Exemplars book (one of my favourite DE resources!) that covers the Eight Guiding Principles. A little Googling also turned up an accessible PDF of the article that Michael just linked. (Total sidebar, but there’s another topic to be had about the value of open-access journals in evaluation–CJPE’s recent return to open access has been a godsend.)

Thank you for weighing in, Michael! This is exactly the kind of dialogue I was hoping a community space like this might afford. I am encountering greater and greater interest in DE, from evaluators and evaluation users, and I’m really interested in how DE in principle is translating into DE in practice, how we’re engaging with and making sense of, in our own ways, the sensitizing concepts of the DE principles in the work we do.

One principle I’ve wrestled with is the idea of the innovation niche (“Elucidate how the change processes and results being evaluated involve innovation and adaptation, the niche of developmental
evaluation”), first because of the word “innovation” and second because of the word “niche”.

With “innovation”, you address it in that chapter–an innovation does not necessarily need to be a big, spangly, disruptive, totally unseen before, paradigm-bending spectacular. An innovation can be contextual, something that is a new and different approach in that setting, even if it is a small and simple shift, even if it might have been “innovated” before elsewhere many times.

The word “niche” though! I wonder about that word. Do we think of it as a “niche”, of having a small and specialized focus? The connotations are dismissive, almost derogatory. I certainly don’t approach my work, or the work of my clients, with the word “niche” in mind–I think of it as part of something vast and significant and interconnected and deeply permeating, not marginal. I’m not even sure if this is a substantive comment or if I’m just being cranky and fussy, but I have always wondered about that word.

ETA: Ah ha! I know what the crux of my complaint is. To me, the use of the word “niche” reinforces the smaller, narrower, and less accommodating interpretation of “innovation”, the one that inspires the reaction of, “Oh, I’m not doing innovation, nothing as special and grand as that. You’ve got to be a certain kind of person to be an innovator”. Still possibly not a substantive issue in the end, but that’s what makes me itchy about it. Would love to know what others think.


#16

I’m using niche in the evolutionary sense of context specialization. Evolutionary biologists describe and understand species as having adapted to ecological niches. It is a descriptive not a judgmental term. It is not about size. The niche of whales is the whole ocean. The innovation niche for DE was meant to contrast it with the dominant model mentality of evaluation which values standardization, replication, routinization, and mechanistic fidelity in contrast to adaptability, responsiveness, agility, and situational and contextual sensitivity. The DE article in AJE is now old enough (after 18 months) that it should be open access.

https://journals.sagepub.com/doi/full/10.1177/1098214015626295


#17

Chris,

You’ve added to my growing list of dataviz distinctive honors. Stephanie Evergreen credits me with the worst PowerPoint slide she’s ever seen. To make an accomplished, creative, and imaginative cartoonist dizzy confirms my niche as dataviz impaired. And this graphic is one of my clearer ones. I’d post the one that Stephanie lauded but my attempt to cut and paste fails. Another impairment.

MQP


#18

No, it’s not accessible.

@clysy - I wonder if it would be possible to have a library here, for people to share personal-copy PDFs (or pre-print proof, etc.)?


#19

@MQP I have to concur with @seaelkins here.

It doesn’t look like I can just get access as a casual visitor. I even registered for a generic account with Sage to see if that made a difference, it did not. You can see in the “My Account” space where it says “You do not have access to this content” and it wants me to purchase the article for $36.

The article might become “open access” by Sage, but they’ll never tell anyone that. And continue to charge for access.

Catherine, it’s a good suggestion. But when I started back up evalcentral the idea was to create an open discussion portal that can be easily accessed, archived, and crawled by search engines. Developing a proper well-designed resource archive is a bit out of scope for me to handle.

For resources like those, I’d suggest https://www.betterevaluation.org/ at the moment. Although anyone is most welcome to share work they’ve created within the context of the forum conversation.


#20

I’ve made the article available here:

What is Essential in
Developmental Evaluation? On Integrity, Fidelity, Adultery, Abstinence,Impotence, Long-Term Commitment, Integrity, and Sensitivity in ImplementingEvaluation Models by
Michael Quinn Patton1