What does it mean to be systematic?


#1

A lot of definitions of evaluation contain the word systematic. The definition I usually use is: Evaluation is a systematic process of determining the value or worth of anything.

I’m curious how you all clarify what it means to be systematic to stakeholders. I usually include the following:

  1. Consistent (e.g., collecting data in a similar matter by using data collection protocols and employing them even when you might get critical feedback)
  2. Incorporating different angles and perspectives (e.g., collecting data from people who have different roles related to a topic or using mix methods approach)
  3. Avoiding Biases (e.g., reflecting on your own biases and being mindful about their influences on our data collection, developing data collection protocols to help mitigate biases, and seeking out potentially critical feedback)

How would you describe being systematic?


#2

Hi Michelle, I think these are good clarifications. The process is definitely something I emphasize. Specifically, that good evaluative thinking is involved, and of course, good evaluative questions. Engaging the community on defining the questions, methods, surveys, and in the interpretation are critical too.

Ann


#3

The first thing that comes to mind for me is that “systematic” can be contrasted with it being a haphazard. When we use the word “evaluate” colloquially, we might be talking about someone evaluating something in a more causal manner (like saying “This was a good movie” or “The restaurant was terrible”), but when we talk about “evaluate” in the sense we are using it here, there is (or should be) a coherent process where we decide what question(s) we want to answer, how we might go about answering them, we use the logic of evaluation, etc. I just googled the definition of systematic and it is “done or acting according to a fixed plan or system; methodical.” So systematic is about having a plan or system (I might take issue with the word “fixed” plan here, as I think you can systematic while having a plan to be flexible and adaptable to the context!).

I think that you could say that an evaluation is “systematic” without, for example, engaging stakeholders in the process, but that probably would not be a very good quality evaluation. That is, “systematic” is part of what an evaluation should be, but on its own it is not sufficient.


#4

Great question! I agree with drbethsnow. An evaluation can be of bad quality, biased in the selection of methods and sources, exclusive, or of a narrow, one-sided perspective - but still be an evaluation. But if it isn’t systematic - if it is not clear what the criteria of value are, and how their fulfillment is judged - then it is not really an evaluation, at least not in the sense of the transdisciplnary professional field we are discussing here.