Need for an Evaluation Blog Archive?


#1

When evalcentral started, it was a blog aggregator.

One the secondary benefits of the structure was the site could keep an ongoing archive of blog posts from across different websites. Its existence actually helped restore at least one evaluation site that experienced a server crash.

I wonder if that secondary benefit is more essential. Past AEA president Molly Engle blogged about Program Evaluation for close to a decade. Visiting today I found the site has been archived. You can find archived versions on archive.org, but while it preserves history, it transforms something that could be useful into a historical document.

http://blogs.oregonstate.edu/programevaluation/

I reached out to ask about getting a copy of the archived version so I might be able to bring it back to life. Curious what everyone else thinks. How important is it to start archiving sites like these and prep them for search/use well after they stop producing new content?


#2

I, for one, really miss the aggregator. It was a one-stop shop to read about evaluation from a lot of different voices.


#3

Follow-up then.

If I bring it back. Who should I include? I always reach out to the administrators and make sure they are okay with me adding their feed.


#4

I hesitate to mention this - because I may be volunteering for something larger than I anticipate - but I conducted an analysis of 10 years of written discourse for an association…which (among other things) created a massive network map. It, too, was a volunteer project (because apparently I get bored without a big research project and want to give back), so I’m speak freely about it and use the methodology here.

I then turned this analysis (and map) into a dynamic viz as a way to look up what discourse topics were discussed, where, and by whom. Kind of like reverse engineering a library catalog by analyzing (quant and qual) the discrete and contextual bits of the material.

If this sounds cool, it’s because, well, it is. There’s probably better documentation now, but I had to fall down many rabbit holes at the time (mostly, interesting). It’s also very intensive to do correctly, but it could create a useful way to increase usability of these blogs.

If this is something people want to use, we can talk about it. I’ll have more (unpaid) time on my hands than I like, and there’s only so much I can do until the garden and trees start doing their thing.


#5

I’m not sure we need that level of analysis, but I agree it would be cool! I’ve thought about doing a study on #EvalTwitter and who is participating, what they are talking about, etc.

I think reaching out and asking people if they want to be included is a good idea. Do you not have your previous list of who was included from a few years ago? I think that would be a good start, but I have a few lists of blog sites in mind already.


#6

Personally would love to see the list you had in mind. I have people to reach out to from before, but I feel somewhat disconnected from newer voices.


#7

It sounds like a big project Jeremy, not sure we have the collection of content ripe for analysis at this point.

I’m in the…

  • Should I invest time in archiving?
  • What to archive?

stage.

Don’t think I have the resource base required for the…
What does the information within this archive tell us about evaluation discourse?

I think at this point, AEA365 would probably be a better content library to analyze if you were to take on such a project. If you reach out to enough people, you’ll get a lot of “that would be cool!” comments. Not sure that would be enough to justify a big time sink though!


#8

Well, mine for one :wink: but others off the top of my head include:


#9

I’m also looking at discourse analysis of the evaluation community for my dissertation… A one-stop shop for such a corpus would be awesome.


#10

The nice thing about the possibility I’ve presented here is that the collection (or, as @awkates calls it in the slang of this methodology, the corpus) would be harvested through a automated process - I’d just need the web addresses of the places we’re interested in examining.

Of course, the automated process needs regulation and all that, but that isn’t the biggest hurdle. I’d be more concerned with building the sample.

@danawanzer, my previous study using this methodology was an association (VSA), and it involved publicly available sources like listserv messages, website text, association documents, and social media postings. I didn’t go elsewhere to private sites since that was outside the scope of the research questions.


#11

It would need to be a constantly growing and expanding corpus. What we could do is decide on a set of criteria and then we can have a crowd-sourced open period for everyone to find and suggest URLs. Then we could have a group of people act as referees to see if we want to include it.

As far as criteria?
-The entire site should be evaluation-focused
-Should be dedicated to writing, theorizing about evaluation. (Not simply organizational documents, e.g. AEA conference website would be excluded)

Just spit-balling. Any other thoughts?


#12

I think this is a good start, yet once we start saying the entire site (or feed) should evaluation focused, I think that will be a tough determination to make…or at least to make in a solid IRR kind of way. Your view (and others from your formal discipline, probably) of the breadth of the evaluation field is likely different from mine. The framework I’ve used in the past allows (mostly) for a more contextual view, even if the corpus is broad, since it employs a healthy mixture of quantitative and qualitative techniques.

But I completely understand if that framework may not match with your desired approach. There are many paths up the mountain.


#13

Roger. Would require some tinkering to come across criteria to be agreed upon. The main thing though would not be to have the right criteria, but to have criteria. Systematizing is key.


#14

Also, I’m very interested in any comparisons that could be made between the discourse that mainstreamers like myself would consider to be squarely evaluation-focused, and that of the more quasi-evaluative discourse of those in user experience, market research, etc.


#15

So, what I’m saying is, a broader corpus could be extremely useful for me.