A brief introduction of the history, applications,, advantages, and ingredients of geospatial impact evaluations.

Geospatial spatial impact evaluations. This is something that here at AidData we've been working on for quite a while. This is a bit of background in AidData. We are a research lab that's based at William and Mary, a university here in Williamsburg, Virginia just a few hours south of Washington, DC.  One of the oldest universities in North America. but we had. we'll have been marrying, and any data have been around since about two thousand and four. we have a professional staff for about thirty five folks. which is a mix of program evaluators and policy analysts Geo. Spatial folks, et cetera.
Our team you're going to be interacting with this week has particular expertise in Gis remote sensing. causal inference. Um! And as you could see, we've partner with lots of different folks over over the years
uh a big part of our focus at AidData in around two thousand and eight nine, and and in the five or six years since then was really the launch of Geo-reference interventions, so identifying the locations of aid projects around the world.
Uh that began really with an an effort to geo-reference The world Banks, Portfolio, especially It's Ida and Ibrd projects
um often that meant drawing from annexes to the project appraisal documents. So you know, really kind of searching through all the project documentation as as best we could, and we've done a lot of this work, together with a lot of undergraduate research assistance here who are helping us come through on this material.
But in the decade or So since then we've geo-referenced over a couple of hundred thousand that a unique projects which are now a geo-reference to to millions of actual locations worth over a trillion dollars.
Some of this effort took a donor specific view, So the blue dots on the screen that you can see are this World Bank portfolio. We now do a reference all bank projects that were approved between one thousand nine hundred and ninety-five and two thousand and fourteen
um, And then some of this work took country specific views. So, for example, the Drc.  Which you could see, we looked at all the different donors in the Drc. And Geo-reference, all of those activities for a different for a distinct time window using the aid information management system that's kept by each of those countries.
So all of this different geo-reference activities that we've now collected and compiled really made us then think, Okay, What more can we do with all of this, now that we have this can create, these great visualizations can think about? Why certain areas are getting this. But why not others?
And naturally one of the things we link that together with was newly available. Geo-reference outcome data on poverty, living conditions, the natural environment
um agricultural outcomes. One of the most immediate things we linked it together with was things like night time, lights and other remotely sensed outcomes, partly because these outcomes are now much more widely available, much more accessible, and much cheaper to access altogether
some of the earliest uses where we applied this geo-reference aid eh activities, together with Geo-reference remote, sensing and survey based activities, was an evaluating infrastructure. So and the images here we're showing an example from Cambodia
where we evaluated a small-scale rural infrastructure project. program rolled out in combination with a variety of donors.  In this particular case it was the swedish eda that funded this particular study.
Sweden had been one of the key donors supporting this project
um and This was a community-driven development project. So
at the top you have the community members who actively discussing which type of infrastructure they most want. Um! Often this was a rural road connecting the village to a nearby secondary or or main road.
Um, and in other cases it might have been small scale irrigation,  And then in the bottom image. You see, we've overlaid nighttime lights over the locations of these individual projects, one
Um, and in this particular case we could trace out the growth of night time lights over time as these projects came online, and and then matured and we can also trace out the consequences for the natural environment, especially for deforestation.
Another example is from the West Bank. in the Palestinian territories, where Usaid had been funding a series of road improvements as well. On the left hand side you could see these individual road segments.
Um that were funded again over time they were rolled out. and we drew buffers around those individual road segments. Obviously you could see some of those buffers overlap, because those road segments, some of them, are very close together and feed into similar main arterial routes.
Um,
Our units of analysis in this particular case, where individual night time lights grid cells.  This data that we used had seven hundred and fifty meter square grid sales for them.
Um! And then we analyze that change in the night time lights output over four year window when these roads were improved. Um! So that let us trace out again the causal inference. Sorry the causal improvement of of the roads on the night time lights impact as well,
moving beyond just infrastructure, though. we've also done this in a number of other sectors.
This particular example we're looking at is from Brazil and the Brazilian Amazon, where indigenous communities had their land rights formally recognized by the Brazilian government with the support of the World Bank and German Kfw.
Um. In this particular case German Kfw sponsored this study that attempted to look at the long term impact of the formalization of these rights on forest conditions, especially deforestation.
And again, the the distinct communities had their rights formalized over different time periods, and so we could compare the changes over that time period using a panel design which we'll talk about in a lot more detail tomorrow.
In this particular case we saw no long term impacts of these community formalizations on the in this sorry, the the natural environment  Other studies since then have used other designs, and detected smaller effects around the margins of the boundaries of some of these communities.
We'll talk more about that tomorrow as well.
Um. And Then, more recently you've done work looking at landmine clearance and its effects on development. In this particular case we complemented all of the nighttime lights, and some of the other remote sensing measures with land use measures drawn from multi spectral imagery.
Um Kunar will tell you a lot more about that in just a little bit today. But um! This particular example is just meant to show that the changes in the amount of built up area, for example around Kabul over time that we could detect with the daytime multi-spectral imagery,
and then link that together with the locations of the distinct Landmine hazardous areas. and their clearance. in this yeah across Afghanistan. But in this particular snapshot around,
Seth will talk to you in a little bit. A lot more about how we actually use that geospatial data about the particular interventions like the clearance of the landlines. So this is just meant to give you kind of a range of the different sectors and types of interventions that we're now studying with all of this,
What does it actually take, though, to do a gie. What do we think about as that kind of key components that distinguish a G from other designs? First of all, very key ingredient, almost always. The first thing that we focus on is the program data we have to have well specified geographic features.
It's. The intervention itself has to have kind of well understood geographic reach or extent.  And then that has to be captured in G. I. That it can be, you know, well interpreted. We also often need details about the timing of the intervention
by individual units. So, as you've just seen in all these examples, we have the staggered roll out of these different interventions across space and across time. And often we're leveraging that staggered design for the causal inference piece.
So those are two very crucial components of the program data that we always need in A.
We then link that, together with Geo-referenced outcomes
some of these outcomes, as you see, are remotely sensed from satellite imagery or other sort sensors.
Some of they are secondary data from things like the demographic and health surveys or living standards. Measurement surveys or primary data collection. That is also Geo-reference that you're commissioning or getting from another partner.
Um a series of key features, though, again, are very important to look at for all of these outcomes.
So you know, almost always we first focus on the spatial resolution and spatial coverage of the data. Is it fine enough to look at the specific extent of the effects that we expect from this particular program.
Um, and how much coverage kind of get us over a particular geography
also really important to think about the temporal resolution and coverage. Is it um? Is it a a recurring enough measurements that we can trace out the effects again over a fine enough time period. And how far back does this measurement go?
Um! That's one of the first kind of constraints. We often face lots of exciting new sensors, and new data has come online in the last five or six years. But of course that means If you want to go back further in time, you have to rely on some of the existing sensors which definitely have some opportunities, but also some constraints.
And then the last piece we have to integrate is our causal inference method, and thinking about how we can leverage often this against spatial and temporal variation in the rollout, maybe using some discontinuities. Matching methods to account for remaining potential confounds,
sometimes using all of these in combination. Some of the things we won't have as much time to talk about, but which we are excited about using more in the future, or also things like synthetic control methods. and kind of other designs that might be combined with some of these features.
Putting all of this together, can often get us at a rigorous design, even when it's impractical or unethical, or just not feasible to randomize
um, partly because often we're using secondary data or data that's already been collected and available.
This can be cheaper and faster to do than leveraging primary data collection, especially if you're doing that over a multi year period,
and partly as a result of the availability of this data we can often do things retrospectively and remotely, all of which, again contributes to the lower cost. Often of doing this,
we because we also often have larger scales with which to study this again. Much of this secondary data, whether it's remotely sensed or survey based, is a pretty wide geographic scope.
Um. And so that lets us better study geographic variation in the impact. So do real heterogeneity, analysis,
and better understand how different parts of the country might be benefiting or not from a particular intervention. Different ethnic groups might be benefiting or not from a particular intervention,
just as the geographic scope of this measurement can get us this heterogeneity analysis. We can also use often the long time scales that are available from this particular, you know, from these sources to look at longer run impacts than we often do in other evaluation designs, you know. Sometimes the evaluations are just things we have to do at the end of the particular project funding
um. But what we'd really like to do is to trace out five a ten years after the fact, or the effects persistent, or, in fact, did it take five years for the effects to really materialize to a scale that would be cost effective? Um,
we can do a lot more of that with
again. Part of what you've seen in the examples are cases where a particular donor funded a specific project and we're looking at the impact of that particular project.
In other cases these are really portfolios where different donors have combined their efforts to mix the funding into a country scale effort. Certainly the case in that Afghanistan landmark clearance,
or the Brazilian indigenous land rights formalization.
And so this lets us scale up also the studies to being portfolio level studies, and really, I think, much more powerful and representative in that sense.
And just want to emphasize. And I don't think I can emphasize this enough, although we'll be talking a lot about quasi experimental designs and remotely sense data. All of this can be paired, together with qualitative work on the ground, and even ourct designs,
all of which, I think is super complementary. It makes all of this all of those individual tools much more powerful when when combined.
Certainly lots of things we can't answer from space. and lots of things we can't answer with the quasi- experimental design. you know all of which I think is is more powerful when we do this in combination.

‍