Posted by Jaideep Gupte
We have now completed four out of nine districts[1] (my earlier post describing our panel study can be found here), although for logistical and security reasons, by the time this is posted, our fieldwork would already have been completed.
In the excitement and busy schedule of our fieldwork, it is easy to forget that some of the areas we are visiting, or travelling through, are not the safest. This afternoon as we were travelling between districts on a quiet stretch of road, our jeep was stopped by a gang of 10-15 young men (some were boys) who demanded a payment to let us through. From what I could see, they were unarmed. It was a routine I had seen before: a few of the boys lay down across the tarmac, a few raised their arms up high as if in prayer, while the others linked arms and crowded the single lane road to bring us to a halt. As soon as we did, the gang held on to the grill of our jeep till we opened the windows to talk. They claimed the payment was to be put towards the upcoming celebrations for Ambedkar Jayanti. However, the annual remembrance celebrations of Dr. Ambedkar, a prominent Dalit leader, were more than a fortnight away. Besides their initial ‘primordial’ display, the gang was very articulate, and talked to us in a calm manner; they even gave us a receipt for our payment. Our seasoned driver handled the negotiation calmly, and told us afterwards that all such proceeds would most likely be used to acquire tharra (moonshine).
Attrition, or the fact that households initially included in the sample are missed in subsequent waves, is the scourge of panel studies. This is because attrition in social surveys is rarely ‘at random’, and can therefore quickly eat away at the validity and reliability of the data. Because our panel study on household vulnerability to civil violence is the first of its kind in India, we had very little prior knowledge to estimate the magnitude of attrition rates we would face. Other panel study’s in India have experienced around 20% attrition,[2] but these are much larger samples over much longer periods of time. We are aiming for half that – a tall ask, specially given the urban and peri-urban, and at times, unstable nature of our sample.
We are tracing 1089 households, but a large part of our study also uses the neighbourhood as unit of analysis. For one, we are interested in seeing how neighbourhood level dynamics interact with household vulnerability. It is therefore important to maintain households as well as neighbourhoods within our sample. For this reason, our tracing protocol is slightly different from a standard household panel in that if we find that a respondent household has moved out of our neighbourhood, it makes little sense in us tracing that household.[3] We treat that household as a lost household, and replace it. We only trace relocated households if they have moved within or near the original neighbourhood. We also often find the exact house, but with nobody at home – in these instances, we wait; as long as our budget allows. In some sites, when neighbours mentioned that members of our respondent household had gone to the market, or were away temporarily, supervisors returned to a house four or five times in a day to check if the family had returned. This is repeated on following days if the team is surveying sites in the same city or district.
Overall, we are managing to keep attrition to a bare minimum – we have only replaced a handful of households. A few lessons we’ve learned along the way:
- Nearly all of our successful traces are due to our GIS information.
- Why GIS and not GPS? GPS is a ‘positioning system’ technology which uses satellites to triangulate position, while GIS refers to tools which map ‘geographic information’, which may or may not use information from satellites. In dense slum areas or inside tenement blocks in state-provided housing projects, for example, GPS is of little use in finding the exact door at which to make our survey call. A few meters here or there, means you end up walking down the wrong lane, and it can be extremely time-consuming to re-trace steps. We therefore relied on GIS maps.
Neighbourhood mapping: addresses in low-income neighbourhoods in India are monotone, particularly so in informal settlements. At times, several hundred families will all identify their residential address by a common local landmark (a shop, mosque, temple, cross-road, or “under flyover bridge” for example). So neighbourhood level mapping becomes essential. During the first wave, we painstakingly mapped each survey site, marking down most visible land-marks and adding in ‘voting booth zone’ demarcation information used by the Election Commission. This information proved to be highly effective on the ground, since even the smallest visible landmarks (‘the Banyan tree’, ‘the well’, or ‘small statue/idol’) are much easier to locate than street names. Local residents often wouldn’t know the names of the streets we were enquiring about, but could easily point out these visible land marks.
- Ensuring adherence to best-practice at the survey site: it can not be stressed enough how arduous tracing households is. The temperature in interior Maharashtra where we are is not helping – it is already above 40oC, and likely to reach 50oC in the coming weeks. We therefore had to think carefully about building in incentives to ensure attrition rates were kept to a minimum, and no households were lost due to inadequacies on our part.
- Replacement households are a good incentive strategy. We are requiring that the size of our sample is maintained through replacement households. Not only do these replacement households function as a second stage baseline for subsequent survey rounds, they also ensure that there is no incentive to report a household as missing. In fact, a completely new household requires more effort on the part of the enumerator since introductions, building rapport, and the household roster all have to be done from scratch. The flip side of this is that for genuine replacements, we need to ensure that the enumerator has adequate support from other team members or the supervisor if necessary.
- Anonymous maps – originally, our plan was to provide the enumeration team with site maps which did not show respondent household codes. The thinking was that households which appeared ‘inaccessible’ or in hard to reach areas, should not automatically be identified as such, and thus create an incentive for them to be marked as missing. However, we quickly learned that this was not working, and was counterproductively increasing the logistical workload for the enumeration team. Providing maps which displayed respondent household codes meant the team knew exactly where to look for each respondent, and this freed up much time for other tracing activities.
Daily data reliability check – We’re using a mobile data capture program on Android tablets to collect our survey data (I’ve described this in my earlier post which can be found here). As soon as the day’s data is downloaded, we begin a series of reliability checks on the data – including ensuring all target households have been found or replaced, changes to household composition have been marked correctly, and several other consistency checks. We often don’t finish before midnight, but we are finding it essential to complete the checks on the day to ensure the task doesn’t pile up, and that any inconsistencies can be corrected before we travel to the next site. We also found it important to work with enumerators to correct mistakes while the interview was still fresh in their memory. We found a clear indication that doing this improved the overall quality of the data day by day. - Enumerator recall. While we placed emphasis on re-hiring much of our team from the first phase, we did not rely on team members to recall the locations of sites or households. When they did remember, it was good to be able to triangulate with the GIS information. But in general, since each enumerator literally visits hundreds of households, and is so focused on the mechanics of the survey instrument, we felt it would over burden them to keep track of household locations as well. It worked well to keep the tracing duties purely within the supervisorial team, while the enumerators were simply shown the doors at which calls were to be conducted.
For the most part, our plans have fallen into place, and fieldwork completion is eminent in the coming weeks. My colleague from IDS, Jean-Pierre Tranchant has joined the travelling party in interior Maharashtra, while Patricia Justino reports that the enumeration team on the western coast has completed its sites. Next, our team moves on to conducting ‘interaction games’ with a sub-set of our respondents. We are all very excited about including this new and innovative method to study cooperation, risk-aversion and trust into our study – Jean-Pierre blogs on this next week, so watch this space!
[1] With Patricia Justino and Jean-Pierre Tranchant. Funded by the European Commission (www.microconflict.eu) and the DFID-ESRC (“Agency and Governance in Contexts of Civil Conflict” http://www.esrc.ac.uk/my-esrc/grants/RES-167-25-0481/read).
[2] For example, the four rounds of the Rural Economic and Demographic Survey (REDS) in India between 1971 and 2006 experienced 17.5% attrition.
[3] Budget constraints meant we could not do neighbourhood level surveys of the new neighbourhoods, and could only contact the relocated household if they had provided a contact mobile phone number to ask a basic set of questions on their circumstance.
MICROCON 