Contrasting methodologies and competencies - the BBC and HRW
Tesfaye Hailu April 12, 2013 email@example.com
In 2010/11 the BBC carried out a detailed analysis of the situation of the media in Somalia. It was a surprisingly detailed piece of work, carried out in the most difficult of circumstances. Indeed, it would be hard to imagine when or where it might be harder to carry out any form of survey than Somalia in 2010-2011, with many areas affected by substantial conflict and levels of violence seriously increased by extremism, a volatile political situation, an almost total lack of government control, and with the country having a thoroughly deserved reputation for being one of the deadliest countries in the world for journalists. In sum it was a region that suffered from near complete lawlessness, a lack of institutions, severely limited infrastructure and almost no resources. Despite all this, the BBC still managed to get into the country and carry out extensive research on the ground, providing a valuable example of the sort of methodology needed to produce reputable research in almost impossibly difficult circumstances.
The BBC’s investigation involved extensive primary research across most of the country as well as outside. It was carefully planned and structured, developed to produce a clear understanding of how things operated at different levels, from the highest levels of legislation and regulation down to individual audience engagement and participation. It included preliminary scoping followed by detailed research on the ground in cooperation with various Somali organizations inside Somalia. It included partnerships with reliable local internal sources, including members of universities, the setting up of audience and journalist focus groups, in-depth interviews with managers, journalists and other staff in local media organizations, and recording of all interviews. Additional knowledge and information was gathered through desk research and consultations with relevant organizations.
It was a complicated and slow process, requiring careful planning, sensitivity and use of researchers who were local to the area. The BBC argues that recruiting local researchers should be a major element of any fieldwork as they alone had an understanding of the context in which they were working and could tailor their approach to the social norms, customs and changing security situations of a specific region. As a result despite all the security issues and suspicions, the BBC managed to recruit hundreds of people for focus groups, using personal networks and contacts with local organizations – all necessary to ensure proper ethical standards of research.
The BBC noted that analysis of its interviews, all recorded and transcribed for later analysis (and/or translation), and available for evaluation and checking by independent sources, involves consideration of the effect that political views, systems and broader social structures make to informants' attitudes, as well as evaluation of the system, of policy and decision makers of wider administrative elements. The BBC also suggests specific identification of key stakeholders and relevant issues are important at every level of investigation. It concluded that a minimum of six months fieldwork was necessary to get realistic evidence in such a controversial area. It actually took well over a year. It also emphasized people must be trained in methodology and techniques of research and this applies not just to people hired locally on the ground, but even more to those coming from the headquarters of outside organizations as these often believe, quite wrongly, they are proficient in the required techniques.
Overall, the BBC’s research included fifteen interviews with experts on the system, dozens of meetings with Media Organizations, TV and Radio, and both print and online organizations as well as relevant journalist associations on operational activities, several dozen interviews with journalists, both in and out of the country, over 30 telephone interviews with radio and TV station managers, heads of newspapers and online organizations, and then specific and detailed case studies of six radio stations (involving in-depth analysis of program schedules, interviews with some 40 journalists and managers in each station as well as the creation of four focus groups, and content analysis of the online and radio output). The study also involved the creation of another 24 focus groups with some hundreds of people around the country. The focus groups which alone involved some hundreds of people carefully recruited through university researchers’ structured recruitment questionnaires to select suitable participants according to age, gender and listening/reading habits. Informed consent was obtained from all participants before recruitment interviews and focus groups were conducted. Participants were briefed on the nature and purpose of the research and consent was given. Discussions were recorded and transcribed for later analysis. The make-up of groups took into account socio-cultural structures of Somali society and previous experience of conducting research in this context.
The contrast with the way HRW operates when 'investigating' an issue is striking and indeed could hardly be greater. HRW gave an account of its own methodology in its January report on the ‘alleged’ violations of the villagization program in Gambella Regional State. One can only say ‘alleged’, because HRW still continues to refuse to allow anyone access to confirm who it has talked to or where, citing supposed security concerns and the supposedly challenging difficulties of research in Ethiopia. For the same supposed reason it fails to identify any sites of ‘claimed' incidents. Given what the BBC managed to achieve in the far more difficult conditions in Somalia especially in areas controlled by Al-Shabaab, this is distinctly unimpressive. It is, after all, quite possible to visit Gambella Regional State freely, to talk to people on the ground and in the villages, both new and old. And people do just that.
In its January report, Human Rights Watch confined itself to a claimed 100 interviews which it says it carried out over four weeks in Ethiopia and a week in the Dadaab refugee camp in Kenya and in Nairobi. It carefully refuses to say how many of these interviews were in fact in Ethiopia and how many were outside the country. Indeed, it deliberately attempts to conceal the certainty that the majority were done outside Ethiopia by adding that the interviewees came “from across the Gambella region” (without actually saying that any of the interviews themselves were actually done within the Gambella region). It also suggests the number was considerable by using the journalistic trick of not specifying actual figures: “including community leaders, students, non-government organization workers, and former government officials”. This sort of phraseology is commonly used to cover a reality of no more than one of each category at most. HRW also claims to have visited a total of five districts (unnamed) and 16 villages (also unmanned and out of many hundreds actually involved). In fact, there is no actual evidence that HRW interviewed more than a tiny fraction of those to whom it claims to have talked to or that it spoke to anyone in the Gambella region itself.
The most notable omission is, as usual, the complete failure to make any effort to establish the political affiliation of any contacts, interviewees or of translators used. This is such a glaring omission that one can only assume it is deliberate. In fact, it is something that is very apparent in all reports that HRW have written on Ethiopia, even those which actually cover overtly political issues including elections. One classic example of this, and of the failure to investigate sources, came in the HRW report on the Ogaden in 2008, another report sourced and ‘researched’ entirely externally. In that, HRW didn't bother to note that almost all its interviewees, many from the Dadaab refugee camp in Kenya, came from the two or three sub-clans of the Mohamed Zubir which provide nearly all support for the Ogaden National Liberation Front, an organization involved in armed struggle and widespread terrorist activity against both the Somali Regional State and the Federal Government. HRW made no effort to check the clan affiliation of its informants; it never even asked if any were members or supporters of the ONLF. Nor apparently did it seem to understand that in a situation with close Somali sub-clan linkages, like that in the Dadaab refugee camp, the fact that two different people tell the same story is no indication of confirmation; rather indeed the reverse.
HRW spends a good deal of effort to try to suggest the extraordinary effort it, and its foreign researchers (no Ethiopians were apparently involved in the writing, editing or reviewing of the report), made to collect its information. It complains this is because of laws limiting NGO activity, restrictions on media activity, intolerance of political dissent, and the intimidation and fear generated by government officials. As so often pointed out before (and repeatedly ignored by HRW), NGOs can and do operate freely in the Gambella Regional State, newspapers in Addis Ababa are often distinctly critical, there are registered opposition political parties and tens of thousands of people have voluntarily moved into improved villages. The political space may not be as wide as some would like, but there is actually no resemblance to the claims and allegations of HRW.
HRW is, of course, handicapped by the fact that it appears to insist on confining its interviews to people that it knows in advance will support previously decided positions. Leading questions are a specialty. Conclusions have been determined in advance; any evidence that disagrees with these is merely ignored or denied. It certainly simplifies the process, of course, even if it leads to conclusions which are unsupported by facts. HRW's brief, however, is investigation of the reality of the situation it ‘knows’ to exist; it is to provide ‘evidence’ for a report whose conclusions HRW might well have written in advance, as is quite clear from discussions with HRW personnel. There is no indication that HRW ever considers its job to be investigation of allegations or claims. This, it appears, is not what an ‘advocacy’ organization does. It is not to establish the truth of the situation – merely to report ‘allegations’, and usually, it might be noted, in such a way as to benefit its own fund-raising. It is also worth noting that recommendations, usually drawn up with an eye on the potential response from fund-raising, totally ignore the extremely serious possible effect on the people at grass-roots levels who would be affected if , for example, HRW managed to bring about its declared aim of ending aid to Ethiopia. HRW may deny any such intention, but this would be the quite deliberate effect of its efforts. Similarly, a recent comment on Rwanda, which underlined the “hubris” of HRA’s chairman, Kenneth Roth, and the organization’s lack of accountability to anyone, pointed out that “Human Rights Watch’s campaign against the government of Rwanda has powerful implications on that country’s tourism, trade, investment and aid – all of which impact significantly on the livelihoods of the people of Rwanda.” The comment asked pointedly: Does HRW care?
The contrast between HRW’s approach to research and the BBC’s leads one to wonder:
- Why HRW never indicates any attempt to contact people inside Ethiopia through Facebook, Twitter or other social media, despite the fact that mobile phones are now relatively widely used in Ethiopia with numbers currently up to 4 million. This may not compare well with Kenya or some other African countries, but it’s a very possible resource.
- Why HRW effectively confines itself to making contact with members of opposition parties, frequently those advocating armed struggle with little or no representation inside the country.
- Why HRW consistently fails to make any effort to check the political affiliation of its sources, or of its interviewees. This, of course, allows it to claim it has interviewed people from all shades of opinion.
- Why HRW does not consider that personal and political interests often take precedence over the provision of objective and unbiased information.
- Why HRW apparently makes no effort to train its researchers - they have so little understanding of proper research techniques they apparently believe that satellite films showing villages that have been burnt at some unspecified point in time, actually “prove” that they were burnt by an identifiable and specific group
- Why HRW has never considered critically evaluating sources and the information provided. Indeed, why it doesn't use empirical information.
- Why HRW has made no effort to carry out even the most basic checks on the reliability of its sources or on the accuracy of the material provided, or, in Ethiopia, made any real effort to work on the ground or use reliable local information,
- Why HRW consistently does no more than merely try to support previously decided conclusions.
- Why HRW consistently denies that any criticisms of its activities deserve attention, and dismisses with apparent contempt any evidence that disagrees with its own conclusions, however well sourced or however accurate: limiting itself to the mantra: we are right – you are wrong.
- Why HRW has never admitted to any error, despite the undoubted fact that it has frequently made mistakes of both fact and analysis, let alone issues of judgment or interpretation.
One could go on, but the failures, underlined by the undoubted excellence of the BBC's approach, are sufficient to make it clear that the level of HRW's work is far below anything that would be acceptable to any serious academic or research organization. HRW can only rely for its reputation on its own pride and on the frequent tendency to assume that human rights advocacy reports are accurate. Some may be, but all too often, as HRW’s own Chairman Emeritus suggested not so long ago, the arrogance of HRW’s current leadership and its continuous methodological failures render its work unpleasantly nasty and ultimately valueless.