The Dangers Of Poor Research

In the last year, Ashoka and Intellecap’s Beyond Profit have both conducted surveys of social entrepreneurs. Ashoka describes itself as the global association of the world’s leading social entrepreneurs, whom it elects as Fellows. The Ashoka survey was conducted to understand how their Fellows have changed systems. This phrasing already suggests a bias that Ashoka Fellows have changed systems, one that is carried through in the way the survey was designed and conducted.

Beyond Profit is the social enterprise magazine of Intellecap, a social investment advisory firm. Their survey was conducted to better understand social enterprises, and the people who lead them, in India. While India is described as having a high degree of social entrepreneurship, it lags behind in research in this field. Therefore, initiatives to conduct research on social entrepreneurship in India are much needed. However, this initiative by Beyond Profit, along with the survey by Ashoka, are marred by weak research methods.

The first limitation to both the Beyond Profit and Ashoka surveys is that neither of them go beyond percentages in reporting on the results. For example, the Beyond Profit survey reports on the percentage of respondents who are men (and women), who fall within a certain age group, and whose work falls within a certain sector. The Ashoka survey says of their Fellows that, “these people are incredibly focused on achieving their goals with 93% pursuing their original objective after 10 years. 80% of them are seen as leaders in their field and 90+% of their ideas are replicated by other groups” (4/6). The Ashoka survey also calculates the percentage of Fellows who have changed the system in one of five ways.

The problem is that we have no way of knowing if these results are due to chance. If the authors had conducted a chi square or t-test on their results, it would give us the probability that these results are due to chance. For example, a probability of 0.001 would mean that the results are highly significant in statistical terms, that is, the results are very probably true.

In addition, these tests only work if you have a random sample, and there is good reason to believe that in neither the Ashoka nor Beyond Profits surveys was this the case. Out of all the Ashoka Fellows elected in 1998,1999, 2003 and 2004, the total number of Fellows with current contact information from those years is 315. 172 of those Fellows returned surveys, and this was used to calculate a response rate of 55%. However, we don’t know how many Fellows current contact information is not available for. This is especially important because it is quite likely that the Fellows for whom current contact information is not available may be those whose social enterprises have closed down or are inactive.

The Beyond Profit survey was conducted using an online platform, and was distributed to Intellecap’s database of social entrepreneurs in India, as well as to the networks of Ashoka, Dasra and Unltd India. The sample size for this survey is 118, as that is the number of people who responded. However, as readers we do not know the universe from which this sample was selected. How many people was the survey distributed to? Do Intellecap’s database, as well as the networks of Ashoka, Dasra and Unltd India, cover all social entrepreneurs in India? Alternatively, was the survey only distributed to a sample of social entrepreneurs to begin with?

As it was up to these social entrepreneurs to respond to the survey, it is quite likely that all those who did are similar to one another in some way. For example, they might be all in a younger age group, and therefore more comfortable with using the Internet to respond to surveys. Therefore this sample is unlikely to be random, and most probably suffers from what is known as self-selection bias.

There is one section in the Beyond Profit survey in which it is acknowledged that those omitted from the survey are likely to have influenced its results. The report states that, “One element to keep in mind is what the data doesn’t tell us. Because we didn’t survey people who almost became entrepreneurs, but didn’t follow through because of negative reactions from family, it is difficult to judge just how prevalent family pressure is” (5/7). However, those omitted from the survey are likely to have influenced all of its results, and this is not acknowledged throughout most of the report.

As long as social enterprises in India do not have their own dedicated legal form(s), it may be difficult to know how many social entrepreneurs there actually are. In a context in which a comprehensive database of social entrepreneurs in India is not available, it makes sense for Intellecap to use their own database and other networks to contact potential respondents. In fact, this is a legitimate research method and is known as snowball sampling. Snowball sampling is suitable for qualitative research, where the main purpose is to gain a rich and complex understanding of a specific social context or phenomenon. The problem is that the Intellecap survey seems to use this method for quantitative research, where the emphasis is on eliciting data that can be generalized to other geographical areas or populations.

For example, the Intellecap survey says that “…there are actually more men than women in social enterprise today” (2/4). Similar statements, which generalize from the sample to the universe of Indian social entrepreneurs, are made throughout the report, including with regard to age, experience, motivation, revenue generation and sector. However, without a random sample, and without testing for the statistical significance of the findings, it is misleading to make these generalizations.

Another limitation that both the Ashoka and Beyond Profit surveys suffer from is the lack of triangulation. In the social sciences, triangulation refers to using more than two methods in a study to double (or triple) check the results. The Ashoka survey, in which Fellows were self-reporting on their achievements, could have certainly benefited from triangulation. In the Beyond Profit survey, triangulation could have been particularly useful in cross-checking certain pieces of information, such as on revenue generation.

A final area in which the Beyond Profit survey errs is in the statement:
…Not surprisingly, people from a for-profit background are more likely to choose a
for-profit structure for their own social enterprise. 63% of respondents who came
from a for-profit business background chose to work in a for-profit structure, while
only 17% of people with non-profit experience switched to a for-profit structure (3/5).
The claim that, “people from a for-profit background are more likely to choose a for-profit structure for their own social enterprise” leads us to believe that there is a correlation between the background of social entrepreneurs and the legal structure they choose for their social enterprises. However, establishing a correlation between these two phenomena requires regression analysis.

The simplest form of regression is linear. If you plot the data collected on a graph, regression analysis will create a single line that best summarizes the distribution of points. The typical distance between the line and all the points indicates whether the regression analysis has captured a relationship that is strong or weak. There is no evidence of regression analysis in the Beyond Profit report.

In addition, the statement discussed above begins with the words “not surprisingly”. This suggests that it is because a social entrepreneur used to work in a for-profit that he / she chose the same legal structure for his / her social enterprise. When one variable (in this case, legal structure of social enterprise) is inferred to be because of another variable (in this case, background of social entrepreneur), this is known as causation. Causation cannot be measured from this study because both the variables were measured together in a setting.

One of my former colleagues described the Ashoka report as impressive and inspiring, and I don’t mean to detract from the achievements of their Fellows by pointing out the weaknesses in the research. A strong research methodology would have made these achievements even more impressive, as they would have been supported by firm evidence.