A sea of blue forget me nots with a few peach and yellow tulips popping up in the middle of one of the flower beds in my back garden.
General

Part 4: Assessing Impact Using CHAOSS Practitioner Guides

This final post in my series about using the CHAOSS Practitioner Guides focuses on determining impact for two specific cases: Funding Impact Measurement and Research Software Impact. If you haven’t already read the previous posts in the series, I encourage you to pause and read those first:

Funding Impact

Much of the critical infrastructure that we all rely on is made up of open source projects that lack the resources to be properly maintained over the long term. Funders need to be able to understand the impacts of past funding in order to secure buy-in for future funding as well as adapting and/or innovating funding approaches whilst mitigating ineffective or even harmful approaches. We all benefit from more public institutions, philanthropic organizations, and companies giving money to open source; when done in a way that positive impact is the ultimate goal and objective.

The Practitioner Guide: Funding Impact Measurement has several things to consider when measuring impact:

  • Start by understanding the context. Accounting for funding objectives, project life stage and social structure, and regional and organizational cost factors is an important first step in measuring funding impact, since it provides important context.
  • Economic, social, and technological impacts can be both positive and negative, direct and indirect, internal (i.e. within a project) and external (i.e. among a project’s ecosystem of dependents and users), and manifest over various time horizons. When assessing the impacts of open source funding, there may be a tendency to focus on technological impacts, which may in part be due to the nature of open source or the relative ease of measuring data from repositories. However, the potential impacts of funding extend beyond the code itself, and it is crucial to consider economic and social impacts on projects and their wider ecosystems of dependents and users.
  • Methods. When it comes to measuring the impacts, funders can consider a breadth of methods for impact measurement. We recommend using a mixed methods approach, which provides a way to combine scalable quantitative measures along with contextual depth from qualitative data to better understand the funding impact.

This guide helps organizations measure the impact of their funding initiatives so that we can increase the funding to open source projects to drive future improvements and allow these projects, and the people working on them, to become healthier and more sustainable over time. Most of this guide came from a paper that I co-wrote with Cailean Osborne, Paul Sharratt, and  Mirko Boehm, so a big thank you to my co-authors and others from the CHAOSS Funding Impact Working Group for supporting this guide!

Research Software Impact

Looking at the impact of research software requires a very different approach from the corporate context, so we have a guide dedicated to that topic: Practitioner Guide: Research Software Impact. Software and open source community work does not have a history of being part of reappointment, tenure, and promotion cases and has not represented a traditional outcome of university research work. To make this even more complicated, there are no consistent international practices for using software identifiers or citing software. A huge thanks to Matt Germonprez and Clare Dillon for writing this one!

The guide highlights the importance of open source software and community work as significant scholarly outputs using several metrics for determining research software impact:

  • Contributor Absence Factor. For researchers and evaluators, the contributor absence factor is helpful in understanding how an open source project is growing to include contributors who are responsible for more than 50% of the code base.
  • New Contributors. A steady influx of new contributors may indicate a healthy, inviting project, while declines can be a signal for potential challenges in engagement or accessibility.
  • Project Velocity. A researcher can use the Project Velocity metric to report project velocity of their own open source project and compare project velocity across a portfolio of projects.
  • Change Requests. The Change Requests metric tracks the proposals for modifications to a project’s source code that have been submitted for review during a given time frame.
  • There are also additional metrics, like software citations and external funding, that may be used if they are available.

The guide focuses on the impact of open source software and/or communities created and maintained by individuals and teams as part of their research role, including evaluators who are part of researcher reappointment, tenure, and promotion cases. This guide focuses on open source software and communities developed and maintained by a researcher or team to recognize the social and technical impact from that work.

Summary

These are very different guides that both focus on assessing impact within slightly niche topics that only apply in specific situations. However, they would also be useful when brainstorming ways to assess impact in other situations. As always, if you want feedback or help with open source strategy, sustainability, governance, or related open source topics, I’m available for consulting engagements. In particular, I’m interested in working with organizations who need help assessing the impact of their open source funding initiatives.

Additional Resources:

As with everything I write, this was written by a human without the use of LLMs or other AI assistance.