Metrics can provide deep insights into our open source projects, but they can also be overwhelming, and because every open source project and every situation is a bit different, metrics require interpretation. And that’s only the first step. Interpreting the metrics and figuring out what they are telling you is only useful if you take the additional step of driving appropriate actions to make decisions and improvements based on those metrics. I’ve been working in open source and using metrics for decades, but for me the metrics have never been what’s important. They are simply the means to an end, a way to make improvements or demonstrate whether we are accomplishing our goals.
This is why I focused on creating the CHAOSS Practitioner Guide series during my three years as CHAOSS Director of Data Science, and I’m proud of what we, as a community, accomplished with these guides. I’ve written quite a few blog posts about the individual guides, but I wanted to take some time here to reflect on the guides and talk about how they can be used together as a cohesive whole. The guides roughly break into two categories, which I’ll address in a series of four blog posts. The first category focuses on health and sustainability of open source projects. This post is about the first category, ‘Building Healthy and Sustainable Open Source Projects’, with part 2 coming next week to cover ‘Additional Sustainability Topics From the CHAOSS Practitioner Guides’. The second category of guides are the more advanced ones dealing with complex topics, like value, viability, and impact, that span multiple projects. Part 3 in the series focuses on value and viability while part 4 will cover determining impact.
Overview
We launched the Practitioner Guide series with a few getting started topics that apply to almost every open source project: contributor sustainability, responsiveness, and organizational participation.
The guides are paired with as ‘Introduction – Things to Think about When Interpreting Metrics’ as a starting guide with more details about how to interpret metrics in light of your unique situation. The introduction has general suggestions about how you should spend time understanding the goals of the project and talking to the people working within the project as key inputs into interpreting metrics. When it comes time to look at the metrics, focusing on trends over time for just a few metrics can help cut through the noise before drilling down into the details where there might be a potential issue. This is why each getting started guide focuses on just a few metrics to get started with tips for expanding to additional metrics as needed.
Every project is a little different, so it’s essential to interpret the metrics in light of a project’s individual needs and ways of operating. It’s also important to remember that metrics are more than just data, they represent actual human beings, so it’s important to be careful about comparing people against each other in ways that might result in the punishment of individuals. The non-human data (e.g., bots, automation, AI) should also be carefully taken into account in your interpretation of metrics. The Introduction – Things to Think about When Interpreting Metrics guide has more details about how to think about metrics and use what you’ve learned productively and ethically.
Contributor Sustainability
The Practitioner Guide: Getting Started with Contributor Sustainability focuses not just on determining whether you have enough contributors to sustain a project over the long-term, it also has tips for increasing contributor sustainability while keeping in mind that new contributors can also create additional burdens on the time of already overloaded maintainers. The guide includes Contributor Absence Factor, Contributors, and Types of Contributions as the key metrics to start investigating contributor sustainability.
To improve contributor sustainability, here are just a few of the recommendations from the guide:
- Start by looking at your existing contributor base to determine whether you have people who are ready to step into a maintainer role, even if they start by maintaining just a sub-project. If there are people who are almost ready, some mentoring or reviewer roles might be a good first step.
- Before starting to recruit new contributors, it might help to review and update existing contributor guides and other onboarding documentation to help people get up to speed quickly while requiring less help from existing maintainers.
- Good first issues and help wanted labels can help new contributors get started on something productive, especially if those issues are created with the information that a new contributor needs to know to accomplish the task.
- Be proactive about asking people for help with specific tasks. Knowing that we’re wanted and appreciated makes us feel good, which can be a strong motivator to contribute to an open source project or to continue contributing.
- Defining the roles and responsibilities for contributors, reviewers, and maintainers as a contributor ladder where contributors can climb up to become reviewers and those reviewers can become maintainers can help with recruiting new people into these roles.
- Think about how people can move into leadership positions to be responsible for documentation, community management, marketing and other important roles.
Responsiveness
Responsiveness is one of the most important factors in attracting newcomers and retaining existing contributors for a project. The Practitioner Guide: Getting Started with Responsiveness uses Time to First Response, Time to Close, and Change Request Closure Ratio as the key metrics to help determine whether a project is responding to requests in a timely manner. It can be tempting to attempt to solve issues with responsiveness by putting more pressure on existing maintainers by asking them to respond more quickly and resolve more contributions, but this rarely solves the long-term problem. It might result in short-term gains, but it could be damaging to the community and the project over time if you are burning out your maintainers by not resolving the underlying problems that are causing the lack of responsiveness in the first place.
Some suggestions for improving responsiveness from the guide include:
- As suggested in the contributor sustainability guide, start by looking at whether you can promote some existing contributors into reviewer or maintainer roles.
- Use the project’s contributing documentation to set expectations about when to expect a response to a contribution.
- Use issue and change request templates to help contributors make good contributions that require less work from maintainers.
- Talk to your maintainers and find out where they are spending too much of their time and focus documentation or recruitment in those targeted areas.
Note that responsiveness is hard to diagnose because many things can impact responsiveness, so don’t be discouraged if your first attempt doesn’t result in improvement.
Organizational Participation
Organizations can have a significant impact on the health and sustainability of an open source project. On the one hand, organizations can help sustain projects over time by employing people to work on projects, but they also introduce risk (e.g., from relicensing, strategy shifts, acquisitions) when one organization is too dominant. From a contribution standpoint, it can be difficult if one organization has all of the influence and other people don’t feel like they are participating as equals.
The Practitioner Guide: Getting Started with Organizational Participation uses three primary metrics, Organizational Influence, Organizational Diversity, and Elephant Factor, to understand organizational participation.
The recommendations for improving organizational participation depend on whether the improvement is coming from the dominant organization. From the dominant organization:
- Contribution documents should be clear about whether contributions are accepted from people outside of the organization and whether those people can move into leadership positions. Transparency and setting expectations can reduce frustration from other contributors.
- Spend some time thinking about why the project isn’t getting contributions from other organizations. Is all of the work happening in the open, and can others easily find decisions and discussions in public channels? If not, some work might be required to redirect private conversations into the public channels.
- Promote opportunities to contribute via social media, conferences, and other channels.
- Use your existing connections to other organizations who are using the project to encourage specific people to contribute.
From outside of the dominant organization:
- Engage in the project to determine whether contributions are really welcome.
- If so, encourage your employees to contribute to the project as part of their time at work. Don’t expect employees to contribute to work projects in their free time.
Being transparent is critical. Saying one thing in the documentation and doing another can damage your organization’s reputation more than just being honest and transparent about how people can (or cannot) contribute to a project. If you are considering using an open source project as a key component of your organization’s products or infrastructure, you should think very carefully about that decision when that project is controlled by a single organization.
Summary
Increasing contributor sustainability, improving responsiveness, and garnering participation from a variety of organizations can all help improve the health and sustainability for an open source project. Check back next week for the second blog post in this series about security, diverse leadership, and sunsetting an open source project. As always, if you want feedback or help with open source strategy, sustainability, governance, or related open source topics, I’m available for consulting engagements.
Additional Resources:
- CHAOSS Practitioner Guides
- Beyond the Repository by Amanda Casari, Julia Ferraioli, and Juniper Lovato
- From Data to Action: Building Healthy and Sustainable Open Source Projects in IEEE’s Computer, pp.74-78.
- Contributor Sustainability Impacts Risk and Adoption of OSS Projects
- Strategy Before Metrics
- Using CHAOSS Practitioner Guides to Improve your OSS Projects
- What can your OSPO do about power dynamics, rug pulls, and other corporate impacts on OSS sustainability?
As with everything I write, this was written by a human without the use of LLMs or other AI assistance.

