General

Strategy Before Metrics

Two chess pieces facing off on a chess board

I’ve been involved with open source project metrics for a very long time, and people often ask me which metrics they should use, but this isn’t really a question that I can answer. What you measure and how you interpret those metrics depend on your goals and what your organization is trying to accomplish. For this blog post, I’m using “organization” loosely. It could mean your company, university, or non-profit, but it might also mean aligning with funding organizations if your OSPO or other open source efforts were funded by another organization. 

The CHAOSS Practitioner Guide: Introduction – Things to Think about When Interpreting Metrics mentions:

“one of the best places to start isn’t actually with the metrics, but by spending some time understanding the overall goals for the project. If the project is primarily driven by one organization or owned by an organization, you should also consider the goals for that organization. By thinking strategically about the overall goals, you’ll be in a better place to decide what you need to measure to determine whether the project is achieving its goals. Open source projects generate a tsunami of data that can be overwhelming, but by focusing on the goals, you can develop a metrics strategy that helps you focus on the metrics that matter most for a particular project.”

It can help to ask yourself, “what is important for my organization or the project?” This often means starting with your team’s open source strategy and aligning it with your organization’s goals. The most important part of putting together strategies and plans for your open source efforts is aligning them with the overall goals for your organization. By taking some time and effort to make sure you support the overall strategy for your organization, then it will be much easier to justify continuing these efforts during the next planning or funding cycle. This will also help you make the case to senior management, executives, funders, and others on the leadership team who aren’t likely to be involved in the low level details. Explaining how your open source contributions support the goals of your organization can help the executive or leadership team understand the strategic importance of this work so that you can continue your work in open source.

Once you have a strategy defined that aligns with the strategy of your organization, then you can figure out what you need to measure to show whether you are achieving your goals. There are a couple of reasons that starting with the goals is important, since metrics can go awry if you aren’t focused on the right things.

  • You won’t know if you are successful if you aren’t measuring the right things. If you aren’t measuring the things that are important for your project or organization, you won’t know if you are making progress in the areas that you care the most about. For example, if you want to improve the performance of a particular piece of open source software, you’ll want to have success criteria and measurement based on specific types of performance. If you want to gain influence within an open source project, maybe you measure increases in contributions or the number of employees moving into positions of influence. 
  • Measurement impacts behavior, and people do different things depending on what you measure. For example, if you publish metrics that focus on the number of comments on issues, you are likely to start getting more comments on issues. If what you are really trying to do is get people to help review and approve contributions, then additional comments on issues might not help as much as looking at reviews on change requests (pull requests / merge requests). 

Once you decide on your success criteria, you need to make sure that you can get the data required to measure it and start measuring it now to get a good baseline. There are plenty of tools available to gather contribution data about open source projects. Some of the commonly used tools can be found in the CHAOSS project, but you can also likely get a pretty good sense for the data by looking at your code repositories and other communication channels. GitHub, for example, has some pretty good data under the insights tab.

After you have your metrics, you need to actually do something with them to show that you are making progress toward accomplishing your goals. Think about which metrics you should be showing to your leadership and which ones should be shared with your team and the broader community. But communicating metrics is much more than just showing some charts or graphs, you also need to interpret those metrics and tell the story about what they mean. The CHAOSS Practitioner Guides can help you think about the interpretation and how you might tell the story about what your metrics mean. Without interpretation and explanation, all you have are numbers, which are way less powerful than the story about what the data means in the context of how it helps your organization achieve their goals. 

Here are a few additional links and resources to help you think about building your metrics strategy and telling the story about what the metrics mean:

Photo by Hassan Pasha on Unsplash.

General

UN Open Source Week Part 2: Digital Sovereignty and Resilience

Row of flags in front of the United Nations Headquarters in New York on a sunny day with green bushes in the foreground and skyscrapers in the background

This is the second, and final, blog post about United Nations Open Source Week, so if you didn’t read part 1, United Nations: OSPOs for Good Day, you might consider pausing and reading that post first, since it provides more context about the event and why the UN cares so much about open source software.

As part of the week’s activities, there were several side events on Friday, and I spent the day in the Digital Sovereignty and Resilience side event. This blog post has a short summary of the presentation, panels, and discussions, so these don’t necessarily represent my views and might contain factual errors. Speaker names and abstracts for each session can be found on the United Nations Open Source Week website on the side events tab.

The Digital Sovereignty and Resilience side event started by looking at ideas for building a sovereign digital workspace with short presentations about various open source workspace solutions. The French Interministerial Digital Directorate (DINUM) talked about how they have collected a set of tools (Le Suite) for their use and are also collaborating with Germany’s Zendis. There were also short presentations about specific projects that can make up a workspace solution: Matrix, Grist, and IREX (Institut du Retour d’EXperience). But it’s more than a set of tools, we need a better vision for how to build an open source digital workspace and integrate it across governments. It’s not about one tool to beat the others, but about working with companies and contributing within open source communities to help make sure that we have integrated solutions. This requires funding, especially when companies are involved. The UN is trying to coordinate across technologies to integrate solutions and improve collaboration, and they are also looking at potentially hosting some open source solutions – starting within the UN, but this is early days for this work, and we would need to expand it beyond internally within the UN to across countries. 

The next session was all about securing the supply chain through global collaboration, which is something we talk about all of the time within OSPOs and in the corporate world, but in light of the CRA, this is increasingly important for all of us. We need more collaboration across governments and other stakeholders (e.g., companies, organizations) to make sure that we can all work together to understand and improve the security of our open source software, and this is something that the UN might be able to help facilitate. We collaborate now, but it’s pretty ad hoc, and we need better public / private relationships / partnerships. We need more funding to be able to sustain these efforts beyond what groups like Alpha Omega and STF / STA are already funding for security initiatives to improve open source security. 

The folks from the Sovereign Tech Agency led a session called, Invisible Work, Critical Code – The Role of Maintainers in Open Source Digital Infrastructure where they talked about how open source is like a sewer, critical infrastructure that needs to be built and maintained over time. In the case of something like public water systems, railways, libraries, schools, and roads, these are usually maintained by governments paid for by our taxes. Digital infrastructure is fragile (obligatory XKCD), despite being the foundation of the digital economy, and governments should also be investing in critical and essential open source digital infrastructure in support of them as public goods. This was followed by a panel, which talked about how maintaining projects go way beyond just code and include things like conflict resolution, mentorship, communication, building community, and other invisible labor. Being inclusive of people with different needs helps level the power imbalances that we so often see in open source and in software in general. Being welcoming for the most vulnerable people creates spaces where people want to contribute and participate. Software is too important to be left to just developers. 

The final session in this Friday side event is Fostering Resiliency in the Digital Public Infrastructure (DPI). DPI is like a public library and needs to be resilient enough to endure over time and across administrations and economic conditions. We need to think about how we can ensure that the 100,000 components embedded in our software are safe and secure when many of these components aren’t actively maintained. This impacts us on a societal scale because these components are all part of our DPI. People need to be able to depend on DPI based on their use case to avoid societal harm. DPI and open source together allow us to recombine and reconfigure to use the software in ways that allow governments to better support their constituencies. Data integrity and trust are important for DPI across the many stakeholders involved in building and maintaining the software. Risk identification, risk understanding, and risk mitigation are steps toward building resilient DPI. Finally, there was an overview of ApeiroRA and the NeoNephos Foundation for next generation cloud infrastructures and services that strengthen digital sovereignty for Europe, but that could also be used by other regions. 

I really enjoyed the time I spent this week! The best part of the week were the conversations, and it was good to get out of my little open source bubble to talk to and better understand people who are working in contexts completely unlike my own. The UN did a great job of bringing people together from industry, government, not for profit organizations, and other groups from around the world with over 40 countries represented, including many from the global south. If we truly want to make open source software better serve the needs of our global communities, we need to collaborate across all of these groups, so a big thank you to the UN for bringing us all together!

General

United Nations: OSPOs for Good Day

United Nations: Welcome to Open Source Week #UNOpenSourceWeek

I was thrilled to be invited to attend the United Nations Open Source Week in New York joining a few hundred people from over 40 countries to talk about open source. 

The OSPOs for Good day on June 18, kicked off with a keynote by Amandeep Singh Gill, Under-Secretary-General and Special Envoy for Digital and Emerging Technologies, UN-ODET, who talked about how the UN sees open source as a foundation for digital transformation, but one that requires the UN to move beyond its historical focus on governments to engage multiple stakeholders. Looking around the room, you could see this reflected in the event attendees who represented governments, companies, universities, not for profit organizations, individuals, and more. Later in the day, Dmitry Mariyasin, Deputy Executive Secretary, UNECE, talked about how he sees open source as an inspiration for how the UN can operate more broadly in their role to create public goods using digital solutions to improve sustainability and increase transparency and trust. 

You can watch the livestream of the event, but here are a few highlights or common themes from the OSPOs for Good day.

  • Historically, many open source conversations were about cost savings, but this wasn’t the case at this event. The conversations have shifted to collaboration, innovation, and using open source to build trust.
  • Several of the people representing governments talked about how having an OSPO helped them build connections to other people involved in open source within their organizations while also allowing them to build connections and partnerships to collaborate and learn from others. Many countries view open source as being critical to their success as a country and in serving the needs and improving the livelihoods of their people.
  • The newly formed Trinidad and Tobago OSPO and Kenya OSPO are part of an initiative to create a replicable, scalable model of OSPOs across the global south with a goal of serving countries beyond just their governments to promote open source. 
  • Governments want to use open source, but struggle with procurement processes, which requires different policies to contribute, add new features, and maintain open source projects. Governments have been more willing to fund new features, but often struggle to fund the invisible maintenance of the software that everyone uses, which creates risks for the public that they represent. Germany’s Sovereign Tech Fund (now under the Sovereign Tech Agency) was created as one way to address these risks and make strategic investments in core digital infrastructure.

So far, I’m really enjoying the event and am looking forward to the next two days!

Update: I’ve followed this up with a second blog post: UN Open Source Week Part 2: Digital Sovereignty and Resilience.

General

Companies Can Mitigate Sustainability Risks

A bunch of people having a great time in a fun picture from CHAOSScon EU

I wrote a blog post earlier this week, Contributor Sustainability Impacts Risk and Adoption of OSS Projects, focused on helping maintainers and open source project leaders understand how companies view risk and how this impacts adoption of their projects. Marko Bevc commented on Bluesky that “there is another side of this coin while companies evaluate the risk (as they should), they should also look into how they are going to support those projects they use (either with contributions or/and other resources – e.g. funding).” This is a really important point, and it’s something that I always bring up when talking to companies about evaluating risk (20 min into this video, for example), so I decided to write a part 2 for the original post, but this time focused on what companies can do to mitigate contributor sustainability risks when adopting open source software.

As part of spending a lot of time over the years thinking about the sustainability of open source projects, I’ve given a bunch of talks (see Additional Resources section below) about how companies can assess the risk and viability of open source projects, but assessing viability is the beginning of the process, not the end. Understanding open source project viability is an ongoing process that needs to be monitored and decisions revisited as projects evolve. As a company, the best way to monitor the ongoing viability of a project is to have your employees contributing and participating within the project. This serves another important purpose beyond just monitoring. By having your employees participating in a project, you can help to continuously improve the viability of that project to increase the chances that it will continue to be viable over the longer-term. 

However, most companies use so many open source projects that you can’t possibly employ contributors to participate in all of them. Generally, I recommend that companies focus their contributions on strategic open source projects that are critical to your ability to deliver customer-facing products or services. For the other projects that you don’t contribute to directly, you might still be able to help them increase their viability in other ways, like through funding, for example. Some companies have funding programs where they fund key dependencies and other projects (e.g., FOSS Funders, Microsoft’s FOSS Fund, Bloomberg’s FOSS Fund). However, it’s also important to think about the impact of providing funding because throwing money at some projects can create friction within the project that can sometimes have a negative impact, while in other projects, funding can make a big difference in increasing viability. We discuss these and other funding issues regularly as part of the CHAOSS Funding Impact Measurement Working Group, and we also maintain a list of resources and research about funding.

To wrap this up, open source sustainability and viability are not something that you can think of as all or nothing. No project is perfect, and each project will have areas within them that are more or less sustainable. Companies can help make projects more sustainable and more viable over time by providing resources, like direct employee contribution and funding for those projects. 

Additional Resources:

General

Contributor Sustainability Impacts Risk and Adoption of OSS Projects

I’ve spent a lot of time over the years thinking about the sustainability of open source projects and the role that contributor sustainability plays in overall project sustainability. When I was co-chair of the CNCF Contributor Strategy Technical Advisory Group, contributor sustainability came up often as a concern for CNCF projects, and the most common question was about how to get more people contributing to our projects. This is a hard problem, but there are some resources at the bottom of this post to help grow your contributor base and increase the sustainability of your open source projects.

What I think many people underestimate is how contributor sustainability is viewed through the lens of risk by companies who are deciding whether to adopt your project. It’s easy to think that your project is different. No one will leave, and the project will be wildly successful forever, but that’s not how many companies think about open source adoption. Some companies think hard about which projects to adopt, especially if those technologies are crucial for delivering solutions to their customers, and would be difficult to replace if the project suddenly wasn’t available. Projects with a single dominant contributor or contributions coming almost entirely from a single company are going to be perceived as riskier and companies will be less likely to adopt or use those projects. This is especially true given the recent wave of companies relicensing open source projects and putting them under proprietary licenses. Put in simple terms, contributor sustainability risk makes it harder to get people to adopt your open source projects.

When I was Director of Open Source Community Strategy at VMware, I would often evaluate the risks of adopting specific open source projects, especially if we were considering building commercial products that incorporated those open source technologies in ways that were critical to delivering products to our customers. Contributor sustainability played a big role in deciding whether we would adopt a project. This was especially true for projects that were more strategically important for us, and which would be hard to replace if the project became unsustainable in the future. Given the choice, we’d select projects with better contributor sustainability, which would be a lower risk for us as a company.

Just last week, I was looking at an open source project where almost all of the contributions came from employees of the company driving the project, and there was a single lead developer who made the vast majority of the contributions and code reviews / approvals. That lead developer and their employer are single points of failure for the project. These single points of failure introduce risk for potential adopters and are likely to cause people to think twice before using a project. If I was a company looking for a solution, I would be unlikely to select a project that could suddenly cease to be updated (including security updates) if something happened to the dominant contributor or the company.

In summary, contributor risk stemming from a single person or a single employer makes your project riskier and less likely to be adopted.

While growing your contributor base is hard work, there are quite a few resources to help you improve contributor sustainability along with gaining a better understanding about how companies think about risk when adopting open source projects. Here are a few of those resources, most of which also have links to additional resources:

Update: You might also be interested in reading this follow up post: Companies Can Mitigate Sustainability Risks

Photo by Jan Kopřiva on Unsplash

General

CHAOSS Data Science Working Group

When I started in the role of Director of Data Science for CHAOSS, one of the first things I did was start the Data Science Working Group (WG) as a way to build community around the data science work that many of us were already doing within the CHAOSS project. I am incredibly proud of what we’ve accomplished in less than 2 years.

Yesterday, we published a CHAOSS blog post about what we’ve been working on lately, but here are a few highlights.

We’ve released 7 Practitioner Guides: Introduction, Contributor Sustainability, Responsiveness, Organizational Participation, Security, Building Diverse Leadership, and Sunsetting an Open Source Project. I’ve covered these in more detail in 2 recent blog posts about Using CHAOSS Practitioner Guides to Improve your OSS Projects and From Data to Action: Building Healthy and Sustainable Open Source Projects.

We are also driving several research projects out of the working group. I’ve already blogged about the Relicensing and Forks research that I’ve been working on, but we also have research looking into projects that move from private ownership into a foundation, archived projects, and a collection of research taxonomies.

You can read the CHAOSS blog post to learn more!

I also wanted to remind people that like all of the CHAOSS working groups, the Data Science WG is open to everyone! All you need to join the Data Science WG is an interest in using data to understand the open source world around us. Most of our work is analysis of data, writing guides, and discussions about using metrics. You don’t need any special skills, and you don’t need to know any advanced statistics, machine learning, or AI. We’re even planning a CHAOSS Data Science Hackathon, which will be  co-located with Open Source Summit North America and CHAOSScon in Denver, CO on June 26, 2025. To learn more, visit our repository, join our meetings, or reach out to us in the #wg-data-science channel in CHAOSS Slack. We hope you’ll join us!

General

From Data to Action: Building Healthy and Sustainable Open Source Projects

Hands holding dirt with a small green flower growing out of it.

Last week, I had an article published in Computer magazine, an IEEE publication: From Data to Action: Building Healthy and Sustainable Open Source Projects (the PDF version is in an easy to read format). One of the primary ways that the CHAOSS project is helping people improve the health and sustainability of their open source projects is via the Practitioner Guide series, which I covered in more detail in the Computer article.

On a related note, we’ve published two new Practitioner Guides this week: Getting Started with Sunsetting an Open Source Project and Getting Started with Building Diverse Leadership! These new guides complement the previously released guides that I talked about in a recent blog post. The rest of this post has a few more details about the Computer magazine article and the two new guides.

In the Computer magazine article, I talked about how the CHAOSS project is providing advice and resources for proactively using metrics to improve open source project health and sustainability before a crisis occurs to make software more sustainable and reliable for everyone. Here’s a short quote from the Computer magazine article:

“Building sustainable open source projects over the long term can be a challenge. Project leaders, maintainers, and contributors are busy people who don’t always have the time to focus on growing a community along with maintaining their software. Using metrics is one way to help identify potential issues and areas where a project can be improved to make it more sustainable over the long term. Metrics are best used if they aren’t used once and never again. By monitoring the data over time, projects can understand trends that might indicate areas for improvement as well as see if those improvements are having the desired effect. Being proactive about improving sustainability before it becomes a crisis can help make open source software more sustainable and reliable for everyone” – Read the rest of the IEEE Computer magazine article for more.

The newest guide in the series, Practitioner Guide: Getting Started with Building Diverse Leadership, was written by Peculiar C. Umeh. It expands on the theme of improving health and sustainability of open source projects by creating a welcoming and inclusive environment that encourages contributions from a wide variety of people. Here’s a quote from the guide:

“A community or project with diverse leadership offers significant advantages because diverse leadership leverages diverse perspectives to build an innovative community, create a welcoming and inclusive environment, and empower individuals from all backgrounds to contribute their unique talents. New and existing contributors feel more included when they can see other people in leadership positions who are like them (Linux Foundation, 2021). When diverse leaders collaborate, their intersection sparks innovation and creates a more harmonious global leadership system. It represents a global and diverse user base, which improves the usability of the project because more users’ voices are represented in decision-making about the project’s design and functionality. It enhances decision-making processes by incorporating various viewpoints and experiences, leading to better problem-solving and more effective strategies. It promotes a culture of inclusion and respect, improving morale and engagement among community members and ultimately contributing to projects’ long-term success and sustainability.” – Read the Practitioner Guide: Getting Started with Building Diverse Leadership for more.

The other new guide in the series, Practitioner Guide: Getting Started with Sunsetting an Open Source Project, is also about making open source more sustainable by being clear about the future of an open source project so that users can make responsible decisions and avoid using open source technologies that are no longer being maintained or updated with security fixes. Here’s a quote from the guide:

“Many open source projects, even widely used ones, become abandoned for a variety of reasons (e.g., evolving interests, family situations, employment changes), but abandonment can be done in a responsible way by proactively sunsetting the project (Miller et al. 2025). Sunsetting is an important consideration for corporate environments where it can be easy to lose track of projects that were created by employees who later walked away from the project and left if abandoned. You don’t want abandoned open source projects with security vulnerabilities sitting in your organization’s source code repositories where someone might trust that project simply because they trust your organization. Finding inactive projects and responsibly sunsetting them is a good business decision and something that many open source teams / Open Source Program Offices (OSPOs) do on a regular basis. It’s important to remember that not every open source project can or should exist forever: technologies evolve, corporate priorities change, and people’s interests change. Part of the beauty of open source is that we work in the open as we innovate, and some of those innovative projects will stand the test of time, while others should be responsibly deprecated via a sunset process. Sunsetting an open source project should take your user’s needs into account, and where possible, offer users time to migrate to a replacement technology. At a minimum, it’s important to signal that the project will no longer be maintained, updated, or have security patches so that users know that they should no longer be using the project.” – Read the Practitioner Guide: Getting Started with Sunsetting an Open Source Project for more.

As always, these CHAOSS guides are under an open source license, so you’re free to use and modify them to meet your needs.

Photo by Jennifer Delmarre on Unsplash.

General

New Power Dynamics in Open Source: Rug Pulls, Relicensing, and Forks

I’ve spent a lot of time over the past year doing research into open source projects that have moved to proprietary licenses and the forks that were the result of those license changes. More recently (starting with a talk at Monki Gras), I’ve been thinking about how the power dynamics within the open source ecosystem have evolved and how rug pulls, relicensing, and forks can shift those power dynamics.

I finally wrote all of this down and turned it into a blog post for The New Stack: Clouds, Code, and Control: The New Open Source Power Struggle. Here’s a short quote from the post:

“With the rise in popularity of large cloud providers, the open source power dynamics are looking kind of similar to the feudalism example I talked about at the beginning of this blog post, but in the open source case, what’s different is that we have ways to shift or flip the power dynamics. A smaller company deciding to move a project away from an open source license can flip the power dynamic and gain power back from those large cloud providers. Still, they also shift the balance of power even further away from contributors and users at the same time when they decide to relicense that project. This encourages those with less power to take collective action to fork a project, flipping the power dynamic in favor of the contributors and users, often including the cloud providers as users. Within the open source world, we are better off than the peasants and serfs because we have certain freedoms that allow us to take collective action to regain power by forking projects when others abuse their power.” – read the rest of the blog post on The New Stack.

If you want to learn more about the research, here are a few places to get started:

Photo by Lance Reis on Unsplash

General

Using CHAOSS Practitioner Guides to Improve your OSS Projects

Within the CHAOSS project, we know that people often struggle to make productive use of the tsunami of data about open source projects. One of my focus areas over the past 2 years within the CHAOSS project has been to develop a series of Practitioner Guides designed to help develop insights that can be used to improve the project health of an open source project. So far, we have 5 guides: Introduction, Contributor Sustainability, Responsiveness, Organizational Participation, and Security with more guides coming soon.

I’ve written about these guides in an OpenSource.net blog post and recorded a CHAOSScast podcast about each guide. I’ve also done quite a few talks related to the topics in these guides, which can be found on my Speaking page. The most recent one was a joint talk with Peculiar C. Umeh at FOSS Backstage with a video that is available to watch.

I won’t go into more detail here, since I’ve already linked to other blog posts, podcasts, and talks on the topic, but I encourage you to have a look at the Practitioner Guides to find ways to make your open source projects healthier and more sustainable!

General

A New Chapter at CHAOSS

For my regularly scheduled (once every year and a half) blog post, I wanted to announce that July 3rd is my last day at VMware, and I will be joining the CHAOSS project as their new Director of Data Science. 

CHAOSS Logo

It was really hard to leave VMware after almost 5 years (including my time at Pivotal). The work was fun, and I worked with so many amazing people that I will miss dearly! But as many of you know, I have a deep passion for data, and in particular open source community metrics, so the opportunity to work full time on the CHAOSS project is the dream job that I just couldn’t turn down. I’ve been working in this space for 10+ years with the CHAOSS project, and before CHAOSS, I was working with Bitergia and a variety of open source tools that later evolved into the software that is now part of the CHAOSS project. I’ll be taking July off and then will be starting my role with CHAOSS in August. A big thank you to the Alfred P. Sloan Foundation for making this possible through the grant that is funding the Director of Data Science position and other CHAOSS project initiatives.

I will be continuing my work on the OpenUK Board and as co-chair for the CNCF Contributor Strategy Technical Advisory Group, which have kept me very busy in addition to my work at VMware and in my role on the CHAOSS Governing Board.

Over the past year and a half, I’ve done quite a few presentations on topics ranging from how companies can work in open source communities to open source health / metrics to leading in open source, which can be found on my Speaking page. The highlight was giving a keynote about growing your contributor base at KubeCon EU in front of an audience of 10,000+, which was amazing and terrifying at the same time! 

In addition to my world tour of conference presentations, I was quoted in a Linux Foundation Diversity Report, won a few awards for my UK work in open source as part of the OpenUK Honours list in 2021 and 2023, and I’ve written a few blog posts since my last post here on my own blog:

On the personal side, Paul and I bought a new house in November, and we have become the people who sit in their back garden and talk about how adorable the squirrels and birds are. Since we live in an area near quite a bit of green space, we have regular visits from foxes and even spotted one badger on our backyard wildlife camera! 

Since I don’t post here often, if you want to keep up with what I’ve been doing, I post occasionally on Mastodon and Instagram.

Open source, research, and other stuff I'm interested in posting.