General

Strategy Before Metrics

Two chess pieces facing off on a chess board

I’ve been involved with open source project metrics for a very long time, and people often ask me which metrics they should use, but this isn’t really a question that I can answer. What you measure and how you interpret those metrics depend on your goals and what your organization is trying to accomplish. For this blog post, I’m using “organization” loosely. It could mean your company, university, or non-profit, but it might also mean aligning with funding organizations if your OSPO or other open source efforts were funded by another organization. 

The CHAOSS Practitioner Guide: Introduction – Things to Think about When Interpreting Metrics mentions:

“one of the best places to start isn’t actually with the metrics, but by spending some time understanding the overall goals for the project. If the project is primarily driven by one organization or owned by an organization, you should also consider the goals for that organization. By thinking strategically about the overall goals, you’ll be in a better place to decide what you need to measure to determine whether the project is achieving its goals. Open source projects generate a tsunami of data that can be overwhelming, but by focusing on the goals, you can develop a metrics strategy that helps you focus on the metrics that matter most for a particular project.”

It can help to ask yourself, “what is important for my organization or the project?” This often means starting with your team’s open source strategy and aligning it with your organization’s goals. The most important part of putting together strategies and plans for your open source efforts is aligning them with the overall goals for your organization. By taking some time and effort to make sure you support the overall strategy for your organization, then it will be much easier to justify continuing these efforts during the next planning or funding cycle. This will also help you make the case to senior management, executives, funders, and others on the leadership team who aren’t likely to be involved in the low level details. Explaining how your open source contributions support the goals of your organization can help the executive or leadership team understand the strategic importance of this work so that you can continue your work in open source.

Once you have a strategy defined that aligns with the strategy of your organization, then you can figure out what you need to measure to show whether you are achieving your goals. There are a couple of reasons that starting with the goals is important, since metrics can go awry if you aren’t focused on the right things.

  • You won’t know if you are successful if you aren’t measuring the right things. If you aren’t measuring the things that are important for your project or organization, you won’t know if you are making progress in the areas that you care the most about. For example, if you want to improve the performance of a particular piece of open source software, you’ll want to have success criteria and measurement based on specific types of performance. If you want to gain influence within an open source project, maybe you measure increases in contributions or the number of employees moving into positions of influence. 
  • Measurement impacts behavior, and people do different things depending on what you measure. For example, if you publish metrics that focus on the number of comments on issues, you are likely to start getting more comments on issues. If what you are really trying to do is get people to help review and approve contributions, then additional comments on issues might not help as much as looking at reviews on change requests (pull requests / merge requests). 

Once you decide on your success criteria, you need to make sure that you can get the data required to measure it and start measuring it now to get a good baseline. There are plenty of tools available to gather contribution data about open source projects. Some of the commonly used tools can be found in the CHAOSS project, but you can also likely get a pretty good sense for the data by looking at your code repositories and other communication channels. GitHub, for example, has some pretty good data under the insights tab.

After you have your metrics, you need to actually do something with them to show that you are making progress toward accomplishing your goals. Think about which metrics you should be showing to your leadership and which ones should be shared with your team and the broader community. But communicating metrics is much more than just showing some charts or graphs, you also need to interpret those metrics and tell the story about what they mean. The CHAOSS Practitioner Guides can help you think about the interpretation and how you might tell the story about what your metrics mean. Without interpretation and explanation, all you have are numbers, which are way less powerful than the story about what the data means in the context of how it helps your organization achieve their goals. 

Here are a few additional links and resources to help you think about building your metrics strategy and telling the story about what the metrics mean:

Photo by Hassan Pasha on Unsplash.