Five Ways to Measure Your Programmers’ Performance
What’s wrong with today’s assessment systems?
Information technology is widely known as an innovative sphere. For people who do not work there, it is often associated with the latest models of computers, new types of chips, processors with a number of cores, cutting-edge mobile technologies, and ultra-modern buildings and offices. However, these things are just the tip of the IT iceberg. Insiders know that the work of a modern programmer resembles the activities of medieval artisans.
Today, the manual work of potters, carpenters, and tailors has been automated: machinery works at a rate of thousands of operations per minute. The accuracy and quality of the automated work is much higher than that done manually. Unfortunately, 8-core processors are still incapable of automating the mental activities of today’s programmers. The tools that provide version control, building, designing, and automated testing just decrease the number of mistakes and defects but do not create software by themselves. And, there is no tool that can increase the speed of a programmer’s code writing by 2x or 3x.
We must admit that the industry developing automation systems for all the others failed to come up with a solution that can automate it. There is just one way to significantly increase software development speed: hiring, retaining, and motivating high performers.
Productivity and efficiency are the most critical factors of any company’s success. However, there is no standard recipe for making employees more efficient. Executives who manage small departments are sure that the performance of each of their subordinates is visible. Normally, we measure someone’s work results in accordance with what we see or what we hear from others. For instance, teammates and QA testers are the people who can say a lot about a programmer.
When a company grows and begins to have dozens or hundreds of employees, or when face-to-face communication among team members is limited, the method of personal feedback as a means of employee evaluation becomes unreliable. As a result, the achievements of employees may go unnoticed, which could prevent them from being able to work their way up the career ladder. People with a similar level of expertise and knowledge may also have different earnings. Such issues may cause conflicts, which can result in declining morale and even lead to employees—including valued specialists—leaving the company.
These issues made our management team at Altoros think about devising an effective, consistent method of employee assessment and motivation. Studying the literature and browsing the Internet did not provide us with a system that could be just plugged in to our organization. We had to develop such a system from scratch. After spending hours studying best practices, conducting meetings, and fostering discussions—as well as some trial and error—we have created a system that, in addition to the aforementioned primary purposes, serves as a motivating tool in the self-development of people. Feedback, including from our clients, has been extremely positive. We expect that this study may appeal to a broad range of specialists and managers working in the area of software development.
Definition of efficiency and performance
Armstrong and Baron (1998) define performance management as a strategic and integrated method of achieving the success of an organization by increasing the quality of employees’ work, as well as the capabilities of both working groups and separate employees. Performance management is:
- Strategic, because it is connected to a broad range of business issues, such as efficient functioning in a business environment, and direction, which should be maintained in order to achieve long-term goals.
- Integrated, because all the elements of an organization, such as departments, teams, and employees, should be managed logically—in order to achieve a common goal.
- Tightly connected with development, which is probably the most important function of performance management. It is impossible to achieve an increase in productivity in an organization without constant improvement of the process.
Alternatively, performance management is a concept of molding a single vision of organization goals and tasks, so that each employee understands his/her personal role in achieving the common results. It also makes it possible to manage and improve the performance of both separate employees and whole organizations.
There are a number of factors that influence work efficiency:
- Personal factors—the knowledge, competency, motivation, and morale of a person
- Leadership factors—the ability of a group leader to inspire, manage, and support his or her team
- Team factors—the ability of colleagues to provide support to each other
- System factors—the system of work and inventory provided by an organization
- Situational factors—internal and external pressure and changes
Why is work efficiency measurement needed?
There are two well-known expressions that characterize performance measurement:
- Those tasks are executed, which are measured.
- You cannot manage the process if you do not measure it.
In order to improve overall performance, one must first begin by establishing the current level of efficiency. Those whose performance is measured must clearly understand the criteria by which they are being assessed, as well as why the measurement process is important. Oakland (1993) suggests that the appropriate performance measurement does the following:
- Provides customer satisfaction
- Provides a basis for comparison
- Provides a “scale” to employees, which allows them to monitor their efficiency
- Highlights quality problems and suggests areas that require immediate attention
- Gives a notion of the costs of low-quality work
- Approves resources usage
- Provides information necessary for further actions that should be taken to increase performance
The results of a survey (Armstrong and Baron, 1998) help to understand the outcomes of a performance management system. Managers of 437 widely known US-based companies were interviewed. 205 companies used performance measurement systems; 232 did not. It was pointed out that the 205 companies that used the systems differed from the others in the following ways:
- They had larger margins, a better cash flow, higher stock exchange ratings, and a higher capitalization.
- They provided a considerable gain in financial activity and performance during the previous three years.
- They had a lower specific increase in the number of employees.
Performance measurement criteria
Armstrong and Baron (1998) point out that the performance measurement must:
- Be connected with the strategic goals of a company or a team.
- Be attributed to the tasks and responsibilities of employees. Measurement is efficient under the condition that personal responsibility is attached to it.
- Be concentrated on measurable results and achievements.
- Be manageable. It must provide information that expectations were justified to a certain extent.
- Correspond to the goals of measurements and data availability as much as possible.
- Provide reliable feedback for further actions.
- Involve all key aspects of work efficiency.
Classification of measures
There are various types of measures, which can be selected in accordance with the criteria described in the previous paragraph. However, the most important characteristics of the units are appropriateness, significance, and precision. Economists suggest that key measures can be targeted at quality, quantity, and profitability.
Metrics can be divided into the following groups:
- Financial (income, added value, profitability, and expenses)
- Results (products sold or services provided, productivity, and the number of new clients)
- Influential (project completion, service level, and innovations)
- Reactions (opinions of teammates and clients)
- Time-related (response time, expenditures of labor, and delivery time)
Numeric expression of measurement results
The measurement results can be expressed in four ways:
- Calculation—quantity of units
- Ratio—quantity of the units related to the maximum possible number of them
- Percentage—proportion of real achievements versus possible ones
- Financial results—gains or losses
Obtaining and analyzing measurement results
1. Work efficiency data
The data can be obtained from the process automation systems. They also allow for comparison between the achieved and expected results, as well as for assessment of alternative ways of acting.
2. Competency levels
Competency levels can be measured by analyzing real and reference behavior. If the factors, which distinguish best performance from average or poor performance, are defined, the competency level can be determined relatively easily. Alternatively, rating scales can be used.
“Benchmarking” means the efficiency measurement of an organization, a group, or an individual, as compared to the best samples in an industry, a company, or a team.
Efficiency management and evolution
On the whole, “evolution” is the development of individuals’ personal abilities to work more efficiently. In most cases, organizations evolve through the personal growth of their employees. Service organizations have no other choice apart from this one. Individual learning does not guarantee obtaining knowledge for the organization; however, the organization will obtain no knowledge without individual education. In order to evolve, a company must stimulate its employees to learn; and efficiency management serves as a powerful tool for motivation.
Measurement action may serve as an educational incentive. Before the action, employees may be asked about the knowledge and experience they would like to obtain. They may also be requested to give their opinions about the necessity of this knowledge and experience to the organization.
The action usually consists of, or is accompanied by, an interview. The interview may serve as the groundwork for assessing the results achieved within a definite period, discovering gaps in knowledge or experience, or formulating a plan for the next period.
Efficiency measurement is extremely useful for creating personal development plans for employees. The plans must include the actions that should be undertaken to achieve better performance and raise competency levels.
Efficiency management and personal compensation
Various researchers have shown that differentiated compensation still remains an important component of many efficiency management schemes. The compensation for efficiency and competencies is desirable because:
- It motivates people to work harder and develop their skills and competencies.
- It spreads a sense of the importance of efficient work.
- It compensates people differently and depends on their work efficiency, their competency, and input.
Armstrong and Murlis (1994) formulated “eight golden rules” of compensation paid to groups and individuals for their efficiency:
- Both teams and individuals must clearly understand their goals and assessment standards.
- They must be able to track their movement toward the goals throughout the period during which they will be assessed.
- They must be able to affect their work efficiency.
- They must understand what compensation they will get for achieving the results. There should be a direct correlation between the efforts and the rewards.
- The reward must follow the achievement immediately.
- The reward should be noticeable.
- The results necessary for the reward receipt must be achievable, but not too easy.
- The rewarding basis must be positively described and easily understood.
A compensation system based on the performance results has the following advantages:
- It motivates people to achieve better performance.
- It gives correct goals orientation.
- It fairly rewards employees for their performance.
- It provides efficient methods of rewarding and achievements recognition.
However, it also has a number of contingent shortcomings:
- The motivation factor is not guaranteed. It is often difficult-to-impossible to correspond to the rewarding criteria.
- Compensation must be based on a definite form of efficiency metrics. The most usual form is rating. However, creating realistic metrics may appear to be complicated. As a result, the metrics may appear unfair, subjective, and inconsistent.
- Such a compensatory system may make salaries grow faster than the benefits of the work improvement results.
- The system may depress teamwork due to its individualistic nature.
- Despite having good metrics, the work may appear to be inefficient and of poor quality.
- The system may make short-term goals more appealing to employees than the long-term ones.
Many researchers have proven that the best results come from those compensation systems that use a competency level for the definition of a base salary and a short-term performance for the bonus calculation. It is even better if a system takes into account not only individual results, but team performance as well.
Overcoming problems of work efficiency measurement
Management of inefficient work is divided into three stages:
- Identify the problem
- Understand the reasons for its origin
- Make an effort to “cure” it
Generally, there are two sides that are responsible for work efficiency—organizations and individuals.
Organizations may create the environment and processes that will solve the problems related to performance. In order to achieve it, they can do the following:
- Create a culture that motivates success. The approach should minimize failures that are the result of passivity. When failures occure, it is important that the organization takes measures that ensure individuals understand what caused the problem, and determine a course of action that can avoid the problem in the future. Elaborate on the performance management processes that set up definite standards for results comparison and shortcomings correction.
- Train managers to handle inefficiency and discipline problems.
- Constantly conduct monitoring of work efficiency and analyze the reasons for failures.
- Stimulate education in the organization.
Employees should be responsible for the efficiency of their work themselves. They must also pay considerable attention to self-education. The managers must assist them by providing coaching and consulting and by preparing personal development plans. The main challenge that managers and the whole organization will face after the introduction of an efficiency measurement system is how to appropriately respond to the low performers, once they have been identified by the system. Luthans and Kreitener (1975) defined five steps of behavior modification:
- Define critical behavior—what individual processes should be modified?
- Measure the frequency of the problem—get clear evidence of the existence of the problem.
- Conduct functional analysis—define what stimuli caused the behavior and/or what kind of reward or punishment influenced the behavior.
- Develop and integrate a correction strategy—it may include positive or negative consequences on the performer.
- Assess the results of a correction—what improvements happened (or did not happen)? What should be done if the correction does not bring results?
Personal performance correction is tightly connected with psychology. Many managers do not like, or are not skilled enough, to criticize their subordinates constructively.
Fletcher and Williams (1992) conclude that the main idea supervisors should convey to employees is one that helps them to modify good performance into better performance, rather than turning bad performance into good performance. They also suggest the following methods of critical discussion:
- Communicate to the assessed person that his sincere admission of his shortcoming is welcomed.
- Let the assessed person suggest a solution for actions correction.
- Provide support to assessed persons. Bring to their attention aspects that you think are acceptable despite being negatively perceived by themselves.
- Be persistent, if an assessed person does not admit the evidence of a problem. Provide details and examples.
- Address your comments to the weaknesses that can be corrected. Do not attack the personality of an assessed person.
- Do not discuss more than two weaknesses during one meeting. People can only bear a limited amount of criticism.
Development of a performance management system
The stages of performance management system development can be generally defined as diagnosis, procedures, design, pilot implementation, and introduction.
Stage #1. Diagnosis
The first stage of developing a successful performance management system (Diagnosis), must include the following considerations:
- General data about…
- …the culture, traditions, and style of management in the organization
- …the structure (meaning the extent of delegation and company hierarchy)
- …processes and systems
- …strategic plans
- …decision-makers and stakeholders
- …the management, measurement, and assessment of work quality
- …work compensation
- …improvement of work efficiency; development of skills and competencies
- …integration of personal and team tasks
- …of tasks that should be done.
- …of reasons why they should be done.
- …of the methods for how they should be done.
Stage #2. Procedures
The method that is considered to be the best by many researchers of personal performance is to assign a working team, which would consist of managers and employees’ representatives. This method allows for various opinions and feedback and fosters a feeling of belonging.
Stage #3. Design
During the design of the performance assessment system, the following aspects should be taken into account:
- Process model—agreement about work efficiency.
- Usage of tasks and competencies.
- The ways in which tasks will be assigned, and which groups and individuals will be able to participate in the formulation of procedures.
- Connection with the performance reward system.
- Methods of measurement.
- Ways in which competencies will be assessed.
- The list of necessary documentation and the managers responsible for its handling.
- How flexible the efficiency management system will be.
Stage #4. Development and implementation
The following points should be defined as a part of the development and implementation program:
- Who will participate in the program and what the responsibilities of each participant will be.
- Timelines of development and implementation.
- Employees’ notifications about plan details and a structure of the compensation system.
- It is crucial that these are done thoroughly and in a timely manner.
- Plan for education.
- Pilot schemes.
- Success criteria.
- Monitoring and assessment of the implementation.
Assessment of the performance management system
In order to assess the performance management system, attention to the following issues should be paid:
- Processes correspond to the management culture.
- Top management participates in the process and supports it.
- Linear managers and employees understand the processes, in general, equally well.
- Performance management processes help unite organizational, group, and personal tasks.
- It is obvious that performance management is helpful in getting both short-term results and long-term development.
- Performance management processes are integrated with the other HR processes.
- Performance management processes are flexible and are efficient under special circumstances.
- All stakeholders of the organization participate in designing, developing, and implementing the system. Project team members include top managers, linear managers, team leaders, and individuals.
- Processes are transparent. They function fairly and without bias.
- Managers and team leaders undertake measures that help team members to understand the strategy, goals, and values of the organization.
- Performance management processes place each employee’s individual needs and company goals in line.
- Employees are able to express their opinions about whether the results expected from them are achievable.
- The primary emphasis is put on employees’ development. Financial reward is the second aspect to be considered.
- There are competency models that are developed by all the stakeholders.
- The goals, gains, and efficiency of the system are communicated to employees in detail.
- Managers, team leaders, and employees are trained to work efficiently.
Efficiency measurement issues in software development
If your software development company is going to introduce a performance measurement system, be prepared to hear a long list of arguments from developers and even managers—a long list of arguments, which are used to justify the impossibility of measuring their performance. Below, there are some examples of arguments.
- “Software developers differ from sales people or factory workers. Their performance cannot be measured by sales volume or by the number of details produced.” Some will go further and declare that, “creative work cannot be measured at all.” However, each work product creates results, and the results can be measured. There are a number of metrics suggested in paragraph 16, which can be used for assessing the performance of programmers.
- “The more the emphasis is put on measuring major metrics, the more likely it is that the small unmeasured elements will be sacrificed.” However, it is impossible to achieve great performance in everything. The system of metrics defines a common set of values for all: clients, company, and individual developers.
- “The quality of work suffers as employees become focused on achieving a grade rather than satisfying clients.” However, customer satisfaction may be used as one of the metrics.
- “Measurement cannot be performed because it cannot take into account the factors that are beyond control.” However, measurement just points out the changes in work efficiency, it does not explain them.
- “Measurement is possible only in a case where the result is measurable.” However, whenever people are expected to perform some work, the extent of the work completion can be assessed.
Efficiency vs. processes
As we may see, nowadays many software development companies invest heavily in processes reengineering and various automation systems. The same goals as in paragraph four apply. Companies tend to:
- Improve customer satisfaction
- Decrease the specific number of employees
- Get larger margins
However, polishing processes cannot take the place of having skilled employees and taking care of programmers’ evolutions. Process management tools cannot automate the creative work of software developers. Nevertheless, automation systems are extremely important and can be helpful in the performance measurement process by providing the necessary numeric data.
Case study: Our performance management system
The expertise of a person, which he or she presents at an interview and on his/her resume, includes those factors that determine whether he/she will get a position in a particular company. These factors will also influence the initial compensation of the person. Unfortunately, when making hiring decisions, we often have to rely on the subjective opinion of an interviewer, which can never be an exact measure of a person’s actual ability to fulfill the requirements of a position. Just as it is impossible for an interviewer to fully evaluate a potential employee, it is also impossible to use a subjective opinion to compare developers, especially if they work in different departments. How can the manager fairly determine the distribution of roles and responsibilities on the project? How can he or she know for certain who is knowledgeable enough to develop a flexible and scalable architecture, or who reads books after work in order to improve his or her knowledge?
A third-party theoretical exam can serve as a fair validation of the theoretical knowledge of a software developer. Certificates from Sun and Microsoft, and online exams at brainbench.com or whizlabs.com, will give you information about the capabilities of developers.
Experience also matters. It is not enough just to read a book and answer the examination questions correctly. There is a lot of information that can be obtained only from practice. Altoros uses a competency scale, which converts years of experience into points used for assessment.
We sometimes face a situation in which a person who is extremely knowledgeable is not productive on projects. There are a lot of possible reasons for a lack of performance, such as fatigability, indiscipline, instability, etc. The next group of metrics displays how a programmer applies the knowledge he or she has.
1. Estimations divergence
The most obvious criterion that directly influences the efficiency of a developer is his or her productivity. But, it is not easy to answer such simple questions as, “How quick is he? How much work did he do?” As it was outlined above, it is incorrect to assess the productivity of a developer by the number of code strings or by bytes of source code written within a time unit. However, it is quite possible to make estimations based on the tasks that are assigned to a developer, and then to calculate how often the actual time spent corresponds to the time planned. In order to make this metric work, both a manager and a developer must agree that the task can be completed within the estimated time period. Statistical data can give answers to the questions below.
- How often does the developer miss deadlines?
- What is the average accuracy of the developer’s original estimations vs. the actual time it took to perform a task or complete a project?
- What is the monthly difference between the estimated time and the time actually spent on carrying out tasks?
2. Distribution of bugs by priority and severity
Every company is interested in having high-quality products. Thousands of software development firms have failed trying to promote buggy applications. The users who installed such products spread their negative feedback throughout the Internet, destroying the reputation of the software vendor.
It is possible to discover the bugs with the help of thorough testing, and then to fix them. However, there is another characteristic of bugs—they cost investors money (Newman, 2002). The higher the severity, the higher the cost.
So, productive developers are those who work without major defects. “It is impossible,” one can say, and we agree with this statement. There are no programmers who work without mistakes. We also could amend it with another saying, “Those people do not make mistakes who do nothing.” However, it is quite possible to compare the bug rates of various developers and understand who is more productive.
We, at Altoros, after analyzing statistical data, use a metric, which is measurable and can be applied as an indicator of a programmer’s quality. The metric is the percentage of bugs of a definite severity or priority in the overall number of bugs produced by the developer.
In the picture above, the most severe bugs are shown in blue. You can see that every developer has a different number of bugs and a different percentage of bugs by severity. James and Nic have a minimal number of bugs. The quantity of defects produced by Shawn is the highest within the group. The difference in the absolute number of defects can be explained by the disparity of periods during which each developer participated in the project, as well as by the nature of work that is performed by a programmer. The absolute number of bugs cannot be compared and used as an assessment criterion. However, we can juxtapose the percentage of the highest severity bugs.
The table above suggests that Nic and Shawn are the most efficient team members because they have the lowest percentage of critical bugs. Also, when evaluating a developer’s performance, it is essential to consider the stage of project development. At an early stage, when the basics of a project structure are being formed, low-priority defects are unlikely to be reported. At a later stage, when the project is quite stable, programmers pay more attention to the user interface. It causes a rise in the quantity of low-priority defects. So, while assessing programmers in accordance with the relevant percentage of bugs’ priority (severity), the stage of the project should be taken into account.
3. The number of regressive bugs. Distribution of regressive bugs by priority and severity.
Imagine a situation in which a team is going to release a product. There are just a few bugs left. Most of them are not critical. The manager is quite optimistic about meeting the deadline. The new build is released; QA team members report that all of the previously known bugs have been fixed. However, there are some new bugs, their number has grown up in comparison with the previous build, and to make matters worse, the number of critical bugs has increased dramatically. The release of the next build brings similar results. It becomes a nightmare for the manager, and the deadline is not met. Have you ever had a similar experience? Unfortunately, there is a class of developers who always seem to produce two critical bugs while fixing a slight glitch. A manager must know about the abilities of his team members to fix defects carefully.
The approach described in the previous paragraph can only be used for regressive bugs. They are especially costly because more time is required to fix them. The increased effort to resolve the regressive bug is caused by the need to analyze the code written some time ago. The higher cost of the regressive bugs deserves the special attention of the programmers. The ability to produce fewer such defects definitely makes a significant contribution to the efficiency of a software product’s development.
4. The number of unsuccessful check-ins
The analysis of the statistical data suggested one more interesting metrics to us. It appears that experienced developers spend less time on build compilation. It often happens that after a check-in, a build fails to be compiled on a server while everything was okay on a developer’s workstation. The developer spends some time in order to find the reason for this issue and often distracts other team members with requests for assistance. The common cause of such a problem is an old version of source code, which a developer has on his workstation. In order to avoid such cases, the developer should check-out the recent version of the source code before merging his changes with it. The metric shows how disciplined the developer is and allows to save troubleshooting time.
5. Compliance with coding standards
Everybody knows how important it is to name procedures and variables correctly; how beneficial object-oriented programming is; how difficult it is to understand the code of a workmate if it does not contain any comments (Source Code Comments). However, programmers often “forget” about the obvious rules of coding, in order to reach short-term goals. The automated system of code check allows not only blocking the attempt to commit the code, which does not comply with a standard but also records such events and then provides the data for assessing the programmers.
Commenting on the code afterwards has proved to be more time-consuming compared with commenting on it immediately upon completion.
Despite being objective and obvious, the metrics described above still do not cover all the aspects of personal performance. This means that the assessment system can be “deceived.” A developer can pay attention just to the aspects of work, which are measured, and not care about anything else. However, there are two groups of people who can hardly be deceived—clients and teammates.
When we designed the assessment system, we never forgot the primary purpose of any service organization—customer satisfaction. And getting the numeric value of this metric is quite simple. Just a few simple questions have to be asked. Setting up a compensation premium proportionate to the level of satisfaction gives very efficient feedback to the project team about the performance and makes everyone think about his or her ability to improve the team results. If quantitative metrics look good and the client is not satisfied, then only the programmers themselves may reveal the hidden problems.
The assessment of team members in accordance with the method of 360 degrees (Edwards and Ewen, 1996), when every person who did some work on the project assesses the others, gives fair information about the real performance of every project participant. We have selected ten personal qualities that we think affect the overall performance most of all:
- Time management
- Ability to understand and convey ideas
- Desire to acquire new knowledge
- Speed of work
- Assistance to other team members
- Quality of work
- Positive thinking
- Independence in work
Like many innovations, setting up a performance management system is often a lengthy and expensive initiative. And a decision whether to invest in it or not, should not be made blindly. The benefits of the systems were described in detail in paragraph 4. Their financial impact will be different for various organizations. Below, is an approximate list of tools, processes, and activities, which should be considered as “expenses.”
- Automation tools, which would allow measuring the quantity of bugs, check-ins, policy violations, etc.
- Time of employees who participate in performance system design, development, and implementation.
- Time for competency scales creation.
- Time for interviewing the employees. (The time of both the interviewer and the interviewee should be taken into account.)
- Time for creation of personal development plans.
- Trainings and seminars.
- System evaluation and correction.
- Examination fees.
- Time for examination.
Keys to success
Armstrong and Baron (1998) are cited in the conclusion. They consider the following factors to be the keys to successful implementation and application of performance management systems:
- Clear understanding of “performance management”
- Understanding of the company’s values and its desire to establish a “performance management culture”
- Laser focus on the individual benefits to employees and their role in the process
Like many innovations, setting up a performance management system is often a lengthy and expensive initiative. However, clear understanding of company’s values and its desire to establish a “performance management culture” can help to succeed in implementation of the system.
Coupled with our time tracking tool TX Chrono, the guidelines described in this study will help you to organize a time-efficient programming process. In brief, TX Chrono is a timesheet tracking application for software development organizations standardizing on Visual Studio .NET. A web-based timetable enables project managers and team members to submit, approve, and analyze timesheets, featuring a set of notification and reporting capabilities.
With the performance management system described in this handbook, organizations can turn their employee review process into a powerful toolset for improving both tactical performance and strategic business results. Though neither can process management tools substitute for skilled employees, nor automate their creative work, they can provide unbiased insights based on measurable data, helping to organize efficient processes.
Let us know if you want to learn more about our performance management system or other tools that can make software development more visible and predictable.