Text 1

Loyola University Publications

  • Loyola University Chicago

  • Loyola eCommons

  • Computer Science: Faculty Publications and Other Works

Authors

  • Nasir U. Eisty, University of Alabama - Tuscaloosa

  • George K. Thiruvathukal, Loyola University Chicago, gkt@cs.luc.edu

  • Jeffrey C. Carver, University of Alabama - Tuscaloosa

Citation

  • N. U. Eisty, G. K. Thiruvathukal, J. C. Carver, "A Survey of Software Metric Use in Research Software Development," 2018 IEEE 14th International Conference on e-Science (e-Science), Amsterdam, Netherlands, 2018, pp. 212-222. doi: 10.1109/eScience.2018.00036

License

  • This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.

Abstract

Background

  • Software libraries, tools, and applications support various disciplines.

  • Software quality and reliability are crucial in this context.

  • Software metrics assess and measure software quality and reliability.

Aims

  • Understand how research software developers use software engineering metrics.

  • Identify how metrics relevant to research software correspond to those used in traditional software engineering.

Method

  • Survey: 129 research software developers were surveyed for their knowledge and usage of software metrics.

  • Demographic influence on metrics usage was analyzed (e.g., project size, role, development stage).

Results

  • Respondents showed general knowledge of software metrics; however, specific knowledge and usage were limited.

  • Most used metrics pertain to performance and testing; code complexity metrics are used less.

Conclusions

  • Interest and value in software metrics is present, yet barriers to implementation exist.

  • Further studies are needed for continuous improvement with metrics.

Introduction

  • Research software engineers (RSEs) develop software to support research across various domains.

  • Software quality affects research conclusions and engineering decisions.

  • Research software engineers prioritize scientific goals over traditional software quality metrics.

  • The paper emphasizes the necessity of understanding metrics for continuous insight into software processes.

Software Metrics

  • Software metrics differentiate between measures (values obtained from metrics) and metrics (functions applied to obtain those values).

  • Two types of metrics:

    • In-process (development-related)

    • Code-oriented (focusing on code complexity)

  • Many research projects use open-source software that includes various software process aspects (e.g., version control, issue tracking).

Research Questions

Metrics

  • RQ1: What is the level of metrics knowledge and use?

  • RQ2: Which metrics are commonly used?

  • RQ3: What is the relationship between metric knowledge and perceived usefulness?

Code Complexity

  • RQ4: Do developers perceive code complexity as a problem?

  • RQ5: Is the use of complexity metrics helpful for managing complexity?

Survey Design

  • The survey contained carefully crafted questions to avoid bias.

  • Solicitation methods targeted a wide range of research software developers.

Survey Results

  • Total respondents: 129

Demographics

  1. Project Types: Majority worked on Scientific Computing Software (79.8%).

  2. Project Size: Most teams were smaller, which may limit metric usage.

  3. Project Role: Higher skew towards technical roles (e.g., developers).

  4. Project Development Stage: Majority were in the released stage, relevant for metric usage evaluation.

Overall Analysis

  • Most respondents reported low knowledge about metrics and perceived low usefulness.

  • A correlation between knowledge and perceived usefulness through statistical testing (p < .01).

Metric Knowledge and Usage

  • Unique Metrics Identified: 89 unique metrics were reported by respondents.

    • Categories include code metrics, process metrics, testing metrics, performance metrics, and recognition metrics.

  • Performance and testing metrics were noted as the most familiar and frequently utilized.

Discussion of Research Questions

  • RQ1: Most reported low metrics knowledge; substantial knowledge exists without corresponding usage.

  • RQ2: Performance metrics and testing metrics are highly recognized and executed. Code metrics showed low application despite high recognition.

  • RQ3: Higher perceived usefulness correlates with increased usage of metrics.

  • RQ4: Respondents recognize code complexity as an issue.

  • RQ5: There was no relationship between perceived complexity issues and helpfulness of complexity metrics.

Threats to Validity

Internal Threats

  • Potential bias from survey design; careful wording used to neutralize bias.

  • Selection bias where some respondents may not fit the research software developer profile.

External Threats

  • Sample may not represent all research software developers; overrepresentation from certain segments.

Construct Threats

  • Possible misunderstanding of survey questions by respondents.

Conclusion

  • The research indicates a gap between knowledge and usage of software metrics among research developers.

  • Performance and testing metrics showed the highest utility.

  • There is a need for continued efforts to improve metrics understanding and application in research software development.

Acknowledgments

  • Acknowledge support from NSF grants.

Appendix: Specific Metrics Identified

  • Lists of metrics organized by categories with knowledge and usage stats.

References

  1. Baxter et al., "The research software engineer."

  2. Carver and Heaton, "Self-perceptions about software engineering: A survey of scientists and engineers."

  3. Carver, "Software engineering for science."

  4. Fenton and Bieman, "Software Metrics: A Rigorous and Practical Approach."

  5. Goble et al., "Engineering Academic Software."

  6. Harmon and Auseklis, "Sustainable it services."

  7. Heaton and Carver, "Claims about the use of software engineering practices in science."

  8. Katz et al., reports from workshops on software sustainability.

  9. McIntosh-Smith et al., "Energy-aware metrics for benchmarking heterogeneous systems."

  10. Munipala and Moore, "Code complexity vs. performance."

  11. Nanthaamornphong and Carver, "Test-driven development in scientific software."

  12. Quinn, "Parallel computing: theory and practice."

  13. Schroeder and Gibson, "A large-scale study of failures in HPC systems."

  14. Stamelos et al., "Code quality analysis in open-source software."

  15. Vitek and Kalibera, "Repeatability, reproducibility, and rigor in systems research."

  16. Zhang et al., "Predicting failures of vision systems."