On the diffusion of test smells and their relationship with test code quality of Java projects

Abstract

Test smells are considered bad practices that can reduce the test code quality, thusharming software testing goals and maintenance activities. Prior studies haveinvestigated the diffusion of test smells and their impact on test code maintainability.However, we cannot directly compare the outcomes of the studies as most of themuse customized datasets. In response, we introduced theTSSM(Test Smells andStructural Metrics) dataset, containing test smells detected using theJNose Testtool and structural metrics (test code and production code) calculated with theCK metricstool of 13,703 open-source Java systems from GitHub. In addition, weperform an empirical study to investigate the relationship between test smells andstructural metrics of test code and the relationship between test smells on a large-scale dataset. We split the projects into three clusters to analyze the distribution oftest smells, the co-occurrences among test smells, and the correlation of test smellsand structural metrics of test code. The ratio of smelly test classes with a specific testsmell is similar among the clusters, but we could observe a significant difference inthe number of test smells among them. The test smellsSleepy Test,Mystery Guest,andResource Optimismrarely occur in the three clusters, and the last two are stronglycorrelated, indicating that those test smells are more severe than others. Our resultspoint out that most test smells have a moderate correlation with high complexity,large size, and coupling of the test code, indicating that they can also negativelyaffect its quality. To support further studies, we made our dataset publicly available.