Exposure assessment to engineered nanomaterials within the French EpiNano program: inter-evaluator reliability study

Objective: In the French epidemiological surveillance program EpiNano, each workstation in the participating companies is randomly assigned a score of potential of exposure to nano-objects and their agglomerates and aggregates (NOAA), assessed by one individual evaluator of the program. Although this exposure assessment is based on a standardized method, the issue of inter-evaluator reliability might be of concern. In this study we compared the results of three evaluators: two experienced evaluators from the EpiNano program team (one epidemiologist and one industrial hygienist), and one non-experienced external evaluator.
Methods: Evaluators carried out an onsite industrial hygiene visit of facility which synthesizes and functionalizes carbon nanotubes and an observation of workstations where NOAA are present using a standardized tool, the Onsite technical logbook. Each evaluator acted at the individual level. Weighted Fleiss kappa (F) and Cohen’s kappa (), inter-class correlation coefficient (ICC) estimated from the one-way random effect model ANOVA, and Spearman correlation coefficients () were calculated to estimate the inter-rater reliability for binary, quantitative, and ordinal semi-quantitative exposure variables, respectively.
Results: The binary exposure variable comparison showed good agreement between the three evaluators (F=0.60, p=0.003), with an excellent agreement between two internal evaluators (=1.00, p=0.004) and a moderate one (=0.46, p=0.07) between external and each internal evaluator. The estimated interclass correlation between three evaluators based on quantitative exposure score was low (ICC=0.032) and Bartlett’s test for equal variance was negative (p=0.09). The highest discrepancy was observed for five of 28 exposure score components (Weight fraction of NOAA in the product, Reduction of exposure by general ventilation, Local control of exposure due to near-field sources, Background emission from deposited contaminants, and Inspections and maintenance of machines). The reliability of the semi-quantitative exposure variable was excellent between the two internal evaluators ( =0.93, p=0.03) and mediocre when these two evaluators’ results were compared with the external evaluator’s results.
Discussion: This work allowed examining possible sources of the inter-evaluator discrepancies and has led to an improvement of the standardized tool (Onsite technical logbook). An explanatory guideline document was elaborated for completing the Onsite technical logbook when implementing the EpiNano method in companies by external or new evaluators.