Background: Artificial Intelligence (AI) in healthcare represents machine-based systems designed to imitate human cognitive abilities, making predictions and recommendations with varying levels of autonomy. As AI-enabled digital health technologies (DHTs) become increasingly prevalent in healthcare settings, questions have emerged regarding appropriate methodological approaches for evaluating their benefits in hospital procurement decisions.

Methods: The study employed a four-step approach: 1) A targeted search in 51 health technology assessment (HTA) institutional webpages to identify methods guidance documents and assessments for AI-enabled DHTs; 2) Analysis of identified methods guidance documents to describe how to assess AI-enabled DHTs' benefits and identification of themes specific for AI; 3) Analysis of identified assessments focusing on applied methods and application areas; and 4) Development of recommendations for Austrian hospitals.

Results: Of 51 HTA institutes, 13 institutes published five methodology documents and 30 HTA reports. The included HTA reports predominantly evaluated AI applications in diagnostics and screening (27/30), particularly in radiology (10/27) and internal medicine (7/27). The radiological AI applications mainly supported image analysis (e.g. computed tomography). In radiotherapy (1/30), AI was investigated for contouring regions to be irradiated. Additional AI applications were evaluated for predictions (2/30) in palliative medicine and patient management. The final decision always remained with the medical professionals - AI served as support for making treatment processes more efficient. The analysis of the methodology documents showed that standard HTA methods are suitable for evaluating certain aspects but should be supplemented with AI-specific aspects. These concern technical (training data, data quality), ethical (algorithmic bias), and organisational aspects (human oversight) as well as post-implementation monitoring. Based on the international methodology documents, a checklist was developed to support decision-makers in implementing AI in the areas of intended purpose, regulatory requirements, HTA evaluation, and monitoring.

Conclusion: Standard HTA methods can serve as a foundation for evaluating AI-enabled DHTs, but must be supplemented with AI-specific considerations, particularly for technical, ethical and organisational aspects. For Austrian decision makers, it is recommended to use existing frameworks for digital health technologies as a starting point, supplemented with AI-specific components. A sophisticated digital infrastructure with high-level interoperability is identified as a prerequisite for successful implementation.