arXiv:2602.10149v3 Announce Type: replace-cross Abstract: Third-Party Risk Assessment (TPRA) relies on large repositories of cybersecurity compliance questions used to assess external suppliers against standards such as ISO/IEC 27001 and NIST. In practice, not all questions are relevant for a specific supplier and selecting questions for a given assessment context remains a manual and time-consuming task. Existing question retrieval approaches based on lexical or semantic similarity can identify topically related questions, but they often fail to capture the underlying assessment intent, including control domain and evaluation scope. To address this limitation, we investigate whether an explicit semantic label space can improve intent-aware TPRA question selection. In particular, we separate label space discovery from large-scale label assignment. We start by discovering overlapping clusters of semantically similar questions and then exploit LLMs to assign unique labels for each cluster. Second, we propagate labels through k-nearest neighbors (kNN) for a larger-scale question annotation. Question retrieval is finally achieved by similarity measure of the query with respect to the extracted labels instead of the questions themselves. This reduces repeated LLM calls while preserving label consistency. Experimental results show that the proposed semi-supervised framework reduces labeling cost and runtime compared with per-question LLM annotation while maintaining label quality and improving efficiency. Furthermore, label-based retrieval achieves better alignment with cybersecurity control domains and assessment scope than similarity-based retrieval, highlighting the value of semantic labels as an intermediate representation.

arXiv:2602.10149v2 Announce Type: replace-cross Abstract: Third-Party Risk Assessment (TPRA) is a core cybersecurity practice for evaluating suppliers against standards such as ISO/IEC 27001 and NIST. TPRA questionnaires are typically drawn from large repositories of security and compliance questions, yet tailoring assessments to organizational needs remains a largely manual process. Existing retrieval approaches rely on keyword or surface-level similarity, which often fails to capture implicit assessment scope and control semantics. This paper explores strategies for organizing and retrieving TPRA cybersecurity questions using semantic labels that describe both control domains and assessment scope. We compare direct question-level labeling with a Large Language Model (LLM) against a hybrid semi-supervised semantic labeling (SSSL) pipeline that clusters questions in embedding space, labels a small representative subset using an LLM, and propagates labels to remaining questions using k-Nearest Neighbors; we also compare downstream retrieval based on direct question similarity versus retrieval in the label space. We find that semantic labels can improve retrieval alignment when labels are discriminative and consistent, and that SSSL can generalize labels from a small labeled subset to large repositories while substantially reducing LLM usage and cost.