arXiv:2605.24213v1 Announce Type: cross Abstract: Evaluation harnesses are software systems that orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting. Despite their critical role in machine learning infrastructure, their operational challenges and engineering concerns have received limited attention so far. We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause. Most harness operational challenges concentrate in the Specification stage (41.4% of issues), where harnesses integrate external models, datasets, and scoring judges. The three most frequent root causes of operational challenges are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), which together account for 61.7% of classified issues, spanning both defects in existing functionality and capability gaps that block intended workflows. Root causes also vary by workflow stage: environment incompatibility and external dependency breakage account for 36.2% of provisioning issues, whereas algorithmic error (25.9%) and validation gap (22.5%) dominate assessment issues. Together, these contributions establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern.

arXiv:2511.01047v3 Announce Type: replace-cross Abstract: Automated program repair (APR) has recently shifted toward large language models and agent-based systems, yet most systems rely on local snapshot context, overlooking repository history. Prior work shows that repository history helps repair single-line bugs, since the last commit touching the buggy line is often the bug-introducing one. In this paper, we investigate whether repository history can also improve agentic APR systems at scale, especially for complex multi-hunk bugs. We present HAFixAgent, a History-Aware Bug-Fixing Agent that injects blame-derived repository heuristics into its repair loop. A preliminary study on 854 Defects4J (Java) and 501 BugsInPy (Python) bugs motivates our design, showing that bug-relevant history is widely available across both benchmarks. Using the same LLM (DeepSeek-V3.2-Exp) for all experiments, including replicated baselines, we show: (1) Effectiveness: HAFixAgent outperforms RepairAgent (+56.6\%) and BIRCH-feedback (+47.1\%) on Defects4J. Historical context further improves repair by +4.4\% on Defects4J and +38.6\% on BugsInPy, especially on single-file multi-hunk (SFMH) bugs. (2) Robustness: under noisy fault localization (+1/+3/+5 line shifts), history provides increasing resilience, maintaining 40 to 56\% success on SFMH bugs where the non-history baseline collapses to 0\%. (3) Efficiency: history does not significantly increase agent steps or token costs on either benchmark.