tio-boot整合yolo

下面是一篇从零开始给初学者看的完整使用文档。 目标：让把 tio-boot + ONNX Runtime + YOLOv8 跑起来，并通过 HTTP 上传图片得到检测结果。

一、项目介绍

这是一个基于 Java 的目标检测服务：

技术栈：

技术作用
tio-bootWeb 服务器（替代 SpringBoot）
OpenCV图片解码、预处理
ONNX Runtime运行 YOLO 神经网络模型
YOLOv8 ONNX目标检测模型
自定义后处理NMS、bbox 映射

最终效果：

客户端上传图片 → 服务器检测 → 返回 JSON 识别框

二、最终接口效果

请求

POST /yolo/object/detection
Content-Type: multipart/form-data

file: 二进制图片

响应

{
    "data": [
        {
            "clsId": 0,
            "label": "person",
            "bbox": [681.3532,337.40396,896.0416,951.0893],
            "confidence": 0.8843324
        }
    ],
    "error": null,
    "msg": null,
    "ok": true,
    "code": 1
}

bbox 含义：

[x1, y1, x2, y2]  → 原图坐标

三、运行前准备

1）准备模型

把 yolov8s.onnx 放到项目目录：

resources/models/yolov8s.onnx

2）加载 OpenCV

启动时执行：

nu.pattern.OpenCV.loadLocally();

否则会报错：

UnsatisfiedLinkError: no opencv_java...

四、核心：检测服务

这个类完成了 90% 工作：

  • 图片解码
  • 预处理 letterbox
  • ONNX 推理
  • NMS 后处理
  • 坐标映射回原图

注意：此类是线程安全的，可并发调用

YoloObjectDetectionService.java

package com.imaginix.mc.cv.services;

import java.nio.FloatBuffer;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

import org.opencv.core.Mat;
import org.opencv.core.MatOfByte;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

import nexus.io.yolo.config.ODConfig;
import nexus.io.yolo.domain.DetectionResult;
import nexus.io.yolo.domain.PreprocessResult;
import nexus.io.yolo.utils.Letterbox;
import nexus.io.yolo.utils.TensorUtils;

import ai.onnxruntime.OnnxTensor;
import ai.onnxruntime.OrtEnvironment;
import ai.onnxruntime.OrtException;
import ai.onnxruntime.OrtSession;

/**
 * YoloDetectionService（返回 List<DetectionResult>）
 *
 * 说明：
 * - 使用单例 OrtEnvironment 与 OrtSession（session.run() 可并发）
 * - 使用固定线程池 + Semaphore 限制并发推理
 * - 输入为图片字节数组（controller 将其解析为 name + bytes 并调用 detect）
 */
public class YoloObjectDetectionService {

  private String modelPath="models/yolov8s.onnx";

  private int threadPoolSize=Runtime.getRuntime().availableProcessors();

  private int concurrentLimit=Runtime.getRuntime().availableProcessors();

  private float confThreshold=0.35f;

  private float nmsThreshold=0.55f;

  private long inferTimeoutMs=30000;

  private OrtEnvironment env;
  private OrtSession session;

  private ExecutorService inferenceExecutor;
  private Semaphore concurrentSemaphore;

  private String[] labels;

  
  @SuppressWarnings("deprecation")
  public YoloObjectDetectionService() {
    this.env = OrtEnvironment.getEnvironment();
    OrtSession.SessionOptions sessionOptions = new OrtSession.SessionOptions();
    try {
      this.session = env.createSession(modelPath, sessionOptions);
    } catch (OrtException e) {
      e.printStackTrace();
    }

    // 读取 model metadata 中的 names（若存在）
    try {
      String meta = session.getMetadata().getCustomMetadata().get("names");
      if (meta != null) {
        meta = meta.substring(1, meta.length() - 1);
        java.util.regex.Pattern p = java.util.regex.Pattern.compile("'([^']*)'");
        java.util.regex.Matcher m = p.matcher(meta);
        java.util.List<String> list = new java.util.ArrayList<>();
        while (m.find()) {
          list.add(m.group(1));
        }
        this.labels = list.toArray(new String[0]);
      } else {
        this.labels = null;
      }
    } catch (Exception e) {
      System.out.println("Failed to read labels from metadata: " + e.getMessage());
      this.labels = null;
    }

    this.inferenceExecutor = Executors.newFixedThreadPool(threadPoolSize, runnable -> {
      Thread t = new Thread(runnable);
      t.setDaemon(true);
      t.setName("yolo-infer-worker-" + t.getId());
      return t;
    });

    this.concurrentSemaphore = new Semaphore(concurrentLimit);
    new ODConfig();

    System.out.printf("YoloDetectionService initialized. model=%s pool=%d concurrentLimit=%d%n",
        modelPath, threadPoolSize, concurrentLimit);
  }


  public void shutdown() {
    if (inferenceExecutor != null) {
      inferenceExecutor.shutdown();
      try {
        if (!inferenceExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
          inferenceExecutor.shutdownNow();
        }
      } catch (InterruptedException e) {
        inferenceExecutor.shutdownNow();
      }
    }
    if (session != null) {
      try {
        session.close();
      } catch (OrtException ignored) {
      }
    }
    if (env != null) {
      env.close();
    }
    System.out.println("YoloDetectionService shutdown complete.");
  }

  /**
   * 主方法：接收文件名与图片字节数组，返回 DetectionResult 列表（bbox 已映射到原图坐标）
   */
  public List<DetectionResult> detect(String name, byte[] imageBytes) throws Exception {
    boolean permit = concurrentSemaphore.tryAcquire(1, 10, TimeUnit.SECONDS);
    if (!permit) {
      throw new RuntimeException("Server busy, please retry later");
    }

    try {
      Callable<List<DetectionResult>> task = () -> {
        long t0 = System.currentTimeMillis();

        Mat img = bytesToMat(imageBytes);
        if (img == null || img.empty()) {
          throw new IllegalArgumentException("Invalid image bytes");
        }

        // 转为 RGB（OpenCV 默认 BGR）
        Mat rgb = img.clone();
        Imgproc.cvtColor(rgb, rgb, Imgproc.COLOR_BGR2RGB);

        // 预处理 -> OnnxTensor
        PreprocessResult pre = preprocess(rgb);

        // 推理
        float[][] rawOutput = null;
        try {
          rawOutput = runModel(pre.tensor);
        } finally {
          if (pre.tensor != null) {
            try {
              pre.tensor.close();
            } catch (Exception ignored) {
            }
          }
        }

        // 后处理：直接生成 DetectionResult（但 bbox 仍在 model-space）
        List<DetectionResult> detectionsModelSpace = postprocessToDetectionResult(rawOutput, labels, confThreshold,
            nmsThreshold);

        // 将 bbox 映射回原始图像坐标并构造最终结果
        List<DetectionResult> results = new ArrayList<>();
        double ratio = pre.ratio;
        double dw = pre.dw;
        double dh = pre.dh;
        double origW = img.width();
        double origH = img.height();

        for (DetectionResult d : detectionsModelSpace) {
          float[] bbox = d.getBbox();
          float x0 = (float) Math.max(0.0, Math.min(origW - 1, (bbox[0] - (float) dw) / (float) ratio));
          float y0 = (float) Math.max(0.0, Math.min(origH - 1, (bbox[1] - (float) dh) / (float) ratio));
          float x1 = (float) Math.max(0.0, Math.min(origW - 1, (bbox[2] - (float) dw) / (float) ratio));
          float y1 = (float) Math.max(0.0, Math.min(origH - 1, (bbox[3] - (float) dh) / (float) ratio));
          // 使用构造函数创建新的 DetectionResult（避免调用空的 setBbox）
          DetectionResult out = new DetectionResult(d.getLabel(), d.getClsId(), new float[] { x0, y0, x1, y1 },
              d.getConfidence());
          results.add(out);
        }

        long t1 = System.currentTimeMillis();
        System.out.printf("detect(%s) done: detections=%d time=%dms%n", name, results.size(), (t1 - t0));

        // 释放 mats
        try {
          img.release();
          rgb.release();
        } catch (Exception ignored) {
        }

        return results;
      };

      Future<List<DetectionResult>> future = inferenceExecutor.submit(task);
      try {
        return future.get(inferTimeoutMs, TimeUnit.MILLISECONDS);
      } catch (TimeoutException te) {
        future.cancel(true);
        throw new RuntimeException("Inference timeout");
      }
    } finally {
      concurrentSemaphore.release();
    }
  }

  // --------------------
  // Helper methods
  // --------------------

  private Mat bytesToMat(byte[] imageBytes) {
    MatOfByte mob = new MatOfByte(imageBytes);
    Mat img = Imgcodecs.imdecode(mob, Imgcodecs.IMREAD_COLOR);
    mob.release();
    return img;
  }

  private PreprocessResult preprocess(Mat imageRgb) throws OrtException {
    PreprocessResult r = new PreprocessResult();

    Letterbox letterbox = new Letterbox(); // 默认 640x640
    Mat letterboxed = letterbox.letterbox(imageRgb);

    r.ratio = letterbox.getRatio();
    r.dw = letterbox.getDw();
    r.dh = letterbox.getDh();
    r.rows = letterbox.getHeight();
    r.cols = letterbox.getWidth();
    r.channels = letterboxed.channels();

    float[] pixels = new float[r.channels * r.rows * r.cols];
    for (int i = 0; i < r.rows; i++) {
      for (int j = 0; j < r.cols; j++) {
        double[] px = letterboxed.get(i, j);
        for (int k = 0; k < r.channels; k++) {
          pixels[r.rows * r.cols * k + i * r.cols + j] = (float) px[k] / 255.0f;
        }
      }
    }

    long[] shape = { 1L, r.channels, r.rows, r.cols };
    r.tensor = OnnxTensor.createTensor(env, FloatBuffer.wrap(pixels), shape);

    return r;
  }

  private float[][] runModel(OnnxTensor tensor) throws OrtException {
    String inputName = session.getInputInfo().keySet().iterator().next();
    HashMap<String, OnnxTensor> inputs = new HashMap<>();
    inputs.put(inputName, tensor);

    try (OrtSession.Result outputs = session.run(inputs)) {
      Object out = outputs.get(0).getValue();
      float[][] raw = ((float[][][]) out)[0];
      return raw;
    }
  }

  /**
   * 后处理：直接构建 DetectionResult（但 bbox 仍为 model-space）
   */
  private List<DetectionResult> postprocessToDetectionResult(float[][] rawOutput, String[] labels, float confThreshold,
      float nmsThreshold) {
    List<DetectionResult> outDetections = new ArrayList<>();

    float[][] outputData = TensorUtils.transposeMatrix(rawOutput);

    Map<Integer, List<float[]>> class2Bbox = new HashMap<>();
    for (float[] row : outputData) {
      if (row.length <= 5)
        continue;
      float[] probs = java.util.Arrays.copyOfRange(row, 4, row.length);
      int cls = TensorUtils.argmax(probs);
      float conf = probs[cls];
      if (conf < confThreshold)
        continue;

      row[4] = conf;
      TensorUtils.xywh2xyxy(row);
      if (row[0] >= row[2] || row[1] >= row[3])
        continue;

      class2Bbox.computeIfAbsent(cls, k -> new ArrayList<>()).add(row);
    }

    for (Map.Entry<Integer, List<float[]>> e : class2Bbox.entrySet()) {
      int cls = e.getKey();
      List<float[]> bboxes = e.getValue();
      List<float[]> kept = TensorUtils.nonMaxSuppression(bboxes, nmsThreshold);
      for (float[] box : kept) {
        String label = labels != null && cls < labels.length ? labels[cls] : String.valueOf(cls);
        float score = box[4];
        float[] bboxXYXY = java.util.Arrays.copyOfRange(box, 0, 4);
        DetectionResult d = new DetectionResult(label, cls, bboxXYXY, score);
        outDetections.add(d);
      }
    }

    return outDetections;
  }
}

五、HTTP 接口

处理上传图片 → 调用检测 → 返回 JSON

YoloObjectDetectionHandler.java

package nexus.io.kit.handler;

import java.util.List;

import com.imaginix.mc.cv.services.YoloObjectDetectionService;
import nexus.io.jfinal.aop.Aop;
import nexus.io.model.body.RespBodyVo;
import nexus.io.model.upload.UploadFile;
import nexus.io.tio.boot.http.TioRequestContext;
import nexus.io.tio.http.common.HttpRequest;
import nexus.io.tio.http.common.HttpResponse;
import nexus.io.tio.http.server.handler.HttpRequestHandler;
import nexus.io.tio.http.server.util.CORSUtils;
import nexus.io.yolo.domain.DetectionResult;

public class YoloObjectDetectionHandler implements HttpRequestHandler {

  YoloObjectDetectionService yoloDetectionService = Aop.get(YoloObjectDetectionService.class);

  @Override
  public HttpResponse handle(HttpRequest httpRequest) throws Exception {
    HttpResponse response = TioRequestContext.getResponse();
    CORSUtils.enableCORS(response);

    UploadFile uploadFile = httpRequest.getUploadFile("file");
    String name = uploadFile.getName();
    byte[] data = uploadFile.getData();

    List<DetectionResult> results = yoloDetectionService.detect(name, data);
    response.body(RespBodyVo.ok(results));
    return response;
  }
}

HttpResponse 获取方式说明

在 tio-boot 的 HttpRequestHandler 中，响应对象有两种常见获取方式：

HttpResponse response = new HttpResponse(httpRequest);

或者：

HttpResponse response = TioRequestContext.getResponse();

两种方式都可以返回响应。new HttpResponse(httpRequest) 会基于当前请求创建一个新的响应对象；TioRequestContext.getResponse() 则从 tio-boot 当前请求上下文中取得框架已经准备好的响应对象。

在 tio-boot 项目中更推荐使用：

HttpResponse response = TioRequestContext.getResponse();

原因是它与 tio-boot 的请求上下文保持一致，适合配合框架内置的上下文能力、拦截器、CORS、统一响应处理等场景使用。例如：

@Override
public HttpResponse handle(HttpRequest httpRequest) throws Exception {
  HttpResponse response = TioRequestContext.getResponse();
  CORSUtils.enableCORS(response);

  UploadFile uploadFile = httpRequest.getUploadFile("file");
  String name = uploadFile.getName();
  byte[] data = uploadFile.getData();

  List<DetectionResult> results = yoloDetectionService.detect(name, data);
  response.body(RespBodyVo.ok(results));
  return response;
}

如果是在更底层的 tio-http-server 用法中，没有经过 tio-boot 的请求上下文，也可以使用：

HttpResponse response = new HttpResponse(httpRequest);

六、路由注册

YoloObjectDetectionHandler yoloObjectDetectionHandler = new YoloObjectDetectionHandler();
r.add("/yolo/object/detection", yoloObjectDetectionHandler);

七、测试接口

curl 测试

curl -X POST http://localhost:8080/yolo/object/detection \
  -F "[email protected]"

八、工作流程（非常重要）

理解后就掌握 YOLO 推理了：

客户端图片
   ↓
tio 接收
   ↓
OpenCV 解码
   ↓
letterbox resize (640x640)
   ↓
归一化 → tensor
   ↓
ONNX Runtime 推理
   ↓
NMS 去重
   ↓
映射回原图坐标
   ↓
JSON 返回

九、并发与性能设计

该示例已经适合生产：

技术作用
线程池多请求并发推理
Semaphore限流防止CPU爆
单例SessionONNX线程安全
超时控制防止卡死

至此已经完成

现在拥有一个：

Java 高性能 YOLOv8 推理 HTTP 服务

可以直接给：

  • 前端上传图片
  • 摄像头抓拍
  • AI网关
  • 边缘计算设备

调用。