源码教程：原生对话框、DOM 弹窗与人机协作

这三类能力的实现对象不同：原生对话框由浏览器事件提供，DOM 弹窗是普通网页节点，人工请求是服务端维护的协作状态。它们不能共用“找到一个确定按钮就点”的逻辑。

1. 原生对话框记录

页面出现 alert、confirm 或 prompt 时，监听器保存最近一次对话框信息并按配置处理。读取命令返回的是记录，不保证对话框现在还在屏幕上。

命令	实现
`get_dialog`、`get_js_dialog`	两个名字调用同一 getDialog 方法；consume 为 true 时读后清除
`clear_dialog`、`clear_js_dialog`	清空最近对话框记录，不是关闭 DOM 窗口
`set_dialog_behavior`	设置后续原生对话框采用接受还是取消策略，参数为 dismiss

通过 seq、timestamp 区分新旧记录。不要在某次提交后读到很久以前的 alert，就误判本次提交结果。修改默认处理方式也不代表回放之前的对话框。

2. DOM 弹窗发现和关闭

get_modals 在页面执行弹窗识别逻辑，结合语义、框架特征及几何线索收集候选，输出标题、按钮及点击坐标。固定页头可能与弹窗一样有较高 z-index，所以不能仅按 z-index 判定。

close_modal 支持 which、title、button 选择目标。默认优先关闭顶部弹窗，使用真实鼠标，并在操作后检查弹窗数量等状态是否真的变化。closed=false 时不应循环重复点击，因为可能产生更多确认层。

{"id":"1001","method":"get_modals","params":{}}

先读取再决定是否关闭；短信验证、条款或业务确认并不一定应被关闭。get_modals 没找到候选也不能证明页面没有任何遮挡，仍可从快照文本和点击命中探针排查。

3. 人工请求的状态机

prompt 与非空 steps 至少提供一个；expiresAt 是毫秒时间戳，设置后优先于 timeoutSeconds。

request_human_input 创建 requestId，保存 prompt、expiresAt 和可选 steps；有图片目标时复用元素截图，可选 OCR。请求可以将当前页提到前台，但不会自动把信息推送到用户正在使用的聊天界面，宿主仍需要展示提示。

submit_human_input 根据 requestId 写入 answer，或根据 stepId/answers 更新多步请求。get_human_input 查询状态，允许短时间等待。多步完成一部分时可返回 partial，未完成且超过 expiresAt 时返回 expired。

创建 → pending
部分步骤答复 → partial（仍有待办）
全部答复 → answered
待答复且超过有效期 → expired

requestId 与浏览器任务 ID 不是同一个标识。用户直接在浏览器里完成操作，不一定同时更新人工请求记录；此时必须读取页面确认真实结果，不能只轮询 pending。

{"id":"1001","method":"request_human_input","params":{"prompt":"请在当前页面完成登录，完成后告知","timeoutSeconds":300}}

后续查询必须使用实际返回的 requestId。本教程不记录真实密码或验证码。过期请求不代表验证成功，不能凭时间流逝继续依赖该验证的步骤。

4. 验证方法

fixture 分别触发 alert 与 DOM 弹窗，验证两套查询不会混淆；再叠加两个 DOM 窗口验证目标选择和关闭回读。人工请求测试覆盖单步、多步部分答复、过期和不存在 requestId。测试“已答复”与网站“已登录”是两个不同断言，不能用一个替代另一个。

注册参数与 Java 入口

以下按 CommandTable 实际读取参数整理。* 表示注册层使用必填读取器；其余字段省略后由服务决定默认行为。带条件的入口仍需满足正文说明，例如上传文件来源、元素定位二选一。外层 id 不重复列出。

命令	params 字段	Java 入口
`get_dialog`	`consume`	`getDialog`
`clear_dialog`	无	`clearDialog`
`get_js_dialog`	`consume`	`getDialog`
`clear_js_dialog`	无	`clearDialog`
`set_dialog_behavior`	`dismiss`	`setDialogBehavior`
`get_modals`	无	`getModals`
`close_modal`	`which`、`title`、`button`	`closeModal`
`request_human_input`	`steps`、`prompt`、`index`、`selector`、`timeoutSeconds`、`expiresAt`、`ocr`、`ocrLanguage`、`inline`、`frame`	`requestHumanInput`
`get_human_input`	`requestId`*、`timeoutSeconds`	`getHumanInput`
`submit_human_input`	`requestId`*、`answer`、`stepId`、`answers`	`submitHumanInput`

当前源码：命令注册与执行

先在本章上半部分理解行为，再按命令展开实现。注册代码说明 JSON 参数如何传给 Java；服务方法展示实际浏览器操作。方法依赖共享类中的字段和辅助函数，不应脱离原类直接粘贴编译。

`get_dialog`

展开参数注册

源码：playwright-server/src/main/java/nexus/io/ai/browser/actions/registry/CommandTable.java。以下为当前实现，可放回原类中阅读；依赖同类字段和辅助方法，并非独立编译单元。

    put("get_dialog", (svc, id, a) -> svc.getDialog(id, a.getBoolean("consume")));

展开 getDialog 实现

源码：playwright-server/src/main/java/nexus/io/ai/browser/service/PlaywrightService.java。以下为当前实现，可放回原类中阅读；依赖同类字段和辅助方法，并非独立编译单元。

  public RespBodyVo getDialog(Long browserId, Boolean consume) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv dialog = inst.lastDialog;
    if (consume != null && consume) {
      inst.lastDialog = null;
    }
    if (dialog == null) {
      return RespBodyVo.ok(Kv.by("dialog", null));
    }
    return RespBodyVo.ok(Kv.by("dialog", dialog));
  }

`clear_dialog`

展开参数注册

    put("clear_dialog", (svc, id, a) -> svc.clearDialog(id));

展开 clearDialog 实现

  public RespBodyVo clearDialog(Long browserId) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    boolean cleared = inst.lastDialog != null;
    inst.lastDialog = null;
    return RespBodyVo.ok(Kv.by("cleared", cleared));
  }

`get_js_dialog`

展开参数注册

    put("get_js_dialog", (svc, id, a) -> svc.getDialog(id, a.getBoolean("consume")));

展开 getDialog 实现

  public RespBodyVo getDialog(Long browserId, Boolean consume) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv dialog = inst.lastDialog;
    if (consume != null && consume) {
      inst.lastDialog = null;
    }
    if (dialog == null) {
      return RespBodyVo.ok(Kv.by("dialog", null));
    }
    return RespBodyVo.ok(Kv.by("dialog", dialog));
  }

`clear_js_dialog`

展开参数注册

    put("clear_js_dialog", (svc, id, a) -> svc.clearDialog(id));

展开 clearDialog 实现

  public RespBodyVo clearDialog(Long browserId) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    boolean cleared = inst.lastDialog != null;
    inst.lastDialog = null;
    return RespBodyVo.ok(Kv.by("cleared", cleared));
  }

`set_dialog_behavior`

展开参数注册

    put("set_dialog_behavior", (svc, id, a) -> svc.setDialogBehavior(id, optBool(a, "dismiss")));

展开 setDialogBehavior 实现

  public RespBodyVo setDialogBehavior(Long browserId, boolean dismiss) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    inst.dismissDialogs = dismiss;
    return RespBodyVo.ok(Kv.by("dismiss", dismiss));
  }

`get_modals`

展开参数注册

    put("get_modals", (svc, id, a) -> svc.getModals(id));

展开 getModals 实现

  public RespBodyVo getModals(Long browserId) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv probe = modalProbe(inst);
    if (probe == null) {
      return RespBodyVo.fail("get_modals 失败：读取页面弹窗失败");
    }
    return RespBodyVo.ok(probe);
  }

展开参数注册

    put("close_modal", (svc, id, a) -> svc.closeModal(id, optStr(a, "which"), optStr(a, "title"),
        optStr(a, "button")));

展开 closeModal 实现

  public RespBodyVo closeModal(Long browserId, String which, String title, String button) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv before = modalProbe(inst);
    if (before == null) {
      return RespBodyVo.fail("close_modal 失败：读取页面弹窗失败");
    }
    int countBefore = asInt(before.get("count"));
    if (countBefore == 0) {
      return RespBodyVo.ok(Kv.by("closed", false).set("countBefore", 0).set("countAfter", 0)
          .set("note", "当前没有可见的 DOM 弹窗,无需关闭"));
    }
    String mode = which == null || which.isBlank() ? "top" : which.trim().toLowerCase(java.util.Locale.ROOT);
    List<Kv> targets = pickModals(before, mode, title);
    if (targets.isEmpty()) {
      return RespBodyVo.fail("close_modal 失败：没有匹配的弹窗（" + (mode.startsWith("class:")
          ? "className 含「" + mode.substring("class:".length()) + "」的弹窗不存在"
          : "标题/文本含「" + title + "」的弹窗不存在")
          + ",当前共 " + countBefore + " 个,用 get_modals 看清单（每项都有 className 与 matchedBy））");
    }
    List<Kv> clicked = new ArrayList<>();
    for (Kv target : targets) {
      Kv hit = clickModalButton(inst, target, button);
      if (hit != null) {
        clicked.add(hit);
      }
      // 关掉一个就重新探一次:数量真的减少了才算成功
      if (!"all".equals(mode)) {
        break;
      }
      Kv now = modalProbe(inst);
      if (now == null || asInt(now.get("count")) == 0) {
        break;
      }
    }
    Kv after = modalProbe(inst);
    int countAfter = after == null ? countBefore : asInt(after.get("count"));
    boolean closed = countAfter < countBefore;
    Kv data = Kv.by("closed", closed).set("countBefore", countBefore).set("countAfter", countAfter)
        .set("clicked", clicked).set("closedAll", countAfter == 0).set("which", mode);
    if (title != null && !title.isBlank()) {
      data.set("title", title);
    }
    if (!closed) {
      data.set("hint", "点了按钮但弹窗数量没减少:该弹窗可能只认真实鼠标事件、或点中的不是它的关闭按钮。"
          + "用 get_modals 拿 closePoint / buttonPoints,再直接 mouse_click 那个坐标");
    }
    return RespBodyVo.ok(data);
  }

`request_human_input`

展开参数注册

    put("request_human_input", (svc, id, a) -> {
      // prompt 与 steps 二选一:两个都没给时在**碰浏览器对象之前**就说清楚是缺哪个参数
      if (a.getJSONArray("steps") == null && optStrRaw(a, "prompt") == null) {
        throw new IllegalArgumentException("缺少参数 prompt");
      }
      return svc.requestHumanInput(id, optStrRaw(a, "prompt"), a.getInteger("index"), optStr(a, "selector"),
          a.getInteger("timeoutSeconds"), stepListOf(a.getJSONArray("steps")), a.getLong("expiresAt"),
          a.getBoolean("ocr"), optStr(a, "ocrLanguage"), a.getBoolean("inline"), optStr(a, "frame"));
    });

展开 requestHumanInput 实现

  public RespBodyVo requestHumanInput(Long browserId, String prompt, Integer index, String selector,
      Integer timeoutSeconds, List<Kv> steps, Long expiresAt, Boolean ocr, String ocrLanguage, Boolean inline,
      String frame) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    if ((prompt == null || prompt.isBlank()) && (steps == null || steps.isEmpty())) {
      return RespBodyVo.fail("request_human_input 需要 prompt,或用 steps 给出待办清单");
    }
    long now = System.currentTimeMillis();
    long deadline;
    if (expiresAt != null && expiresAt > 0) {
      deadline = expiresAt;
    } else {
    int ttl = timeoutSeconds == null || timeoutSeconds <= 0 ? DEFAULT_HUMAN_TIMEOUT_SECONDS : timeoutSeconds;
      deadline = now + ttl * 1_000L;
    }
    String requestId = "hr-" + inst.humanSeq.incrementAndGet() + "-" + SnowflakeIdUtils.id();
    Kv request = Kv.by("requestId", requestId).set("prompt", prompt).set("status", "pending").set("answer", null)
        .set("createdAt", now).set("expiresAt", deadline);
    inst.humanRequests.put(requestId, request);
    // 需要人工介入时把页面带到最前,人才能直接看到验证码/表单
    activate(inst.page);
    Kv data = Kv.by("requestId", requestId).set("prompt", prompt).set("expiresAt", deadline)
        .set("expiresInSeconds", Math.max(0, (deadline - now) / 1000)).set("url", inst.page.url());
    if (deadline <= now) {
      // 过期时刻已经过去了:直接说清楚,别让人等一个永远不会来的答复
      request.set("status", "expired");
      data.set("status", "expired").set("note", "expiresAt 已经过去了,这个请求没有生效;请重新发起");
      return RespBodyVo.ok(data);
    }

    boolean wantOcr = Boolean.TRUE.equals(ocr);
    boolean wantInline = inline == null || inline;
    // 第一个要看图的目标:老写法(index/selector)或 steps 里第一个带目标的步骤
    Integer shotIndex = index;
    String shotSelector = selector;
    String shotPrompt = prompt;
    // 验证码/二维码经常在跨域 iframe 里(实测企业微信登录页的二维码就在 iframe 中),
    // 不给 frame 时 selector 只在顶层文档找,回执里 imageUrl 会是空的 —— 所以这里一路透传下去
    String shotFrame = frame;
    if (steps != null && !steps.isEmpty()) {
      List<Kv> items = new ArrayList<>();
      int seq = 0;
      for (Kv step : steps) {
        String stepId = "s" + (++seq);
        Kv item = Kv.by("stepId", stepId).set("prompt", step.getStr("prompt")).set("status", "pending")
            .set("answer", null);
        if (step.get("index") != null) {
          item.set("index", step.get("index"));
        }
        if (step.getStr("selector") != null) {
          item.set("selector", step.getStr("selector"));
        }
        if (step.getStr("frame") != null) {
          item.set("frame", step.getStr("frame"));
        }
        items.add(item);
      }
      request.set("steps", items).set("status", "pending");
      data.set("steps", items).set("stepCount", items.size());
      if (shotPrompt == null || shotPrompt.isBlank()) {
        shotPrompt = steps.get(0).getStr("prompt");
      }
      if (shotIndex == null && (shotSelector == null || shotSelector.isBlank())) {
        Object stepIndex = steps.get(0).get("index");
        String stepSelector = steps.get(0).getStr("selector");
        shotIndex = stepIndex instanceof Number ? ((Number) stepIndex).intValue() : null;
        shotSelector = stepSelector;
        if (shotFrame == null || shotFrame.isBlank()) {
          shotFrame = steps.get(0).getStr("frame");
        }
      }
      data.set("note", "这是一次带多步待办的请求:让人一次做完,再用 submit_human_input 按 stepId 逐个回填;"
          + "全部回填后 get_human_input 的 status 会变成 answered");
    }

    if (shotIndex != null || (shotSelector != null && !shotSelector.isEmpty())) {
      // 请人看验证码/二维码是「确实必须看图」的场景:落盘 + 内联 + 可 GET 的 URL 一起给,
      // 读不了图的模型至少还能把 imageUrl 贴给用户,或走 OCR
      String path = defaultShotPath(inst);
      RespBodyVo shot = elementScreenshot(inst, shotIndex, shotSelector, path, wantInline, shotFrame);
      if (shot.isOk() && shot.getData() instanceof Kv) {
        Kv shotData = (Kv) shot.getData();
        if (wantInline) {
          data.set("imageBase64", shotData.getStr("base64"));
        }
        data.set("imageSize", shotData.get("size"))
            .set("imagePath", shotData.getStr("path"))
            .set("imageUrl", shotData.getStr("url"))
            // 截图取自哪个 frame/选择器:跨域 iframe 里截图时,这是判断「到底截到没有」的唯一线索
            .set("imageTarget", shotData.getStr("target"));
      } else {
        data.set("imageError", shot.getMsg());
      }
    }

    if (wantOcr) {
      String imagePath = data.getStr("imagePath");
      if (imagePath == null || imagePath.isEmpty()) {
        data.set("ocrError", "ocr:true 需要 index / selector(或 steps 里的第一个带目标的步骤)来指定要读的图");
      } else {
        Kv read = WindowsOcr.read(Paths.get(imagePath), ocrLanguage);
        data.set("ocr", read);
        if (Boolean.TRUE.equals(read.get("ok"))) {
          data.set("ocrText", read.getStr("text"));
        } else {
          data.set("ocrError", read.getStr("error"));
        }
      }
    }
    return RespBodyVo.ok(data);
  }

`get_human_input`

展开参数注册

    put("get_human_input",
        (svc, id, a) -> svc.getHumanInput(id, reqStr(a, "requestId"), a.getInteger("timeoutSeconds")));

展开 getHumanInput 实现

  public RespBodyVo getHumanInput(Long browserId, String requestId, Integer timeoutSeconds) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv request = inst.humanRequests.get(requestId);
    if (request == null) {
      return RespBodyVo.fail("get_human_input 没有这个请求: " + requestId);
    }
    long deadline = timeoutSeconds == null || timeoutSeconds <= 0 ? 0
        : System.currentTimeMillis() + timeoutSeconds * 1_000L;
    while ("pending".equals(humanStatus(request)) && System.currentTimeMillis() < deadline) {
      sleepQuietly(500);
    }
    String status = humanStatus(request);
    Kv data = Kv.by("requestId", requestId).set("status", status).set("answer", request.get("answer"))
        .set("prompt", request.get("prompt")).set("expiresAt", request.get("expiresAt"));
    List<Kv> steps = stepList(request);
    if (!steps.isEmpty()) {
      data.set("steps", steps);
      boolean anyAnswered = false;
      for (Kv step : steps) {
        if ("answered".equals(step.getStr("status"))) {
          anyAnswered = true;
          break;
        }
      }
      if (anyAnswered && "pending".equals(status)) {
        data.set("status", "partial");
        status = "partial";
      }
    }
    if ("expired".equals(status)) {
      data.set("expired", true)
          .set("hint", "这个人工请求已经过期:二维码/短信码这类有短时效的凭证通常也已经失效,"
              + "请重新发起 request_human_input(可以用 expiresAt 明确告诉它什么时候过期)");
    }
    return RespBodyVo.ok(data);
  }

`submit_human_input`

展开参数注册

    put("submit_human_input", (svc, id, a) -> svc.submitHumanInput(id, reqStr(a, "requestId"),
        optStrRaw(a, "answer"), optStr(a, "stepId"), a.getJSONObject("answers")));

展开 submitHumanInput 实现

  public RespBodyVo submitHumanInput(Long browserId, String requestId, String answer, String stepId,
      JSONObject answers) {
    BrowserInstance inst = INSTANCES.get(browserId);
    if (inst == null) {
      return notFound(browserId);
    }
    Kv request = inst.humanRequests.get(requestId);
    if (request == null) {
      return RespBodyVo.fail("submit_human_input 没有这个请求: " + requestId);
    }
    long now = System.currentTimeMillis();
    List<Kv> steps = stepList(request);
    // 多步待办:必须按 stepId / answers 逐条回填 —— 直接给一个 answer 说不清它对应哪一步,静默接受
    // 只会让剩下的步骤一直挂在 pending 上,所以这里明确报错并给出待办清单
    if (!steps.isEmpty()) {
      Map<String, String> filled = new LinkedHashMap<>();
      if (answers != null) {
        for (String key : answers.keySet()) {
          filled.put(key, answers.getString(key));
        }
      }
      if (stepId != null) {
        filled.put(stepId, answer);
      }
      if (filled.isEmpty()) {
        List<String> pendingIds = new ArrayList<>();
        for (Kv step : steps) {
          pendingIds.add(step.getStr("stepId") + "(" + step.getStr("prompt") + ")");
        }
        return RespBodyVo.fail("submit_human_input 这条请求有 " + steps.size() + " 个待办步骤,"
            + "请用 stepId 指定回填哪一步,或用 answers 一次回填多步。待办:" + String.join(" / ", pendingIds));
      }
      List<String> unknown = new ArrayList<>();
      for (Map.Entry<String, String> entry : filled.entrySet()) {
        Kv target = null;
        for (Kv step : steps) {
          if (entry.getKey().equals(step.getStr("stepId"))) {
            target = step;
            break;
          }
        }
        if (target == null) {
          unknown.add(entry.getKey());
          continue;
        }
        target.set("answer", entry.getValue()).set("status", "answered").set("answeredAt", now);
      }
      if (!unknown.isEmpty()) {
        return RespBodyVo.fail("submit_human_input 不认识的 stepId: " + String.join(" / ", unknown));
      }
      boolean allDone = true;
      List<Kv> pendingSteps = new ArrayList<>();
      for (Kv step : steps) {
        if (!"answered".equals(step.getStr("status"))) {
          allDone = false;
          pendingSteps.add(Kv.by("stepId", step.getStr("stepId")).set("prompt", step.getStr("prompt")));
        }
      }
      if (allDone) {
        request.set("status", "answered").set("answeredAt", now);
      }
      Kv data = Kv.by("requestId", requestId)
          .set("status", allDone ? "answered" : "partial").set("steps", steps);
      if (!allDone) {
        data.set("pendingSteps", pendingSteps)
            .set("note", "还有 " + pendingSteps.size() + " 步没回填;全部回填后 status 才是 answered");
      }
      return RespBodyVo.ok(data);
    }
    if (answer == null) {
      return RespBodyVo.fail("submit_human_input 需要 answer(或用 stepId / answers 逐条回填)");
    }
    request.set("answer", answer).set("status", "answered").set("answeredAt", now);
    return RespBodyVo.ok(Kv.by("requestId", requestId).set("status", "answered").set("answer", answer));
  }