diff --git a/.claude/specs/feishu-account-personalization/design.md b/.claude/specs/feishu-account-personalization/design.md new file mode 100644 index 0000000..95f9b50 --- /dev/null +++ b/.claude/specs/feishu-account-personalization/design.md @@ -0,0 +1,251 @@ +# 设计文档 + +## 概述 + +本设计在现有 FastAPI 模块化单体内增加飞书终端用户身份、个人画像与持久化订阅三个 +业务域。内部 API 继续使用服务 API Key；飞书终端权限只能来自已经通过验签或 SDK +验证的事件，身份主键固定为 `(tenant_key, open_id)`。 + +平台只写自身数据库。旧业务 MySQL 保持 SELECT-only。个人定时提示词不允许调用工具、 +读取公司报表或写入问答历史；群订阅也不加载创建者个人画像。 + +## 架构设计 + +### 系统架构图 + +```mermaid +flowchart LR + F["飞书 webhook / 长连接"] --> V["事件验证与解密边界"] + V --> I["feishu_users
身份与权限"] + I --> C["飞书命令编排"] + C --> P["personalization
规则/偏好/会话"] + C --> S["subscriptions
计划/投递"] + C --> A["ai_agent
受限问答"] + API["内部 API Key"] --> I + API --> S + DB[("平台数据库")] --- I + DB --- P + DB --- S + T["每分钟持久化扫描"] --> S + S --> A + S --> FC["FeishuClient"] +``` + +### 数据流图 + +```mermaid +sequenceDiagram + participant F as 飞书 + participant E as 事件服务 + participant U as 用户服务 + participant C as 命令服务 + participant P as 个性化服务 + participant A as AI + + F->>E: 已签名消息事件 + E->>E: 验签/解密、事件去重 + E->>U: tenant_key + sender open_id + U->>U: 查找或自动注册普通用户 + U-->>E: FeishuPrincipal + E->>C: principal + chat context + mentions + C->>P: 仅加载 principal 所有的数据 + P->>A: 安全规则→公司规则→个人规则→请求→偏好/记忆→历史 + A-->>C: 回答或明确不可用 + C-->>F: 原会话回复 +``` + +## 组件与接口 + +### `app/modules/feishu_users` + +- `models.py` 定义 `FeishuUser`，公开随机 `code`，内部使用整数主键。 +- `principal.py` 定义不可伪造的 `FeishuPrincipal`，包含用户编号、租户、open_id、角色、 + 状态、当前 chat_id/chat_type 和结构化 mentions。 +- `services/identity.py` 只接收验证边界传入的身份，负责首次注册、初始管理员引导、 + 最后活跃时间和旧自选认领。 +- `services/management.py` 负责列表、角色/状态修改、最后管理员保护和审计。 +- `routes.py` 暴露服务 API Key 保护的用户管理接口；正文中的 actor/open_id 不参与授权。 + +### `app/modules/personalization` + +- `UserPreference` 保存白名单类别：`language`、`tone`、`detail`、`topic`、`interest`。 +- `AIConversation` 唯一标识“用户 + 私聊/群聊”；`AIConversationMessage` 保存消息， + 每次读写时清理 30 天外记录并裁剪到最近 20 轮。 +- `services/preferences.py` 提供显式偏好管理和真实 AI 可用时的结构化提取；敏感类别及 + 密钥样式在入库前拒绝。 +- `services/conversations.py` 提供历史加载、追加、重置。 +- `services/context.py` 按固定优先级组装问答上下文。 +- `services/erasure.py` 实现一次性确认码和事务性“忘记我”，审计只保留随机匿名主体。 + +### `app/modules/subscriptions` + +- `services/schedule_parser.py` 实现受控中文语法，不使用不确定的自由文本推断。 +- `services/subscriptions.py` 管理创建、列表、暂停、恢复、退订、时区和安静时段。 +- `services/scanner.py` 每分钟领取到期订阅，使用行锁（支持时）与唯一投递键创建投递， + 并推进订阅的下一执行时间。 +- `services/delivery.py` 生成受限 AI 内容并发送；个人订阅仅使用个人上下文，群订阅仅使用 + 系统/公司规则。HTTP 错误、429、5xx 和飞书业务 `code != 0` 都是失败。 +- `tasks/` 和 `task_queue/` 仅作为执行适配层；关闭 Celery 时由扫描器直接处理持久化投递。 + +### 共享集成 + +- `FeishuEventService` 在验证后、注册事件前解析 `tenant_key/open_id`；缺失身份不创建数据。 +- `FeishuCommandService.handle_text` 接收 `FeishuPrincipal | None`。内部预览接口不构造终端 + 用户身份，因此只能预览不涉及个人数据的安全路径。 +- 管理员能力由统一权限守卫保护：公司规则、公司/财务/风险/考勤命令、用户管理、群订阅。 +- `FeishuClient.send_message` 接收租户键和可选稳定 `uuid`，并把非零业务码转换为可分类失败。 +- 自建应用使用 `/auth/v3/tenant_access_token/internal`；商店应用先使用最近一次已验证的 + `app_ticket` 获取 `app_access_token`，再按目标 `tenant_key` 获取 + `tenant_access_token`。应用令牌按应用缓存，租户令牌按 `(app_id, tenant_key)` 隔离缓存。 +- 只有通过 webhook 验真或长连接 SDK 验证的 `app_ticket` 事件可以轮换持久化票据； + 环境变量票据仅作为启动兜底，票据和访问令牌不得进入日志、审计或响应。 +- 命令回复、卡片、图片和个人/群订阅都显式携带事件主体的 `tenant_key`；固定默认群报表 + 使用 `FEISHU_DEFAULT_TENANT_KEY`，不得复用另一租户的令牌。 +- AI adapter context 增加每用户 session id；`noop` 明确返回不可用且不写会话、偏好或记忆。 + +## 数据模型 + +### `FeishuUser` + +- `id`, `code`, `tenant_key`, `open_id`, `union_id`, `user_id` +- `role` (`user|admin`), `status` (`active|disabled`) +- `timezone`, `quiet_hours_start`, `quiet_hours_end` +- `last_active_at`, `created_at`, `updated_at` +- 唯一约束：`(tenant_key, open_id)`；`code` 全局唯一。 + +### `FeishuAdminBootstrapTombstone` + +- 仅保存带域隔离的 `tenant_key + open_id` 不可逆摘要和创建时间。 +- 不保存原始飞书身份、owner、角色或其他画像；仅用于阻止已执行“忘记我”的 + 配置初始管理员在重新联系时被自动重授管理员。 + +### `UserPreference` + +- `id`, `code`, `owner_id`, `category`, `value`, `source`, `created_at`, `updated_at` +- 唯一约束：`(owner_id, category, normalized_value)`。 + +### `AIConversation` / `AIConversationMessage` + +- 会话：`id`, `code`, `owner_id`, `chat_type`, `chat_key`, `created_at`, `updated_at` +- 消息：`id`, `conversation_id`, `role`, `content`, `created_at` +- 唯一约束：`(owner_id, chat_type, chat_key)`。 + +### 现有模型扩展 + +- `AIMemoryEntry.owner_id` 可空；公司规则 `owner_id=NULL`，个人规则/记忆必须有 owner。 +- `AIMemoryEntry.kind` 区分 `company_rule|personal_rule|memory`，旧显式规则迁为 + `company_rule/legacy_company`，旧无所有者自动记忆归档。 +- 指纹唯一性改为 `(owner_id, fingerprint)`，公司数据使用空 owner 的独立范围。 +- `MarketWatchlist.owner_id` 可空；旧 actor 暂存为 legacy claim key，首次注册时认领， + 认领前个人查询不可见。 + +### `PushSubscription` + +- `id`, `code`, `owner_id`, `target_type` (`user|chat`), `target_id` +- `prompt`, `schedule_type`, `schedule_config`, `timezone`, `next_run_at` +- `status`, `consented_at`, `last_run_at`, `created_at`, `updated_at` +- 群目标仅从当前已验证群事件写入。 + +### `PushDelivery` + +- `id`, `code`, `subscription_id`, `scheduled_for`, `idempotency_key`, `message_uuid` +- `status`, `attempt_count`, `next_attempt_at`, `provider_message_id` +- `last_error`, `sent_at`, `created_at`, `updated_at` +- `idempotency_key` 和 `message_uuid` 全局唯一。 + +### `FeishuAppTicket` + +- `id`, `app_id`, `app_ticket`, `received_at`, `updated_at` +- `app_id` 全局唯一；只保留当前有效票据，不保留票据历史。 + +## 业务流程 + +### 身份与权限 + +```mermaid +flowchart TD + E["收到事件"] --> V{"来源已验证？"} + V -- 否 --> R["拒绝且不创建个人数据"] + V -- 是 --> K{"tenant_key/open_id 完整？"} + K -- 否 --> R + K -- 是 --> U["查找或注册普通用户"] + U --> B["应用 FEISHU_ADMIN_IDENTITIES 引导"] + B --> S{"用户有效且有权限？"} + S -- 否 --> D["拒绝并审计"] + S -- 是 --> C["执行命令"] +``` + +初始管理员配置按 `tenant_key:open_id` 精确匹配，多条配置均可各自完成首次引导。若该 +配置身份曾执行“忘记我”，系统在同一删除事务内保留不可逆 bootstrap tombstone；再次 +联系时只注册为普通用户。飞书管理员命令只读取事件中的结构化 mention open_id；内部 +API 修改由服务 principal 审计。降级或停用前锁定目标并统计有效管理员，禁止移除最后 +一个。 + +### 问答与偏好 + +1. 验证 principal 并按用户 + chat 建立会话。 +2. 加载启用的公司规则、当前用户个人规则、偏好/兴趣、相关个人记忆和最近历史。 +3. 按固定顺序传给 AI；使用由用户与会话派生的 provider session id。 +4. 真实 AI 成功后保存本轮消息，再做一次受限偏好提取；定时任务跳过两步。 +5. 每次读写裁剪到最近 20 轮并删除 30 天前消息。 + +### 计划解析 + +支持： + +- 单次：今天/明天 HH:mm，`YYYY-MM-DD HH:mm` +- 周期：每天、工作日、每周一至周日、每月 1-31 号 +- 间隔：每隔 N 分钟/小时，折算后不得短于 15 分钟 + +解析输出 `schedule_type + schedule_config + timezone + next_run_at(UTC)`。日期不存在、时间 +已过、模糊表达、无效 IANA 时区均拒绝。月末没有目标日期时跳到下个有效月份。 + +### 扫描、投递与重试 + +```mermaid +flowchart TD + S["每分钟扫描 next_run_at<=now"] --> L["领取订阅并创建唯一投递"] + L --> Q{"用户/订阅/频控/安静时段允许？"} + Q -- 否 --> P["跳过或延后，并推进计划"] + Q -- 是 --> G["生成受限内容"] + G --> F["以 delivery UUID 发送飞书"] + F --> O{"HTTP 与业务 code 成功？"} + O -- 是 --> X["标记 sent"] + O -- 可重试 --> B["1/5/15 分钟后 retry"] + O -- 最终失败 --> Z["标记 failed"] +``` + +投递尝试总计最多四次（初次 + 三次重试）。扫描器重启后继续处理 +`pending/retry` 状态。唯一投递键由 `subscription_id + scheduled_for` 派生；同一键重复任务 +返回已有结果，不再次调用飞书。 + +### 忘记我 + +首次命令仅保存哈希确认码与短期过期时间。确认后在一个事务内删除个人规则/记忆、偏好、 +兴趣、会话、订阅及投递中的个人内容，删除身份映射；相关审计 actor/target 替换为随机 +匿名标识，并清空目标、请求、响应和请求 ID。若被删除身份仍属于初始管理员配置，仅保留 +不可逆 bootstrap tombstone 防止重新授予；用户再次联系时按新普通用户注册。删除成功 +确认消息不再额外写入包含目标摘要或提供方响应的发送审计。 + +## 错误处理 + +- 未验证事件、缺身份、停用用户、权限不足：拒绝且只审计最小元数据。 +- AI 未配置或 `noop`：返回“AI 当前不可用”，不创建虚假回答、历史、偏好或记忆。 +- 计划解析失败、额度超限、手填群 id：使用可操作示例响应且不部分写入。 +- 飞书 429/5xx/超时及业务非零码：分类为可重试；其他 4xx 为最终失败。 +- 缺少飞书凭据且存在启用订阅：readiness 返回 degraded，不影响基础 health。 +- 商店应用缺少可用 `app_ticket` 或默认租户时 readiness 返回 degraded；自建应用若启用订阅 + 涉及多个租户时 readiness 返回 degraded，避免把单租户令牌错误用于其他租户。 +- 数据库竞争：依赖唯一约束兜底；冲突后回滚到保存点并读取已存在投递。 + +## 测试策略 + +- 单元测试：身份解析、权限矩阵、敏感偏好过滤、上下文顺序、会话裁剪、计划解析、 + 安静时段、下一执行时间和重试分类。 +- 服务测试：两租户/两用户隔离、最后管理员保护、群订阅绑定、忘记我级联与匿名审计。 +- 并发/幂等测试：多扫描器只产生一条投递、相同 UUID 不重复发送、业务非零码不成功。 +- API/事件测试：验签失败不注册、内部 API Key 保护、正文 actor 不参与权限。 +- 多租户认证测试：A/B 租户令牌缓存隔离、`app_ticket` 只能由已验证事件更新、文本/卡片/ + 图片/订阅均使用目标租户、固定群任务使用默认租户。 +- 迁移测试：Alembic head 与元数据一致，旧规则/记忆/自选按既定策略迁移。 +- 回归测试：现有固定群报表调度与现有内部接口继续工作。 diff --git a/.claude/specs/feishu-account-personalization/requirements.md b/.claude/specs/feishu-account-personalization/requirements.md new file mode 100644 index 0000000..b517cac --- /dev/null +++ b/.claude/specs/feishu-account-personalization/requirements.md @@ -0,0 +1,213 @@ +# 需求文档 + +## 简介 + +本功能为飞书用户提供基于飞书账号身份的安全问答、个人规则、长期记忆、 +兴趣偏好和定时推送能力。飞书账号是终端用户权限的唯一来源；内部 API Key +继续用于服务间认证，不得代表或伪造终端用户。 + +本功能只允许写入平台自身的身份、权限、偏好、订阅、记忆、审计和投递状态。 +现有业务 MySQL 必须保持 SELECT-only，不得增加业务数据回写、审批、付款、 +绩效定级、交易或其他越权动作。 + +## 已确认边界 + +- 单个飞书应用可以服务一个或多个飞书租户。 +- 飞书用户身份使用 `tenant_key + open_id` 唯一确定。 +- `union_id` 和飞书 `user_id` 可以作为关联标识，但不得单独作为默认权限主键。 +- `chat_id` 只表示会话或群推送目标，不表示用户身份。 +- 个人数据默认不共享；公司级规则必须经过管理员权限控制。 +- 用户规则和兴趣属于低信任数据，不能覆盖系统安全、只读、权限和审计规则。 +- 本阶段不建设终端用户 Web 页面，不同步修改飞书通讯录，也不回写任何业务系统。 + +## 需求列表 + +### 需求 1：飞书账号身份认证 + +**用户故事：** 作为飞书用户，我希望系统准确识别我的飞书账号，以便所有问答、 +规则和订阅都归属于我本人。 + +#### 验收条件 + +1. WHEN 系统收到已经通过飞书验证的 webhook 或长连接消息事件 THEN 系统 SHALL + 从事件中提取 `tenant_key` 和发送者 `open_id`，并解析为唯一用户身份。 +2. IF 消息事件缺少 `tenant_key`、`open_id` 或未通过飞书来源验证 THEN 系统 SHALL + 拒绝执行用户命令，且不得创建任何个人数据。 +3. WHEN 一个合法飞书账号首次与机器人交互 THEN 系统 SHALL 为其建立默认普通用户身份， + 且不得自动授予管理权限。 +4. WHERE 不同租户出现相同 `open_id` THEN 系统 SHALL 将其识别为不同用户。 +5. WHEN 系统处理群聊消息 THEN 系统 SHALL 以消息发送者账号作为用户身份，而不是以 + `chat_id` 或群成员身份作为用户身份。 +6. IF 飞书用户被停用 THEN 系统 SHALL 拒绝其所有受保护命令并记录拒绝审计。 + +### 需求 2：基于飞书账号的角色与权限 + +**用户故事：** 作为管理员，我希望权限绑定到飞书账号，以便普通用户只能操作自己的数据， +而受信任人员才能使用公司级能力。 + +#### 验收条件 + +1. WHEN 新飞书用户首次注册 THEN 系统 SHALL 仅授予普通用户角色。 +2. WHEN 用户执行命令 THEN 系统 SHALL 根据该飞书身份的有效角色和权限决定是否允许执行。 +3. IF 普通用户尝试创建、修改或停用公司级规则 THEN 系统 SHALL 拒绝操作并记录审计。 +4. IF 用户尝试访问未授权的公司、项目、财务、风险或审计能力 THEN 系统 SHALL + 返回不泄露受保护数据的拒绝响应。 +5. WHEN 内部服务管理用户角色或状态 THEN 系统 SHALL 要求有效的内部服务认证， + 且角色变更 SHALL 记录操作者、目标飞书用户、变更前后状态和时间。 +6. IF 请求正文提供 `actor`、`open_id` 或其他可伪造身份字段 THEN 系统 SHALL + 忽略这些字段作为权限依据。 + +### 需求 3：个人规则和记忆隔离 + +**用户故事：** 作为飞书用户，我希望我的规则和历史记忆只影响我自己的回答， +以免其他用户看到或受其影响。 + +#### 验收条件 + +1. WHEN 用户创建规则、偏好或 AI 记忆 THEN 系统 SHALL 将记录关联到当前飞书用户所有者。 +2. WHEN 系统列出、召回、更新、启停或删除个人规则与记忆 THEN 系统 SHALL + 强制按当前用户所有者过滤。 +3. IF 用户提交其他用户的数据编号 THEN 系统 SHALL 返回未找到或无权限， + 且不得泄露该记录是否存在。 +4. WHEN 两个用户保存相同内容 THEN 系统 SHALL 分别保存或去重到各自所有者范围内， + 不得复用另一用户的记录。 +5. WHEN 用户在群聊中创建个人规则或产生记忆 THEN 系统 SHALL 仍将其仅归属于发送者。 +6. WHERE 公司级规则被应用 THEN 系统 SHALL 仅使用已经启用且具有管理员来源的公司规则。 +7. WHEN 迁移现有无所有者的规则和记忆 THEN 系统 SHALL 不得把旧数据自动暴露给任意个人； + 无法安全归属的数据 SHALL 进入停用或待审状态。 + +### 需求 4：个人规则、偏好和兴趣管理 + +**用户故事：** 作为飞书用户，我希望通过飞书告诉机器人我的规则、表达偏好和兴趣， +并能随时查看或撤销，以便后续交流符合我的要求。 + +#### 验收条件 + +1. WHEN 普通用户发送学习规则命令 THEN 系统 SHALL 默认创建个人规则，而不是公司规则。 +2. WHEN 有权限的管理员明确发送公司规则命令 THEN 系统 SHALL 创建公司级规则并记录审计。 +3. WHEN 用户设置语言、语气、内容详略、关注主题或其他偏好 THEN 系统 SHALL + 以结构化个人偏好保存并关联当前用户。 +4. WHEN 用户添加或移除兴趣 THEN 系统 SHALL 更新当前用户的兴趣数据， + 且现有个人股票自选 SHALL 作为市场兴趣来源之一。 +5. WHEN 普通问答中出现语言、语气、详略或内容主题等白名单偏好线索且真实 AI + 提供方可用 THEN 系统 SHALL 静默提取并永久保存非敏感偏好；IF 内容涉及密钥、 + 健康、宗教、政治、性取向、绩效或财务秘密 THEN 系统 SHALL 拒绝保存。 +6. WHEN 用户请求查看、启停、修改或删除个人规则与偏好 THEN 系统 SHALL + 仅操作当前用户的数据并返回明确结果。 +7. IF 用户规则与安全、权限、只读或公司规则冲突 THEN 系统 SHALL 忽略冲突部分并保留审计。 + +### 需求 5：个性化 AI 问答 + +**用户故事：** 作为飞书用户，我希望 AI 根据我的规则、偏好、兴趣和相关记忆回答， +同时不混入其他用户的信息。 + +#### 验收条件 + +1. WHEN 用户发起 AI 问答 THEN 系统 SHALL 按当前飞书身份加载公司规则、个人规则、 + 个人偏好、个人兴趣和当前用户的相关记忆。 +2. WHEN 系统组装 AI 上下文 THEN 系统 SHALL 按照“系统安全与只读规则、公司规则、 + 个人明确规则、当前请求、偏好与兴趣、相关记忆”的优先顺序处理。 +3. IF AI 提供方支持会话标识 THEN 系统 SHALL 为不同飞书用户使用隔离的会话标识， + 不得复用全局用户会话。 +4. IF AI 提供方不可用或仍为占位提供方 THEN 系统 SHALL 返回明确的不可用响应， + 不得把占位内容当作真实个性化答案。 +5. WHEN 问答来自群聊 THEN 系统 SHALL 回复原会话，但个人规则、偏好和记忆仍只属于发送者。 +6. IF 个人规则试图要求调用未授权工具、修改业务数据或绕过安全控制 THEN 系统 SHALL + 拒绝执行该要求。 + +### 需求 6：个人推送订阅 + +**用户故事：** 作为飞书用户，我希望按自己的兴趣和时间订阅消息，并能暂停或退订， +以便只收到需要的内容。 + +#### 验收条件 + +1. WHEN 用户创建订阅 THEN 系统 SHALL 保存订阅主题、飞书接收身份、计划时间、时区、 + 安静时段、启用状态和用户同意时间。 +2. WHEN 用户未明确创建或同意订阅 THEN 系统 SHALL 不得主动向该用户发送个人推送。 +3. WHEN 用户查看订阅 THEN 系统 SHALL 仅返回当前用户的订阅。 +4. WHEN 用户暂停、恢复或退订 THEN 系统 SHALL 立即更新当前用户订阅状态并记录审计。 +5. IF 用户尝试修改其他用户订阅 THEN 系统 SHALL 拒绝且不泄露目标订阅信息。 +6. WHEN 订阅主题支持个人兴趣过滤 THEN 系统 SHALL 使用当前用户明确保存的兴趣生成内容。 +7. WHILE 当前时间位于用户安静时段内 THEN 系统 SHALL 延后非紧急推送，不得直接发送。 +8. IF 订阅时间、时区、主题或接收目标无效 THEN 系统 SHALL 拒绝创建并返回可操作的错误说明。 +9. WHEN 用户以受控中文时间语法创建订阅且解析成功 THEN 系统 SHALL 立即启用订阅， + 返回标准化计划、下次执行时间和暂停命令。 +10. IF 订阅间隔短于 15 分钟、用户已有 50 个启用订阅或当日投递将超过 96 条 + THEN 系统 SHALL 拒绝创建或跳过投递并说明限制。 +11. WHEN 普通用户创建订阅 THEN 系统 SHALL 固定绑定其本人 `open_id`； + WHEN 管理员在群内创建群订阅 THEN 系统 SHALL 仅绑定当前事件的 `chat_id`， + 且 SHALL 拒绝用户手工提供原始 `chat_id`。 + +### 需求 7：可靠的个性化定时投递 + +**用户故事：** 作为订阅用户，我希望消息按时且不重复地送达，并在临时失败后安全重试。 + +#### 验收条件 + +1. WHEN 个人订阅到期 THEN 系统 SHALL 通过持久化订阅状态发现任务并生成投递请求， + 不得仅依赖进程内临时任务。 +2. WHEN 系统发送个人消息 THEN 系统 SHALL 默认使用当前用户的飞书 `open_id` + 作为接收目标。 +3. WHEN 系统处理同一用户、同一订阅和同一时间窗口 THEN 系统 SHALL 只创建一次有效投递。 +4. IF 飞书返回 HTTP 错误、限流响应或非零业务状态码 THEN 系统 SHALL 将投递标记为失败， + 不得记录为成功。 +5. IF 投递失败属于可重试错误 THEN 系统 SHALL 按受限退避策略重试； + 超过最大次数后 SHALL 进入最终失败状态。 +6. WHEN 投递成功 THEN 系统 SHALL 保存飞书响应标识、发送时间和最终状态。 +7. IF 多个调度器或工作进程同时扫描同一订阅 THEN 系统 SHALL 防止重复生成或重复发送消息。 +8. WHILE `READ_ONLY_MODE=true` THEN 系统 SHALL 允许平台自身的订阅与投递状态写入， + 但 SHALL 保持旧业务数据库只读。 + +### 需求 8：飞书自助命令 + +**用户故事：** 作为飞书用户，我希望直接通过机器人管理个人规则、偏好和订阅， +无需访问后台页面。 + +#### 验收条件 + +1. WHEN 用户请求帮助 THEN 系统 SHALL 返回其当前权限允许使用的命令说明。 +2. WHEN 用户管理个人规则 THEN 系统 SHALL 支持创建、查看、修改、启停和删除。 +3. WHEN 用户管理偏好或兴趣 THEN 系统 SHALL 支持设置、查看和删除。 +4. WHEN 用户管理订阅 THEN 系统 SHALL 支持创建、查看、暂停、恢复和退订。 +5. WHEN 管理员执行公司级命令 THEN 系统 SHALL 在执行前验证其飞书账号权限。 +6. IF 命令格式错误 THEN 系统 SHALL 返回示例格式，且不得产生部分写入。 +7. IF 命令涉及敏感信息、密钥或禁止内容 THEN 系统 SHALL 拒绝保存并给出安全提示。 +8. WHEN 管理员通过飞书管理用户角色或状态 THEN 系统 SHALL 只接受当前验证事件中的 + 结构化 `@用户` 身份，不得接受文本伪造的 `open_id`。 +9. IF 操作会停用或降级最后一个有效管理员 THEN 系统 SHALL 拒绝操作。 + +### 需求 9：隐私、审计和用户控制 + +**用户故事：** 作为飞书用户和审计人员，我希望个性化数据受到保护且关键操作可追踪。 + +#### 验收条件 + +1. WHEN 系统认证用户、拒绝权限、修改规则、修改偏好、修改订阅或执行投递 THEN 系统 SHALL + 记录必要的审计元数据。 +2. WHEN 系统记录审计或错误 THEN 系统 SHALL 避免保存访问令牌、密钥和不必要的完整个人问答内容。 +3. WHEN 用户请求删除个人画像和记忆 THEN 系统 SHALL 删除或匿名化可删除的个人数据， + 停止其订阅，并保留满足审计要求的最小不可逆证据。 +4. WHEN 用户请求查看自己的个性化数据 THEN 系统 SHALL 返回其规则、偏好、兴趣和订阅摘要。 +5. IF 内部服务查询个人数据 THEN 系统 SHALL 根据服务权限限制数据范围并记录访问审计。 +6. WHEN 用户发送“忘记我” THEN 系统 SHALL 返回短时有效的一次性确认码； + WHEN 同一用户发送“确认忘记我 <码>”且验证码有效 THEN 系统 SHALL 在一个事务中 + 删除其个人数据、停用订阅、移除身份映射并将审计主体替换为随机匿名标识。 + +### 需求 10：兼容性、迁移与验证 + +**用户故事：** 作为维护人员，我希望现有固定群推送和内部 API 保持可用， +同时安全升级到飞书用户权限体系。 + +#### 验收条件 + +1. WHEN 数据库升级 THEN 系统 SHALL 通过 Alembic 创建身份、权限、偏好、订阅和投递所需结构。 +2. WHEN 升级现有数据库 THEN 系统 SHALL 保留已有审计、报表推送和业务只读数据。 +3. WHEN 现有固定群定时任务运行 THEN 系统 SHALL 保持原有功能，且不得被误认为个人订阅。 +4. WHEN 内部 API 使用 API Key 调用 THEN 系统 SHALL 保持服务级认证， + 但不得通过请求参数伪造飞书终端用户权限。 +5. WHEN 自动化测试运行 THEN 系统 SHALL 覆盖两个不同飞书用户的数据隔离、 + 越权拒绝、角色权限、个人问答上下文、订阅生命周期、安静时段、幂等投递、 + 飞书非零业务码和数据库迁移一致性。 +6. WHEN 完整验证运行 THEN Ruff、Python 编译检查、Alembic 元数据一致性测试和全部 pytest + 测试 SHALL 通过。 diff --git a/.claude/specs/feishu-account-personalization/tasks.md b/.claude/specs/feishu-account-personalization/tasks.md new file mode 100644 index 0000000..4734dad --- /dev/null +++ b/.claude/specs/feishu-account-personalization/tasks.md @@ -0,0 +1,70 @@ +# 实现任务 + +- [x] 1. 建立飞书用户身份与权限域 + - 新增 `app/modules/feishu_users/` 的模型、schema、principal、身份与管理服务。 + - 实现首次注册、初始管理员引导、角色/状态修改、最后管理员保护与审计。 + - 增加用户管理 API 和身份/权限测试。 + - _Requirements: 1, 2, 8.5, 8.8, 8.9, 9.1, 10.4_ + +- [x] 2. 将验证后的飞书事件接入用户主体 + - 扩展 webhook/长连接事件提取 `tenant_key`、sender ids、chat type 和 mentions。 + - 只有验证后的事件才能解析或创建 `FeishuPrincipal`；群聊始终使用发送者身份。 + - 把 principal 传入命令处理并在停用/缺身份时安全拒绝。 + - _Requirements: 1, 2.2, 2.4, 8.1, 9.1_ + +- [x] 3. 实现个人规则、偏好、兴趣和记忆隔离 + - 新增 `app/modules/personalization/` 偏好模型与服务。 + - 为 AI 规则/记忆和市场自选增加 owner，按 owner 查询、去重和安全认领。 + - 将现有 AI rule API 固定为公司规则入口；飞书个人/公司规则命令分别授权。 + - 实现个人规则、偏好、兴趣 CRUD 命令及跨用户隔离测试。 + - _Requirements: 3, 4, 8.2, 8.3, 8.5, 8.7, 9.4_ + +- [x] 4. 实现会话与个性化 AI 问答 + - 新增会话和消息模型、20 轮裁剪、30 天清理与重置命令。 + - 按既定优先级组装公司规则、个人规则、请求、偏好/兴趣、记忆和会话历史。 + - 隔离 provider session；`noop`/AI 不可用时不写历史、偏好或记忆。 + - 实现白名单自动偏好提取与敏感画像拒绝。 + - _Requirements: 3, 4.5, 5, 8.3, 8.7, 9.2_ + +- [x] 5. 实现订阅与受控中文计划解析 + - 新增订阅/投递模型、schema 和服务。 + - 支持单次、每天、工作日、每周、每月和间隔计划以及 IANA 时区。 + - 实现 15 分钟下限、50 个启用订阅、每日 96 条、安静时段。 + - 实现私聊固定 open_id、管理员当前群绑定以及订阅生命周期命令/API。 + - _Requirements: 6, 8.4, 8.6, 8.7, 10.4_ + +- [x] 6. 实现持久化扫描、幂等投递和重试 + - 每分钟扫描到期订阅，通过行锁和唯一投递键领取并推进下一执行时间。 + - 为 Feishu client 增加 `open_id`、消息 UUID 和业务状态校验。 + - 实现个人/群订阅上下文边界、1/5/15 分钟重试和无 Celery 持久化恢复。 + - 接入 scheduler、Celery task/queue helper 并保持固定群报表任务不变。 + - _Requirements: 6.7, 7, 10.3_ + +- [x] 7. 实现用户数据查看、忘记我和运行可观测性 + - 实现 `我的数据`、二次确认、事务删除和审计匿名化。 + - 增加用户/订阅/投递查询 API、活跃用户和投递状态指标。 + - 启用订阅但缺少飞书凭据时 readiness 返回 degraded。 + - _Requirements: 6.2, 9, 10.2, 10.4_ + +- [x] 8. 完成配置、Alembic 迁移与全量验证 + - 增加安全关闭的功能开关、管理员身份配置并更新 `.env.example`。 + - 创建迁移并处理旧公司规则、旧自动记忆和旧自选延迟认领。 + - 补齐身份、隔离、权限、计划、并发、投递、删除和迁移回归测试。 + - 运行 Ruff、compileall、完整 pytest 和迁移一致性检查。 + - _Requirements: 3.7, 10_ + +```mermaid +flowchart LR + T1["1 身份权限"] --> T2["2 事件主体"] + T1 --> T3["3 个人数据隔离"] + T2 --> T4["4 个性化问答"] + T3 --> T4 + T1 --> T5["5 订阅与计划"] + T5 --> T6["6 扫描投递"] + T3 --> T7["7 删除与可观测性"] + T5 --> T7 + T2 --> T8["8 迁移与验证"] + T4 --> T8 + T6 --> T8 + T7 --> T8 +``` diff --git a/alembic/env.py b/alembic/env.py index 0eec56f..ad710c2 100644 --- a/alembic/env.py +++ b/alembic/env.py @@ -10,7 +10,10 @@ from app.modules.audit import models as audit_models from app.modules.business import models as business_models from app.modules.events import models as event_models from app.modules.feishu import models as feishu_models +from app.modules.feishu_users import models as feishu_user_models from app.modules.observability import models as observability_models +from app.modules.personalization import models as personalization_models +from app.modules.subscriptions import models as subscription_models from app.modules.workflows import models as workflow_models config = context.config @@ -28,7 +31,10 @@ _REGISTERED_MODEL_MODULES = ( business_models, event_models, feishu_models, + feishu_user_models, observability_models, + personalization_models, + subscription_models, workflow_models, ) diff --git a/alembic/versions/202607260001_v3_runtime_reliability.py b/alembic/versions/202607260001_v3_runtime_reliability.py new file mode 100644 index 0000000..f61b526 --- /dev/null +++ b/alembic/versions/202607260001_v3_runtime_reliability.py @@ -0,0 +1,51 @@ +"""Harden V3 heartbeat identity and concurrent updates. + +Revision ID: 202607260001 +Revises: 202607150001 +Create Date: 2026-07-26 +""" + +from alembic import op +import sqlalchemy as sa + + +revision = "202607260001" +down_revision = "202607150001" +branch_labels = None +depends_on = None + +HEARTBEAT_TABLE = "system_heartbeats" +HEARTBEAT_IDENTITY_CONSTRAINT = "uq_system_heartbeat_component_instance" + + +def upgrade() -> None: + op.execute( + sa.text( + """ + DELETE FROM system_heartbeats + WHERE id IN ( + SELECT id + FROM ( + SELECT + id, + ROW_NUMBER() OVER ( + PARTITION BY component, instance_id + ORDER BY last_seen_at DESC, id DESC + ) AS duplicate_rank + FROM system_heartbeats + ) ranked + WHERE duplicate_rank > 1 + ) + """ + ) + ) + with op.batch_alter_table(HEARTBEAT_TABLE) as batch_op: + batch_op.create_unique_constraint( + HEARTBEAT_IDENTITY_CONSTRAINT, + ["component", "instance_id"], + ) + + +def downgrade() -> None: + with op.batch_alter_table(HEARTBEAT_TABLE) as batch_op: + batch_op.drop_constraint(HEARTBEAT_IDENTITY_CONSTRAINT, type_="unique") diff --git a/alembic/versions/202607260002_v3_workflow_memory_idempotency.py b/alembic/versions/202607260002_v3_workflow_memory_idempotency.py new file mode 100644 index 0000000..8871ea2 --- /dev/null +++ b/alembic/versions/202607260002_v3_workflow_memory_idempotency.py @@ -0,0 +1,64 @@ +"""Add V3 workflow-event and AI-memory idempotency keys. + +Revision ID: 202607260002 +Revises: 202607260001 +Create Date: 2026-07-26 +""" + +from alembic import op +import sqlalchemy as sa + + +revision = "202607260002" +down_revision = "202607260001" +branch_labels = None +depends_on = None + +AI_MEMORY_TABLE = "ai_memory_entries" +AI_MEMORY_FINGERPRINT_INDEX = "ix_ai_memory_entries_fingerprint" +WORKFLOW_ACTION_TABLE = "workflow_actions" +WORKFLOW_SOURCE_EVENT_INDEX = "ix_workflow_actions_source_event_id" +WORKFLOW_SOURCE_EVENT_FOREIGN_KEY = "fk_workflow_actions_source_event_id" + + +def upgrade() -> None: + with op.batch_alter_table(AI_MEMORY_TABLE) as batch_op: + batch_op.add_column( + sa.Column("fingerprint", sa.String(length=64), nullable=True) + ) + batch_op.create_index( + AI_MEMORY_FINGERPRINT_INDEX, + ["fingerprint"], + unique=True, + ) + + with op.batch_alter_table(WORKFLOW_ACTION_TABLE) as batch_op: + batch_op.add_column( + sa.Column("source_event_id", sa.String(length=64), nullable=True) + ) + batch_op.create_index( + WORKFLOW_SOURCE_EVENT_INDEX, + ["source_event_id"], + unique=True, + ) + batch_op.create_foreign_key( + WORKFLOW_SOURCE_EVENT_FOREIGN_KEY, + "domain_events", + ["source_event_id"], + ["event_id"], + ondelete="RESTRICT", + ) + + +def downgrade() -> None: + with op.batch_alter_table(WORKFLOW_ACTION_TABLE) as batch_op: + batch_op.drop_constraint( + WORKFLOW_SOURCE_EVENT_FOREIGN_KEY, + type_="foreignkey", + ) + batch_op.drop_index(WORKFLOW_SOURCE_EVENT_INDEX) + batch_op.drop_column("source_event_id") + + with op.batch_alter_table(AI_MEMORY_TABLE) as batch_op: + batch_op.drop_index(AI_MEMORY_FINGERPRINT_INDEX) + batch_op.drop_column("fingerprint") diff --git a/alembic/versions/202607260003_feishu_personalization_subscriptions.py b/alembic/versions/202607260003_feishu_personalization_subscriptions.py new file mode 100644 index 0000000..d077200 --- /dev/null +++ b/alembic/versions/202607260003_feishu_personalization_subscriptions.py @@ -0,0 +1,481 @@ +"""Add Feishu identities, personalization, and durable subscriptions. + +Revision ID: 202607260003 +Revises: 202607260002 +Create Date: 2026-07-26 +""" + +from alembic import op +import sqlalchemy as sa + + +revision = "202607260003" +down_revision = "202607260002" +branch_labels = None +depends_on = None + +AI_MEMORY_TABLE = "ai_memory_entries" +MARKET_WATCHLIST_TABLE = "market_watchlists" + + +def _create_indexes( + table_name: str, + definitions: tuple[tuple[str, tuple[str, ...], bool], ...], +) -> None: + for name, columns, unique in definitions: + op.create_index(op.f(name), table_name, list(columns), unique=unique) + + +def upgrade() -> None: + _create_feishu_users() + _upgrade_ai_memory() + _upgrade_market_watchlists() + _create_personalization_tables() + _create_subscription_tables() + + +def _create_feishu_users() -> None: + op.create_table( + "feishu_users", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=64), nullable=False), + sa.Column("tenant_key", sa.String(length=128), nullable=False), + sa.Column("open_id", sa.String(length=128), nullable=False), + sa.Column("union_id", sa.String(length=128), nullable=True), + sa.Column("user_id", sa.String(length=128), nullable=True), + sa.Column("role", sa.String(length=32), nullable=False), + sa.Column("status", sa.String(length=32), nullable=False), + sa.Column("timezone", sa.String(length=64), nullable=False), + sa.Column("quiet_hours_start", sa.Time(), nullable=True), + sa.Column("quiet_hours_end", sa.Time(), nullable=True), + sa.Column("last_active_at", sa.DateTime(), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "tenant_key", + "open_id", + name="uq_feishu_user_tenant_open_id", + ), + ) + _create_indexes( + "feishu_users", + ( + ("ix_feishu_users_code", ("code",), True), + ("ix_feishu_users_created_at", ("created_at",), False), + ("ix_feishu_users_last_active_at", ("last_active_at",), False), + ("ix_feishu_users_open_id", ("open_id",), False), + ("ix_feishu_users_role", ("role",), False), + ("ix_feishu_users_status", ("status",), False), + ("ix_feishu_users_tenant_key", ("tenant_key",), False), + ("ix_feishu_users_union_id", ("union_id",), False), + ("ix_feishu_users_user_id", ("user_id",), False), + ), + ) + + +def _upgrade_ai_memory() -> None: + op.drop_index( + "ix_ai_memory_entries_fingerprint", + table_name=AI_MEMORY_TABLE, + ) + with op.batch_alter_table(AI_MEMORY_TABLE) as batch_op: + batch_op.add_column( + sa.Column("owner_id", sa.Integer(), nullable=True) + ) + batch_op.add_column( + sa.Column( + "kind", + sa.String(length=32), + nullable=False, + server_default="memory", + ) + ) + batch_op.create_foreign_key( + "fk_ai_memory_entries_owner_id", + "feishu_users", + ["owner_id"], + ["id"], + ondelete="CASCADE", + ) + batch_op.create_unique_constraint( + "uq_ai_memory_owner_fingerprint", + ["owner_id", "fingerprint"], + ) + batch_op.create_index( + "ix_ai_memory_entries_fingerprint", + ["fingerprint"], + unique=False, + ) + batch_op.create_index( + "ix_ai_memory_entries_owner_id", + ["owner_id"], + unique=False, + ) + batch_op.create_index( + "ix_ai_memory_entries_kind", + ["kind"], + unique=False, + ) + + connection = op.get_bind() + connection.execute( + sa.text( + """ + UPDATE ai_memory_entries + SET + owner_id = NULL, + kind = 'company_rule', + source = 'legacy_company', + status = 'active' + WHERE source = 'user_rule' + """ + ) + ) + connection.execute( + sa.text( + """ + UPDATE ai_memory_entries + SET kind = 'memory', status = 'archived' + WHERE owner_id IS NULL AND source IN ('auto', 'hermes') + """ + ) + ) + with op.batch_alter_table(AI_MEMORY_TABLE) as batch_op: + batch_op.alter_column( + "kind", + existing_type=sa.String(length=32), + nullable=False, + server_default=None, + ) + + +def _upgrade_market_watchlists() -> None: + with op.batch_alter_table(MARKET_WATCHLIST_TABLE) as batch_op: + batch_op.drop_constraint( + "uq_market_watchlist_actor_symbol", + type_="unique", + ) + batch_op.add_column(sa.Column("owner_id", sa.Integer(), nullable=True)) + batch_op.create_foreign_key( + "fk_market_watchlists_owner_id", + "feishu_users", + ["owner_id"], + ["id"], + ondelete="CASCADE", + ) + batch_op.create_unique_constraint( + "uq_market_watchlist_owner_symbol", + ["owner_id", "symbol"], + ) + batch_op.create_index( + "ix_market_watchlists_owner_id", + ["owner_id"], + unique=False, + ) + + +def _create_personalization_tables() -> None: + op.create_table( + "user_preferences", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=64), nullable=False), + sa.Column("owner_id", sa.Integer(), nullable=False), + sa.Column("category", sa.String(length=32), nullable=False), + sa.Column("value", sa.Text(), nullable=False), + sa.Column("normalized_value", sa.String(length=1000), nullable=False), + sa.Column("source", sa.String(length=32), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["owner_id"], + ["feishu_users.id"], + name="fk_user_preferences_owner_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "owner_id", + "category", + "normalized_value", + name="uq_user_preference_owner_category_value", + ), + ) + _create_indexes( + "user_preferences", + ( + ("ix_user_preferences_category", ("category",), False), + ("ix_user_preferences_code", ("code",), True), + ("ix_user_preferences_created_at", ("created_at",), False), + ("ix_user_preferences_owner_id", ("owner_id",), False), + ("ix_user_preferences_source", ("source",), False), + ), + ) + + op.create_table( + "ai_conversations", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=64), nullable=False), + sa.Column("owner_id", sa.Integer(), nullable=False), + sa.Column("chat_type", sa.String(length=32), nullable=False), + sa.Column("chat_key", sa.String(length=256), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["owner_id"], + ["feishu_users.id"], + name="fk_ai_conversations_owner_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "owner_id", + "chat_type", + "chat_key", + name="uq_ai_conversation_owner_chat", + ), + ) + _create_indexes( + "ai_conversations", + ( + ("ix_ai_conversations_chat_key", ("chat_key",), False), + ("ix_ai_conversations_chat_type", ("chat_type",), False), + ("ix_ai_conversations_code", ("code",), True), + ("ix_ai_conversations_created_at", ("created_at",), False), + ("ix_ai_conversations_owner_id", ("owner_id",), False), + ("ix_ai_conversations_updated_at", ("updated_at",), False), + ), + ) + + op.create_table( + "ai_conversation_messages", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("conversation_id", sa.Integer(), nullable=False), + sa.Column("role", sa.String(length=32), nullable=False), + sa.Column("content", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["conversation_id"], + ["ai_conversations.id"], + name="fk_ai_conversation_messages_conversation_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + ) + _create_indexes( + "ai_conversation_messages", + ( + ( + "ix_ai_conversation_messages_conversation_id", + ("conversation_id",), + False, + ), + ("ix_ai_conversation_messages_created_at", ("created_at",), False), + ("ix_ai_conversation_messages_role", ("role",), False), + ), + ) + + op.create_table( + "personal_data_erasure_requests", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("owner_id", sa.Integer(), nullable=False), + sa.Column("token_hash", sa.String(length=64), nullable=False), + sa.Column("expires_at", sa.DateTime(), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["owner_id"], + ["feishu_users.id"], + name="fk_personal_data_erasure_requests_owner_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + ) + _create_indexes( + "personal_data_erasure_requests", + ( + ( + "ix_personal_data_erasure_requests_created_at", + ("created_at",), + False, + ), + ( + "ix_personal_data_erasure_requests_expires_at", + ("expires_at",), + False, + ), + ( + "ix_personal_data_erasure_requests_owner_id", + ("owner_id",), + True, + ), + ), + ) + + +def _create_subscription_tables() -> None: + op.create_table( + "push_subscriptions", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=64), nullable=False), + sa.Column("owner_id", sa.Integer(), nullable=False), + sa.Column("target_type", sa.String(length=16), nullable=False), + sa.Column("target_id", sa.String(length=256), nullable=False), + sa.Column("prompt", sa.Text(), nullable=False), + sa.Column("schedule_type", sa.String(length=32), nullable=False), + sa.Column("schedule_config", sa.JSON(), nullable=False), + sa.Column("timezone", sa.String(length=64), nullable=False), + sa.Column("next_run_at", sa.DateTime(), nullable=True), + sa.Column("status", sa.String(length=32), nullable=False), + sa.Column("consented_at", sa.DateTime(), nullable=False), + sa.Column("last_run_at", sa.DateTime(), nullable=True), + sa.Column("locked_by", sa.String(length=128), nullable=True), + sa.Column("locked_until", sa.DateTime(), nullable=True), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["owner_id"], + ["feishu_users.id"], + name="fk_push_subscriptions_owner_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + ) + _create_indexes( + "push_subscriptions", + ( + ("ix_push_subscriptions_code", ("code",), True), + ("ix_push_subscriptions_created_at", ("created_at",), False), + ("ix_push_subscriptions_locked_by", ("locked_by",), False), + ("ix_push_subscriptions_locked_until", ("locked_until",), False), + ("ix_push_subscriptions_next_run_at", ("next_run_at",), False), + ("ix_push_subscriptions_owner_id", ("owner_id",), False), + ("ix_push_subscriptions_schedule_type", ("schedule_type",), False), + ("ix_push_subscriptions_status", ("status",), False), + ("ix_push_subscriptions_target_type", ("target_type",), False), + ), + ) + + op.create_table( + "push_deliveries", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=64), nullable=False), + sa.Column("subscription_id", sa.Integer(), nullable=False), + sa.Column("scheduled_for", sa.DateTime(), nullable=False), + sa.Column("idempotency_key", sa.String(length=64), nullable=False), + sa.Column("message_uuid", sa.String(length=36), nullable=False), + sa.Column("status", sa.String(length=32), nullable=False), + sa.Column("attempt_count", sa.Integer(), nullable=False), + sa.Column("next_attempt_at", sa.DateTime(), nullable=True), + sa.Column("rendered_content", sa.Text(), nullable=True), + sa.Column("provider_message_id", sa.String(length=256), nullable=True), + sa.Column("last_error", sa.Text(), nullable=True), + sa.Column("locked_by", sa.String(length=128), nullable=True), + sa.Column("locked_until", sa.DateTime(), nullable=True), + sa.Column("sent_at", sa.DateTime(), nullable=True), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint( + ["subscription_id"], + ["push_subscriptions.id"], + name="fk_push_deliveries_subscription_id", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "subscription_id", + "scheduled_for", + name="uq_push_delivery_subscription_schedule", + ), + ) + _create_indexes( + "push_deliveries", + ( + ("ix_push_deliveries_code", ("code",), True), + ("ix_push_deliveries_created_at", ("created_at",), False), + ( + "ix_push_deliveries_idempotency_key", + ("idempotency_key",), + True, + ), + ("ix_push_deliveries_locked_by", ("locked_by",), False), + ("ix_push_deliveries_locked_until", ("locked_until",), False), + ("ix_push_deliveries_message_uuid", ("message_uuid",), True), + ( + "ix_push_deliveries_next_attempt_at", + ("next_attempt_at",), + False, + ), + ( + "ix_push_deliveries_provider_message_id", + ("provider_message_id",), + False, + ), + ("ix_push_deliveries_scheduled_for", ("scheduled_for",), False), + ("ix_push_deliveries_sent_at", ("sent_at",), False), + ("ix_push_deliveries_status", ("status",), False), + ( + "ix_push_deliveries_subscription_id", + ("subscription_id",), + False, + ), + ), + ) + + +def downgrade() -> None: + op.drop_table("push_deliveries") + op.drop_table("push_subscriptions") + op.drop_table("personal_data_erasure_requests") + op.drop_table("ai_conversation_messages") + op.drop_table("ai_conversations") + op.drop_table("user_preferences") + + with op.batch_alter_table(MARKET_WATCHLIST_TABLE) as batch_op: + batch_op.drop_constraint( + "uq_market_watchlist_owner_symbol", + type_="unique", + ) + batch_op.drop_constraint( + "fk_market_watchlists_owner_id", + type_="foreignkey", + ) + batch_op.drop_index("ix_market_watchlists_owner_id") + batch_op.drop_column("owner_id") + batch_op.create_unique_constraint( + "uq_market_watchlist_actor_symbol", + ["actor", "symbol"], + ) + + op.execute( + sa.text( + """ + UPDATE ai_memory_entries + SET source = 'user_rule' + WHERE owner_id IS NULL + AND kind = 'company_rule' + AND source = 'legacy_company' + """ + ) + ) + with op.batch_alter_table(AI_MEMORY_TABLE) as batch_op: + batch_op.drop_constraint( + "uq_ai_memory_owner_fingerprint", + type_="unique", + ) + batch_op.drop_constraint( + "fk_ai_memory_entries_owner_id", + type_="foreignkey", + ) + batch_op.drop_index("ix_ai_memory_entries_fingerprint") + batch_op.drop_index("ix_ai_memory_entries_kind") + batch_op.drop_index("ix_ai_memory_entries_owner_id") + batch_op.drop_column("kind") + batch_op.drop_column("owner_id") + batch_op.create_index( + "ix_ai_memory_entries_fingerprint", + ["fingerprint"], + unique=True, + ) + + op.drop_table("feishu_users") diff --git a/alembic/versions/202607260004_feishu_app_tickets.py b/alembic/versions/202607260004_feishu_app_tickets.py new file mode 100644 index 0000000..cbd9ce4 --- /dev/null +++ b/alembic/versions/202607260004_feishu_app_tickets.py @@ -0,0 +1,43 @@ +"""Persist verified Feishu app tickets. + +Revision ID: 202607260004 +Revises: 202607260003 +Create Date: 2026-07-26 +""" + +from alembic import op +import sqlalchemy as sa + + +revision = "202607260004" +down_revision = "202607260003" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.create_table( + "feishu_app_tickets", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("app_id", sa.String(length=128), nullable=False), + sa.Column("app_ticket", sa.Text(), nullable=False), + sa.Column("received_at", sa.DateTime(), nullable=False), + sa.Column("updated_at", sa.DateTime(), nullable=False), + sa.PrimaryKeyConstraint("id"), + ) + op.create_index( + op.f("ix_feishu_app_tickets_app_id"), + "feishu_app_tickets", + ["app_id"], + unique=True, + ) + op.create_index( + op.f("ix_feishu_app_tickets_received_at"), + "feishu_app_tickets", + ["received_at"], + unique=False, + ) + + +def downgrade() -> None: + op.drop_table("feishu_app_tickets") diff --git a/alembic/versions/202607260005_feishu_admin_bootstrap_tombstones.py b/alembic/versions/202607260005_feishu_admin_bootstrap_tombstones.py new file mode 100644 index 0000000..8e00d72 --- /dev/null +++ b/alembic/versions/202607260005_feishu_admin_bootstrap_tombstones.py @@ -0,0 +1,41 @@ +"""Prevent erased initial administrators from being bootstrapped again. + +Revision ID: 202607260005 +Revises: 202607260004 +Create Date: 2026-07-27 +""" + +from alembic import op +import sqlalchemy as sa + + +revision = "202607260005" +down_revision = "202607260004" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.create_table( + "feishu_admin_bootstrap_tombstones", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("identity_hash", sa.String(length=64), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.PrimaryKeyConstraint("id"), + ) + op.create_index( + op.f("ix_feishu_admin_bootstrap_tombstones_created_at"), + "feishu_admin_bootstrap_tombstones", + ["created_at"], + unique=False, + ) + op.create_index( + op.f("ix_feishu_admin_bootstrap_tombstones_identity_hash"), + "feishu_admin_bootstrap_tombstones", + ["identity_hash"], + unique=True, + ) + + +def downgrade() -> None: + op.drop_table("feishu_admin_bootstrap_tombstones") diff --git a/app/api/router.py b/app/api/router.py index aee621f..9e152ad 100644 --- a/app/api/router.py +++ b/app/api/router.py @@ -8,11 +8,13 @@ from app.modules.business.routes import router as business_router from app.modules.dashboard.routes import router as dashboard_router from app.modules.events.routes import router as events_router from app.modules.feishu.routes import router as feishu_router +from app.modules.feishu_users.routes import router as feishu_users_router from app.modules.legacy_mysql.routes import router as legacy_mysql_router from app.modules.market.routes import router as market_router from app.modules.observability.routes import router as observability_router from app.modules.reports.routes import router as reports_router from app.modules.risk.routes import router as risk_router +from app.modules.subscriptions.routes import router as subscriptions_router from app.modules.workflows.routes import router as workflows_router api_router = APIRouter() @@ -29,11 +31,21 @@ api_router.include_router(business_router, prefix="/business", tags=["business"] api_router.include_router(dashboard_router, prefix="/dashboard", tags=["dashboard"]) api_router.include_router(legacy_mysql_router, prefix="/integrations/mysql", tags=["mysql"]) api_router.include_router(feishu_router, prefix="/integrations/feishu", tags=["feishu"]) +api_router.include_router( + feishu_users_router, + prefix="/integrations/feishu", + tags=["feishu-users"], +) api_router.include_router(ai_router, prefix="/ai", tags=["ai"]) api_router.include_router(ai_memory_router, prefix="/ai", tags=["ai-memory"]) api_router.include_router(reports_router, prefix="/reports", tags=["reports"]) api_router.include_router(market_router, prefix="/market", tags=["market"]) api_router.include_router(risk_router, prefix="/risks", tags=["risks"]) +api_router.include_router( + subscriptions_router, + prefix="/subscriptions", + tags=["subscriptions"], +) api_router.include_router(audit_router, prefix="/audit", tags=["audit"]) api_router.include_router(events_router, prefix="/events", tags=["events"]) api_router.include_router(workflows_router, prefix="/workflows", tags=["workflows"]) diff --git a/app/application/delivery/subscriptions.py b/app/application/delivery/subscriptions.py new file mode 100644 index 0000000..fe4615c --- /dev/null +++ b/app/application/delivery/subscriptions.py @@ -0,0 +1,76 @@ +from typing import Any + +from app.core.constants import ActorValue +from app.core.database import SessionLocal +from app.modules.ai_agent.constants import AIResponseKey +from app.modules.ai_agent.service import AIService +from app.modules.feishu.service import FeishuService +from app.modules.subscriptions.services import ( + DeliveryGenerationRequest, + DeliverySendRequest, + DeliveryService, + RetryableDeliveryError, + SubscriptionScanner, +) + + +class AISubscriptionGenerator: + """Generate side-effect-free subscription content through the configured AI.""" + + def __init__(self, ai: AIService): + self.ai = ai + + def generate(self, request: DeliveryGenerationRequest) -> str: + result = self.ai.generate_scheduled( + request.prompt, + owner_id=request.owner_id, + group=request.use_company_rules and not request.use_personal_context, + actor=ActorValue.SCHEDULER, + ) + if not result.get(AIResponseKey.OK): + raise RetryableDeliveryError("AI provider is unavailable") + return str(result[AIResponseKey.ANSWER]) + + +class FeishuSubscriptionSender: + """Send a delivery with the stable Feishu UUID supplied by durable state.""" + + def __init__(self, feishu: FeishuService): + self.feishu = feishu + + def send(self, request: DeliverySendRequest) -> dict[str, Any]: + return self.feishu.send_text( + request.text, + receive_id=request.receive_id, + receive_id_type=request.receive_id_type, + actor=ActorValue.SCHEDULER, + uuid=request.uuid, + tenant_key=request.tenant_key, + ) + + +def run_subscription_cycle( + *, + actor: str = ActorValue.SCHEDULER, +) -> dict[str, Any]: + """Materialize due windows and process pending/retry deliveries.""" + + _ = actor + db = SessionLocal() + try: + created = SubscriptionScanner(db).scan_due() + delivery_service = DeliveryService( + db, + generator=AISubscriptionGenerator(AIService(db)), + sender=FeishuSubscriptionSender(FeishuService(db)), + ) + processed = delivery_service.process_due() + return { + "created": [item.code for item in created], + "processed": [ + {"code": item.code, "status": item.status} + for item in processed + ], + } + finally: + db.close() diff --git a/app/application/events/dispatch.py b/app/application/events/dispatch.py index 9f870ab..646514b 100644 --- a/app/application/events/dispatch.py +++ b/app/application/events/dispatch.py @@ -3,9 +3,12 @@ from typing import Any from uuid import uuid4 from fastapi import HTTPException, status -from sqlalchemy import or_, select +from sqlalchemy import or_, select, update -from app.application.events.handlers import EventHandlerMixin +from app.application.events.handlers import ( + EventHandlerMixin, + UnsupportedEventTypeError, +) from app.core.config import get_settings from app.core.constants import ActorValue from app.core.http.pagination import bounded_limit @@ -44,12 +47,11 @@ class EventDispatchService(EventHandlerMixin): worker_id: str | None = None, preclaimed: bool = False, ) -> DomainEvent: - lock_owner = worker_id or f"api:{uuid4().hex}" - record = ( - self.get_event(event_id) - if preclaimed - else self._claim_event(event_id, lock_owner) - ) + if preclaimed: + lock_owner = worker_id or "" + else: + lock_owner = f"{worker_id or 'api'}:{uuid4().hex}" + record = self.get_event(event_id) if preclaimed else self._claim_event(event_id, lock_owner) if record.status == EventStatus.PROCESSED: return record if preclaimed and record.locked_by != lock_owner: @@ -63,30 +65,31 @@ class EventDispatchService(EventHandlerMixin): try: self._handle_event(record) except Exception as exc: - retryable = record.attempts < self._max_attempts(record) - record.status = EventStatus.PENDING if retryable else EventStatus.FAILED - record.last_error = str(exc) - record.next_attempt_at = ( - utc_now() + timedelta(seconds=settings.event_dispatch_retry_delay_seconds) - if retryable - else None + self.db.rollback() + record = self.get_event(event_id) + retryable = not isinstance( + exc, UnsupportedEventTypeError + ) and record.attempts < self._max_attempts(record) + return self._finalize_event( + event_id, + lock_owner, + status_value=(EventStatus.PENDING if retryable else EventStatus.FAILED), + last_error=f"{type(exc).__name__}: {exc}"[:2000], + next_attempt_at=( + utc_now() + timedelta(seconds=settings.event_dispatch_retry_delay_seconds) + if retryable + else None + ), + processed_at=None, ) - record.locked_by = None - record.locked_until = None - self.db.commit() - self.db.refresh(record) - self._audit_dispatch(record) - return record - record.status = EventStatus.PROCESSED - record.last_error = None - record.processed_at = utc_now() - record.next_attempt_at = None - record.locked_by = None - record.locked_until = None - self.db.commit() - self.db.refresh(record) - self._audit_dispatch(record) - return record + return self._finalize_event( + event_id, + lock_owner, + status_value=EventStatus.PROCESSED, + last_error=None, + next_attempt_at=None, + processed_at=utc_now(), + ) def dispatch_pending( self, @@ -95,7 +98,7 @@ class EventDispatchService(EventHandlerMixin): ) -> list[dict[str, Any]]: now = utc_now() stmt = ( - select(DomainEvent) + select(DomainEvent.event_id) .where( DomainEvent.status == EventStatus.PENDING, or_( @@ -113,38 +116,51 @@ class EventDispatchService(EventHandlerMixin): ) .order_by(DomainEvent.id.asc()) .limit(bounded_limit(limit)) - .with_for_update(skip_locked=True) ) - records = list(self.db.execute(stmt).scalars()) - lock_owner = worker_id or f"worker:{uuid4().hex}" - locked_until = now + timedelta( - seconds=get_settings().event_dispatch_lock_seconds - ) - for record in records: - record.locked_by = lock_owner - record.locked_until = locked_until - record.attempts += 1 - self.db.commit() - return [ - _serialize_event( - self.dispatch_event( - record.event_id, + event_ids = list(self.db.execute(stmt).scalars()) + lock_owner = f"{worker_id or 'worker'}:{uuid4().hex}" + dispatched: list[dict[str, Any]] = [] + for event_id in event_ids: + try: + record = self.dispatch_event( + event_id, worker_id=lock_owner, - preclaimed=True, ) - ) - for record in records - ] + except HTTPException as exc: + if exc.status_code == status.HTTP_409_CONFLICT: + continue + raise + dispatched.append(_serialize_event(record)) + return dispatched def retry_event(self, event_id: str, actor: str = ActorValue.API) -> DomainEvent: - record = self.get_event(event_id) + record = self.db.execute( + select(DomainEvent).where(DomainEvent.event_id == event_id).with_for_update() + ).scalar_one_or_none() + if record is None: + self.db.rollback() + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=EventErrorDetail.EVENT_NOT_FOUND, + ) + now = utc_now() + if ( + record.locked_by is not None + and record.locked_until is not None + and record.locked_until > now + ): + self.db.rollback() + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=EventErrorDetail.EVENT_LOCKED, + ) if record.status == EventStatus.PROCESSED: + self.db.rollback() raise HTTPException( status_code=status.HTTP_409_CONFLICT, detail=EventErrorDetail.EVENT_NOT_RETRYABLE, ) record.status = EventStatus.PENDING - record.actor = actor record.attempts = 0 record.max_attempts = record.max_attempts or get_settings().event_dispatch_max_attempts record.last_error = None @@ -155,8 +171,48 @@ class EventDispatchService(EventHandlerMixin): self.db.refresh(record) return record - def _audit_dispatch(self, record: DomainEvent) -> None: - AuditService(self.db).log( + def _finalize_event( + self, + event_id: str, + lock_owner: str, + *, + status_value: str, + last_error: str | None, + next_attempt_at: Any, + processed_at: Any, + ) -> DomainEvent: + result = self.db.execute( + update(DomainEvent) + .where( + DomainEvent.event_id == event_id, + DomainEvent.locked_by == lock_owner, + DomainEvent.status == EventStatus.PENDING, + ) + .values( + status=status_value, + last_error=last_error, + next_attempt_at=next_attempt_at, + processed_at=processed_at, + locked_by=None, + locked_until=None, + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=EventErrorDetail.EVENT_LOCKED, + ) + self.db.expire_all() + record = self.get_event(event_id) + self._stage_dispatch_audit(record) + self.db.commit() + self.db.refresh(record) + return record + + def _stage_dispatch_audit(self, record: DomainEvent) -> None: + AuditService(self.db).record( AuditLogCreate( actor=record.actor, source=AuditSource.EVENTS, @@ -198,9 +254,7 @@ class EventDispatchService(EventHandlerMixin): detail=EventErrorDetail.EVENT_LOCKED, ) record.locked_by = lock_owner - record.locked_until = now + timedelta( - seconds=get_settings().event_dispatch_lock_seconds - ) + record.locked_until = now + timedelta(seconds=get_settings().event_dispatch_lock_seconds) record.status = EventStatus.PENDING record.attempts += 1 self.db.commit() diff --git a/app/application/events/handlers.py b/app/application/events/handlers.py index 54dbea4..70f44d2 100644 --- a/app/application/events/handlers.py +++ b/app/application/events/handlers.py @@ -6,6 +6,10 @@ from app.modules.events.constants import ( from app.modules.events.models import DomainEvent +class UnsupportedEventTypeError(ValueError): + """Raised when no application handler is registered for an event type.""" + + class EventHandlerMixin: def _handle_event(self, record: DomainEvent) -> None: if record.event_type == EventType.RISK_ACTION_RECORDED: @@ -30,6 +34,9 @@ class EventHandlerMixin: if record.event_type == EventType.ENTERPRISE_ANALYTICS_GENERATED: self._handle_enterprise_analytics_event(record) return + raise UnsupportedEventTypeError( + f"Unsupported domain event type: {record.event_type}" + ) def _handle_risk_action(self, record: DomainEvent) -> None: from app.modules.risk.constants import RiskEventActionValue @@ -53,6 +60,7 @@ class EventHandlerMixin: actor=record.actor, payload=payload, commit=False, + source_event_id=record.event_id, ) def _handle_report_event(self, record: DomainEvent) -> None: @@ -125,4 +133,5 @@ class EventHandlerMixin: actor=record.actor, payload=record.payload or {}, commit=False, + source_event_id=record.event_id, ) diff --git a/app/application/feishu/commands.py b/app/application/feishu/commands.py index 8c86888..4a25760 100644 --- a/app/application/feishu/commands.py +++ b/app/application/feishu/commands.py @@ -6,15 +6,23 @@ from sqlalchemy.orm import Session from app.application.feishu.delivery import send_card_if_configured, send_text_if_configured from app.application.feishu.handlers import ( + handle_admin_command, handle_finance_command, handle_market_command, + handle_personal_data_command, + handle_personalization_command, handle_rule_command, + handle_subscription_command, + is_admin_command, + is_company_rule_command, ) from app.application.feishu.results import command_result +from app.core.config import get_settings from app.core.constants import ActorValue from app.modules.ai_agent.constants import AIResponseKey from app.modules.ai_agent.service import AIService -from app.modules.audit.constants import AuditSource +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate from app.modules.feishu.constants import ( FEISHU_AI_REPLY_TITLE, FEISHU_MENTION_PATTERN, @@ -25,6 +33,12 @@ from app.modules.feishu.constants import ( FeishuReplyType, ) from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.constants import ( + FEISHU_USER_TARGET_TYPE, + FeishuCapability, + FeishuUserAuditAction, +) +from app.modules.feishu_users.principal import FeishuMention, FeishuPrincipal from app.modules.reports.constants import ReportResponseKey from app.modules.reports.services import ReportService @@ -34,6 +48,18 @@ ATTENDANCE_KEYWORDS = ("打卡", "考勤", "attendance") RISK_KEYWORDS = ("风险", "预警", "risk") AI_COMMAND_PREFIXES = ("问 ", "ai ", "AI ", "/ask ") DEFAULT_AI_PROMPT = "请说明你能做什么。" +PERMISSION_DENIED_TITLE = "权限不足" +COMPANY_RULE_COMMAND_PREFIXES = ( + "学习公司规则", + "查看公司规则", + "修改公司规则", + "启用公司规则", + "停用公司规则", + "删除公司规则", + "学习公司市场规则", + "查看公司市场规则", +) +FINANCE_COMMAND_PREFIXES = ("资金需求", "未来30天资金需求", "项目资金 ") def _parse_content_text(content: Any) -> str: @@ -70,12 +96,14 @@ class FeishuCommandService: self.feishu = FeishuService(db) def extract_event_command(self, payload: dict[str, Any]) -> dict[str, Any] | None: + header = payload.get(FeishuPayloadKey.HEADER) or {} event = payload.get(FeishuPayloadKey.EVENT) or {} message = event.get(FeishuPayloadKey.MESSAGE) or {} if not message: return None - text = _clean_command_text(_parse_content_text(message.get(FeishuPayloadKey.CONTENT))) - if not text: + raw_text = _parse_content_text(message.get(FeishuPayloadKey.CONTENT)) + command_text = _clean_command_text(raw_text) + if not command_text: return None sender = event.get(FeishuPayloadKey.SENDER) or {} sender_id = sender.get(FeishuPayloadKey.SENDER_ID) or {} @@ -85,9 +113,18 @@ class FeishuCommandService: or ActorValue.FEISHU ) return { - FeishuCommandKey.TEXT: text, + FeishuCommandKey.TEXT: ( + raw_text + if get_settings().feishu_user_features_enabled + else command_text + ), FeishuCommandKey.CHAT_ID: message.get(FeishuCommandKey.CHAT_ID), + FeishuCommandKey.CHAT_TYPE: message.get(FeishuPayloadKey.CHAT_TYPE), FeishuCommandKey.ACTOR: actor, + FeishuCommandKey.MENTIONS: _parse_mentions( + message.get(FeishuPayloadKey.MENTIONS), + str(header.get(FeishuPayloadKey.TENANT_KEY) or ""), + ), } def handle_text( @@ -96,14 +133,99 @@ class FeishuCommandService: chat_id: str | None = None, actor: str = ActorValue.FEISHU, auto_reply: bool = True, + principal: FeishuPrincipal | None = None, + tenant_key: str | None = None, ) -> dict[str, Any]: + if tenant_key is not None: + self.feishu.set_tenant_key(tenant_key) + raw_text = text or "" command_text = _clean_command_text(text) lowered = command_text.lower() + if get_settings().feishu_user_features_enabled: + if principal is None: + return self._permission_denied( + chat_id=chat_id, + actor=actor, + auto_reply=False, + principal=None, + reason="missing_verified_identity", + content="无法确认飞书账号身份，已拒绝执行该命令。", + ) + self.feishu.set_tenant_key(principal.tenant_key) + chat_id = principal.chat_id or chat_id + actor = principal.user_code + if not principal.is_active: + return self._permission_denied( + chat_id=chat_id, + actor=actor, + auto_reply=auto_reply, + principal=principal, + reason="disabled_user", + content="当前飞书账号已停用，请联系管理员。", + ) + required_capability = _required_capability(command_text) + if ( + required_capability is not None + and not principal.has_capability(required_capability) + ): + return self._permission_denied( + chat_id=chat_id, + actor=actor, + auto_reply=auto_reply, + principal=principal, + reason=f"missing_capability:{required_capability}", + content="当前飞书账号无权使用该公司级功能。", + ) + admin_result = handle_admin_command( + self.db, + self.feishu, + raw_text=raw_text, + command_text=command_text, + principal=principal, + auto_reply=auto_reply, + ) + if admin_result is not None: + return admin_result + for handler in ( + handle_subscription_command, + handle_personal_data_command, + handle_personalization_command, + ): + result = handler( + self.db, + self.feishu, + command_text, + principal, + auto_reply, + ) + if result is not None: + return result + + rule_result = handle_rule_command( + self.db, + self.feishu, + command_text, + chat_id, + actor, + auto_reply, + principal=principal, + ) + if rule_result is not None: + return rule_result + market_result = handle_market_command( + self.db, + self.feishu, + command_text, + chat_id, + actor, + auto_reply, + principal=principal, + ) + if market_result is not None: + return market_result for handler in ( - handle_rule_command, handle_finance_command, - handle_market_command, ): result = handler( self.db, @@ -115,11 +237,65 @@ class FeishuCommandService: ) if result is not None: return result + if not get_settings().feishu_user_features_enabled: + for handler in (handle_rule_command, handle_market_command): + result = handler( + self.db, + self.feishu, + command_text, + chat_id, + actor, + auto_reply, + ) + if result is not None: + return result report_result = self._handle_report_command(command_text, chat_id, actor, auto_reply) if report_result is not None: return report_result - return self._handle_ai_command(command_text, lowered, chat_id, actor, auto_reply) + return self._handle_ai_command( + command_text, + lowered, + chat_id, + actor, + auto_reply, + principal=principal, + ) + + def _permission_denied( + self, + *, + chat_id: str | None, + actor: str, + auto_reply: bool, + principal: FeishuPrincipal | None, + reason: str, + content: str, + ) -> dict[str, Any]: + self.feishu.audit.log( + AuditLogCreate( + actor=actor, + source=AuditSource.FEISHU, + action=FeishuUserAuditAction.PERMISSION_DENIED, + target_type=FEISHU_USER_TARGET_TYPE, + target_id=principal.user_code if principal else None, + risk_level=AuditRiskLevel.MEDIUM, + response_payload={"result": "denied", "reason": reason}, + status="denied", + ) + ) + response = ( + send_text_if_configured(self.feishu, chat_id, content, actor) + if auto_reply + else None + ) + return command_result( + FeishuCommandName.PERMISSION_DENIED, + FeishuReplyType.TEXT, + PERMISSION_DENIED_TITLE, + content, + response, + ) def _handle_report_command( self, @@ -170,6 +346,7 @@ class FeishuCommandService: chat_id: str | None, actor: str, auto_reply: bool, + principal: FeishuPrincipal | None = None, ) -> dict[str, Any]: prompt = command_text for prefix in AI_COMMAND_PREFIXES: @@ -178,12 +355,23 @@ class FeishuCommandService: break if not prompt: prompt = DEFAULT_AI_PROMPT - ai_result = AIService(self.db).ask( - prompt, - context={}, - actor=actor, - source=AuditSource.FEISHU, - ) + if get_settings().feishu_user_features_enabled and principal is not None: + chat_type, chat_key = _principal_chat_context(principal) + ai_result = AIService(self.db).ask_personalized( + principal.owner_id, + chat_type, + chat_key, + prompt, + actor=actor, + source=AuditSource.FEISHU, + ) + else: + ai_result = AIService(self.db).ask( + prompt, + context={}, + actor=actor, + source=AuditSource.FEISHU, + ) content = ai_result[AIResponseKey.ANSWER] is_explicit_ai = any( command_text.startswith(prefix) or lowered.startswith(prefix) @@ -199,3 +387,76 @@ class FeishuCommandService: content, response, ) + + +def _parse_mentions(value: Any, tenant_key: str) -> tuple[FeishuMention, ...]: + if not isinstance(value, list): + return () + mentions: list[FeishuMention] = [] + for item in value: + if not isinstance(item, dict): + continue + mention_id = item.get(FeishuPayloadKey.ID) or {} + if not isinstance(mention_id, dict): + mention_id = {} + mentions.append( + FeishuMention( + key=_optional_text(item.get(FeishuPayloadKey.KEY)), + name=_optional_text(item.get(FeishuPayloadKey.NAME)), + tenant_key=( + _optional_text(item.get(FeishuPayloadKey.TENANT_KEY)) + or tenant_key + or None + ), + open_id=_optional_text( + mention_id.get(FeishuPayloadKey.OPEN_ID) + or item.get(FeishuPayloadKey.OPEN_ID) + ), + union_id=_optional_text( + mention_id.get(FeishuPayloadKey.UNION_ID) + or item.get(FeishuPayloadKey.UNION_ID) + ), + user_id=_optional_text( + mention_id.get(FeishuPayloadKey.USER_ID) + or item.get(FeishuPayloadKey.USER_ID) + ), + ) + ) + return tuple(mentions) + + +def _optional_text(value: Any) -> str | None: + text = str(value or "").strip() + return text or None + + +def _required_capability(command_text: str) -> FeishuCapability | None: + if is_admin_command(command_text): + return FeishuCapability.USER_ADMINISTRATION + if ( + command_text.startswith(COMPANY_RULE_COMMAND_PREFIXES) + or is_company_rule_command(command_text) + ): + return FeishuCapability.COMPANY_RULES + if command_text.startswith(FINANCE_COMMAND_PREFIXES): + return FeishuCapability.COMPANY_REPORTS + if any( + keyword in command_text + for keyword in ( + *DAILY_REPORT_KEYWORDS, + *PROJECT_WEEKLY_KEYWORDS, + *ATTENDANCE_KEYWORDS, + *RISK_KEYWORDS, + ) + ): + return FeishuCapability.COMPANY_REPORTS + return None + + +def _principal_chat_context(principal: FeishuPrincipal) -> tuple[str, str]: + chat_type = principal.chat_type or "p2p" + if chat_type in {"group", "group_chat"}: + if not principal.chat_id: + raise ValueError("Verified group chat is missing chat_id") + return chat_type, principal.chat_id + return chat_type, principal.chat_id or principal.open_id diff --git a/app/application/feishu/delivery.py b/app/application/feishu/delivery.py index 0815615..20d0e30 100644 --- a/app/application/feishu/delivery.py +++ b/app/application/feishu/delivery.py @@ -9,13 +9,20 @@ def send_text_if_configured( chat_id: str | None, text: str, actor: str, + *, + record_audit: bool = True, ) -> dict[str, Any] | None: """Send a text reply only when Feishu credentials are configured.""" settings = get_settings() if not (settings.feishu_app_id and settings.feishu_app_secret): return None - return feishu.send_text(text, receive_id=chat_id, actor=actor) + return feishu.send_text( + text, + receive_id=chat_id, + actor=actor, + record_audit=record_audit, + ) def send_card_if_configured( diff --git a/app/application/feishu/events.py b/app/application/feishu/events.py index e8438c4..79a6494 100644 --- a/app/application/feishu/events.py +++ b/app/application/feishu/events.py @@ -1,12 +1,20 @@ +from dataclasses import replace from typing import Any +from fastapi import HTTPException, status from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import Session from app.application.feishu.commands import FeishuCommandService +from app.core.config import get_settings from app.core.constants import ActorValue from app.modules.audit.constants import AuditAction, AuditSource from app.modules.audit.schemas import AuditLogCreate +from app.modules.feishu.app_tickets import ( + APP_TICKET_EVENT_TYPE, + APP_TICKET_PAYLOAD_KEY, + FeishuAppTicketService, +) from app.modules.feishu.constants import ( FeishuCommandKey, FeishuEventReceiptKey, @@ -16,6 +24,8 @@ from app.modules.feishu.constants import ( ) from app.modules.feishu.models import FeishuEventReceipt from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.principal import FeishuMention, FeishuPrincipal +from app.modules.feishu_users.services import FeishuIdentityService FEISHU_EVENT_ACTIONS = { FeishuEventSource.WEBHOOK: AuditAction.FEISHU_WEBHOOK_EVENT, @@ -38,10 +48,33 @@ class FeishuEventService: auto_reply: bool = True, ) -> dict[str, Any]: self.feishu.verify_event(payload) + return self._handle_verified_event(payload, source, auto_reply) + + def _handle_verified_event( + self, + payload: dict[str, Any], + source: str | FeishuEventSource, + auto_reply: bool = True, + ) -> dict[str, Any]: + """Handle an event after an HTTP verifier or the Feishu SDK accepted it.""" + challenge = payload.get(FeishuPayloadKey.CHALLENGE) if challenge: return {FeishuResponseKey.CHALLENGE: challenge} source_value = _normalize_source(source) + if _event_type(payload) == APP_TICKET_EVENT_TYPE: + return self._handle_app_ticket_event(payload, source_value) + user_features_enabled = get_settings().feishu_user_features_enabled + command = ( + self.commands.extract_event_command(payload) + if user_features_enabled + else None + ) + principal = ( + self._resolve_principal(payload, command) + if user_features_enabled and command + else None + ) event_identity = _event_identity(payload, source) if event_identity and not self._register_event(event_identity): return { @@ -51,7 +84,7 @@ class FeishuEventService: } self.feishu.audit.log( AuditLogCreate( - actor=ActorValue.FEISHU, + actor=principal.user_code if principal else ActorValue.FEISHU, source=AuditSource.FEISHU, action=FEISHU_EVENT_ACTIONS[source_value], target_type=source_value, @@ -60,18 +93,32 @@ class FeishuEventService: if event_identity else None ), - request_payload=_audit_event_metadata(payload), + request_payload=_audit_event_metadata( + payload, + include_open_id=not user_features_enabled, + include_identity_context=user_features_enabled, + ), response_payload={FeishuResponseKey.ACCEPTED: True}, ) ) - command = self.commands.extract_event_command(payload) + if command is None: + command = self.commands.extract_event_command(payload) if not command: return {FeishuResponseKey.OK: True, FeishuResponseKey.HANDLED: False} result = self.commands.handle_text( command[FeishuCommandKey.TEXT], chat_id=command[FeishuCommandKey.CHAT_ID], - actor=command[FeishuCommandKey.ACTOR], + actor=( + principal.user_code + if principal + else ( + ActorValue.FEISHU + if user_features_enabled + else command[FeishuCommandKey.ACTOR] + ) + ), auto_reply=auto_reply, + principal=principal, ) return { FeishuResponseKey.OK: True, @@ -79,6 +126,97 @@ class FeishuEventService: FeishuResponseKey.RESULT: result, } + def _handle_app_ticket_event( + self, + payload: dict[str, Any], + source: FeishuEventSource, + ) -> dict[str, Any]: + settings = get_settings() + configured_app_id = str(settings.feishu_app_id or "").strip() + if not configured_app_id: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="FEISHU_APP_ID is required for app ticket events", + ) + + app_id, ticket = _app_ticket_fields(payload) + if not app_id or not ticket: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Invalid Feishu app ticket event", + ) + if app_id != configured_app_id: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Feishu app ticket app_id does not match configured application", + ) + + event_identity = _event_identity(payload, source) + if event_identity is None: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Feishu app ticket event identity is required", + ) + if not self._register_event(event_identity): + return { + FeishuResponseKey.OK: True, + FeishuResponseKey.HANDLED: False, + FeishuResponseKey.DUPLICATE: True, + } + + FeishuAppTicketService(self.db).store_verified(app_id, ticket) + self.feishu.audit.log( + AuditLogCreate( + actor=ActorValue.FEISHU, + source=AuditSource.FEISHU, + action=FEISHU_EVENT_ACTIONS[source], + target_type=source, + target_id=str(event_identity[FeishuEventReceiptKey.EVENT_KEY]), + request_payload=_audit_event_metadata(payload), + response_payload={FeishuResponseKey.ACCEPTED: True}, + ) + ) + return { + FeishuResponseKey.OK: True, + FeishuResponseKey.HANDLED: True, + } + + def _resolve_principal( + self, + payload: dict[str, Any], + command: dict[str, Any], + ) -> FeishuPrincipal | None: + header = payload.get(FeishuPayloadKey.HEADER) or {} + event = payload.get(FeishuPayloadKey.EVENT) or {} + sender = event.get(FeishuPayloadKey.SENDER) or {} + sender_id = sender.get(FeishuPayloadKey.SENDER_ID) or {} + tenant_key = str(header.get(FeishuPayloadKey.TENANT_KEY) or "").strip() + open_id = str(sender_id.get(FeishuPayloadKey.OPEN_ID) or "").strip() + if not tenant_key or not open_id: + return None + principal = FeishuIdentityService(self.db).resolve_or_register( + tenant_key=tenant_key, + open_id=open_id, + union_id=sender_id.get(FeishuPayloadKey.UNION_ID), + user_id=sender_id.get(FeishuPayloadKey.USER_ID), + ) + mentions_value = command.get(FeishuCommandKey.MENTIONS) + mentions = ( + tuple( + mention + for mention in mentions_value + if isinstance(mention, FeishuMention) + ) + if isinstance(mentions_value, (list, tuple)) + else () + ) + return replace( + principal, + chat_id=command.get(FeishuCommandKey.CHAT_ID), + chat_type=command.get(FeishuCommandKey.CHAT_TYPE), + mentions=mentions, + ) + def _register_event(self, event_identity: dict[str, str | None]) -> bool: receipt = FeishuEventReceipt( event_key=str(event_identity[FeishuEventReceiptKey.EVENT_KEY]), @@ -86,16 +224,21 @@ class FeishuEventService: event_id=event_identity.get(FeishuEventReceiptKey.EVENT_ID), message_id=event_identity.get(FeishuEventReceiptKey.MESSAGE_ID), ) - self.db.add(receipt) try: - self.db.flush() + with self.db.begin_nested(): + self.db.add(receipt) + self.db.flush() except IntegrityError: - self.db.rollback() return False return True -def _audit_event_metadata(payload: dict[str, Any]) -> dict[str, Any]: +def _audit_event_metadata( + payload: dict[str, Any], + *, + include_open_id: bool = True, + include_identity_context: bool = False, +) -> dict[str, Any]: """Keep webhook audit evidence without storing message content or tokens.""" header = payload.get(FeishuPayloadKey.HEADER) or {} @@ -103,15 +246,23 @@ def _audit_event_metadata(payload: dict[str, Any]) -> dict[str, Any]: message = event.get(FeishuPayloadKey.MESSAGE) or {} sender = event.get(FeishuPayloadKey.SENDER) or {} sender_id = sender.get(FeishuPayloadKey.SENDER_ID) or {} - return { + metadata = { "schema": payload.get("schema"), FeishuPayloadKey.EVENT_ID: header.get(FeishuPayloadKey.EVENT_ID), - FeishuPayloadKey.EVENT_TYPE: header.get(FeishuPayloadKey.EVENT_TYPE), + FeishuPayloadKey.EVENT_TYPE: _event_type(payload), FeishuPayloadKey.MESSAGE_ID: message.get(FeishuPayloadKey.MESSAGE_ID), FeishuCommandKey.CHAT_ID: message.get(FeishuCommandKey.CHAT_ID), FeishuPayloadKey.MESSAGE_TYPE: message.get(FeishuPayloadKey.MESSAGE_TYPE), - FeishuPayloadKey.OPEN_ID: sender_id.get(FeishuPayloadKey.OPEN_ID), } + app_id, _ = _app_ticket_fields(payload) + if app_id: + metadata[FeishuPayloadKey.APP_ID] = app_id + if include_identity_context: + metadata[FeishuPayloadKey.TENANT_KEY] = header.get(FeishuPayloadKey.TENANT_KEY) + metadata[FeishuCommandKey.CHAT_TYPE] = message.get(FeishuPayloadKey.CHAT_TYPE) + if include_open_id: + metadata[FeishuPayloadKey.OPEN_ID] = sender_id.get(FeishuPayloadKey.OPEN_ID) + return metadata def _normalize_source(source: str | FeishuEventSource) -> FeishuEventSource: @@ -126,15 +277,21 @@ def _event_identity( header = payload.get(FeishuPayloadKey.HEADER) or {} event = payload.get(FeishuPayloadKey.EVENT) or {} message = event.get(FeishuPayloadKey.MESSAGE) or {} - event_id = header.get(FeishuPayloadKey.EVENT_ID) + event_id = ( + header.get(FeishuPayloadKey.EVENT_ID) + or payload.get(FeishuPayloadKey.EVENT_ID) + or payload.get(FeishuPayloadKey.UUID) + or event.get(FeishuPayloadKey.UUID) + ) message_id = message.get(FeishuPayloadKey.MESSAGE_ID) stable_id = event_id or message_id if not stable_id: return None - event_type = header.get(FeishuPayloadKey.EVENT_TYPE) + event_type = _event_type(payload) + app_id, _ = _app_ticket_fields(payload) + tenant_key = header.get(FeishuPayloadKey.TENANT_KEY) or app_id or "unknown-tenant" event_key = ":".join( - str(part) - for part in (source_value, event_type or FeishuPayloadKey.EVENT, stable_id) + str(part) for part in (tenant_key, event_type or FeishuPayloadKey.EVENT, stable_id) ) return { FeishuEventReceiptKey.EVENT_KEY: event_key, @@ -142,3 +299,41 @@ def _event_identity( FeishuEventReceiptKey.EVENT_ID: str(event_id) if event_id else None, FeishuEventReceiptKey.MESSAGE_ID: str(message_id) if message_id else None, } + + +def _event_type(payload: dict[str, Any]) -> str: + header = payload.get(FeishuPayloadKey.HEADER) or {} + return str( + header.get(FeishuPayloadKey.EVENT_TYPE) + or payload.get(FeishuPayloadKey.EVENT_TYPE) + or payload.get("type") + or "" + ).strip() + + +def _app_ticket_fields(payload: dict[str, Any]) -> tuple[str, str]: + header = payload.get(FeishuPayloadKey.HEADER) + event = payload.get(FeishuPayloadKey.EVENT) + data = payload.get(FeishuPayloadKey.DATA) + candidates = [ + value + for value in (event, data, header, payload) + if isinstance(value, dict) + ] + app_id = next( + ( + str(candidate.get(FeishuPayloadKey.APP_ID) or "").strip() + for candidate in candidates + if candidate.get(FeishuPayloadKey.APP_ID) + ), + "", + ) + ticket = next( + ( + str(candidate.get(APP_TICKET_PAYLOAD_KEY) or "").strip() + for candidate in candidates + if candidate.get(APP_TICKET_PAYLOAD_KEY) + ), + "", + ) + return app_id, ticket diff --git a/app/application/feishu/handlers/__init__.py b/app/application/feishu/handlers/__init__.py index cb17ca9..29b8b2d 100644 --- a/app/application/feishu/handlers/__init__.py +++ b/app/application/feishu/handlers/__init__.py @@ -1,9 +1,29 @@ +from app.application.feishu.handlers.admin import ( + handle_admin_command, + is_admin_command, +) from app.application.feishu.handlers.finance import handle_finance_command from app.application.feishu.handlers.market import handle_market_command -from app.application.feishu.handlers.rules import handle_rule_command +from app.application.feishu.handlers.personalization import ( + handle_personalization_command, +) +from app.application.feishu.handlers.personal_data import ( + handle_personal_data_command, +) +from app.application.feishu.handlers.rules import ( + handle_rule_command, + is_company_rule_command, +) +from app.application.feishu.handlers.subscriptions import handle_subscription_command __all__ = [ + "handle_admin_command", "handle_finance_command", "handle_market_command", + "handle_personalization_command", + "handle_personal_data_command", "handle_rule_command", + "handle_subscription_command", + "is_admin_command", + "is_company_rule_command", ] diff --git a/app/application/feishu/handlers/admin.py b/app/application/feishu/handlers/admin.py new file mode 100644 index 0000000..d2752c2 --- /dev/null +++ b/app/application/feishu/handlers/admin.py @@ -0,0 +1,192 @@ +from typing import Any + +from fastapi import HTTPException +from sqlalchemy.orm import Session + +from app.application.feishu.delivery import send_text_if_configured +from app.application.feishu.results import command_result +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.constants import ( + FEISHU_USER_TARGET_TYPE, + FeishuUserAuditAction, + FeishuUserRole, + FeishuUserStatus, +) +from app.modules.feishu_users.principal import FeishuMention, FeishuPrincipal +from app.modules.feishu_users.services import ( + FeishuIdentityService, + FeishuUserManagementService, +) + +ADMIN_COMMAND_TITLE = "飞书用户管理" +ADMIN_COMMAND_HELP = ( + "用户管理命令必须使用一个真实的飞书 @用户：\n" + "设为管理员 @用户\n" + "设为普通用户 @用户\n" + "停用用户 @用户\n" + "启用用户 @用户" +) + +_ADMIN_COMMANDS: dict[str, tuple[FeishuCommandName, dict[str, str], str]] = { + "设为管理员": ( + FeishuCommandName.USER_SET_ADMIN, + {"role": FeishuUserRole.ADMIN}, + "已设为管理员", + ), + "设为普通用户": ( + FeishuCommandName.USER_SET_USER, + {"role": FeishuUserRole.USER}, + "已设为普通用户", + ), + "停用用户": ( + FeishuCommandName.USER_DISABLE, + {"status": FeishuUserStatus.DISABLED}, + "已停用", + ), + "启用用户": ( + FeishuCommandName.USER_ENABLE, + {"status": FeishuUserStatus.ACTIVE}, + "已启用", + ), +} + + +def is_admin_command(command_text: str) -> bool: + return any(command_text.startswith(prefix) for prefix in _ADMIN_COMMANDS) + + +def handle_admin_command( + db: Session, + feishu: FeishuService, + *, + raw_text: str, + command_text: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any] | None: + """Manage a user selected only from verified structured mention metadata.""" + + matched = next( + ( + (prefix, definition) + for prefix, definition in _ADMIN_COMMANDS.items() + if command_text.startswith(prefix) + ), + None, + ) + if matched is None: + return None + command_prefix, definition = matched + command, changes, success_text = definition + target = _target_mention(raw_text, command_prefix, principal.mentions) + if target is None or not target.open_id: + _audit_denied(db, principal, "missing_or_ambiguous_structured_mention") + return _result( + feishu, + command, + ADMIN_COMMAND_HELP, + principal, + auto_reply, + ) + target_tenant = target.tenant_key or principal.tenant_key + if target_tenant != principal.tenant_key: + _audit_denied(db, principal, "cross_tenant_target") + return _result( + feishu, + command, + "只能管理当前租户内通过飞书 @ 提及的用户。", + principal, + auto_reply, + ) + + target_principal = FeishuIdentityService(db).resolve_or_register( + tenant_key=target_tenant, + open_id=target.open_id, + union_id=target.union_id, + user_id=target.user_id, + actor=principal.user_code, + ) + try: + updated = FeishuUserManagementService(db).update_user( + target_principal.user_code, + changes=changes, + actor=principal.user_code, + ) + display_name = target.name or updated.code + content = f"{display_name} {success_text}。" + except HTTPException as exc: + if exc.status_code == 409: + content = "操作已拒绝：不能停用或降级最后一个有效管理员。" + else: + content = "用户状态未修改，请确认目标用户后重试。" + return _result(feishu, command, content, principal, auto_reply) + + +def _target_mention( + raw_text: str, + command_text: str, + mentions: tuple[FeishuMention, ...], +) -> FeishuMention | None: + command_index = raw_text.find(command_text) + if command_index >= 0: + command_tail = raw_text[command_index + len(command_text) :] + candidates = [ + mention + for mention in mentions + if mention.key and mention.key in command_tail + ] + if len(candidates) == 1: + return candidates[0] + return None + if len(mentions) == 1: + return mentions[0] + return None + + +def _audit_denied( + db: Session, + principal: FeishuPrincipal, + reason: str, +) -> None: + AuditService(db).record( + AuditLogCreate( + actor=principal.user_code, + source=AuditSource.FEISHU, + action=FeishuUserAuditAction.UPDATE_DENIED, + target_type=FEISHU_USER_TARGET_TYPE, + risk_level=AuditRiskLevel.HIGH, + response_payload={"result": "denied", "reason": reason}, + status="denied", + ) + ) + db.commit() + + +def _result( + feishu: FeishuService, + command: FeishuCommandName, + content: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any]: + response = ( + send_text_if_configured( + feishu, + principal.chat_id, + content, + principal.user_code, + ) + if auto_reply + else None + ) + return command_result( + command, + FeishuReplyType.TEXT, + ADMIN_COMMAND_TITLE, + content, + response, + ) diff --git a/app/application/feishu/handlers/market.py b/app/application/feishu/handlers/market.py index 19f3a95..ab4ed98 100644 --- a/app/application/feishu/handlers/market.py +++ b/app/application/feishu/handlers/market.py @@ -9,6 +9,7 @@ from app.application.feishu.results import command_result from app.core.config import get_settings from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.principal import FeishuPrincipal from app.modules.market.chart import render_market_chart from app.modules.market.service import MarketService @@ -38,6 +39,7 @@ def handle_market_command( chat_id: str | None, actor: str, auto_reply: bool, + principal: FeishuPrincipal | None = None, ) -> dict[str, Any] | None: """Handle market analysis and watchlist commands.""" @@ -54,6 +56,31 @@ def handle_market_command( and not comparison ): return None + service = MarketService(db) + owner_id = principal.owner_id if principal is not None else None + if add: + item = service.add_watchlist(actor, add.group(1), owner_id=owner_id) + return _text_result( + feishu, + FeishuCommandName.WATCHLIST_ADD, + "自选股", + f"已加入自选：{item['symbol']}", + chat_id, + actor, + auto_reply, + ) + if text == "查看自选": + items = service.watchlist(actor, owner_id=owner_id) + content = "自选股：" + ("、".join(item["symbol"] for item in items) or "暂无") + return _text_result( + feishu, + FeishuCommandName.WATCHLIST_LIST, + "自选股", + content, + chat_id, + actor, + auto_reply, + ) if not get_settings().market_analysis_enabled: return _text_result( feishu, @@ -65,40 +92,6 @@ def handle_market_command( auto_reply, ) - service = MarketService(db) - if add: - if get_settings().read_only_mode: - return _text_result( - feishu, - FeishuCommandName.WATCHLIST_ADD, - "自选股", - "当前为只读模式，不能修改自选股。请由管理员启用操作后重试。", - chat_id, - actor, - auto_reply, - ) - item = service.add_watchlist(actor, add.group(1)) - return _text_result( - feishu, - FeishuCommandName.WATCHLIST_ADD, - "自选股", - f"已加入自选：{item['symbol']}", - chat_id, - actor, - auto_reply, - ) - if text == "查看自选": - items = service.watchlist(actor) - content = "自选股：" + ("、".join(item["symbol"] for item in items) or "暂无") - return _text_result( - feishu, - FeishuCommandName.WATCHLIST_LIST, - "自选股", - content, - chat_id, - actor, - auto_reply, - ) if text == "最新公告": items = service.announcements(limit=10)["items"] content = ( diff --git a/app/application/feishu/handlers/personal_data.py b/app/application/feishu/handlers/personal_data.py new file mode 100644 index 0000000..25875b1 --- /dev/null +++ b/app/application/feishu/handlers/personal_data.py @@ -0,0 +1,267 @@ +import re +from collections import Counter +from typing import Any + +from fastapi import HTTPException +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.application.feishu.delivery import send_text_if_configured +from app.application.feishu.personal_data import FeishuPersonalDataService +from app.application.feishu.results import command_result +from app.modules.ai_memory.constants import AIMemoryKind +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.business.models import MarketWatchlist +from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.personalization.constants import PreferenceCategory +from app.modules.personalization.models import AIConversation +from app.modules.personalization.services import PreferenceService +from app.modules.subscriptions.models import PushSubscription + +PERSONAL_DATA_TITLE = "我的数据" +_CONFIRM_PATTERN = re.compile(r"^确认忘记我\s+([A-Fa-f0-9]{8})$") +_COMMAND_PREFIXES = ("我的数据", "忘记我", "确认忘记我") + + +def handle_personal_data_command( + db: Session, + feishu: FeishuService, + command_text: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any] | None: + """Handle private data summaries and two-step account erasure.""" + + text = command_text.strip() + if not text.startswith(_COMMAND_PREFIXES): + return None + principal.require_active() + if principal.chat_type in {"group", "group_chat"}: + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_SUMMARY, + "为保护个人信息，请私聊机器人使用“我的数据”或“忘记我”。", + principal, + auto_reply, + ) + if text == "我的数据": + content = _summary(db, principal) + _audit_summary(db, principal) + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_SUMMARY, + content, + principal, + auto_reply, + ) + if text == "忘记我": + confirmation = FeishuPersonalDataService(db).request_confirmation(principal) + _audit_erasure_request(db, principal) + content = ( + "该操作会永久删除你的个人规则、记忆、偏好、兴趣、" + "会话、订阅和飞书身份映射。\n" + f"确认码：{confirmation.confirmation_code}\n" + f"有效期至：{confirmation.expires_at.strftime('%Y-%m-%d %H:%M:%S')} UTC\n" + f"确认命令：确认忘记我 {confirmation.confirmation_code}" + ) + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_ERASURE_REQUEST, + content, + principal, + auto_reply, + ) + + match = _CONFIRM_PATTERN.fullmatch(text) + if match is None: + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_ERASURE_CONFIRM, + "格式不正确。请先发送“忘记我”获取一次性确认码。", + principal, + auto_reply, + ) + try: + erased = FeishuPersonalDataService(db).confirm(principal, match.group(1)) + except HTTPException as exc: + db.rollback() + content = ( + "不能删除最后一个有效管理员，请先设置另一名管理员。" + if exc.status_code == 409 + else "确认码无效或已过期，请重新发送“忘记我”。" + ) + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_ERASURE_CONFIRM, + content, + principal, + auto_reply, + ) + return _result( + feishu, + FeishuCommandName.PERSONAL_DATA_ERASURE_CONFIRM, + "你的个人数据和飞书身份映射已删除。再次联系时会创建新的普通用户身份。", + principal, + auto_reply, + audit_actor=erased.anonymous_id, + ) + + +def _summary(db: Session, principal: FeishuPrincipal) -> str: + owner_id = principal.owner_id + preferences = PreferenceService(db).list_preferences(owner_id) + interest_categories = { + PreferenceCategory.TOPIC, + PreferenceCategory.INTEREST, + } + profile_preferences = [ + item for item in preferences if item["category"] not in interest_categories + ] + interest_preferences = [ + item for item in preferences if item["category"] in interest_categories + ] + watchlist = list( + db.execute( + select(MarketWatchlist.symbol) + .where( + MarketWatchlist.owner_id == owner_id, + MarketWatchlist.enabled.is_(True), + ) + .order_by(MarketWatchlist.symbol.asc()) + ).scalars() + ) + subscription_statuses = Counter( + db.execute( + select(PushSubscription.status).where( + PushSubscription.owner_id == owner_id + ) + ).scalars() + ) + personal_rule_count = _memory_count( + db, + owner_id, + AIMemoryKind.PERSONAL_RULE, + ) + memory_count = _memory_count(db, owner_id, AIMemoryKind.MEMORY) + conversation_count = int( + db.scalar( + select(func.count()) + .select_from(AIConversation) + .where(AIConversation.owner_id == owner_id) + ) + or 0 + ) + preference_text = "、".join( + f"{item['category']}={item['value']}" for item in profile_preferences[:10] + ) + interest_values = [ + *(str(item["value"]) for item in interest_preferences), + *watchlist, + ] + interest_text = "、".join(interest_values[:10]) + subscriptions_text = "、".join( + f"{status} {count}" for status, count in sorted(subscription_statuses.items()) + ) + return "\n".join( + [ + "我的个人数据摘要：", + f"个人规则：{personal_rule_count} 条", + f"偏好：{preference_text or '暂无'}", + f"兴趣与自选：{interest_text or '暂无'}", + f"个人记忆：{memory_count} 条", + f"会话：{conversation_count} 个", + f"订阅：{subscriptions_text or '暂无'}", + "可发送“我的偏好”“查看规则”“我的订阅”查看明细。", + ] + ) + + +def _memory_count( + db: Session, + owner_id: int, + kind: str, +) -> int: + return int( + db.scalar( + select(func.count()) + .select_from(AIMemoryEntry) + .where( + AIMemoryEntry.owner_id == owner_id, + AIMemoryEntry.kind == kind, + ) + ) + or 0 + ) + + +def _audit_summary(db: Session, principal: FeishuPrincipal) -> None: + _audit( + db, + principal, + action="personalization.data.summary", + risk_level=AuditRiskLevel.LOW, + ) + + +def _audit_erasure_request(db: Session, principal: FeishuPrincipal) -> None: + _audit( + db, + principal, + action="personalization.erasure.request", + risk_level=AuditRiskLevel.HIGH, + ) + + +def _audit( + db: Session, + principal: FeishuPrincipal, + *, + action: str, + risk_level: str, +) -> None: + AuditService(db).record( + AuditLogCreate( + actor=principal.user_code, + source=AuditSource.FEISHU, + action=action, + target_type="feishu-user", + target_id=principal.user_code, + risk_level=risk_level, + response_payload={"result": "success"}, + ) + ) + db.commit() + + +def _result( + feishu: FeishuService, + command: FeishuCommandName, + content: str, + principal: FeishuPrincipal, + auto_reply: bool, + *, + audit_actor: str | None = None, +) -> dict[str, Any]: + response = ( + send_text_if_configured( + feishu, + principal.chat_id, + content, + audit_actor or principal.user_code, + record_audit=audit_actor is None, + ) + if auto_reply + else None + ) + return command_result( + command, + FeishuReplyType.TEXT, + PERSONAL_DATA_TITLE, + content, + response, + ) diff --git a/app/application/feishu/handlers/personalization.py b/app/application/feishu/handlers/personalization.py new file mode 100644 index 0000000..c356e16 --- /dev/null +++ b/app/application/feishu/handlers/personalization.py @@ -0,0 +1,300 @@ +import re +from typing import Any + +from fastapi import HTTPException +from sqlalchemy.orm import Session + +from app.application.feishu.delivery import send_text_if_configured +from app.application.feishu.results import command_result +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.personalization.constants import PreferenceCategory +from app.modules.personalization.services import ConversationService, PreferenceService + +PERSONALIZATION_TITLE = "个人设置" +PREFERENCE_HELP = ( + "偏好指令格式：\n" + "记住偏好 语言：中文\n" + "记住偏好 语气：简洁\n" + "记住偏好 详略：详细\n" + "关注主题：人工智能\n" + "我的偏好\n" + "删除偏好 <偏好编号>" +) +_CATEGORY_LABELS = { + "语言": PreferenceCategory.LANGUAGE, + "语气": PreferenceCategory.TONE, + "详略": PreferenceCategory.DETAIL, + "主题": PreferenceCategory.TOPIC, + "兴趣": PreferenceCategory.INTEREST, +} +_CATEGORY_NAMES = { + PreferenceCategory.LANGUAGE: "语言", + PreferenceCategory.TONE: "语气", + PreferenceCategory.DETAIL: "详略", + PreferenceCategory.TOPIC: "关注主题", + PreferenceCategory.INTEREST: "兴趣", +} +_SET_PATTERN = re.compile( + r"^记住偏好\s+(语言|语气|详略|主题|兴趣)\s*[：:]\s*(.+)$" +) +_TOPIC_PATTERN = re.compile(r"^关注主题\s*[：:]?\s*(.+)$") +_DELETE_PATTERN = re.compile( + r"^删除偏好\s+(PREF-[A-Za-z0-9-]+)$", + re.IGNORECASE, +) +_LIST_COMMANDS = {"我的偏好", "查看偏好", "我的兴趣", "查看兴趣"} +_HELP_COMMANDS = {"帮助", "使用帮助", "命令帮助"} +_COMMAND_PREFIXES = ( + "记住偏好", + "关注主题", + "我的偏好", + "查看偏好", + "我的兴趣", + "查看兴趣", + "删除偏好", + "重置对话", + *_HELP_COMMANDS, +) + + +def handle_personalization_command( + db: Session, + feishu: FeishuService, + command_text: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any] | None: + """Handle owner-scoped preferences, interests, help, and conversation reset.""" + + text = command_text.strip() + if not text.startswith(_COMMAND_PREFIXES): + return None + principal.require_active() + if text in _HELP_COMMANDS: + return _result( + feishu, + FeishuCommandName.HELP, + _help_content(principal), + principal, + auto_reply, + ) + if text == "重置对话": + chat_type, chat_key = _conversation_key(principal) + reset = ConversationService(db).reset( + principal.owner_id, + chat_type, + chat_key, + ) + _audit( + db, + principal, + action="personalization.conversation.reset", + target_type="ai-conversation", + response={"reset": reset}, + ) + content = "当前会话已重置。" if reset else "当前会话没有可清除的历史。" + return _result( + feishu, + FeishuCommandName.CONVERSATION_RESET, + content, + principal, + auto_reply, + ) + + service = PreferenceService(db) + set_match = _SET_PATTERN.fullmatch(text) + topic_match = _TOPIC_PATTERN.fullmatch(text) + delete_match = _DELETE_PATTERN.fullmatch(text) + try: + if set_match: + category = _CATEGORY_LABELS[set_match.group(1)] + record = service.upsert( + principal.owner_id, + category, + set_match.group(2), + ) + _audit_preference(db, principal, "upsert", record) + content = ( + "偏好已保存。\n" + f"编号：{record['code']}\n" + f"类别：{_category_name(record['category'])}\n" + f"内容：{record['value']}" + ) + command = FeishuCommandName.PREFERENCE_SET + elif topic_match: + record = service.upsert( + principal.owner_id, + PreferenceCategory.TOPIC, + topic_match.group(1), + ) + _audit_preference(db, principal, "upsert", record) + content = ( + "关注主题已保存。\n" + f"编号：{record['code']}\n" + f"主题：{record['value']}" + ) + command = FeishuCommandName.PREFERENCE_SET + elif text in _LIST_COMMANDS: + records = service.list_preferences(principal.owner_id) + _audit( + db, + principal, + action="personalization.preference.list", + target_type="user-preference", + response={"count": len(records)}, + ) + content = _preference_list(records) + command = FeishuCommandName.PREFERENCE_LIST + elif delete_match: + code = delete_match.group(1) + service.delete(principal.owner_id, code) + _audit( + db, + principal, + action="personalization.preference.delete", + target_type="user-preference", + target_id=code, + response={"deleted": True}, + ) + content = "偏好已删除。" + command = FeishuCommandName.PREFERENCE_DELETE + else: + content = PREFERENCE_HELP + command = FeishuCommandName.PREFERENCE_SET + except HTTPException as exc: + db.rollback() + if exc.status_code == 404: + content = "没有找到该偏好，请先发送“我的偏好”确认编号。" + elif "Sensitive preference" in str(exc.detail): + content = "该内容可能涉及敏感个人信息或密钥，已拒绝保存。" + else: + content = f"偏好指令未执行，请检查格式。\n\n{PREFERENCE_HELP}" + command = ( + FeishuCommandName.PREFERENCE_DELETE + if delete_match + else FeishuCommandName.PREFERENCE_SET + ) + return _result(feishu, command, content, principal, auto_reply) + + +def _preference_list(records: list[dict[str, Any]]) -> str: + if not records: + return "当前没有已保存的偏好或兴趣。\n\n" + PREFERENCE_HELP + lines = ["我的偏好与兴趣："] + for record in records: + lines.append( + f"{record['code']}｜{_category_name(record['category'])}" + f"｜{record['value']}" + ) + return "\n".join(lines) + + +def _help_content(principal: FeishuPrincipal) -> str: + lines = [ + "你可以使用：", + "问 <问题>", + "学习规则：<个人规则>；查看/修改/启用/停用/删除规则", + "记住偏好、关注主题、我的偏好、删除偏好", + "加入自选 <股票代码>；查看自选", + "订阅 <自然语言时间>：<提示词>；我的订阅、暂停、恢复、退订", + "设置时区、设置/关闭安静时段", + "重置对话、我的数据、忘记我", + ] + if principal.is_admin: + lines.extend( + [ + "", + "管理员还可以使用：", + "日报、周报、财务、风险和考勤查询", + "学习/查看/修改/启停/删除公司规则", + "在当前群创建群订阅", + "通过真实 @用户 管理角色和状态", + ] + ) + return "\n".join(lines) + + +def _conversation_key(principal: FeishuPrincipal) -> tuple[str, str]: + chat_type = principal.chat_type or "p2p" + is_group = chat_type in {"group", "group_chat"} + chat_key = principal.chat_id if is_group else (principal.chat_id or principal.open_id) + if not chat_key: + raise HTTPException(status_code=422, detail="Missing Feishu chat identity") + return chat_type, chat_key + + +def _audit_preference( + db: Session, + principal: FeishuPrincipal, + action: str, + record: dict[str, Any], +) -> None: + _audit( + db, + principal, + action=f"personalization.preference.{action}", + target_type="user-preference", + target_id=str(record["code"]), + response={"category": record["category"]}, + ) + + +def _audit( + db: Session, + principal: FeishuPrincipal, + *, + action: str, + target_type: str, + target_id: str | None = None, + response: dict[str, Any] | None = None, +) -> None: + AuditService(db).record( + AuditLogCreate( + actor=principal.user_code, + source=AuditSource.FEISHU, + action=action, + target_type=target_type, + target_id=target_id, + risk_level=AuditRiskLevel.LOW, + response_payload=response, + ) + ) + db.commit() + + +def _category_name(value: str) -> str: + try: + return _CATEGORY_NAMES[PreferenceCategory(value)] + except ValueError: + return value + + +def _result( + feishu: FeishuService, + command: FeishuCommandName, + content: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any]: + response = ( + send_text_if_configured( + feishu, + principal.chat_id, + content, + principal.user_code, + ) + if auto_reply + else None + ) + return command_result( + command, + FeishuReplyType.TEXT, + PERSONALIZATION_TITLE, + content, + response, + ) diff --git a/app/application/feishu/handlers/rules.py b/app/application/feishu/handlers/rules.py index a5d09e7..75afb9f 100644 --- a/app/application/feishu/handlers/rules.py +++ b/app/application/feishu/handlers/rules.py @@ -11,32 +11,60 @@ from app.modules.ai_memory.constants import AIMemoryStatus from app.modules.ai_memory.service import AIMemoryService from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.constants import FeishuCapability +from app.modules.feishu_users.principal import FeishuPrincipal RULE_TITLE = "AI 学习规则" RULE_CREATE_PATTERN = re.compile(r"^学习规则(?:\s+(\d{1,3}))?\s*[：:]\s*(.*)$") MARKET_RULE_CREATE_PATTERN = re.compile( r"^学习市场规则(?:\s+(\d{1,3}))?\s*[：:]\s*(.*)$" ) +RULE_UPDATE_PATTERN = re.compile( + r"^修改规则\s+(MEM-[A-Za-z0-9-]+)(?:\s+(\d{1,3}))?\s*[：:]\s*(.*)$", + re.IGNORECASE, +) RULE_DISABLE_PATTERN = re.compile(r"^停用规则\s+(MEM-[A-Za-z0-9-]+)$", re.IGNORECASE) RULE_ENABLE_PATTERN = re.compile(r"^启用规则\s+(MEM-[A-Za-z0-9-]+)$", re.IGNORECASE) +RULE_DELETE_PATTERN = re.compile(r"^删除规则\s+(MEM-[A-Za-z0-9-]+)$", re.IGNORECASE) RULE_LIST_COMMANDS = {"查看规则", "规则列表", "查看市场规则"} RULE_COMMAND_PREFIXES = ( + "学习公司市场规则", + "学习公司规则", + "查看公司市场规则", + "查看公司规则", + "修改公司规则", + "启用公司规则", + "停用公司规则", + "删除公司规则", "学习市场规则", "学习规则", "查看市场规则", "查看规则", "规则列表", + "修改规则", "停用规则", "启用规则", + "删除规则", ) RULE_COMMAND_HELP = ( "规则指令格式：\n" - "学习规则：<规则内容>\n" "学习规则 80：<规则内容>\n" - "学习市场规则 80：<仅用于市场分析的规则内容>\n" "查看规则\n" - "停用规则 <规则编号>\n" - "启用规则 <规则编号>" + "修改规则 <编号> 80：<新内容>\n" + "启用规则 <编号>\n" + "停用规则 <编号>\n" + "删除规则 <编号>" +) + +_COMPANY_REPLACEMENTS = ( + ("学习公司市场规则", "学习市场规则"), + ("查看公司市场规则", "查看市场规则"), + ("学习公司规则", "学习规则"), + ("查看公司规则", "查看规则"), + ("修改公司规则", "修改规则"), + ("启用公司规则", "启用规则"), + ("停用公司规则", "停用规则"), + ("删除公司规则", "删除规则"), ) @@ -47,29 +75,40 @@ def handle_rule_command( chat_id: str | None, actor: str, auto_reply: bool, + principal: FeishuPrincipal | None = None, ) -> dict[str, Any] | None: - """Handle persistent AI rule commands.""" + """Handle company or owner-scoped persistent AI rule commands.""" if not command_text.startswith(RULE_COMMAND_PREFIXES): return None - command = _command_name(command_text) - if command in { - FeishuCommandName.RULE_CREATE, - FeishuCommandName.RULE_DISABLE, - FeishuCommandName.RULE_ENABLE, - } and get_settings().read_only_mode: - return _result( - feishu, - command, - "当前为只读模式，不能新增或修改学习规则。请由管理员启用操作后重试。", - chat_id, - actor, - auto_reply, - ) + normalized, company_rule = _normalize_company_command(command_text) + owner_id: int | None = None + if get_settings().feishu_user_features_enabled: + if principal is None: + return _result( + feishu, + FeishuCommandName.PERMISSION_DENIED, + "个人规则只能由已验证的飞书账号管理。", + chat_id, + actor, + auto_reply, + ) + if company_rule: + principal.require_capability(FeishuCapability.COMPANY_RULES) + else: + owner_id = principal.owner_id + command = _command_name(normalized) content = RULE_COMMAND_HELP try: - command, content = _execute(db, command_text, command, actor) + command, content = _execute( + db, + normalized, + command, + actor, + owner_id=owner_id, + company_rule=company_rule or owner_id is None, + ) except HTTPException as exc: detail = str(exc.detail) if "secret-like" in detail: @@ -83,21 +122,65 @@ def handle_rule_command( return _result(feishu, command, content, chat_id, actor, auto_reply) +def is_company_rule_command(command_text: str) -> bool: + return any(command_text.startswith(prefix) for prefix, _ in _COMPANY_REPLACEMENTS) + + +def _normalize_company_command(command_text: str) -> tuple[str, bool]: + for prefix, replacement in _COMPANY_REPLACEMENTS: + if command_text.startswith(prefix): + return replacement + command_text[len(prefix) :], True + return command_text, False + + def _execute( db: Session, command_text: str, command: FeishuCommandName, actor: str, + *, + owner_id: int | None, + company_rule: bool, ) -> tuple[FeishuCommandName, str]: market_create_match = MARKET_RULE_CREATE_PATTERN.fullmatch(command_text) create_match = market_create_match or RULE_CREATE_PATTERN.fullmatch(command_text) + update_match = RULE_UPDATE_PATTERN.fullmatch(command_text) disable_match = RULE_DISABLE_PATTERN.fullmatch(command_text) enable_match = RULE_ENABLE_PATTERN.fullmatch(command_text) + delete_match = RULE_DELETE_PATTERN.fullmatch(command_text) memory = AIMemoryService(db) if create_match: - return command, _create_rule(memory, create_match, market_create_match is not None, actor) + return ( + command, + _create_rule( + memory, + create_match, + market_create_match is not None, + actor, + owner_id, + company_rule, + ), + ) if command_text in RULE_LIST_COMMANDS: - return FeishuCommandName.RULE_LIST, _list_rules(memory, command_text) + return ( + FeishuCommandName.RULE_LIST, + _list_rules(memory, command_text, owner_id, company_rule), + ) + if update_match: + priority = int(update_match.group(2) or 50) + content = update_match.group(3).strip() + if not content: + return FeishuCommandName.RULE_UPDATE, "规则内容不能为空。" + rule = memory.update_rule( + code=update_match.group(1), + content=content, + priority=priority, + tags=None, + enabled=None, + actor=actor, + owner_id=owner_id, + ) + return FeishuCommandName.RULE_UPDATE, _rule_state(rule, "已修改") if disable_match or enable_match: enabled = enable_match is not None match = enable_match or disable_match @@ -108,16 +191,16 @@ def _execute( tags=None, enabled=enabled, actor=actor, + owner_id=owner_id, ) state = "已启用" if enabled else "已停用" return ( FeishuCommandName.RULE_ENABLE if enabled else FeishuCommandName.RULE_DISABLE, - f"规则{state}。\n" - f"编号：{rule['code']}\n" - f"优先级：{rule['importance']}\n" - f"范围：{rule['scope']} / {rule['subject']}\n" - f"状态：{state}", + _rule_state(rule, state), ) + if delete_match: + memory.delete_rule(delete_match.group(1), actor=actor, owner_id=owner_id) + return FeishuCommandName.RULE_DELETE, "规则已删除。" return command, RULE_COMMAND_HELP @@ -126,6 +209,8 @@ def _create_rule( match: re.Match[str], market_rule: bool, actor: str, + owner_id: int | None, + company_rule: bool, ) -> str: priority = int(match.group(1) or 50) content = match.group(2).strip() @@ -136,29 +221,39 @@ def _create_rule( rule = memory.create_rule( content=content, scope="market" if market_rule else "global", - subject="market" if market_rule else "company", + subject=( + "market" + if market_rule + else ("company" if company_rule else "personal") + ), priority=priority, - tags=["feishu", *(["market"] if market_rule else [])], + tags=[ + "feishu", + "company" if company_rule else "personal", + *(["market"] if market_rule else []), + ], actor=actor, + owner_id=owner_id, ) - return ( - "规则已学习。\n" - f"编号：{rule['code']}\n" - f"优先级：{rule['importance']}\n" - f"范围：{rule['scope']} / {rule['subject']}\n" - "状态：已启用" - ) + return _rule_state(rule, "已学习") -def _list_rules(memory: AIMemoryService, command_text: str) -> str: +def _list_rules( + memory: AIMemoryService, + command_text: str, + owner_id: int | None, + company_rule: bool, +) -> str: rules = memory.list_rules( scope="market" if command_text == "查看市场规则" else None, status_filter=AIMemoryStatus.ACTIVE, limit=20, + owner_id=owner_id, ) if not rules: - return "当前没有已启用的学习规则。" - lines = ["当前已启用的学习规则："] + target = "公司" if company_rule else "个人" + return f"当前没有已启用的{target}规则。" + lines = ["当前已启用的公司规则：" if company_rule else "当前已启用的个人规则："] for rule in rules: rule_text = str(rule["content"]) if len(rule_text) > 80: @@ -170,11 +265,25 @@ def _list_rules(memory: AIMemoryService, command_text: str) -> str: return "\n\n".join(lines) +def _rule_state(rule: dict[str, Any], state: str) -> str: + return ( + f"规则{state}。\n" + f"编号：{rule['code']}\n" + f"优先级：{rule['importance']}\n" + f"范围：{rule['scope']} / {rule['subject']}\n" + f"状态：{state}" + ) + + def _command_name(command_text: str) -> FeishuCommandName: if command_text.startswith("停用规则"): return FeishuCommandName.RULE_DISABLE if command_text.startswith("启用规则"): return FeishuCommandName.RULE_ENABLE + if command_text.startswith("修改规则"): + return FeishuCommandName.RULE_UPDATE + if command_text.startswith("删除规则"): + return FeishuCommandName.RULE_DELETE if command_text.startswith(("查看市场规则", "查看规则", "规则列表")): return FeishuCommandName.RULE_LIST return FeishuCommandName.RULE_CREATE diff --git a/app/application/feishu/handlers/subscriptions.py b/app/application/feishu/handlers/subscriptions.py new file mode 100644 index 0000000..ca0540b --- /dev/null +++ b/app/application/feishu/handlers/subscriptions.py @@ -0,0 +1,321 @@ +import re +from datetime import UTC, datetime +from typing import Any +from zoneinfo import ZoneInfo + +from fastapi import HTTPException +from sqlalchemy.orm import Session + +from app.application.feishu.delivery import send_text_if_configured +from app.application.feishu.results import command_result +from app.modules.feishu.constants import FeishuCommandName, FeishuReplyType +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.subscriptions.constants import ( + DAILY_DELIVERY_LIMIT_REACHED, + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services import ( + ScheduleParseError, + SubscriptionManagementService, + parse_schedule, +) + +SUBSCRIPTION_TITLE = "订阅管理" +SUBSCRIPTION_HELP = ( + "订阅指令格式：\n" + "订阅 每天 09:00：提示词\n" + "订阅 工作日 18:00：提示词\n" + "订阅 每周一 09:00：提示词\n" + "订阅 每月1号 09:00：提示词\n" + "订阅 每隔30分钟：提示词\n" + "我的订阅\n" + "暂停订阅 <订阅编号>\n" + "恢复订阅 <订阅编号>\n" + "退订 <订阅编号>\n" + "设置时区 Asia/Shanghai\n" + "设置安静时段 22:00-07:00\n" + "关闭安静时段" +) +_LIST_COMMANDS = {"我的订阅", "查看订阅"} +_PAUSE_PATTERN = re.compile( + r"^(?:暂停订阅|停用订阅)\s+(SUB-[A-Za-z0-9-]+)$", + re.IGNORECASE, +) +_RESUME_PATTERN = re.compile( + r"^(?:恢复订阅|启用订阅)\s+(SUB-[A-Za-z0-9-]+)$", + re.IGNORECASE, +) +_CANCEL_PATTERN = re.compile( + r"^(?:退订|取消订阅)\s+(SUB-[A-Za-z0-9-]+)$", + re.IGNORECASE, +) +_TIMEZONE_PATTERN = re.compile(r"^设置时区\s+(\S+)$") +_QUIET_PATTERN = re.compile( + r"^设置安静时段\s+(\d{1,2}(?:[:：]\d{1,2}))" + r"\s*(?:-|~|至|到)\s*(\d{1,2}(?:[:：]\d{1,2}))$" +) +_CLOSE_QUIET_COMMAND = "关闭安静时段" +_COMMAND_PREFIXES = ( + "订阅", + "我的订阅", + "查看订阅", + "暂停订阅", + "停用订阅", + "恢复订阅", + "启用订阅", + "退订", + "取消订阅", + "设置时区", + "设置安静时段", + "关闭安静时段", +) + + +def handle_subscription_command( + db: Session, + feishu: FeishuService, + command_text: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any] | None: + """Handle self-service subscriptions for a verified Feishu principal.""" + + text = command_text.strip() + if not text.startswith(_COMMAND_PREFIXES): + return None + service = SubscriptionManagementService(db) + command = _command_name(text) + try: + content = _execute(service, text, principal) + except HTTPException as exc: + db.rollback() + content = _error_content(exc) + return _result(feishu, command, content, principal, auto_reply) + + +def _execute( + service: SubscriptionManagementService, + command_text: str, + principal: FeishuPrincipal, +) -> str: + if command_text.startswith("订阅"): + parts = _create_parts(command_text, principal.timezone) + if parts is None: + return f"无法识别订阅时间或提示词。\n\n{SUBSCRIPTION_HELP}" + schedule_expression, prompt = parts + if principal.chat_type in {"group", "group_chat"}: + subscription, schedule = service.create_group( + principal, + schedule_expression, + prompt, + ) + else: + subscription, schedule = service.create_private( + principal, + schedule_expression, + prompt, + ) + return ( + "订阅已启用。\n" + f"编号：{subscription.code}\n" + f"计划：{schedule.display}\n" + f"时区：{schedule.timezone}\n" + f"下次执行：{_format_next(schedule.next_run_at, schedule.timezone)}\n" + f"暂停命令：暂停订阅 {subscription.code}" + ) + if command_text in _LIST_COMMANDS: + return _list_content( + service.list_for_owner(principal), + service.latest_deliveries_for_owner(principal), + ) + + pause_match = _PAUSE_PATTERN.fullmatch(command_text) + if pause_match: + record = service.pause(principal, pause_match.group(1)) + return f"订阅已暂停。\n编号：{record.code}\n恢复命令：恢复订阅 {record.code}" + + resume_match = _RESUME_PATTERN.fullmatch(command_text) + if resume_match: + record = service.resume(principal, resume_match.group(1)) + return ( + "订阅已恢复。\n" + f"编号：{record.code}\n" + f"下次执行：{_format_next(record.next_run_at, record.timezone)}\n" + f"暂停命令：暂停订阅 {record.code}" + ) + + cancel_match = _CANCEL_PATTERN.fullmatch(command_text) + if cancel_match: + record = service.cancel(principal, cancel_match.group(1)) + return f"已退订。\n编号：{record.code}" + + timezone_match = _TIMEZONE_PATTERN.fullmatch(command_text) + if timezone_match: + owner = service.set_timezone(principal, timezone_match.group(1)) + return f"时区已设置为 {owner.timezone}。" + + quiet_match = _QUIET_PATTERN.fullmatch(command_text) + if quiet_match: + owner = service.set_quiet_hours( + principal, + quiet_match.group(1), + quiet_match.group(2), + ) + return ( + "安静时段已设置。\n" + f"{owner.quiet_hours_start.strftime('%H:%M')}" + f"-{owner.quiet_hours_end.strftime('%H:%M')}" + ) + + if command_text == _CLOSE_QUIET_COMMAND: + service.clear_quiet_hours(principal) + return "安静时段已关闭。" + return SUBSCRIPTION_HELP + + +def _create_parts(command_text: str, timezone_name: str) -> tuple[str, str] | None: + payload = command_text.removeprefix("订阅").strip() + separator_indexes = [ + index for index, character in enumerate(payload) if character in {":", "："} + ] + for index in reversed(separator_indexes): + schedule_expression = payload[:index].strip() + prompt = payload[index + 1 :].strip() + if not schedule_expression or not prompt: + continue + try: + parse_schedule(schedule_expression, timezone_name) + except ScheduleParseError: + continue + return schedule_expression, prompt + return None + + +def _list_content( + records: list[PushSubscription], + latest_deliveries: dict[int, PushDelivery], +) -> str: + if not records: + return "当前没有订阅。\n\n" + SUBSCRIPTION_HELP.splitlines()[0] + lines = ["我的订阅："] + for record in records: + prompt = record.prompt if len(record.prompt) <= 40 else f"{record.prompt[:40]}…" + target = ( + "私聊" + if record.target_type == SubscriptionTargetType.USER + else "当前群" + ) + lines.append( + f"{record.code}｜{_status_name(record.status)}｜{target}\n" + f"{_schedule_name(record)}｜下次 {_format_next(record.next_run_at, record.timezone)}\n" + f"{prompt}{_delivery_note(latest_deliveries.get(record.id))}" + ) + return "\n\n".join(lines) + + +def _schedule_name(record: PushSubscription) -> str: + config = record.schedule_config + if record.schedule_type == SubscriptionScheduleType.ONCE: + return "单次" + if record.schedule_type == SubscriptionScheduleType.INTERVAL: + return f"每隔 {config['minutes']} 分钟" + clock = f"{int(config['hour']):02d}:{int(config['minute']):02d}" + if record.schedule_type == SubscriptionScheduleType.DAILY: + return f"每天 {clock}" + if record.schedule_type == SubscriptionScheduleType.WEEKDAY: + return f"工作日 {clock}" + if record.schedule_type == SubscriptionScheduleType.WEEKLY: + names = "一二三四五六日" + return f"每周{names[int(config['weekday'])]} {clock}" + return f"每月 {config['day']} 号 {clock}" + + +def _status_name(status_value: str) -> str: + return { + PushSubscriptionStatus.ACTIVE: "已启用", + PushSubscriptionStatus.PAUSED: "已暂停", + PushSubscriptionStatus.CANCELLED: "已退订", + PushSubscriptionStatus.COMPLETED: "已完成", + }.get(status_value, status_value) + + +def _delivery_note(delivery: PushDelivery | None) -> str: + if delivery is None: + return "" + if ( + delivery.status == PushDeliveryStatus.SKIPPED + and delivery.last_error == DAILY_DELIVERY_LIMIT_REACHED + ): + return "\n最近投递：因每日最多 96 条限制已跳过" + if delivery.status == PushDeliveryStatus.RETRY: + return "\n最近投递：发送失败，正在按 1/5/15 分钟重试" + if delivery.status == PushDeliveryStatus.FAILED: + return "\n最近投递：重试后仍失败，请联系管理员" + if delivery.status == PushDeliveryStatus.SKIPPED: + return "\n最近投递：因账号、订阅状态或安静时段限制已跳过" + return "" + + +def _format_next(value: datetime | None, timezone_name: str) -> str: + if value is None: + return "无" + aware = value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC) + return aware.astimezone(ZoneInfo(timezone_name)).strftime("%Y-%m-%d %H:%M") + + +def _command_name(command_text: str) -> FeishuCommandName: + if command_text in _LIST_COMMANDS: + return FeishuCommandName.SUBSCRIPTION_LIST + if _PAUSE_PATTERN.fullmatch(command_text): + return FeishuCommandName.SUBSCRIPTION_PAUSE + if _RESUME_PATTERN.fullmatch(command_text): + return FeishuCommandName.SUBSCRIPTION_RESUME + if _CANCEL_PATTERN.fullmatch(command_text): + return FeishuCommandName.SUBSCRIPTION_CANCEL + if _TIMEZONE_PATTERN.fullmatch(command_text): + return FeishuCommandName.SUBSCRIPTION_TIMEZONE + if _QUIET_PATTERN.fullmatch(command_text) or command_text == _CLOSE_QUIET_COMMAND: + return FeishuCommandName.SUBSCRIPTION_QUIET_HOURS + return FeishuCommandName.SUBSCRIPTION_CREATE + + +def _error_content(exc: HTTPException) -> str: + detail = str(exc.detail) + if exc.status_code == 404: + return "没有找到该订阅，请先发送“我的订阅”确认编号。" + if exc.status_code == 409 and "50" in detail: + return "已达到最多 50 个启用订阅，请先暂停或退订现有订阅。" + if exc.status_code == 403: + return "当前飞书账号无权执行该订阅操作。" + return f"订阅指令未执行：{detail}\n\n{SUBSCRIPTION_HELP}" + + +def _result( + feishu: FeishuService, + command: FeishuCommandName, + content: str, + principal: FeishuPrincipal, + auto_reply: bool, +) -> dict[str, Any]: + response = ( + send_text_if_configured( + feishu, + principal.chat_id, + content, + principal.user_code, + ) + if auto_reply + else None + ) + return command_result( + command, + FeishuReplyType.TEXT, + SUBSCRIPTION_TITLE, + content, + response, + ) diff --git a/app/application/feishu/personal_data.py b/app/application/feishu/personal_data.py new file mode 100644 index 0000000..937bddf --- /dev/null +++ b/app/application/feishu/personal_data.py @@ -0,0 +1,324 @@ +from collections.abc import Sequence + +from fastapi import HTTPException, status +from sqlalchemy import delete, select +from sqlalchemy.orm import Session + +from app.core.config import get_settings +from app.core.utils.time import utc_now +from app.modules.audit.constants import AuditRiskLevel, AuditSource, AuditStatus +from app.modules.audit.models import AuditLog +from app.modules.feishu_users.constants import ( + FEISHU_USER_NOT_FOUND, + LAST_ACTIVE_ADMIN_ERROR, + FeishuCapability, + FeishuUserRole, + FeishuUserStatus, + parse_admin_identities, +) +from app.modules.feishu_users.bootstrap import admin_bootstrap_identity_hash +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.personalization.schemas import ErasureConfirmation, ErasureResult +from app.modules.personalization.services.erasure import ( + ErasureHook, + PersonalDataErasureService, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription + +PERSONAL_DATA_ERASURE_ACTION = "feishu.user.personal_data_erased" + + +class FeishuPersonalDataService: + """Coordinate Feishu identity erasure across personal-data domains.""" + + def __init__( + self, + db: Session, + *, + extra_hooks: Sequence[ErasureHook] = (), + ) -> None: + self.db = db + self.extra_hooks = tuple(extra_hooks) + self.erasure = PersonalDataErasureService(db) + + def request_confirmation( + self, + principal: FeishuPrincipal, + ) -> ErasureConfirmation: + """Issue a short-lived confirmation code for the authenticated user.""" + + principal.require_capability(FeishuCapability.PERSONAL_DATA) + user = self._find_principal_user(principal) + if user.status != FeishuUserStatus.ACTIVE: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu user is disabled", + ) + return self.erasure.request_confirmation(user.id) + + def confirm( + self, + principal: FeishuPrincipal, + confirmation_code: str, + ) -> ErasureResult: + """Confirm and erase the authenticated user's personal data.""" + + principal.require_capability(FeishuCapability.PERSONAL_DATA) + user = self._lock_principal_user(principal) + if user.status != FeishuUserStatus.ACTIVE: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu user is disabled", + ) + return self._confirm_and_erase( + user, + confirmation_code, + audit_source=AuditSource.FEISHU, + ) + + def erase_by_user_code(self, code: str, *, actor: str) -> ErasureResult: + """Erase a user selected by an authenticated internal service.""" + + if not actor.strip(): + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Authenticated service actor is required", + ) + user = self._lock_user(code=code) + confirmation = self.erasure.request_confirmation(user.id) + + # request_confirmation commits by design. Lock and re-check the last-admin + # invariant in the deletion transaction before any personal row is removed. + user = self._lock_user(code=code) + return self._confirm_and_erase( + user, + confirmation.confirmation_code, + audit_source=AuditSource.API, + ) + + def _find_principal_user(self, principal: FeishuPrincipal) -> FeishuUser: + user = self.db.execute( + select(FeishuUser).where( + FeishuUser.id == principal.owner_id, + FeishuUser.code == principal.user_code, + FeishuUser.tenant_key == principal.tenant_key, + FeishuUser.open_id == principal.open_id, + ) + ).scalar_one_or_none() + if user is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=FEISHU_USER_NOT_FOUND, + ) + return user + + def _lock_principal_user(self, principal: FeishuPrincipal) -> FeishuUser: + return self._lock_user( + owner_id=principal.owner_id, + code=principal.user_code, + tenant_key=principal.tenant_key, + open_id=principal.open_id, + ) + + def _lock_user( + self, + *, + owner_id: int | None = None, + code: str | None = None, + tenant_key: str | None = None, + open_id: str | None = None, + ) -> FeishuUser: + active_admin_ids = list( + self.db.execute( + select(FeishuUser.id) + .where( + FeishuUser.role == FeishuUserRole.ADMIN, + FeishuUser.status == FeishuUserStatus.ACTIVE, + ) + .order_by(FeishuUser.id.asc()) + .with_for_update() + ).scalars() + ) + filters = [] + if owner_id is not None: + filters.append(FeishuUser.id == owner_id) + if code is not None: + filters.append(FeishuUser.code == code) + if tenant_key is not None: + filters.append(FeishuUser.tenant_key == tenant_key) + if open_id is not None: + filters.append(FeishuUser.open_id == open_id) + user = self.db.execute( + select(FeishuUser).where(*filters).with_for_update() + ).scalar_one_or_none() + if user is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=FEISHU_USER_NOT_FOUND, + ) + if ( + user.role == FeishuUserRole.ADMIN + and user.status == FeishuUserStatus.ACTIVE + and active_admin_ids == [user.id] + ): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=LAST_ACTIVE_ADMIN_ERROR, + ) + return user + + def _confirm_and_erase( + self, + user: FeishuUser, + confirmation_code: str, + *, + audit_source: str, + ) -> ErasureResult: + identifiers = tuple( + sorted( + { + value + for value in ( + user.code, + user.open_id, + user.union_id, + user.user_id, + ) + if value + }, + key=len, + reverse=True, + ) + ) + configured_admins = parse_admin_identities( + getattr(get_settings(), "feishu_admin_identities", ()) + ) + bootstrap_identity_hash = ( + admin_bootstrap_identity_hash(user.tenant_key, user.open_id) + if (user.tenant_key, user.open_id) in configured_admins + else None + ) + + def delete_subscriptions( + db: Session, + owner_id: int, + _anonymous_id: str, + ) -> dict[str, int]: + subscription_ids = select(PushSubscription.id).where( + PushSubscription.owner_id == owner_id + ) + deliveries = _row_count( + db.execute( + delete(PushDelivery).where( + PushDelivery.subscription_id.in_(subscription_ids) + ) + ).rowcount + ) + subscriptions = _row_count( + db.execute( + delete(PushSubscription).where( + PushSubscription.owner_id == owner_id + ) + ).rowcount + ) + return { + "deliveries": deliveries, + "subscriptions": subscriptions, + } + + def finalize_identity( + db: Session, + _owner_id: int, + anonymous_id: str, + ) -> dict[str, int]: + # Flush the pending confirmation-request deletion before removing + # the identity that owns it. + db.flush() + anonymized_logs = _anonymize_audit_logs( + db, + identifiers=identifiers, + anonymous_id=anonymous_id, + ) + if bootstrap_identity_hash is not None: + existing_tombstone = db.scalar( + select(FeishuAdminBootstrapTombstone.id).where( + FeishuAdminBootstrapTombstone.identity_hash + == bootstrap_identity_hash + ) + ) + if existing_tombstone is None: + db.add( + FeishuAdminBootstrapTombstone( + identity_hash=bootstrap_identity_hash + ) + ) + db.delete(user) + db.flush() + db.add( + AuditLog( + actor=anonymous_id, + source=audit_source, + action=PERSONAL_DATA_ERASURE_ACTION, + target_type=None, + target_id=None, + risk_level=AuditRiskLevel.HIGH, + request_payload=None, + response_payload=None, + status=AuditStatus.SUCCESS, + request_id=None, + created_at=utc_now(), + ) + ) + return { + "audit_logs_anonymized": anonymized_logs, + "identity": 1, + } + + return self.erasure.confirm_and_erase( + user.id, + confirmation_code, + before_hooks=(delete_subscriptions,), + extra_hooks=(*self.extra_hooks, finalize_identity), + ) + + +def _anonymize_audit_logs( + db: Session, + *, + identifiers: Sequence[str], + anonymous_id: str, +) -> int: + if not identifiers: + return 0 + changed_count = 0 + for record in db.execute(select(AuditLog)).scalars(): + searchable_values = ( + record.actor, + record.target_id, + record.request_payload, + record.response_payload, + ) + if not any( + identifier in value + for value in searchable_values + if value is not None + for identifier in identifiers + ): + continue + record.actor = anonymous_id + record.target_type = None + record.target_id = None + record.request_payload = None + record.response_payload = None + record.request_id = None + changed_count += 1 + db.flush() + return changed_count + + +def _row_count(value: int | None) -> int: + return max(0, int(value or 0)) diff --git a/app/application/pipelines/lifecycle.py b/app/application/pipelines/lifecycle.py index 265259e..7e5bd3d 100644 --- a/app/application/pipelines/lifecycle.py +++ b/app/application/pipelines/lifecycle.py @@ -5,7 +5,6 @@ from sqlalchemy import select from sqlalchemy.orm import Session from app.core.config import get_settings -from app.core.security import business_mutations_enabled from app.core.constants import ActorValue from app.core.utils.time import utc_now from app.application.delivery import ReportDeliveryService @@ -66,12 +65,6 @@ class LifecyclePipelineService: force: bool = False, actor: str = ActorValue.SCHEDULER, ) -> dict[str, Any]: - if not business_mutations_enabled(): - return { - "period_key": self.period_key(report_type), - "deduplicated": False, - "status": "operations_disabled", - } workflow, period_key, deduplicated = self.prepare(report_type, actor, force) if deduplicated: return { diff --git a/app/application/pipelines/market.py b/app/application/pipelines/market.py index 5634933..c2a89e0 100644 --- a/app/application/pipelines/market.py +++ b/app/application/pipelines/market.py @@ -6,7 +6,6 @@ from sqlalchemy.orm import Session from app.core.config import get_settings from app.core.constants import ActorValue -from app.core.security import business_mutations_enabled from app.core.utils.time import utc_now from app.modules.feishu.service import FeishuService from app.modules.market.chart import render_market_chart @@ -51,12 +50,6 @@ class MarketPipelineService: ) -> dict[str, Any]: target = reference_date or date.today() period_key = self.period_key(report_type, target) - if not business_mutations_enabled(): - return { - "period_key": period_key, - "status": "operations_disabled", - "deduplicated": False, - } existing = self.find(period_key) if existing is not None and existing.status == WorkflowStatus.COMPLETED and not force: return { diff --git a/app/application/scheduling/scheduler.py b/app/application/scheduling/scheduler.py index 3150055..645a723 100644 --- a/app/application/scheduling/scheduler.py +++ b/app/application/scheduling/scheduler.py @@ -46,14 +46,23 @@ def create_scheduler(app: FastAPI | None = None) -> Any: enqueue_market_report, enqueue_project_weekly_push, enqueue_risk_progress_push, + enqueue_subscription_cycle, enqueue_work_daily_push, enqueue_work_weekly_push, ) from app.modules.observability.service import ObservabilityService + from app.modules.personalization.services import ConversationService from app.modules.reports.services import ReportService settings = get_settings() - scheduler = BackgroundScheduler(timezone="Asia/Shanghai") + scheduler = BackgroundScheduler( + timezone="Asia/Shanghai", + job_defaults={ + "coalesce": True, + "max_instances": 1, + "misfire_grace_time": 300, + }, + ) def deliver_report( state_key: str, @@ -189,7 +198,19 @@ def create_scheduler(app: FastAPI | None = None) -> Any: finally: db.close() - if settings.lifecycle_pipeline_enabled and not settings.read_only_mode: + def run_subscription_cycle() -> None: + dispatch = enqueue_subscription_cycle(actor=ActorValue.SCHEDULER) + _set_state(app, "last_subscription_cycle", dispatch) + + def run_personalization_retention_cleanup() -> None: + db = SessionLocal() + try: + deleted = ConversationService(db).cleanup_expired_globally() + _set_state(app, "last_personalization_retention_cleanup", deleted) + finally: + db.close() + + if settings.lifecycle_pipeline_enabled: scheduler.add_job( run_daily_lifecycle, trigger="cron", @@ -266,7 +287,7 @@ def create_scheduler(app: FastAPI | None = None) -> Any: id="event_dispatch", replace_existing=True, ) - if settings.market_analysis_enabled and not settings.read_only_mode: + if settings.market_analysis_enabled: scheduler.add_job( run_market_premarket, trigger="cron", @@ -301,9 +322,23 @@ def create_scheduler(app: FastAPI | None = None) -> Any: id="scheduler_heartbeat", replace_existing=True, ) + if settings.feishu_user_features_enabled: + scheduler.add_job( + run_subscription_cycle, + trigger="interval", + minutes=1, + id="subscription_delivery_cycle", + replace_existing=True, + ) + scheduler.add_job( + run_personalization_retention_cleanup, + trigger="interval", + minutes=1, + id="personalization_retention_cleanup", + replace_existing=True, + ) if ( not settings.lifecycle_pipeline_enabled - and not settings.read_only_mode and settings.legacy_sync_enabled and settings.legacy_project_query ): @@ -317,7 +352,6 @@ def create_scheduler(app: FastAPI | None = None) -> Any: ) if ( not settings.lifecycle_pipeline_enabled - and not settings.read_only_mode and settings.legacy_sync_enabled and settings.legacy_task_query ): diff --git a/app/core/background/task_queue/__init__.py b/app/core/background/task_queue/__init__.py index 477f221..46306de 100644 --- a/app/core/background/task_queue/__init__.py +++ b/app/core/background/task_queue/__init__.py @@ -12,6 +12,7 @@ from app.tasks.constants import ( TASK_RUN_LIFECYCLE, TASK_RUN_MARKET_CLOSE, TASK_RUN_MARKET_REPORT, + TASK_RUN_SUBSCRIPTION_CYCLE, ) from app.core.background.task_queue.dispatcher import dispatch_task from app.core.background.task_queue.events import enqueue_event_dispatch @@ -30,6 +31,7 @@ from app.core.background.task_queue.reports import ( enqueue_work_weekly_push, ) from app.core.background.task_queue.risk import enqueue_risk_event_generation +from app.core.background.task_queue.subscriptions import enqueue_subscription_cycle __all__ = [ @@ -46,6 +48,7 @@ __all__ = [ "TASK_RUN_LIFECYCLE", "TASK_RUN_MARKET_CLOSE", "TASK_RUN_MARKET_REPORT", + "TASK_RUN_SUBSCRIPTION_CYCLE", "dispatch_task", "enqueue_attendance_summary_push", "enqueue_daily_brief_push", @@ -58,6 +61,7 @@ __all__ = [ "enqueue_project_weekly_push", "enqueue_risk_progress_push", "enqueue_risk_event_generation", + "enqueue_subscription_cycle", "enqueue_work_daily_push", "enqueue_work_weekly_push", ] diff --git a/app/core/background/task_queue/lifecycle.py b/app/core/background/task_queue/lifecycle.py index 7ec7d7b..36b2cc7 100644 --- a/app/core/background/task_queue/lifecycle.py +++ b/app/core/background/task_queue/lifecycle.py @@ -5,6 +5,7 @@ from app.core.config import get_settings from app.core.constants import ActorValue from app.core.database import SessionLocal from app.application.pipelines import LifecyclePipelineService +from app.modules.workflows.constants import WorkflowStatus, WorkflowType def enqueue_lifecycle_report( @@ -28,16 +29,28 @@ def enqueue_lifecycle_report( if get_settings().task_queue_enabled: from app.tasks import celery_app - result = celery_app.signature( - TASK_RUN_LIFECYCLE, - kwargs={ - "report_type": report_type, - "receive_id": receive_id, - "receive_id_type": receive_id_type, - "force": force, - "actor": actor, - }, - ).apply_async() + try: + result = celery_app.signature( + TASK_RUN_LIFECYCLE, + kwargs={ + "report_type": report_type, + "receive_id": receive_id, + "receive_id_type": receive_id_type, + "force": force, + "actor": actor, + }, + ).apply_async() + except Exception as exc: + service.workflows.start_or_update( + workflow_type=WorkflowType.LIFECYCLE_REPORT, + aggregate_type="report_period", + aggregate_id=period_key, + status_value=WorkflowStatus.FAILED, + action="enqueue_failed", + actor=actor, + payload={"error": str(exc)[:2000]}, + ) + raise return { "workflow_code": workflow.code, "period_key": period_key, diff --git a/app/core/background/task_queue/reports.py b/app/core/background/task_queue/reports.py index 553d0ad..51d6a81 100644 --- a/app/core/background/task_queue/reports.py +++ b/app/core/background/task_queue/reports.py @@ -1,5 +1,6 @@ from collections.abc import Callable from typing import Any +from uuid import uuid4 from app.tasks.constants import ( TASK_PUSH_ATTENDANCE_SUMMARY, @@ -173,37 +174,44 @@ def _queue_celery_report_push( from app.modules.reports.services import ReportService from app.tasks import celery_app + task_id = uuid4().hex db = SessionLocal() try: - push_run = ReportService(db).create_push_run( + service = ReportService(db) + push_run = service.create_push_run( report_type=report_type, title=title, receive_id=receive_id, receive_id_type=receive_id_type, actor=actor, status=ReportPushStatus.QUEUED, + task_id=task_id, ) - async_result = celery_app.signature( - task_name, - kwargs={ - "receive_id": receive_id, - "receive_id_type": receive_id_type, - "actor": actor, - "push_run_code": push_run.code, - }, - ).apply_async() - ReportService(db).update_push_run( - push_run.code, - ReportPushStatus.QUEUED, - task_id=async_result.id, - ) + try: + celery_app.signature( + task_name, + kwargs={ + "receive_id": receive_id, + "receive_id_type": receive_id_type, + "actor": actor, + "push_run_code": push_run.code, + }, + ).apply_async(task_id=task_id) + except Exception as exc: + service.update_push_run( + push_run.code, + ReportPushStatus.FAILED, + task_id=task_id, + error_message=str(exc), + ) + raise finally: db.close() return { "queued": True, "mode": "celery", "task_name": task_name, - "task_id": async_result.id, + "task_id": task_id, "push_run_code": push_run.code, } diff --git a/app/core/background/task_queue/subscriptions.py b/app/core/background/task_queue/subscriptions.py new file mode 100644 index 0000000..0f433f2 --- /dev/null +++ b/app/core/background/task_queue/subscriptions.py @@ -0,0 +1,14 @@ +from app.application.delivery.subscriptions import run_subscription_cycle +from app.core.background.task_queue.dispatcher import dispatch_task +from app.core.constants import ActorValue +from app.tasks.constants import TASK_RUN_SUBSCRIPTION_CYCLE + + +def enqueue_subscription_cycle( + actor: str = ActorValue.SCHEDULER, +) -> dict: + return dispatch_task( + TASK_RUN_SUBSCRIPTION_CYCLE, + {"actor": actor}, + lambda: run_subscription_cycle(actor=actor), + ) diff --git a/app/core/config/settings.py b/app/core/config/settings.py index 281fec8..28f14ad 100644 --- a/app/core/config/settings.py +++ b/app/core/config/settings.py @@ -1,6 +1,7 @@ import json +import os from functools import lru_cache -from typing import Annotated, Any +from typing import Annotated, Any, Literal from pydantic import Field, field_validator, model_validator from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict @@ -12,11 +13,22 @@ from app.core.constants import ( DEFAULT_OPENCLAW_ACTION_JSON, ) +_DOTENV_DISABLED = os.getenv("COMPANY_AI_DISABLE_DOTENV", "").strip().lower() in { + "1", + "true", + "yes", + "on", +} + class Settings(BaseSettings): """Runtime settings loaded from environment variables and `.env`.""" - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + model_config = SettingsConfigDict( + env_file=None if _DOTENV_DISABLED else ".env", + env_file_encoding="utf-8", + extra="ignore", + ) app_name: str = "Company AI Management Platform" app_env: str = "local" @@ -45,9 +57,14 @@ class Settings(BaseSettings): feishu_base_url: str = "https://open.feishu.cn/open-apis" feishu_app_id: str | None = None feishu_app_secret: str | None = None + feishu_app_type: Literal["self", "store"] = "self" + feishu_app_ticket: str | None = None feishu_verification_token: str | None = None feishu_encrypt_key: str | None = None feishu_default_chat_id: str | None = None + feishu_default_tenant_key: str | None = None + feishu_user_features_enabled: bool = False + feishu_admin_identities: Annotated[list[str], NoDecode] = Field(default_factory=list) model_provider: str = DEFAULT_MODEL_PROVIDER openclaw_base_url: str = "http://127.0.0.1:2070" openclaw_http_url: str | None = None @@ -109,6 +126,7 @@ class Settings(BaseSettings): event_dispatch_lock_seconds: int = 300 event_dispatch_cron_minute: str = "*/5" heartbeat_interval_seconds: int = 60 + heartbeat_retention_seconds: int = Field(default=86400, ge=1) ai_memory_enabled: bool = True ai_memory_auto_write_enabled: bool = True ai_memory_recall_limit: int = 5 @@ -135,6 +153,8 @@ class Settings(BaseSettings): ai_memory_forbidden_keys: list[str] = Field( default_factory=lambda: [ "authorization", + "app_access_token", + "app_ticket", "api_key", "apikey", "access_token", @@ -147,6 +167,7 @@ class Settings(BaseSettings): "direct_llm_api_key", "market_data_token", "feishu_app_secret", + "feishu_app_ticket", "feishu_verification_token", ] ) @@ -166,11 +187,20 @@ class Settings(BaseSettings): return [str(item).strip() for item in data if str(item).strip()] return [item.strip() for item in text.split(",") if item.strip()] + @field_validator( + "feishu_app_type", + mode="before", + ) + @classmethod + def normalize_feishu_app_type(cls, value: Any) -> str: + return str(value or "self").strip().lower() + @field_validator( "openclaw_allowed_tools", "openclaw_allowed_actions", "ai_memory_forbidden_keys", "ai_memory_blocked_content_terms", + "feishu_admin_identities", mode="before", ) @classmethod @@ -240,19 +270,49 @@ class Settings(BaseSettings): def validate_production_safety(self) -> "Settings": if self.app_env.lower() not in {"prod", "production"}: return self + def _enabled_keys( + legacy_key: str | None, + configured_keys: list[dict[str, Any]], + ) -> set[str]: + values: set[str] = set() + if legacy_key and legacy_key.strip(): + values.add(legacy_key) + for item in configured_keys: + enabled = item.get("enabled", True) + if not isinstance(enabled, bool): + enabled = str(enabled).strip().lower() not in { + "0", "false", "no", "off", "disabled" + } + key = item.get("key") + if enabled and key is not None and str(key).strip(): + values.add(str(key)) + return values + errors: list[str] = [] + api_key_values = _enabled_keys(self.api_key, self.api_keys) + audit_key_values = _enabled_keys(self.audit_api_key, self.audit_api_keys) if self.database_url.startswith("sqlite"): errors.append("DATABASE_URL must use PostgreSQL in production") - if not self.api_key and not any(item.get("key") for item in self.api_keys): + if not api_key_values: errors.append("API_KEY or API_KEYS is required in production") - if not self.audit_api_key and not any( - item.get("key") for item in self.audit_api_keys - ): + if not audit_key_values: errors.append("AUDIT_API_KEY or AUDIT_API_KEYS is required in production") + if api_key_values & audit_key_values: + errors.append( + "API_KEY/API_KEYS and AUDIT_API_KEY/AUDIT_API_KEYS cannot overlap in production" + ) if "*" in self.cors_origins: errors.append("CORS_ORIGINS cannot contain '*' in production") if self.debug: errors.append("DEBUG must be false in production") + if not self.mask_sensitive_responses: + errors.append("MASK_SENSITIVE_RESPONSES must be true in production") + if not self.read_only_mode: + errors.append("READ_ONLY_MODE must be true in production") + if self.feishu_user_features_enabled and not self.feishu_admin_identities: + errors.append( + "FEISHU_ADMIN_IDENTITIES is required when Feishu user features are enabled" + ) if errors: raise ValueError("; ".join(errors)) return self diff --git a/app/core/http/masking.py b/app/core/http/masking.py index 6ec8603..41bfc5a 100644 --- a/app/core/http/masking.py +++ b/app/core/http/masking.py @@ -6,26 +6,61 @@ from app.core.config import get_settings MASKED_VALUE = "[MASKED]" SENSITIVE_RESPONSE_KEYS = frozenset( { + "access_token", "account_number", "api_key", + "apikey", + "app_access_token", + "app_ticket", + "audit_api_key", + "authorization", "bank_account", "card_no", + "client_secret", + "cookie", "direct_llm_api_key", "email", + "encrypt_key", "feishu_app_secret", + "feishu_app_ticket", + "feishu_encrypt_key", "feishu_verification_token", "hermes_api_key", "id_card", + "market_data_token", "mobile", + "openclaw_api_key", "openclaw_gateway_token", "password", "payment_account", "phone", + "private_key", + "refresh_token", "secret", + "secret_key", + "set-cookie", + "set_cookie", "tenant_access_token", "token", + "x-api-key", + "x-audit-api-key", + "x_api_key", + "x_audit_api_key", } ) +NORMALIZED_SENSITIVE_RESPONSE_KEYS = frozenset( + "".join(character for character in key.casefold() if character.isalnum()) + for key in SENSITIVE_RESPONSE_KEYS +) + + +def is_sensitive_key(key: str) -> bool: + """Return whether a key matches a built-in sensitive field name.""" + + normalized = "".join( + character for character in key.casefold() if character.isalnum() + ) + return normalized in NORMALIZED_SENSITIVE_RESPONSE_KEYS def mask_configured(value: Any, domain: str | None = None) -> Any: @@ -61,7 +96,7 @@ def mask_sensitive( def _should_mask(key: str, domain: str | None, configured_fields: set[str]) -> bool: field = key.lower() - if field in SENSITIVE_RESPONSE_KEYS or field in configured_fields: + if is_sensitive_key(key) or field in configured_fields: return True if f"*.{field}" in configured_fields: return True diff --git a/app/core/security/__init__.py b/app/core/security/__init__.py index 3f9cd96..60d10e1 100644 --- a/app/core/security/__init__.py +++ b/app/core/security/__init__.py @@ -1,21 +1,7 @@ from app.core.security.api_keys import ApiPrincipal, require_api_key, require_audit_api_key -from app.core.security.operation_guard import ( - READ_ONLY_OPERATION_DISABLED, - require_operations_enabled, -) -from app.core.security.operation_policy import ( - OperationsDisabledError, - business_mutations_enabled, - ensure_business_mutations_enabled, -) __all__ = [ "ApiPrincipal", - "OperationsDisabledError", - "READ_ONLY_OPERATION_DISABLED", - "business_mutations_enabled", - "ensure_business_mutations_enabled", "require_api_key", "require_audit_api_key", - "require_operations_enabled", ] diff --git a/app/core/security/operation_guard.py b/app/core/security/operation_guard.py deleted file mode 100644 index a62bfa8..0000000 --- a/app/core/security/operation_guard.py +++ /dev/null @@ -1,16 +0,0 @@ -from fastapi import HTTPException, status - -from app.core.security.operation_policy import business_mutations_enabled - - -READ_ONLY_OPERATION_DISABLED = "This service is read-only; data mutation operations are disabled" - - -def require_operations_enabled(detail: str = READ_ONLY_OPERATION_DISABLED) -> None: - """Reject mutation-oriented endpoints when the product is running read-only.""" - - if not business_mutations_enabled(): - raise HTTPException( - status_code=status.HTTP_405_METHOD_NOT_ALLOWED, - detail=detail, - ) diff --git a/app/core/security/operation_policy.py b/app/core/security/operation_policy.py deleted file mode 100644 index 1c9a09c..0000000 --- a/app/core/security/operation_policy.py +++ /dev/null @@ -1,20 +0,0 @@ -from app.core.config import get_settings - - -class OperationsDisabledError(RuntimeError): - """Raised when a business mutation is attempted in read-only mode.""" - - -def business_mutations_enabled() -> bool: - """Return whether platform-owned business ledgers may be mutated.""" - - return not get_settings().read_only_mode - - -def ensure_business_mutations_enabled() -> None: - """Enforce read-only policy outside the HTTP transport layer.""" - - if not business_mutations_enabled(): - raise OperationsDisabledError( - "Business mutations are disabled while READ_ONLY_MODE is enabled" - ) diff --git a/app/modules/ai_agent/adapters/common.py b/app/modules/ai_agent/adapters/common.py index 5853e2f..bc3d3d1 100644 --- a/app/modules/ai_agent/adapters/common.py +++ b/app/modules/ai_agent/adapters/common.py @@ -1,12 +1,13 @@ +import json from typing import Any from fastapi import HTTPException, status from app.modules.ai_agent.constants import ( - CHAT_USER_CONTENT_TEMPLATE, COMPANY_MANAGEMENT_SYSTEM_INSTRUCTIONS, UNEXPECTED_HERMES_RESPONSE, AIChatRole, + AIContextKey, AIErrorKey, AIHttpPayloadKey, AIResponseKey, @@ -41,6 +42,7 @@ def _chat_completion_payload(response: Any, error_key: AIErrorKey) -> dict[str, def _chat_messages(prompt: str, context: dict[str, Any] | None = None) -> list[dict[str, str]]: + request_context = context or {} return [ { AIHttpPayloadKey.ROLE: AIChatRole.SYSTEM, @@ -48,14 +50,57 @@ def _chat_messages(prompt: str, context: dict[str, Any] | None = None) -> list[d }, { AIHttpPayloadKey.ROLE: AIChatRole.USER, - AIHttpPayloadKey.CONTENT: CHAT_USER_CONTENT_TEMPLATE.format( - context=context or {}, - task=prompt, - ), + AIHttpPayloadKey.CONTENT: _ordered_context(prompt, request_context), }, ] +def _ordered_context(prompt: str, context: dict[str, Any]) -> str: + """Serialize trusted personalization layers in their required precedence.""" + + company_rules = context.get(AIContextKey.COMPANY_RULES) + if company_rules is None: + company_rules = context.get(AIContextKey.USER_RULES) or [] + controlled_keys = { + AIContextKey.USER_RULES, + AIContextKey.COMPANY_RULES, + AIContextKey.PERSONAL_RULES, + AIContextKey.PREFERENCES, + AIContextKey.INTERESTS, + AIContextKey.LOCAL_MEMORY, + AIContextKey.CONVERSATION_HISTORY, + AIContextKey.PROVIDER_SESSION_ID, + AIContextKey.ALLOW_PROVIDER_MEMORY, + } + current_context = { + str(key): value for key, value in context.items() if key not in controlled_keys + } + sections = [ + ("公司规则", company_rules), + ("个人规则", context.get(AIContextKey.PERSONAL_RULES) or []), + ( + "当前请求", + { + "prompt": prompt, + "context": current_context, + }, + ), + ( + "个人偏好与兴趣", + { + "preferences": context.get(AIContextKey.PREFERENCES) or [], + "interests": context.get(AIContextKey.INTERESTS) or [], + }, + ), + ("个人相关记忆", context.get(AIContextKey.LOCAL_MEMORY) or []), + ("当前会话历史", context.get(AIContextKey.CONVERSATION_HISTORY) or []), + ] + return "\n\n".join( + f"{title}:\n{json.dumps(value, ensure_ascii=False, default=str)}" + for title, value in sections + ) + + def _service_root(base_url: str, suffix: str) -> str: root = base_url.rstrip("/") normalized_suffix = suffix.rstrip("/") diff --git a/app/modules/ai_agent/adapters/hermes.py b/app/modules/ai_agent/adapters/hermes.py index 7c5e8ff..6aa4b94 100644 --- a/app/modules/ai_agent/adapters/hermes.py +++ b/app/modules/ai_agent/adapters/hermes.py @@ -15,6 +15,7 @@ from app.modules.ai_agent.constants import ( AUTHORIZATION_BEARER_TEMPLATE, UNEXPECTED_HERMES_RESPONSE, AIErrorKey, + AIContextKey, AIHttpHeader, AIHttpPath, AIHttpPayloadKey, @@ -38,11 +39,16 @@ class HermesAdapter(AIAdapter): headers[AIHttpHeader.AUTHORIZATION] = AUTHORIZATION_BEARER_TEMPLATE.format( token=self.settings.hermes_api_key ) - if self.settings.hermes_session_id: - headers[AIHttpHeader.HERMES_SESSION_ID] = self.settings.hermes_session_id + request_context = context or {} + session_id = ( + request_context.get(AIContextKey.PROVIDER_SESSION_ID) + or self.settings.hermes_session_id + ) + if session_id: + headers[AIHttpHeader.HERMES_SESSION_ID] = str(session_id) payload = { AIHttpPayloadKey.MODEL: self.settings.hermes_model, - AIHttpPayloadKey.MESSAGES: _chat_messages(prompt, context), + AIHttpPayloadKey.MESSAGES: _chat_messages(prompt, request_context), AIHttpPayloadKey.STREAM: False, } with httpx.Client(timeout=300, trust_env=False) as client: diff --git a/app/modules/ai_agent/adapters/openclaw_hermes.py b/app/modules/ai_agent/adapters/openclaw_hermes.py index 6f98a1c..9274509 100644 --- a/app/modules/ai_agent/adapters/openclaw_hermes.py +++ b/app/modules/ai_agent/adapters/openclaw_hermes.py @@ -32,7 +32,14 @@ class OpenClawHermesAdapter(AIAdapter): def ask(self, prompt: str, context: dict[str, Any] | None = None) -> dict[str, Any]: base_context = context or {} - recall = self._recall_memory(prompt, base_context) + allow_provider_memory = bool( + base_context.get(AIContextKey.ALLOW_PROVIDER_MEMORY, True) + ) + recall = ( + self._recall_memory(prompt, base_context) + if allow_provider_memory + else {AIResponseKey.ANSWER: "", AIResponseKey.RAW: {}} + ) openclaw = self._openclaw_context(base_context) hermes_context = { **base_context, @@ -41,10 +48,14 @@ class OpenClawHermesAdapter(AIAdapter): AIContextKey.OPENCLAW: openclaw, } hermes_result = self.hermes.ask(prompt, hermes_context) - remember = self._remember_interaction( - prompt, - base_context, - hermes_result[AIResponseKey.ANSWER], + remember = ( + self._remember_interaction( + prompt, + base_context, + hermes_result[AIResponseKey.ANSWER], + ) + if allow_provider_memory + else {AIResponseKey.OK: False, AIResponseKey.RAW: {}} ) return { AIResponseKey.ANSWER: hermes_result[AIResponseKey.ANSWER], diff --git a/app/modules/ai_agent/constants.py b/app/modules/ai_agent/constants.py index 1f8fce9..97d9c46 100644 --- a/app/modules/ai_agent/constants.py +++ b/app/modules/ai_agent/constants.py @@ -60,6 +60,14 @@ class AIContextKey(StrEnum): REQUEST_CONTEXT = "request_context" ASSISTANT_ANSWER = "assistant_answer" USER_RULES = "user_rules" + COMPANY_RULES = "company_rules" + PERSONAL_RULES = "personal_rules" + PREFERENCES = "preferences" + INTERESTS = "interests" + CONVERSATION_HISTORY = "conversation_history" + PROVIDER_SESSION_ID = "provider_session_id" + ALLOW_PROVIDER_MEMORY = "allow_provider_memory" + EXECUTION_MODE = "execution_mode" class AIMemoryMode(StrEnum): @@ -67,6 +75,14 @@ class AIMemoryMode(StrEnum): WRITE = "memory_write" +class AIExecutionMode(StrEnum): + INTERNAL = "internal" + PERSONALIZED = "personalized" + PREFERENCE_EXTRACTION = "preference_extraction" + SCHEDULED_PRIVATE = "scheduled_private" + SCHEDULED_GROUP = "scheduled_group" + + class AIHttpPath(StrEnum): CHAT_COMPLETIONS = "/chat/completions" HEALTH = "/health" @@ -95,13 +111,6 @@ class AIHttpPayloadKey(StrEnum): MESSAGE = "message" -class AIToolAuditKey(StrEnum): - TOOL = "tool" - ACTION = "action" - ARGS = "args" - SESSION_KEY = "session_key" - - class AIChatRole(StrEnum): SYSTEM = "system" USER = "user" @@ -133,16 +142,25 @@ CHAT_USER_CONTENT_TEMPLATE = "Context:\n{context}\n\nTask:\n{task}" AUTHORIZATION_BEARER_TEMPLATE = "Bearer {token}" NOOP_PROVIDER_ANSWER = ( "AI provider is not configured yet. This is a deterministic placeholder. " - "Set MODEL_PROVIDER to openclaw_hermes, openclaw, hermes, or direct_llm " - "after credentials are ready." + "Set MODEL_PROVIDER to openclaw_hermes, hermes, or direct_llm after " + "credentials are ready." +) +AI_UNAVAILABLE_ANSWER = "AI 当前不可用，请稍后重试。" +PREFERENCE_EXTRACTION_INSTRUCTIONS = ( + "Extract only durable user communication preferences from the supplied user text. " + "Allowed categories are language, tone, detail, topic, and interest. " + "Return strict JSON only in this shape: " + '{"preferences":[{"category":"language","value":"中文"}]}. ' + "Return an empty preferences list when there is no durable preference. " + "Never return secrets, credentials, health, religion, politics, sexual orientation, " + "performance, compensation, or confidential financial information." ) OPENCLAW_TOOL_COMPLETED_ANSWER = "OpenClaw tool invocation completed." DIRECT_LLM_API_KEY_MISSING = "DIRECT_LLM_API_KEY is not configured" UNEXPECTED_HERMES_RESPONSE = "Unexpected chat completion response" OPENCLAW_CHAT_PROVIDER_REQUIRED = ( - "OpenClaw Gateway is not configured as a chat provider. " - "Provide context.openclaw_tool for /tools/invoke, or use " - "MODEL_PROVIDER=hermes/openclaw_hermes for AI answers." + "OpenClaw Gateway is not exposed as a tool-execution provider. " + "Use MODEL_PROVIDER=hermes, openclaw_hermes, or direct_llm for AI answers." ) OPENCLAW_TOOL_NOT_ALLOWED = "OpenClaw tool is not allowed" OPENCLAW_ACTION_NOT_ALLOWED = "OpenClaw action is not allowed" diff --git a/app/modules/ai_agent/routes.py b/app/modules/ai_agent/routes.py index f8bacb7..b4fdf3a 100644 --- a/app/modules/ai_agent/routes.py +++ b/app/modules/ai_agent/routes.py @@ -2,14 +2,13 @@ from fastapi import APIRouter, Depends from sqlalchemy.orm import Session from app.core.database import get_db -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.audit.constants import AuditSource from app.modules.ai_agent.schemas import ( AIAskRequest, AIAskResponse, DraftPolicyRequest, InvestmentResearchRequest, - OpenClawToolInvokeRequest, ) from app.modules.ai_agent.service import AIService @@ -38,22 +37,6 @@ def provider_health( return AIService(db).provider_health(actor=principal.actor) -@router.post("/openclaw/tools/invoke") -def invoke_openclaw_tool( - payload: OpenClawToolInvokeRequest, - db: Session = Depends(get_db), - principal: ApiPrincipal = Depends(require_api_key), -) -> dict: - require_operations_enabled() - return AIService(db).invoke_openclaw_tool( - tool=payload.tool, - action=payload.action, - args=payload.args, - session_key=payload.session_key, - actor=principal.actor, - ) - - @router.post("/draft-policy", response_model=AIAskResponse) def draft_policy( payload: DraftPolicyRequest, diff --git a/app/modules/ai_agent/schemas.py b/app/modules/ai_agent/schemas.py index f57e03e..5b76777 100644 --- a/app/modules/ai_agent/schemas.py +++ b/app/modules/ai_agent/schemas.py @@ -3,7 +3,7 @@ from typing import Any from pydantic import BaseModel, Field from app.core.constants import ActorValue -from app.modules.ai_agent.constants import AIDefault, AIRiskPreference +from app.modules.ai_agent.constants import AIRiskPreference from app.modules.audit.constants import AuditSource @@ -15,17 +15,11 @@ class AIAskRequest(BaseModel): class AIAskResponse(BaseModel): + ok: bool = True provider: str answer: str raw: dict[str, Any] = Field(default_factory=dict) - - -class OpenClawToolInvokeRequest(BaseModel): - tool: str - action: str = AIDefault.ACTION_JSON - args: dict[str, Any] = Field(default_factory=dict) - session_key: str = AIDefault.SESSION_KEY_MAIN - actor: str = ActorValue.API + error: str | None = None class DraftPolicyRequest(BaseModel): diff --git a/app/modules/ai_agent/service.py b/app/modules/ai_agent/service.py index 5c9b669..9498c0e 100644 --- a/app/modules/ai_agent/service.py +++ b/app/modules/ai_agent/service.py @@ -1,20 +1,23 @@ from typing import Any +from fastapi import HTTPException, status from sqlalchemy.orm import Session from app.core.constants import ActorValue from app.core.config import get_settings from app.modules.ai_agent.adapters import HermesAdapter, OpenClawAdapter, get_adapter from app.modules.ai_agent.constants import ( - AIDefault, + AI_UNAVAILABLE_ANSWER, AI_AUDIT_MAX_DEPTH, AI_AUDIT_MAX_SEQUENCE_ITEMS, AI_AUDIT_MAX_TEXT_LENGTH, AI_AUDIT_REDACTED_VALUE, AI_AUDIT_SENSITIVE_KEYS, AI_AUDIT_TRUNCATED_VALUE, - AIToolAuditKey, + COMPANY_MANAGEMENT_SYSTEM_INSTRUCTIONS, + PREFERENCE_EXTRACTION_INSTRUCTIONS, AIContextKey, + AIExecutionMode, AIProviderName, AIRequestKey, AIResponseKey, @@ -30,6 +33,13 @@ from app.modules.audit.constants import ( ) from app.modules.audit.schemas import AuditLogCreate from app.modules.audit.service import AuditService +from app.modules.personalization.services import ( + ConversationService, + PersonalizationContext, + PersonalizationContextService, + PreferenceService, +) +from app.modules.personalization.services.preferences import contains_preference_signal class AIService: @@ -46,15 +56,30 @@ class AIService: actor: str = ActorValue.API, source: str = AuditSource.API, ) -> dict[str, Any]: + request_context = dict(context or {}) + _validate_external_context(request_context) adapter = get_adapter() - original_context = context or {} - adapter_context = dict(original_context) + if _is_unavailable_adapter(adapter): + response = _unavailable_response(adapter.provider_name) + self._audit_ai_response( + actor=actor, + source=source, + request_payload={ + AIRequestKey.PROMPT: prompt, + AIRequestKey.CONTEXT: request_context, + }, + response=response, + ) + return response + + adapter_context = dict(request_context) memory_service = AIMemoryService(self.db) - memory_scope = _memory_scope(original_context) - memory_subject = _memory_subject(original_context) + memory_scope = _memory_scope(request_context) + memory_subject = _memory_subject(request_context) user_rules = memory_service.active_rules( scope=memory_scope, subject=memory_subject, + owner_id=None, ) if user_rules: adapter_context[AIContextKey.USER_RULES] = user_rules @@ -63,6 +88,7 @@ class AIService: scope=memory_scope, subject=memory_subject, actor=actor, + owner_id=None, ) if local_memory: adapter_context[AIContextKey.LOCAL_MEMORY] = local_memory @@ -73,9 +99,10 @@ class AIService: raw[AIResponseKey.LOCAL_MEMORY] = local_memory memory_record = memory_service.auto_write( prompt=prompt, - context=original_context, + context=request_context, answer=answer, actor=actor, + owner_id=None, ) if memory_record is not None: raw[AIResponseKey.MEMORY_WRITE] = { @@ -83,10 +110,253 @@ class AIService: AIMemoryPayloadKey.STATUS: memory_record.status, } response = { + AIResponseKey.OK: True, AIResponseKey.PROVIDER: adapter.provider_name, AIResponseKey.ANSWER: answer, AIResponseKey.RAW: raw, } + self._audit_ai_response( + actor=actor, + source=source, + request_payload={ + AIRequestKey.PROMPT: prompt, + AIRequestKey.CONTEXT: request_context, + }, + response=response, + ) + return response + + def ask_personalized( + self, + owner_id: int, + chat_type: str, + chat_key: str, + prompt: str, + actor: str = ActorValue.FEISHU, + source: str = AuditSource.FEISHU, + scope: str = AIMemoryScope.USER, + subject: str | None = None, + ) -> dict[str, Any]: + """Answer with one verified owner's isolated personalization context.""" + + _validate_owner_id(owner_id) + _validate_prompt(prompt) + session_id = ConversationService.provider_session_id( + owner_id, + chat_type, + chat_key, + ) + adapter = get_adapter() + if _is_unavailable_adapter(adapter): + response = _unavailable_response(adapter.provider_name) + self._audit_ai_response( + actor=actor, + source=source, + request_payload={ + "owner_id": owner_id, + "chat_type": chat_type, + AIContextKey.EXECUTION_MODE: AIExecutionMode.PERSONALIZED, + }, + response=response, + ) + return response + + memory_subject = subject or f"owner:{owner_id}" + personalization = PersonalizationContextService(self.db).build( + owner_id=owner_id, + request=prompt, + system_constraints=COMPANY_MANAGEMENT_SYSTEM_INSTRUCTIONS, + chat_type=chat_type, + chat_key=chat_key, + scope=scope, + subject=memory_subject, + actor=actor, + include_company_rules=True, + include_personal_context=True, + include_history=True, + ) + adapter_context = _adapter_context( + personalization, + execution_mode=AIExecutionMode.PERSONALIZED, + allow_provider_memory=False, + provider_session_id=session_id, + ) + result = adapter.ask(prompt, adapter_context) + answer = _required_answer(result) + raw = dict(result.get(AIResponseKey.RAW, {})) + + ConversationService(self.db).record_turn( + owner_id, + chat_type, + chat_key, + user_content=prompt, + assistant_content=answer, + provider_name=str(adapter.provider_name), + ) + memory_record = AIMemoryService(self.db).auto_write( + prompt=prompt, + context={ + AIMemoryPayloadKey.SCOPE: scope, + AIMemoryPayloadKey.SUBJECT: memory_subject, + }, + answer=answer, + actor=actor, + owner_id=owner_id, + ) + if memory_record is not None: + raw[AIResponseKey.MEMORY_WRITE] = { + AIMemoryPayloadKey.CODE: memory_record.code, + AIMemoryPayloadKey.STATUS: memory_record.status, + } + saved_preferences = self._extract_preferences( + adapter=adapter, + owner_id=owner_id, + user_text=prompt, + provider_session_id=session_id, + ) + if saved_preferences: + raw["preferences_saved"] = [ + { + "code": item["code"], + "category": item["category"], + } + for item in saved_preferences + ] + + response = { + AIResponseKey.OK: True, + AIResponseKey.PROVIDER: adapter.provider_name, + AIResponseKey.ANSWER: answer, + AIResponseKey.RAW: raw, + } + self._audit_ai_response( + actor=actor, + source=source, + request_payload={ + "owner_id": owner_id, + "chat_type": chat_type, + AIContextKey.EXECUTION_MODE: AIExecutionMode.PERSONALIZED, + }, + response={ + AIResponseKey.OK: True, + AIResponseKey.PROVIDER: adapter.provider_name, + }, + ) + return response + + def generate_scheduled( + self, + prompt: str, + owner_id: int | None, + group: bool, + actor: str = "subscription-system", + ) -> dict[str, Any]: + """Generate side-effect-free scheduled content within its target boundary.""" + + _validate_prompt(prompt) + if not group: + if owner_id is None: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Private scheduled generation requires an owner", + ) + _validate_owner_id(owner_id) + + adapter = get_adapter() + execution_mode = ( + AIExecutionMode.SCHEDULED_GROUP + if group + else AIExecutionMode.SCHEDULED_PRIVATE + ) + if _is_unavailable_adapter(adapter): + response = _unavailable_response(adapter.provider_name) + self._audit_ai_response( + actor=actor, + source=AuditSource.FEISHU, + request_payload={AIContextKey.EXECUTION_MODE: execution_mode}, + response=response, + ) + return response + + context_service = PersonalizationContextService(self.db) + if group: + personalization = context_service.build_group_scheduled( + request=prompt, + system_constraints=COMPANY_MANAGEMENT_SYSTEM_INSTRUCTIONS, + ) + else: + personalization = context_service.build_private_scheduled( + owner_id=owner_id, + request=prompt, + system_constraints=COMPANY_MANAGEMENT_SYSTEM_INSTRUCTIONS, + actor=actor, + scope=AIMemoryScope.USER, + subject=f"owner:{owner_id}", + ) + adapter_context = _adapter_context( + personalization, + execution_mode=execution_mode, + allow_provider_memory=False, + provider_session_id=None, + ) + result = adapter.ask(prompt, adapter_context) + response = { + AIResponseKey.OK: True, + AIResponseKey.PROVIDER: adapter.provider_name, + AIResponseKey.ANSWER: _required_answer(result), + AIResponseKey.RAW: dict(result.get(AIResponseKey.RAW, {})), + } + self._audit_ai_response( + actor=actor, + source=AuditSource.FEISHU, + request_payload={AIContextKey.EXECUTION_MODE: execution_mode}, + response={ + AIResponseKey.OK: True, + AIResponseKey.PROVIDER: adapter.provider_name, + }, + ) + return response + + def _extract_preferences( + self, + *, + adapter: Any, + owner_id: int, + user_text: str, + provider_session_id: str, + ) -> list[dict[str, Any]]: + if not contains_preference_signal(user_text): + return [] + extraction_context = { + "preference_source_text": user_text, + AIContextKey.PROVIDER_SESSION_ID: f"{provider_session_id}-preferences", + AIContextKey.ALLOW_PROVIDER_MEMORY: False, + AIContextKey.EXECUTION_MODE: AIExecutionMode.PREFERENCE_EXTRACTION, + } + try: + extraction = adapter.ask( + PREFERENCE_EXTRACTION_INSTRUCTIONS, + extraction_context, + ) + structured_payload = extraction.get(AIResponseKey.ANSWER, "") + return PreferenceService(self.db).save_auto_extraction( + owner_id, + provider_name=str(adapter.provider_name), + user_text=user_text, + structured_payload=structured_payload, + ) + except Exception: + # Preference inference must never block or change the primary answer. + return [] + + def _audit_ai_response( + self, + *, + actor: str, + source: str, + request_payload: dict[str, Any], + response: dict[str, Any], + ) -> None: self.audit.log( AuditLogCreate( actor=actor, @@ -94,14 +364,10 @@ class AIService: action=AuditAction.AI_ASK, target_type=AuditTargetType.AI, risk_level=AuditRiskLevel.MEDIUM, - request_payload=_audit_safe_payload({ - AIRequestKey.PROMPT: prompt, - AIRequestKey.CONTEXT: context or {}, - }), + request_payload=_audit_safe_payload(request_payload), response_payload=_audit_safe_payload(response), ) ) - return response def run_skill( self, @@ -139,35 +405,6 @@ class AIService: ) return response - def invoke_openclaw_tool( - self, - tool: str, - action: str = AIDefault.ACTION_JSON, - args: dict[str, Any] | None = None, - session_key: str = AIDefault.SESSION_KEY_MAIN, - actor: str = ActorValue.API, - ) -> dict[str, Any]: - result = OpenClawAdapter(get_settings()).invoke_tool(tool, action, args or {}, session_key) - response = {AIResponseKey.PROVIDER: AIProviderName.OPENCLAW, AIResponseKey.RESULT: result} - self.audit.log( - AuditLogCreate( - actor=actor, - source=AuditSource.OPENCLAW, - action=AuditAction.OPENCLAW_TOOLS_INVOKE, - target_type=AuditTargetType.OPENCLAW_TOOL, - target_id=tool, - risk_level=AuditRiskLevel.HIGH, - request_payload=_audit_safe_payload({ - AIToolAuditKey.TOOL: tool, - AIToolAuditKey.ACTION: action, - AIToolAuditKey.ARGS: args or {}, - AIToolAuditKey.SESSION_KEY: session_key, - }), - response_payload=_audit_safe_payload(result), - ) - ) - return response - @staticmethod def _health_result(check: Any) -> dict[str, Any]: try: @@ -247,3 +484,103 @@ def _memory_scope(context: dict[str, Any]) -> str: def _memory_subject(context: dict[str, Any]) -> str | None: value = context.get(AIContextKey.MEMORY_SUBJECT) or context.get(AIMemoryPayloadKey.SUBJECT) return str(value) if value else None + + +def _validate_external_context(context: dict[str, Any]) -> None: + controlled_keys = { + AIContextKey.OPENCLAW_TOOL, + AIContextKey.OPENCLAW_ACTION, + AIContextKey.OPENCLAW_ARGS, + AIContextKey.OPENCLAW_SESSION_KEY, + AIContextKey.AGENT_PIPELINE, + AIContextKey.HERMES_MEMORY, + AIContextKey.LOCAL_MEMORY, + AIContextKey.OPENCLAW, + AIContextKey.MODE, + AIContextKey.USER_PROMPT, + AIContextKey.REQUEST_CONTEXT, + AIContextKey.ASSISTANT_ANSWER, + AIContextKey.USER_RULES, + AIContextKey.COMPANY_RULES, + AIContextKey.PERSONAL_RULES, + AIContextKey.PREFERENCES, + AIContextKey.INTERESTS, + AIContextKey.CONVERSATION_HISTORY, + AIContextKey.PROVIDER_SESSION_ID, + AIContextKey.ALLOW_PROVIDER_MEMORY, + AIContextKey.EXECUTION_MODE, + "system_constraints", + "current_request", + "personal_memory", + "owner_id", + "tenant_key", + "open_id", + } + supplied = {str(key) for key in context} + if supplied.intersection(str(key) for key in controlled_keys): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Controlled AI context fields are not allowed", + ) + + +def _adapter_context( + personalization: PersonalizationContext, + *, + execution_mode: AIExecutionMode, + allow_provider_memory: bool, + provider_session_id: str | None, +) -> dict[str, Any]: + context: dict[str, Any] = { + AIContextKey.COMPANY_RULES: personalization.company_rules, + AIContextKey.PERSONAL_RULES: personalization.personal_rules, + AIContextKey.PREFERENCES: personalization.preferences, + AIContextKey.INTERESTS: personalization.interests, + AIContextKey.LOCAL_MEMORY: personalization.personal_memory, + AIContextKey.CONVERSATION_HISTORY: personalization.conversation_history, + } + if provider_session_id: + context[AIContextKey.PROVIDER_SESSION_ID] = provider_session_id + context[AIContextKey.ALLOW_PROVIDER_MEMORY] = allow_provider_memory + context[AIContextKey.EXECUTION_MODE] = execution_mode + return context + + +def _is_unavailable_adapter(adapter: Any) -> bool: + return str(getattr(adapter, "provider_name", "")).strip().lower() == AIProviderName.NOOP + + +def _unavailable_response(provider_name: Any) -> dict[str, Any]: + return { + AIResponseKey.OK: False, + AIResponseKey.PROVIDER: str(provider_name or AIProviderName.NOOP), + AIResponseKey.ANSWER: AI_UNAVAILABLE_ANSWER, + AIResponseKey.RAW: {"reason": "provider_unavailable"}, + AIResponseKey.ERROR: "AI provider unavailable", + } + + +def _required_answer(result: dict[str, Any]) -> str: + answer = str(result.get(AIResponseKey.ANSWER) or "").strip() + if not answer: + raise HTTPException( + status_code=status.HTTP_502_BAD_GATEWAY, + detail="AI provider returned an empty answer", + ) + return answer + + +def _validate_owner_id(owner_id: int) -> None: + if owner_id <= 0: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Valid AI owner is required", + ) + + +def _validate_prompt(prompt: str) -> None: + if not str(prompt).strip(): + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="AI prompt is required", + ) diff --git a/app/modules/ai_memory/constants.py b/app/modules/ai_memory/constants.py index 9b3a236..a11cfc7 100644 --- a/app/modules/ai_memory/constants.py +++ b/app/modules/ai_memory/constants.py @@ -19,6 +19,13 @@ class AIMemorySource(StrEnum): HERMES = "hermes" API = "api" USER_RULE = "user_rule" + LEGACY_COMPANY = "legacy_company" + + +class AIMemoryKind(StrEnum): + COMPANY_RULE = "company_rule" + PERSONAL_RULE = "personal_rule" + MEMORY = "memory" class AIMemoryResponseKey(StrEnum): diff --git a/app/modules/ai_memory/models.py b/app/modules/ai_memory/models.py index 9117068..52c4543 100644 --- a/app/modules/ai_memory/models.py +++ b/app/modules/ai_memory/models.py @@ -1,19 +1,46 @@ from datetime import datetime -from sqlalchemy import JSON, DateTime, Integer, String, Text -from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy import JSON, DateTime, ForeignKey, Integer, String, Text, UniqueConstraint +from sqlalchemy.orm import Mapped, mapped_column, relationship from app.core.constants import ActorValue from app.core.database import Base from app.core.utils.time import utc_now -from app.modules.ai_memory.constants import AIMemoryScope, AIMemorySource, AIMemoryStatus +from app.modules.feishu_users.models import FeishuUser +from app.modules.ai_memory.constants import ( + AIMemoryKind, + AIMemoryScope, + AIMemorySource, + AIMemoryStatus, +) class AIMemoryEntry(Base): __tablename__ = "ai_memory_entries" + __table_args__ = ( + UniqueConstraint( + "owner_id", + "fingerprint", + name="uq_ai_memory_owner_fingerprint", + ), + ) id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) code: Mapped[str] = mapped_column(String(64), unique=True, index=True) + fingerprint: Mapped[str | None] = mapped_column( + String(64), nullable=True, index=True + ) + owner_id: Mapped[int | None] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + nullable=True, + index=True, + ) + owner: Mapped[FeishuUser | None] = relationship() + kind: Mapped[str] = mapped_column( + String(32), + default=AIMemoryKind.MEMORY, + index=True, + ) scope: Mapped[str] = mapped_column(String(64), default=AIMemoryScope.GLOBAL, index=True) subject: Mapped[str] = mapped_column(String(128), index=True) content: Mapped[str] = mapped_column(Text) diff --git a/app/modules/ai_memory/routes.py b/app/modules/ai_memory/routes.py index 1cbd0d3..b8999d7 100644 --- a/app/modules/ai_memory/routes.py +++ b/app/modules/ai_memory/routes.py @@ -2,7 +2,7 @@ from fastapi import APIRouter, Depends, Query from sqlalchemy.orm import Session from app.core.database import get_db -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.ai_memory.constants import AIMemoryResponseKey, AIMemoryStatus from app.modules.ai_memory.schemas import ( AIMemoryRecallRequest, @@ -22,14 +22,15 @@ def list_memory( limit: int = Query(default=100, ge=1, le=500), db: Session = Depends(get_db), ) -> dict: - return { - AIMemoryResponseKey.ITEMS: AIMemoryService(db).list_entries( - scope=scope, - subject=subject, - status_filter=status, - limit=limit, - ) - } + items = AIMemoryService(db).list_entries( + scope=scope, + subject=subject, + status_filter=status, + limit=limit, + owner_id=None, + ) + db.commit() + return {AIMemoryResponseKey.ITEMS: items} @router.post("/memory/recall") @@ -44,6 +45,7 @@ def recall_memory( subject=payload.subject, limit=payload.limit, actor=principal.actor, + owner_id=None, ) return {AIMemoryResponseKey.ITEMS: items} @@ -62,6 +64,7 @@ def list_rules( subject=subject, status_filter=status, limit=limit, + owner_id=None, ) } @@ -72,7 +75,6 @@ def create_rule( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return { AIMemoryResponseKey.DATA: AIMemoryService(db).create_rule( content=payload.content, @@ -81,6 +83,7 @@ def create_rule( priority=payload.priority, tags=payload.tags, actor=principal.actor, + owner_id=None, ) } @@ -92,7 +95,6 @@ def update_rule( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return { AIMemoryResponseKey.DATA: AIMemoryService(db).update_rule( code=code, @@ -101,5 +103,20 @@ def update_rule( tags=payload.tags, enabled=payload.enabled, actor=principal.actor, + owner_id=None, ) } + + +@router.delete("/rules/{code}") +def delete_rule( + code: str, + db: Session = Depends(get_db), + principal: ApiPrincipal = Depends(require_api_key), +) -> dict: + AIMemoryService(db).delete_rule( + code=code, + actor=principal.actor, + owner_id=None, + ) + return {AIMemoryResponseKey.DATA: {"code": code, "deleted": True}} diff --git a/app/modules/ai_memory/schemas.py b/app/modules/ai_memory/schemas.py index fde5084..516d87d 100644 --- a/app/modules/ai_memory/schemas.py +++ b/app/modules/ai_memory/schemas.py @@ -14,6 +14,8 @@ class AIMemoryRecallRequest(BaseModel): class AIMemoryRead(BaseModel): code: str + owner_id: int | None + kind: str scope: str subject: str content: str diff --git a/app/modules/ai_memory/service.py b/app/modules/ai_memory/service.py index 3783925..4d5d23d 100644 --- a/app/modules/ai_memory/service.py +++ b/app/modules/ai_memory/service.py @@ -1,9 +1,11 @@ from datetime import datetime, timedelta +from hashlib import sha256 from typing import Any from uuid import uuid4 from fastapi import HTTPException, status -from sqlalchemy import func, or_, select +from sqlalchemy import delete, func, or_, select, update +from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import Session from app.core.config import get_settings @@ -15,6 +17,7 @@ from app.modules.ai_memory.constants import ( AI_MEMORY_MAX_CONTENT_LENGTH, AI_MEMORY_MAX_SUMMARY_LENGTH, AI_MEMORY_MIN_AUTO_WRITE_LENGTH, + AIMemoryKind, AIMemoryPayloadKey, AIMemoryScope, AIMemorySource, @@ -50,10 +53,15 @@ class AIMemoryService: subject: str | None = None, status_filter: str = AIMemoryStatus.ACTIVE, limit: int = 100, + owner_id: int | None = None, ) -> list[dict[str, Any]]: + self._archive_expired() stmt = ( select(AIMemoryEntry) - .where(AIMemoryEntry.status == status_filter) + .where( + AIMemoryEntry.status == status_filter, + _owner_filter(owner_id), + ) .order_by(AIMemoryEntry.importance.desc(), AIMemoryEntry.id.desc()) .limit(bounded_limit(limit)) ) @@ -70,16 +78,20 @@ class AIMemoryService: subject: str | None = None, limit: int | None = None, actor: str = ActorValue.API, + owner_id: int | None = None, ) -> list[dict[str, Any]]: settings = get_settings() if not settings.ai_memory_enabled: return [] limit_value = bounded_limit(limit or settings.ai_memory_recall_limit) + self._archive_expired() now = utc_now() stmt = ( select(AIMemoryEntry) .where( AIMemoryEntry.status == AIMemoryStatus.ACTIVE, + AIMemoryEntry.kind == AIMemoryKind.MEMORY, + _owner_filter(owner_id), or_(AIMemoryEntry.expires_at.is_(None), AIMemoryEntry.expires_at > now), AIMemoryEntry.scope.in_({AIMemoryScope.GLOBAL, scope}), ) @@ -95,8 +107,6 @@ class AIMemoryService: ) candidates = list(self.db.execute(stmt).scalars()) items = [item for item in candidates if _matches_query(item, query)] - if not items: - items = candidates[:limit_value] items = items[:limit_value] for item in items: item.last_used_at = now @@ -126,11 +136,13 @@ class AIMemoryService: context: dict[str, Any], answer: str, actor: str = ActorValue.API, + owner_id: int | None = None, ) -> AIMemoryEntry | None: settings = get_settings() if not settings.ai_memory_enabled or not settings.ai_memory_auto_write_enabled: return None content = _build_memory_content(prompt, context, answer) + self._archive_expired() if len(content) < AI_MEMORY_MIN_AUTO_WRITE_LENGTH: return None scope = str(context.get(AIMemoryPayloadKey.SCOPE) or AIMemoryText.DEFAULT_SCOPE) @@ -143,39 +155,60 @@ class AIMemoryService: }, settings.ai_memory_forbidden_keys, ): + safe_content = str(AIMemoryText.REJECTED_SECRET) record = self._create_entry( scope=scope, subject=subject, - content=str(AIMemoryText.REJECTED_SECRET), - summary=str(AIMemoryText.REJECTED_SECRET), + content=safe_content, + summary=safe_content, tags=[str(AIMemoryText.AUTO_TAG)], source=AIMemorySource.AUTO, importance=0, status_value=AIMemoryStatus.REJECTED, actor=actor, + fingerprint=_memory_fingerprint( + owner_id, + scope, + subject, + content, + AIMemoryStatus.REJECTED, + ), + owner_id=owner_id, + kind=AIMemoryKind.MEMORY, expires_at=utc_now() + timedelta(days=settings.ai_memory_auto_write_ttl_days), ) return record if _contains_blocked_content(content, settings.ai_memory_blocked_content_terms): + safe_content = str(AIMemoryText.REJECTED_SENSITIVE_FACT) return self._create_entry( scope=scope, subject=subject, - content=str(AIMemoryText.REJECTED_SENSITIVE_FACT), - summary=str(AIMemoryText.REJECTED_SENSITIVE_FACT), + content=safe_content, + summary=safe_content, tags=[str(AIMemoryText.AUTO_TAG)], source=AIMemorySource.AUTO, importance=0, status_value=AIMemoryStatus.REJECTED, actor=actor, + fingerprint=_memory_fingerprint( + owner_id, + scope, + subject, + content, + AIMemoryStatus.REJECTED, + ), + owner_id=owner_id, + kind=AIMemoryKind.MEMORY, expires_at=utc_now() + timedelta(days=settings.ai_memory_auto_write_ttl_days), ) summary = _truncate(answer, AI_MEMORY_MAX_SUMMARY_LENGTH) + stored_content = _truncate(content, AI_MEMORY_MAX_CONTENT_LENGTH) record = self._create_entry( scope=scope, subject=subject, - content=_truncate(content, AI_MEMORY_MAX_CONTENT_LENGTH), + content=stored_content, summary=summary, tags=[str(AIMemoryText.AUTO_TAG)], source=AIMemorySource.AUTO, @@ -183,6 +216,15 @@ class AIMemoryService: status_value=AIMemoryStatus.ACTIVE, actor=actor, expires_at=utc_now() + timedelta(days=settings.ai_memory_auto_write_ttl_days), + fingerprint=_memory_fingerprint( + owner_id, + scope, + subject, + stored_content, + AIMemoryStatus.ACTIVE, + ), + owner_id=owner_id, + kind=AIMemoryKind.MEMORY, ) return record @@ -192,10 +234,19 @@ class AIMemoryService: subject: str | None = None, status_filter: str | None = None, limit: int = 100, + owner_id: int | None = None, ) -> list[dict[str, Any]]: + kind = ( + AIMemoryKind.COMPANY_RULE + if owner_id is None + else AIMemoryKind.PERSONAL_RULE + ) stmt = ( select(AIMemoryEntry) - .where(AIMemoryEntry.source == AIMemorySource.USER_RULE) + .where( + AIMemoryEntry.kind == kind, + _owner_filter(owner_id), + ) .order_by(AIMemoryEntry.importance.desc(), AIMemoryEntry.id.asc()) .limit(bounded_limit(limit)) ) @@ -212,11 +263,19 @@ class AIMemoryService: scope: str = AIMemoryScope.GLOBAL, subject: str | None = None, limit: int = 50, + owner_id: int | None = None, ) -> list[dict[str, Any]]: + self._archive_expired() + kind = ( + AIMemoryKind.COMPANY_RULE + if owner_id is None + else AIMemoryKind.PERSONAL_RULE + ) stmt = ( select(AIMemoryEntry) .where( - AIMemoryEntry.source == AIMemorySource.USER_RULE, + AIMemoryEntry.kind == kind, + _owner_filter(owner_id), AIMemoryEntry.status == AIMemoryStatus.ACTIVE, AIMemoryEntry.scope.in_({AIMemoryScope.GLOBAL, scope}), ) @@ -249,6 +308,7 @@ class AIMemoryService: priority: int, tags: list[str] | None, actor: str, + owner_id: int | None = None, ) -> dict[str, Any]: self._validate_rule(content, priority) record = self._create_entry( @@ -262,6 +322,12 @@ class AIMemoryService: status_value=AIMemoryStatus.ACTIVE, actor=actor, audit_action=AuditAction.AI_RULE_CREATE, + owner_id=owner_id, + kind=( + AIMemoryKind.COMPANY_RULE + if owner_id is None + else AIMemoryKind.PERSONAL_RULE + ), ) return serialize_model(record) @@ -273,11 +339,18 @@ class AIMemoryService: tags: list[str] | None, enabled: bool | None, actor: str, + owner_id: int | None = None, ) -> dict[str, Any]: + kind = ( + AIMemoryKind.COMPANY_RULE + if owner_id is None + else AIMemoryKind.PERSONAL_RULE + ) record = self.db.execute( select(AIMemoryEntry).where( AIMemoryEntry.code == code, - AIMemoryEntry.source == AIMemorySource.USER_RULE, + AIMemoryEntry.kind == kind, + _owner_filter(owner_id), ) ).scalar_one_or_none() if record is None: @@ -312,6 +385,50 @@ class AIMemoryService: self.db.refresh(record) return serialize_model(record) + def delete_rule( + self, + code: str, + actor: str, + owner_id: int | None = None, + ) -> None: + """Delete a rule only within the requested company or personal owner scope.""" + + kind = ( + AIMemoryKind.COMPANY_RULE + if owner_id is None + else AIMemoryKind.PERSONAL_RULE + ) + record = self.db.execute( + select(AIMemoryEntry).where( + AIMemoryEntry.code == code, + AIMemoryEntry.kind == kind, + _owner_filter(owner_id), + ) + ).scalar_one_or_none() + if record is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="AI rule not found") + self.db.delete(record) + self.audit.record( + AuditLogCreate( + actor=actor, + source=AuditSource.AI_MEMORY, + action=AuditAction.AI_RULE_UPDATE, + target_type=AuditTargetType.AI_MEMORY, + target_id=code, + risk_level=AuditRiskLevel.MEDIUM, + response_payload={"deleted": True}, + ) + ) + self.db.commit() + + def delete_owner_entries(self, owner_id: int) -> int: + """Stage deletion of all personal rules and memories for an owner.""" + + result = self.db.execute( + delete(AIMemoryEntry).where(AIMemoryEntry.owner_id == owner_id) + ) + return max(0, int(result.rowcount or 0)) + def _validate_rule(self, content: str, priority: int) -> None: if not AI_USER_RULE_MIN_PRIORITY <= priority <= AI_USER_RULE_MAX_PRIORITY: raise HTTPException( @@ -325,11 +442,46 @@ class AIMemoryService: ) def count_by_status(self) -> dict[str, int]: + self._archive_expired() rows = self.db.execute( select(AIMemoryEntry.status, func.count()).group_by(AIMemoryEntry.status) ).all() return {str(status_value): int(count) for status_value, count in rows} + def _archive_expired(self) -> int: + now = utc_now() + result = self.db.execute( + update(AIMemoryEntry) + .where( + AIMemoryEntry.status.in_( + { + AIMemoryStatus.ACTIVE, + AIMemoryStatus.REJECTED, + } + ), + AIMemoryEntry.expires_at.is_not(None), + AIMemoryEntry.expires_at <= now, + ) + .values( + status=AIMemoryStatus.ARCHIVED, + updated_at=now, + ) + ) + archived = max(0, int(result.rowcount or 0)) + return archived + + def _find_by_fingerprint( + self, + owner_id: int | None, + fingerprint: str, + ) -> AIMemoryEntry | None: + return self.db.execute( + select(AIMemoryEntry).where( + AIMemoryEntry.fingerprint == fingerprint, + _owner_filter(owner_id), + ) + ).scalar_one_or_none() + def _create_entry( self, scope: str, @@ -343,12 +495,23 @@ class AIMemoryService: actor: str, audit_action: str = AuditAction.AI_MEMORY_WRITE, expires_at: datetime | None = None, + fingerprint: str | None = None, + owner_id: int | None = None, + kind: str = AIMemoryKind.MEMORY, ) -> AIMemoryEntry: + if fingerprint: + existing = self._find_by_fingerprint(owner_id, fingerprint) + if existing is not None: + return self._reuse_entry(existing, status_value, expires_at) + record = AIMemoryEntry( code=( f"{AI_MEMORY_CODE_PREFIX}-{utc_now():%Y%m%d%H%M%S%f}-" f"{uuid4().hex[:8]}" ), + fingerprint=fingerprint, + owner_id=owner_id, + kind=kind, scope=scope, subject=subject, content=content, @@ -360,8 +523,20 @@ class AIMemoryService: actor=actor, expires_at=expires_at, ) - self.db.add(record) - self.db.flush() + if fingerprint: + try: + with self.db.begin_nested(): + self.db.add(record) + self.db.flush() + except IntegrityError: + existing = self._find_by_fingerprint(owner_id, fingerprint) + if existing is None: + raise + return self._reuse_entry(existing, status_value, expires_at) + else: + self.db.add(record) + self.db.flush() + self.audit.record( AuditLogCreate( actor=actor, @@ -397,6 +572,21 @@ class AIMemoryService: self.db.refresh(record) return record + def _reuse_entry( + self, + record: AIMemoryEntry, + status_value: str, + expires_at: datetime | None, + ) -> AIMemoryEntry: + if record.status != AIMemoryStatus.ARCHIVED: + return record + record.status = status_value + record.expires_at = expires_at + record.updated_at = utc_now() + self.db.commit() + self.db.refresh(record) + return record + def _build_memory_content(prompt: str, context: dict[str, Any], answer: str) -> str: context_text = ", ".join( @@ -427,6 +617,24 @@ def _contains_blocked_content(value: str, blocked_terms: list[str]) -> bool: return any(term.lower() in lowered for term in blocked_terms if term.strip()) +def _memory_fingerprint( + owner_id: int | None, + scope: str, + subject: str, + content: str, + status_value: str, +) -> str: + owner_key = "company" if owner_id is None else f"owner:{owner_id}" + value = "\0".join((owner_key, scope, subject, status_value, content)) + return sha256(value.encode("utf-8")).hexdigest() + + +def _owner_filter(owner_id: int | None): + if owner_id is None: + return AIMemoryEntry.owner_id.is_(None) + return AIMemoryEntry.owner_id == owner_id + + def _matches_query(entry: AIMemoryEntry, query: str) -> bool: query_text = query.lower().strip() if not query_text: diff --git a/app/modules/audit/constants.py b/app/modules/audit/constants.py index 9bf845f..d63efdb 100644 --- a/app/modules/audit/constants.py +++ b/app/modules/audit/constants.py @@ -4,7 +4,6 @@ from enum import StrEnum class AuditAction(StrEnum): AI_ASK = "ai.ask" AI_PROVIDER_HEALTH = "ai.provider_health" - OPENCLAW_TOOLS_INVOKE = "openclaw.tools.invoke" GENERATE_EVENTS = "generate_events" FEISHU_WEBHOOK_EVENT = "webhook_event" FEISHU_LONG_CONNECTION_EVENT = "long_connection_event" @@ -34,7 +33,6 @@ class AuditRiskLevel(StrEnum): class AuditSource(StrEnum): API = "api" - OPENCLAW = "openclaw" RISK = "risk" FEISHU = "feishu" LEGACY_MYSQL = "legacy_mysql" @@ -47,7 +45,6 @@ class AuditSource(StrEnum): class AuditTargetType(StrEnum): AI = "ai" - OPENCLAW_TOOL = "openclaw_tool" RISK_EVENTS = "risk-events" WORK_REPORTS = "work-reports" ENTERPRISE_ANALYTICS = "enterprise-analytics" @@ -62,21 +59,3 @@ class AuditStatus(StrEnum): AUDIT_REDACTED_VALUE = "[REDACTED]" -AUDIT_SENSITIVE_KEYS = frozenset( - { - "authorization", - "api_key", - "apikey", - "access_token", - "tenant_access_token", - "token", - "secret", - "password", - "openclaw_gateway_token", - "hermes_api_key", - "direct_llm_api_key", - "market_data_token", - "feishu_app_secret", - "feishu_verification_token", - } -) diff --git a/app/modules/audit/service.py b/app/modules/audit/service.py index 3ef76c2..3a2625a 100644 --- a/app/modules/audit/service.py +++ b/app/modules/audit/service.py @@ -4,9 +4,10 @@ from typing import Any from sqlalchemy import select from sqlalchemy.orm import Session +from app.core.http.masking import is_sensitive_key from app.core.http.pagination import bounded_limit from app.core.http.request_context import get_request_id -from app.modules.audit.constants import AUDIT_REDACTED_VALUE, AUDIT_SENSITIVE_KEYS +from app.modules.audit.constants import AUDIT_REDACTED_VALUE from app.modules.audit.models import AuditLog from app.modules.audit.schemas import AuditLogCreate @@ -16,7 +17,7 @@ def _redact(value: Any) -> Any: safe: dict[str, Any] = {} for key, item in value.items(): key_text = str(key) - if key_text.lower() in AUDIT_SENSITIVE_KEYS: + if is_sensitive_key(key_text): safe[key_text] = AUDIT_REDACTED_VALUE else: safe[key_text] = _redact(item) @@ -34,6 +35,12 @@ def _dump(value: Any | None) -> str | None: if value is None: return None if isinstance(value, str): + try: + parsed = json.loads(value) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(parsed, (dict, list)): + return json.dumps(_redact(parsed), ensure_ascii=False, default=str) return value return json.dumps(_redact(value), ensure_ascii=False, default=str) diff --git a/app/modules/business/models/market.py b/app/modules/business/models/market.py index 5346a62..ea0a7d0 100644 --- a/app/modules/business/models/market.py +++ b/app/modules/business/models/market.py @@ -1,10 +1,20 @@ from datetime import date, datetime from decimal import Decimal -from sqlalchemy import Boolean, Date, DateTime, Integer, Numeric, String, UniqueConstraint -from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy import ( + Boolean, + Date, + DateTime, + ForeignKey, + Integer, + Numeric, + String, + UniqueConstraint, +) +from sqlalchemy.orm import Mapped, mapped_column, relationship from app.core.database import Base +from app.modules.feishu_users.models import FeishuUser from app.modules.business.models.common import TimestampMixin @@ -80,8 +90,16 @@ class MarketAnnouncement(Base, TimestampMixin): class MarketWatchlist(Base, TimestampMixin): __tablename__ = "market_watchlists" - __table_args__ = (UniqueConstraint("actor", "symbol", name="uq_market_watchlist_actor_symbol"),) + __table_args__ = ( + UniqueConstraint("owner_id", "symbol", name="uq_market_watchlist_owner_symbol"), + ) id: Mapped[int] = mapped_column(Integer, primary_key=True) + owner_id: Mapped[int | None] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + nullable=True, + index=True, + ) + owner: Mapped[FeishuUser | None] = relationship() actor: Mapped[str] = mapped_column(String(128), index=True) symbol: Mapped[str] = mapped_column(String(32), index=True) enabled: Mapped[bool] = mapped_column(Boolean, default=True, index=True) diff --git a/app/modules/dashboard/routes.py b/app/modules/dashboard/routes.py index 4a57357..20826e3 100644 --- a/app/modules/dashboard/routes.py +++ b/app/modules/dashboard/routes.py @@ -15,4 +15,6 @@ def dashboard_summary( principal: ApiPrincipal = Depends(require_api_key), ) -> dict: _ = principal - return mask_configured(DashboardService(db).summary()) + result = mask_configured(DashboardService(db).summary()) + db.commit() + return result diff --git a/app/modules/dashboard/service.py b/app/modules/dashboard/service.py index 76cfd77..991da11 100644 --- a/app/modules/dashboard/service.py +++ b/app/modules/dashboard/service.py @@ -4,8 +4,7 @@ from sqlalchemy import func, select from sqlalchemy.orm import Session from app.modules.ai_memory.constants import AIMemoryStatus -from app.modules.ai_memory.models import AIMemoryEntry -from app.modules.audit.models import AuditLog +from app.modules.ai_memory.service import AIMemoryService from app.modules.business.constants import DONE_STATUSES, PROJECT_CLOSED_STATUSES, StatusValue from app.modules.business.models import ( LegacySyncRun, @@ -53,7 +52,8 @@ class DashboardService: WorkflowInstance, WorkflowInstance.status == WorkflowStatus.FAILED, ) - active_ai_memory = self._count(AIMemoryEntry, AIMemoryEntry.status == AIMemoryStatus.ACTIVE) + memory_counts = AIMemoryService(self.db).count_by_status() + active_ai_memory = memory_counts.get(AIMemoryStatus.ACTIVE, 0) heartbeat_summary = ObservabilityService(self.db).heartbeat_summary() latest_reports = self.db.execute( select(WorkReport).order_by(WorkReport.id.desc()).limit(5) @@ -64,9 +64,6 @@ class DashboardService: latest_sync_runs = self.db.execute( select(LegacySyncRun).order_by(LegacySyncRun.id.desc()).limit(10) ).scalars() - latest_audit_logs = self.db.execute( - select(AuditLog).order_by(AuditLog.id.desc()).limit(10) - ).scalars() risk_summary = self.risks.summary() return { "metrics": { @@ -94,7 +91,6 @@ class DashboardService: "latest_reports": [serialize_model(item) for item in latest_reports], "latest_push_runs": [serialize_model(item) for item in latest_push_runs], "latest_sync_runs": [serialize_model(item) for item in latest_sync_runs], - "latest_audit_logs": [serialize_model(item) for item in latest_audit_logs], } def _count(self, model: type, *conditions: Any) -> int: diff --git a/app/modules/events/services/query.py b/app/modules/events/services/query.py index 984c694..e01d268 100644 --- a/app/modules/events/services/query.py +++ b/app/modules/events/services/query.py @@ -1,7 +1,9 @@ from typing import Any +from uuid import uuid4 from fastapi import HTTPException, status from sqlalchemy import func, select +from sqlalchemy.exc import IntegrityError from app.core.config import get_settings from app.core.constants import ActorValue @@ -35,7 +37,10 @@ class EventQueryMixin: settings = get_settings() now = utc_now() record = DomainEvent( - event_id=f"{EVENT_CODE_PREFIX}-{now:%Y%m%d%H%M%S%f}", + event_id=( + f"{EVENT_CODE_PREFIX}-{now:%Y%m%d%H%M%S%f}-" + f"{uuid4().hex[:8]}" + ), event_type=event_type, source=source, aggregate_type=aggregate_type, @@ -46,8 +51,20 @@ class EventQueryMixin: next_attempt_at=now, max_attempts=settings.event_dispatch_max_attempts, ) - self.db.add(record) - self.db.flush() + if not idempotency_key: + self.db.add(record) + self.db.flush() + return record + + try: + with self.db.begin_nested(): + self.db.add(record) + self.db.flush() + except IntegrityError: + existing = self._find_idempotent_event(idempotency_key) + if existing is None: + raise + return existing return record def emit( diff --git a/app/modules/feishu/app_tickets.py b/app/modules/feishu/app_tickets.py new file mode 100644 index 0000000..9d8bdcc --- /dev/null +++ b/app/modules/feishu/app_tickets.py @@ -0,0 +1,63 @@ +from sqlalchemy import select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.core.utils.time import utc_now +from app.modules.feishu.models import FeishuAppTicket + +APP_TICKET_EVENT_TYPE = "app_ticket" +APP_TICKET_PAYLOAD_KEY = "app_ticket" + + +class FeishuAppTicketService: + """Persist the latest ticket received through a verified Feishu event.""" + + def __init__(self, db: Session): + self.db = db + + def get_ticket(self, app_id: str) -> str | None: + app_id_value = str(app_id).strip() + if not app_id_value: + return None + return self.db.scalar( + select(FeishuAppTicket.app_ticket).where( + FeishuAppTicket.app_id == app_id_value + ) + ) + + def store_verified(self, app_id: str, ticket: str) -> FeishuAppTicket: + app_id_value = str(app_id).strip() + ticket_value = str(ticket).strip() + if not app_id_value or not ticket_value: + raise ValueError("Verified Feishu app ticket fields are required") + + now = utc_now() + record = self.db.execute( + select(FeishuAppTicket) + .where(FeishuAppTicket.app_id == app_id_value) + .with_for_update() + ).scalar_one_or_none() + if record is None: + record = FeishuAppTicket( + app_id=app_id_value, + app_ticket=ticket_value, + received_at=now, + updated_at=now, + ) + try: + with self.db.begin_nested(): + self.db.add(record) + self.db.flush() + except IntegrityError: + record = self.db.execute( + select(FeishuAppTicket) + .where(FeishuAppTicket.app_id == app_id_value) + .with_for_update() + ).scalar_one() + + record.app_ticket = ticket_value + record.received_at = now + record.updated_at = now + self.db.commit() + self.db.refresh(record) + return record diff --git a/app/modules/feishu/client.py b/app/modules/feishu/client.py index 5b47c42..b4b694d 100644 --- a/app/modules/feishu/client.py +++ b/app/modules/feishu/client.py @@ -1,67 +1,185 @@ import json import time +from threading import RLock from typing import Any import httpx from fastapi import HTTPException, status +from sqlalchemy.orm import Session -from app.core.constants import BEARER_TOKEN_TEMPLATE, HttpHeader from app.core.config import get_settings +from app.core.constants import BEARER_TOKEN_TEMPLATE, HttpHeader from app.modules.feishu.constants import ( + FEISHU_APP_TICKET_MISSING, + FEISHU_APP_TOKEN_PATH, FEISHU_AUTH_MISSING, FEISHU_DEFAULT_TOKEN_EXPIRE_SECONDS, - FEISHU_MESSAGE_PATH, FEISHU_IMAGE_PATH, + FEISHU_MESSAGE_PATH, FEISHU_RECEIVE_ID_MISSING, + FEISHU_STORE_TENANT_TOKEN_PATH, FEISHU_SUCCESS_CODE, + FEISHU_TENANT_KEY_MISSING, FEISHU_TENANT_TOKEN_PATH, FEISHU_TOKEN_EXPIRE_SAFETY_SECONDS, + FeishuAppType, FeishuMessageType, FeishuPayloadKey, FeishuReceiveIdType, ) +from app.modules.feishu.errors import FeishuAPIError + +_RETRYABLE_PROVIDER_CODES = frozenset( + { + 99991400, + 99991401, + 99991402, + 99991403, + } +) +_RETRYABLE_PROVIDER_TERMS = ( + "rate limit", + "too many request", + "temporar", + "timeout", + "busy", + "限流", + "频率", + "超时", + "繁忙", +) class FeishuClient: - """Small Feishu Open Platform client for tenant token and message APIs.""" + """Small Feishu Open Platform client with tenant-isolated token caches.""" - def __init__(self) -> None: + def __init__(self, db: Session | None = None) -> None: self.settings = get_settings() + self.db = db self._tenant_access_token: str | None = None self._token_expires_at: float = 0 + self._app_access_tokens: dict[str, tuple[str, float]] = {} + self._store_tenant_access_tokens: dict[tuple[str, str], tuple[str, float]] = {} + self._token_lock = RLock() def _is_configured(self) -> bool: return bool(self.settings.feishu_app_id and self.settings.feishu_app_secret) - def _get_tenant_access_token(self) -> str: + def _get_tenant_access_token(self, tenant_key: str | None = None) -> str: if not self._is_configured(): raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=FEISHU_AUTH_MISSING, ) - if self._tenant_access_token and time.time() < self._token_expires_at: - return self._tenant_access_token + if self.settings.feishu_app_type == FeishuAppType.STORE: + return self._get_store_tenant_access_token(tenant_key) + return self._get_self_tenant_access_token() - url = f"{self.settings.feishu_base_url}{FEISHU_TENANT_TOKEN_PATH}" - payload = { - FeishuPayloadKey.APP_ID: self.settings.feishu_app_id, - FeishuPayloadKey.APP_SECRET: self.settings.feishu_app_secret, - } - with httpx.Client(timeout=20) as client: - response = client.post(url, json=payload) - response.raise_for_status() - data = response.json() - if data.get(FeishuPayloadKey.CODE) != FEISHU_SUCCESS_CODE: - raise HTTPException( - status_code=status.HTTP_502_BAD_GATEWAY, - detail={FeishuPayloadKey.FEISHU_ERROR: data}, + def _get_self_tenant_access_token(self) -> str: + with self._token_lock: + if ( + self._tenant_access_token + and time.time() < self._token_expires_at + ): + return self._tenant_access_token + data = self._post( + f"{self.settings.feishu_base_url}{FEISHU_TENANT_TOKEN_PATH}", + operation="Feishu self tenant token request", + json={ + FeishuPayloadKey.APP_ID: self.settings.feishu_app_id, + FeishuPayloadKey.APP_SECRET: self.settings.feishu_app_secret, + }, ) - self._tenant_access_token = data[FeishuPayloadKey.TENANT_ACCESS_TOKEN] - expire_seconds = int( - data.get(FeishuPayloadKey.EXPIRE, FEISHU_DEFAULT_TOKEN_EXPIRE_SECONDS) - ) - self._token_expires_at = time.time() + expire_seconds - FEISHU_TOKEN_EXPIRE_SAFETY_SECONDS - return self._tenant_access_token + token = self._required_token( + data, + FeishuPayloadKey.TENANT_ACCESS_TOKEN, + "Feishu self tenant token response", + ) + self._tenant_access_token = token + self._token_expires_at = self._expires_at(data) + return token + + def _get_store_app_access_token(self) -> str: + app_id = str(self.settings.feishu_app_id) + with self._token_lock: + cached = self._get_cached(self._app_access_tokens, app_id) + if cached is not None: + return cached + app_ticket = self._get_app_ticket(app_id) + data = self._post( + f"{self.settings.feishu_base_url}{FEISHU_APP_TOKEN_PATH}", + operation="Feishu store app token request", + json={ + FeishuPayloadKey.APP_ID: app_id, + FeishuPayloadKey.APP_SECRET: self.settings.feishu_app_secret, + FeishuPayloadKey.APP_TICKET: app_ticket, + }, + ) + token = self._required_token( + data, + FeishuPayloadKey.APP_ACCESS_TOKEN, + "Feishu store app token response", + ) + self._app_access_tokens[app_id] = (token, self._expires_at(data)) + return token + + def _get_store_tenant_access_token(self, tenant_key: str | None) -> str: + normalized_tenant_key = str(tenant_key or "").strip() + if not normalized_tenant_key: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=FEISHU_TENANT_KEY_MISSING, + ) + app_id = str(self.settings.feishu_app_id) + cache_key = (app_id, normalized_tenant_key) + with self._token_lock: + cached = self._get_cached( + self._store_tenant_access_tokens, + cache_key, + ) + if cached is not None: + return cached + app_access_token = self._get_store_app_access_token() + data = self._post( + f"{self.settings.feishu_base_url}{FEISHU_STORE_TENANT_TOKEN_PATH}", + operation="Feishu store tenant token request", + json={ + FeishuPayloadKey.APP_ACCESS_TOKEN: app_access_token, + FeishuPayloadKey.TENANT_KEY: normalized_tenant_key, + }, + ) + token = self._required_token( + data, + FeishuPayloadKey.TENANT_ACCESS_TOKEN, + "Feishu store tenant token response", + ) + self._store_tenant_access_tokens[cache_key] = ( + token, + self._expires_at(data), + ) + return token + + def _get_app_ticket(self, app_id: str) -> str: + ticket: Any = None + if self.db is not None: + from app.modules.feishu.app_tickets import FeishuAppTicketService + + ticket = FeishuAppTicketService(self.db).get_ticket(app_id) + if ticket is not None and not isinstance(ticket, str): + ticket = getattr(ticket, "app_ticket", None) or getattr( + ticket, + "ticket", + None, + ) + normalized = str(ticket or "").strip() + if not normalized: + normalized = str(self.settings.feishu_app_ticket or "").strip() + if not normalized: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail=FEISHU_APP_TICKET_MISSING, + ) + return normalized def send_message( self, @@ -69,77 +187,207 @@ class FeishuClient: receive_id_type: str, msg_type: str, content: dict[str, Any], + uuid: str | None = None, + tenant_key: str | None = None, ) -> dict[str, Any]: - token = self._get_tenant_access_token() - url = f"{self.settings.feishu_base_url}{FEISHU_MESSAGE_PATH}" + token = self._get_tenant_access_token(tenant_key) headers = {HttpHeader.AUTHORIZATION: BEARER_TOKEN_TEMPLATE.format(token=token)} - params = {FeishuPayloadKey.RECEIVE_ID_TYPE: receive_id_type} payload = { FeishuPayloadKey.RECEIVE_ID: receive_id, FeishuPayloadKey.MESSAGE_TYPE: msg_type, FeishuPayloadKey.CONTENT: json.dumps(content, ensure_ascii=False), } - with httpx.Client(timeout=20) as client: - response = client.post(url, headers=headers, params=params, json=payload) - response.raise_for_status() - data = response.json() - return data + if uuid: + payload[FeishuPayloadKey.UUID] = uuid + return self._post( + f"{self.settings.feishu_base_url}{FEISHU_MESSAGE_PATH}", + operation="Feishu message request", + headers=headers, + params={FeishuPayloadKey.RECEIVE_ID_TYPE: receive_id_type}, + json=payload, + ) def send_text( self, text: str, receive_id: str | None = None, receive_id_type: str = FeishuReceiveIdType.CHAT_ID, - ) -> dict: - chat_id = receive_id or self.settings.feishu_default_chat_id - if not chat_id: + uuid: str | None = None, + tenant_key: str | None = None, + ) -> dict[str, Any]: + target_id = receive_id or self.settings.feishu_default_chat_id + if not target_id: raise HTTPException( status_code=status.HTTP_400_BAD_REQUEST, detail=FEISHU_RECEIVE_ID_MISSING, ) + resolved_tenant_key = tenant_key + if receive_id is None and not resolved_tenant_key: + resolved_tenant_key = self.settings.feishu_default_tenant_key return self.send_message( - chat_id, + target_id, receive_id_type, FeishuMessageType.TEXT, {FeishuPayloadKey.TEXT: text}, + uuid, + resolved_tenant_key, ) def upload_image( self, image: bytes, filename: str = "lifecycle-report.png", + tenant_key: str | None = None, ) -> dict[str, Any]: - token = self._get_tenant_access_token() - url = f"{self.settings.feishu_base_url}{FEISHU_IMAGE_PATH}" + resolved_tenant_key = tenant_key or self.settings.feishu_default_tenant_key + token = self._get_tenant_access_token(resolved_tenant_key) headers = {HttpHeader.AUTHORIZATION: BEARER_TOKEN_TEMPLATE.format(token=token)} - with httpx.Client(timeout=30) as client: - response = client.post( - url, - headers=headers, - data={FeishuPayloadKey.IMAGE_TYPE: FeishuPayloadKey.MESSAGE}, - files={ - FeishuPayloadKey.IMAGE: (filename, image, "image/png"), - }, - ) - response.raise_for_status() - data = response.json() - if data.get(FeishuPayloadKey.CODE) != FEISHU_SUCCESS_CODE: - raise HTTPException( - status_code=status.HTTP_502_BAD_GATEWAY, - detail={FeishuPayloadKey.FEISHU_ERROR: data}, - ) - return data + return self._post( + f"{self.settings.feishu_base_url}{FEISHU_IMAGE_PATH}", + operation="Feishu image upload request", + timeout=30, + headers=headers, + data={FeishuPayloadKey.IMAGE_TYPE: FeishuPayloadKey.MESSAGE}, + files={ + FeishuPayloadKey.IMAGE: (filename, image, "image/png"), + }, + ) def send_card( self, card: dict[str, Any], receive_id: str | None = None, receive_id_type: str = FeishuReceiveIdType.CHAT_ID, - ) -> dict: - chat_id = receive_id or self.settings.feishu_default_chat_id - if not chat_id: + uuid: str | None = None, + tenant_key: str | None = None, + ) -> dict[str, Any]: + target_id = receive_id or self.settings.feishu_default_chat_id + if not target_id: raise HTTPException( status_code=status.HTTP_400_BAD_REQUEST, detail=FEISHU_RECEIVE_ID_MISSING, ) - return self.send_message(chat_id, receive_id_type, FeishuMessageType.INTERACTIVE, card) + resolved_tenant_key = tenant_key + if receive_id is None and not resolved_tenant_key: + resolved_tenant_key = self.settings.feishu_default_tenant_key + return self.send_message( + target_id, + receive_id_type, + FeishuMessageType.INTERACTIVE, + card, + uuid, + resolved_tenant_key, + ) + + def _post( + self, + url: str, + *, + operation: str, + timeout: int = 20, + **kwargs: Any, + ) -> dict[str, Any]: + try: + with httpx.Client(timeout=timeout) as client: + response = client.post(url, **kwargs) + except httpx.HTTPError: + raise FeishuAPIError( + f"{operation} failed", + retryable=True, + ) from None + if not status.HTTP_200_OK <= response.status_code < status.HTTP_300_MULTIPLE_CHOICES: + raise FeishuAPIError( + f"{operation} returned an HTTP error", + retryable=_is_retryable_http_status(response.status_code), + http_status=response.status_code, + ) + try: + data = response.json() + except ValueError: + raise FeishuAPIError( + f"{operation} response was not valid JSON", + retryable=True, + http_status=response.status_code, + ) from None + if not isinstance(data, dict): + raise FeishuAPIError( + f"{operation} response was not a JSON object", + retryable=True, + http_status=response.status_code, + ) + provider_code = data.get(FeishuPayloadKey.CODE) + if provider_code != FEISHU_SUCCESS_CODE: + raise FeishuAPIError( + f"{operation} returned a non-zero business code", + retryable=_is_retryable_business_error(data), + http_status=response.status_code, + provider_code=provider_code, + provider_response={ + FeishuPayloadKey.CODE: provider_code, + }, + ) + return data + + @staticmethod + def _required_token( + data: dict[str, Any], + key: FeishuPayloadKey, + operation: str, + ) -> str: + token = data.get(key) + if not isinstance(token, str) or not token.strip(): + raise FeishuAPIError( + f"{operation} did not include the required credential", + retryable=True, + provider_code=data.get(FeishuPayloadKey.CODE), + provider_response={ + FeishuPayloadKey.CODE: data.get(FeishuPayloadKey.CODE), + }, + ) + return token + + @staticmethod + def _expires_at(data: dict[str, Any]) -> float: + try: + expire_seconds = int( + data.get( + FeishuPayloadKey.EXPIRE, + FEISHU_DEFAULT_TOKEN_EXPIRE_SECONDS, + ) + ) + except (TypeError, ValueError): + expire_seconds = FEISHU_DEFAULT_TOKEN_EXPIRE_SECONDS + usable_seconds = max( + 1, + expire_seconds - FEISHU_TOKEN_EXPIRE_SAFETY_SECONDS, + ) + return time.time() + usable_seconds + + @staticmethod + def _get_cached( + cache: dict[Any, tuple[str, float]], + key: Any, + ) -> str | None: + entry = cache.get(key) + if entry is None: + return None + token, expires_at = entry + if time.time() < expires_at: + return token + cache.pop(key, None) + return None + + +def _is_retryable_http_status(status_code: int) -> bool: + return ( + status_code == status.HTTP_429_TOO_MANY_REQUESTS + or status_code >= status.HTTP_500_INTERNAL_SERVER_ERROR + ) + + +def _is_retryable_business_error(data: dict[str, Any]) -> bool: + code = data.get(FeishuPayloadKey.CODE) + if code in _RETRYABLE_PROVIDER_CODES: + return True + message = str(data.get("msg") or "").casefold() + return any(term in message for term in _RETRYABLE_PROVIDER_TERMS) diff --git a/app/modules/feishu/constants.py b/app/modules/feishu/constants.py index ed158b4..f83a8bf 100644 --- a/app/modules/feishu/constants.py +++ b/app/modules/feishu/constants.py @@ -3,6 +3,12 @@ from enum import StrEnum class FeishuReceiveIdType(StrEnum): CHAT_ID = "chat_id" + OPEN_ID = "open_id" + + +class FeishuAppType(StrEnum): + SELF = "self" + STORE = "store" class FeishuMessageType(StrEnum): @@ -16,24 +22,30 @@ class FeishuEventSource(StrEnum): class FeishuPayloadKey(StrEnum): + APP_ACCESS_TOKEN = "app_access_token" APP_ID = "app_id" APP_SECRET = "app_secret" + APP_TICKET = "app_ticket" CARD = "card" CHALLENGE = "challenge" + CHAT_TYPE = "chat_type" CODE = "code" CONFIG = "config" CONTENT = "content" DIV = "div" DATA = "data" ELEMENTS = "elements" + ENCRYPT = "encrypt" EXPIRE = "expire" FEISHU_ERROR = "feishu_error" HEADER = "header" IMAGE = "image" IMAGE_KEY = "image_key" IMAGE_TYPE = "image_type" + ID = "id" IMG = "img" IMG_KEY = "img_key" + KEY = "key" ALT = "alt" EVENT = "event" EVENT_ID = "event_id" @@ -42,6 +54,8 @@ class FeishuPayloadKey(StrEnum): MESSAGE = "message" MESSAGE_ID = "message_id" MESSAGE_TYPE = "msg_type" + MENTIONS = "mentions" + NAME = "name" OPEN_ID = "open_id" PLAIN_TEXT = "plain_text" RECEIVE_ID = "receive_id" @@ -50,17 +64,23 @@ class FeishuPayloadKey(StrEnum): SENDER_ID = "sender_id" TAG = "tag" TENANT_ACCESS_TOKEN = "tenant_access_token" + TENANT_KEY = "tenant_key" TEXT = "text" TITLE = "title" TOKEN = "token" + UNION_ID = "union_id" USER_ID = "user_id" + UUID = "uuid" WIDE_SCREEN_MODE = "wide_screen_mode" class FeishuCommandKey(StrEnum): TEXT = "text" CHAT_ID = "chat_id" + CHAT_TYPE = "chat_type" ACTOR = "actor" + MENTIONS = "mentions" + PRINCIPAL = "principal" class FeishuResponseKey(StrEnum): @@ -85,10 +105,32 @@ class FeishuCommandResultKey(StrEnum): class FeishuCommandName(StrEnum): + PERMISSION_DENIED = "permission_denied" + HELP = "help" + USER_SET_ADMIN = "user_set_admin" + USER_SET_USER = "user_set_user" + USER_DISABLE = "user_disable" + USER_ENABLE = "user_enable" + PREFERENCE_SET = "preference_set" + PREFERENCE_LIST = "preference_list" + PREFERENCE_DELETE = "preference_delete" + CONVERSATION_RESET = "conversation_reset" + PERSONAL_DATA_SUMMARY = "personal_data_summary" + PERSONAL_DATA_ERASURE_REQUEST = "personal_data_erasure_request" + PERSONAL_DATA_ERASURE_CONFIRM = "personal_data_erasure_confirm" + SUBSCRIPTION_CREATE = "subscription_create" + SUBSCRIPTION_LIST = "subscription_list" + SUBSCRIPTION_PAUSE = "subscription_pause" + SUBSCRIPTION_RESUME = "subscription_resume" + SUBSCRIPTION_CANCEL = "subscription_cancel" + SUBSCRIPTION_TIMEZONE = "subscription_timezone" + SUBSCRIPTION_QUIET_HOURS = "subscription_quiet_hours" RULE_CREATE = "rule_create" RULE_LIST = "rule_list" RULE_DISABLE = "rule_disable" RULE_ENABLE = "rule_enable" + RULE_UPDATE = "rule_update" + RULE_DELETE = "rule_delete" FINANCE_NEEDS = "finance_needs" PROJECT_FINANCE = "project_finance" MARKET_OVERVIEW = "market_overview" @@ -123,11 +165,15 @@ class FeishuCardKey(StrEnum): VALUE = "value" +FEISHU_APP_TOKEN_PATH = "/auth/v3/app_access_token" +FEISHU_STORE_TENANT_TOKEN_PATH = "/auth/v3/tenant_access_token" FEISHU_TENANT_TOKEN_PATH = "/auth/v3/tenant_access_token/internal" FEISHU_MESSAGE_PATH = "/im/v1/messages" FEISHU_IMAGE_PATH = "/im/v1/images" FEISHU_DEFAULT_OPEN_API_DOMAIN = "https://open.feishu.cn" FEISHU_AUTH_MISSING = "Feishu app credentials are not configured" +FEISHU_APP_TICKET_MISSING = "Feishu store app ticket is not available" +FEISHU_TENANT_KEY_MISSING = "tenant_key is required for Feishu store apps" FEISHU_VERIFICATION_TOKEN_REQUIRED = "FEISHU_VERIFICATION_TOKEN is required" FEISHU_INVALID_TOKEN = "Invalid Feishu token" FEISHU_RECEIVE_ID_MISSING = "receive_id or FEISHU_DEFAULT_CHAT_ID is required" diff --git a/app/modules/feishu/errors.py b/app/modules/feishu/errors.py new file mode 100644 index 0000000..6505cd5 --- /dev/null +++ b/app/modules/feishu/errors.py @@ -0,0 +1,29 @@ +from typing import Any + +from fastapi import HTTPException, status + + +class FeishuAPIError(HTTPException): + """Normalized outbound Feishu failure with retry classification.""" + + def __init__( + self, + detail: str, + *, + retryable: bool, + http_status: int | None = None, + provider_code: int | str | None = None, + provider_response: dict[str, Any] | None = None, + ) -> None: + super().__init__( + status_code=( + status.HTTP_503_SERVICE_UNAVAILABLE + if retryable + else status.HTTP_502_BAD_GATEWAY + ), + detail=detail, + ) + self.retryable = retryable + self.http_status = http_status + self.provider_code = provider_code + self.provider_response = provider_response or {} diff --git a/app/modules/feishu/event_verification.py b/app/modules/feishu/event_verification.py new file mode 100644 index 0000000..f7a98a3 --- /dev/null +++ b/app/modules/feishu/event_verification.py @@ -0,0 +1,131 @@ +import base64 +import json +import time +from hashlib import sha256 +from secrets import compare_digest +from typing import Any, Mapping + +from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes +from cryptography.hazmat.primitives.padding import PKCS7 +from fastapi import HTTPException, status + +from app.core.config import get_settings +from app.modules.feishu.constants import FeishuPayloadKey + +_SIGNATURE_MAX_AGE_SECONDS = 300 +_SIGNATURE_HEADER = "x-lark-signature" +_TIMESTAMP_HEADER = "x-lark-request-timestamp" +_NONCE_HEADER = "x-lark-request-nonce" + + +class FeishuWebhookVerifier: + """Verify, decrypt, and normalize an HTTP webhook before business handling.""" + + def verify( + self, + raw_body: bytes, + headers: Mapping[str, str], + ) -> dict[str, Any]: + settings = get_settings() + if settings.feishu_encrypt_key: + self._verify_signature(raw_body, headers, settings.feishu_encrypt_key) + payload = self._load_json(raw_body) + if FeishuPayloadKey.ENCRYPT in payload: + if not settings.feishu_encrypt_key: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="FEISHU_ENCRYPT_KEY is required for encrypted webhooks", + ) + payload = self._decrypt( + str(payload[FeishuPayloadKey.ENCRYPT]), + settings.feishu_encrypt_key, + ) + self._verify_token(payload, settings.feishu_verification_token) + return payload + + @staticmethod + def _load_json(raw_body: bytes) -> dict[str, Any]: + try: + payload = json.loads(raw_body) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Invalid Feishu webhook JSON", + ) from exc + if not isinstance(payload, dict): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Invalid Feishu webhook payload", + ) + return payload + + @staticmethod + def _verify_signature( + raw_body: bytes, + headers: Mapping[str, str], + encrypt_key: str, + ) -> None: + normalized = {str(key).lower(): str(value) for key, value in headers.items()} + timestamp = normalized.get(_TIMESTAMP_HEADER) + nonce = normalized.get(_NONCE_HEADER) + signature = normalized.get(_SIGNATURE_HEADER) + if not timestamp or not nonce or not signature: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Missing Feishu webhook signature headers", + ) + try: + request_time = int(timestamp) + except ValueError as exc: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid Feishu webhook timestamp", + ) from exc + if abs(int(time.time()) - request_time) > _SIGNATURE_MAX_AGE_SECONDS: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Expired Feishu webhook signature", + ) + signed = ( + timestamp.encode("utf-8") + + nonce.encode("utf-8") + + encrypt_key.encode("utf-8") + + raw_body + ) + expected = sha256(signed).hexdigest() + if not compare_digest(signature, expected): + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid Feishu webhook signature", + ) + + @staticmethod + def _decrypt(encrypted: str, encrypt_key: str) -> dict[str, Any]: + try: + key = sha256(encrypt_key.encode("utf-8")).digest() + encrypted_bytes = base64.b64decode(encrypted, validate=True) + decryptor = Cipher(algorithms.AES(key), modes.CBC(key[:16])).decryptor() + padded = decryptor.update(encrypted_bytes) + decryptor.finalize() + unpadder = PKCS7(algorithms.AES.block_size).unpadder() + cleartext = unpadder.update(padded) + unpadder.finalize() + except (ValueError, TypeError) as exc: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid encrypted Feishu webhook", + ) from exc + return FeishuWebhookVerifier._load_json(cleartext) + + @staticmethod + def _verify_token(payload: dict[str, Any], expected: str | None) -> None: + if not expected: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="FEISHU_VERIFICATION_TOKEN is required", + ) + header = payload.get(FeishuPayloadKey.HEADER) or {} + token = payload.get(FeishuPayloadKey.TOKEN) or header.get(FeishuPayloadKey.TOKEN) + if not token or not compare_digest(str(token), expected): + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid Feishu token", + ) diff --git a/app/modules/feishu/long_connection.py b/app/modules/feishu/long_connection.py index 0c83e85..bdd389e 100644 --- a/app/modules/feishu/long_connection.py +++ b/app/modules/feishu/long_connection.py @@ -31,10 +31,18 @@ def _sdk_event_to_payload(event: Any) -> dict[str, Any]: def _handle_message_event(event: Any) -> None: + _handle_verified_sdk_event(event) + + +def _handle_app_ticket_event(event: Any) -> None: + _handle_verified_sdk_event(event) + + +def _handle_verified_sdk_event(event: Any) -> None: payload = _sdk_event_to_payload(event) db = SessionLocal() try: - result = FeishuEventService(db).handle_event( + result = FeishuEventService(db)._handle_verified_event( payload, source=FeishuEventSource.LONG_CONNECTION, auto_reply=True, @@ -62,6 +70,7 @@ def run_long_connection() -> None: settings.feishu_verification_token or "", ) .register_p2_im_message_receive_v1(_handle_message_event) + .register_p1_customized_event("app_ticket", _handle_app_ticket_event) .build() ) client = lark.ws.Client( diff --git a/app/modules/feishu/models.py b/app/modules/feishu/models.py index 6c4ed2e..bb691f6 100644 --- a/app/modules/feishu/models.py +++ b/app/modules/feishu/models.py @@ -1,6 +1,6 @@ from datetime import datetime -from sqlalchemy import DateTime, Integer, String +from sqlalchemy import DateTime, Integer, String, Text from sqlalchemy.orm import Mapped, mapped_column from app.core.database import Base @@ -16,3 +16,17 @@ class FeishuEventReceipt(Base): event_id: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) message_id: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) received_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + + +class FeishuAppTicket(Base): + __tablename__ = "feishu_app_tickets" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + app_id: Mapped[str] = mapped_column(String(128), unique=True, index=True) + app_ticket: Mapped[str] = mapped_column(Text) + received_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) diff --git a/app/modules/feishu/routes.py b/app/modules/feishu/routes.py index f5ad2af..6aeecb9 100644 --- a/app/modules/feishu/routes.py +++ b/app/modules/feishu/routes.py @@ -1,12 +1,11 @@ -from typing import Any - -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, Request from sqlalchemy.orm import Session from app.core.database import get_db from app.core.security import ApiPrincipal, require_api_key from app.application.feishu import FeishuCommandService, FeishuEventService from app.modules.feishu.constants import FeishuEventSource, FeishuPayloadKey, FeishuResponseKey +from app.modules.feishu.event_verification import FeishuWebhookVerifier from app.modules.feishu.schemas import ( FeishuCardMessage, FeishuCommandRequest, @@ -20,10 +19,11 @@ router = APIRouter() @router.post("/webhook") -def feishu_webhook(payload: dict[str, Any], db: Session = Depends(get_db)) -> dict: +async def feishu_webhook(request: Request, db: Session = Depends(get_db)) -> dict: """Handle Feishu webhook challenge and text command events.""" - return FeishuEventService(db).handle_event( + payload = FeishuWebhookVerifier().verify(await request.body(), request.headers) + return FeishuEventService(db)._handle_verified_event( payload, source=FeishuEventSource.WEBHOOK, auto_reply=True, @@ -41,6 +41,7 @@ def send_text( receive_id=payload.receive_id, receive_id_type=payload.receive_id_type, actor=principal.actor, + tenant_key=payload.tenant_key, ) return { FeishuResponseKey.OK: result.get(FeishuPayloadKey.CODE) == 0, @@ -59,6 +60,7 @@ def send_card( receive_id=payload.receive_id, receive_id_type=payload.receive_id_type, actor=principal.actor, + tenant_key=payload.tenant_key, ) return { FeishuResponseKey.OK: result.get(FeishuPayloadKey.CODE) == 0, @@ -82,4 +84,5 @@ def preview_command( chat_id=payload.chat_id, actor=principal.actor, auto_reply=payload.auto_reply, + tenant_key=payload.tenant_key, ) diff --git a/app/modules/feishu/schemas.py b/app/modules/feishu/schemas.py index 96377f3..ed40212 100644 --- a/app/modules/feishu/schemas.py +++ b/app/modules/feishu/schemas.py @@ -12,12 +12,14 @@ class FeishuTextMessage(BaseModel): description="chat_id or open_id depending on type.", ) receive_id_type: str = FeishuReceiveIdType.CHAT_ID + tenant_key: str | None = Field(default=None, max_length=128) text: str class FeishuCardMessage(BaseModel): receive_id: str | None = None receive_id_type: str = FeishuReceiveIdType.CHAT_ID + tenant_key: str | None = Field(default=None, max_length=128) card: dict[str, Any] @@ -38,6 +40,7 @@ class FeishuSendResult(BaseModel): class FeishuCommandRequest(BaseModel): text: str chat_id: str | None = None + tenant_key: str | None = Field(default=None, max_length=128) actor: str = ActorValue.API auto_reply: bool = False diff --git a/app/modules/feishu/service.py b/app/modules/feishu/service.py index bc47444..625bd64 100644 --- a/app/modules/feishu/service.py +++ b/app/modules/feishu/service.py @@ -1,3 +1,4 @@ +from hashlib import sha256 from secrets import compare_digest from typing import Any @@ -22,10 +23,18 @@ from app.modules.feishu.constants import ( class FeishuService: """Send Feishu messages and record audit entries for outbound actions.""" - def __init__(self, db: Session): + def __init__(self, db: Session, tenant_key: str | None = None): self.db = db self.audit = AuditService(db) - self.client = FeishuClient() + self.client = FeishuClient(db) + self.tenant_key = _optional_text(tenant_key) or _optional_text( + get_settings().feishu_default_tenant_key + ) + + def set_tenant_key(self, tenant_key: str | None) -> None: + """Set the default tenant used by subsequent outbound operations.""" + + self.tenant_key = _optional_text(tenant_key) def verify_event(self, payload: dict[str, Any]) -> None: settings = get_settings() @@ -49,21 +58,32 @@ class FeishuService: receive_id: str | None = None, receive_id_type: str = FeishuReceiveIdType.CHAT_ID, actor: str = ActorValue.SYSTEM, + uuid: str | None = None, + tenant_key: str | None = None, + record_audit: bool = True, ) -> dict[str, Any]: - result = self.client.send_text(text, receive_id, receive_id_type) - self.audit.log( - AuditLogCreate( - actor=actor, - source=AuditSource.FEISHU, - action=AuditAction.FEISHU_SEND_TEXT, - request_payload={ - FeishuPayloadKey.RECEIVE_ID: receive_id, - FeishuPayloadKey.RECEIVE_ID_TYPE: receive_id_type, - FeishuPayloadKey.TEXT: text, - }, - response_payload=result, - ) + result = self.client.send_text( + text, + receive_id, + receive_id_type, + uuid, + tenant_key=self._resolve_tenant_key(tenant_key), ) + if record_audit: + self.audit.log( + AuditLogCreate( + actor=actor, + source=AuditSource.FEISHU, + action=AuditAction.FEISHU_SEND_TEXT, + request_payload={ + "receive_target_hash": _target_fingerprint(receive_id), + FeishuPayloadKey.RECEIVE_ID_TYPE: receive_id_type, + "content_length": len(text), + FeishuPayloadKey.UUID: uuid, + }, + response_payload=result, + ) + ) return result def send_card( @@ -72,17 +92,26 @@ class FeishuService: receive_id: str | None = None, receive_id_type: str = FeishuReceiveIdType.CHAT_ID, actor: str = ActorValue.SYSTEM, + uuid: str | None = None, + tenant_key: str | None = None, ) -> dict[str, Any]: - result = self.client.send_card(card, receive_id, receive_id_type) + result = self.client.send_card( + card, + receive_id, + receive_id_type, + uuid, + tenant_key=self._resolve_tenant_key(tenant_key), + ) self.audit.log( AuditLogCreate( actor=actor, source=AuditSource.FEISHU, action=AuditAction.FEISHU_SEND_CARD, request_payload={ - FeishuPayloadKey.RECEIVE_ID: receive_id, + "receive_target_hash": _target_fingerprint(receive_id), FeishuPayloadKey.RECEIVE_ID_TYPE: receive_id_type, - FeishuPayloadKey.CARD: card, + "card_element_count": len(card.get(FeishuPayloadKey.ELEMENTS) or []), + FeishuPayloadKey.UUID: uuid, }, response_payload=result, ) @@ -93,8 +122,12 @@ class FeishuService: self, image: bytes, actor: str = ActorValue.SYSTEM, + tenant_key: str | None = None, ) -> dict[str, Any]: - result = self.client.upload_image(image) + result = self.client.upload_image( + image, + tenant_key=self._resolve_tenant_key(tenant_key), + ) self.audit.log( AuditLogCreate( actor=actor, @@ -106,6 +139,9 @@ class FeishuService: ) return result + def _resolve_tenant_key(self, tenant_key: str | None) -> str | None: + return _optional_text(tenant_key) or self.tenant_key + @staticmethod def build_basic_card( title: str, @@ -144,3 +180,14 @@ class FeishuService: }, FeishuPayloadKey.ELEMENTS: elements, } + + +def _target_fingerprint(receive_id: str | None) -> str | None: + if not receive_id: + return None + return sha256(receive_id.encode("utf-8")).hexdigest()[:16] + + +def _optional_text(value: str | None) -> str | None: + text = str(value or "").strip() + return text or None diff --git a/app/modules/feishu_users/__init__.py b/app/modules/feishu_users/__init__.py new file mode 100644 index 0000000..94043d9 --- /dev/null +++ b/app/modules/feishu_users/__init__.py @@ -0,0 +1,22 @@ +from app.modules.feishu_users.constants import ( + FeishuCapability, + FeishuUserRole, + FeishuUserStatus, +) +from app.modules.feishu_users.models import FeishuUser +from app.modules.feishu_users.principal import FeishuMention, FeishuPrincipal +from app.modules.feishu_users.services import ( + FeishuIdentityService, + FeishuUserManagementService, +) + +__all__ = [ + "FeishuCapability", + "FeishuIdentityService", + "FeishuMention", + "FeishuPrincipal", + "FeishuUser", + "FeishuUserManagementService", + "FeishuUserRole", + "FeishuUserStatus", +] diff --git a/app/modules/feishu_users/bootstrap.py b/app/modules/feishu_users/bootstrap.py new file mode 100644 index 0000000..586a90f --- /dev/null +++ b/app/modules/feishu_users/bootstrap.py @@ -0,0 +1,21 @@ +from hashlib import sha256 + +_ADMIN_BOOTSTRAP_TOMBSTONE_DOMAIN = ( + b"company-ai-platform:feishu-admin-bootstrap-tombstone:v1\0" +) + + +def admin_bootstrap_identity_hash(tenant_key: str, open_id: str) -> str: + """Return a domain-separated digest used only to prevent admin re-grants.""" + + tenant_bytes = tenant_key.encode("utf-8") + open_id_bytes = open_id.encode("utf-8") + identity = b"".join( + ( + len(tenant_bytes).to_bytes(4, "big"), + tenant_bytes, + len(open_id_bytes).to_bytes(4, "big"), + open_id_bytes, + ) + ) + return sha256(_ADMIN_BOOTSTRAP_TOMBSTONE_DOMAIN + identity).hexdigest() diff --git a/app/modules/feishu_users/constants.py b/app/modules/feishu_users/constants.py new file mode 100644 index 0000000..8368dbb --- /dev/null +++ b/app/modules/feishu_users/constants.py @@ -0,0 +1,86 @@ +from enum import StrEnum +from typing import Iterable + + +class FeishuUserRole(StrEnum): + USER = "user" + ADMIN = "admin" + + +class FeishuUserStatus(StrEnum): + ACTIVE = "active" + DISABLED = "disabled" + + +class FeishuCapability(StrEnum): + PERSONAL_AI = "personal_ai" + PERSONAL_DATA = "personal_data" + PRIVATE_SUBSCRIPTION = "private_subscription" + PERSONAL_MARKET = "personal_market" + COMPANY_REPORTS = "company_reports" + COMPANY_RULES = "company_rules" + USER_ADMINISTRATION = "user_administration" + GROUP_SUBSCRIPTION = "group_subscription" + + +class FeishuUserAuditAction(StrEnum): + REGISTER = "feishu.user.register" + AUTHENTICATE = "feishu.user.authenticate" + PERMISSION_DENIED = "feishu.permission.denied" + LIST = "feishu.user.list" + READ = "feishu.user.read" + UPDATE = "feishu.user.update" + UPDATE_DENIED = "feishu.user.update_denied" + + +FEISHU_USER_CODE_PREFIX = "FSU" +FEISHU_USER_TARGET_TYPE = "feishu-user" +DEFAULT_FEISHU_USER_TIMEZONE = "Asia/Shanghai" +LAST_ACTIVE_ADMIN_ERROR = "The last active Feishu administrator cannot be changed" +FEISHU_USER_NOT_FOUND = "Feishu user not found" +INVALID_FEISHU_IDENTITY = "tenant_key and open_id are required" +INVALID_FEISHU_TIMEZONE = "Invalid IANA timezone" +INVALID_QUIET_HOURS = "quiet_hours_start and quiet_hours_end must both be set or cleared" +INVALID_ADMIN_IDENTITY = ( + "FEISHU_ADMIN_IDENTITIES entries must use tenant_key:open_id format" +) + +_USER_CAPABILITIES = frozenset( + { + FeishuCapability.PERSONAL_AI, + FeishuCapability.PERSONAL_DATA, + FeishuCapability.PRIVATE_SUBSCRIPTION, + FeishuCapability.PERSONAL_MARKET, + } +) +_ADMIN_CAPABILITIES = frozenset(FeishuCapability) + + +def capabilities_for_role(role: str | FeishuUserRole) -> frozenset[FeishuCapability]: + """Return the fixed capability set for a Feishu user role.""" + + if FeishuUserRole(role) == FeishuUserRole.ADMIN: + return _ADMIN_CAPABILITIES + return _USER_CAPABILITIES + + +def parse_admin_identities( + value: str | Iterable[str] | None, +) -> frozenset[tuple[str, str]]: + """Parse exact tenant/open-id pairs used only for initial administrator creation.""" + + if value is None: + return frozenset() + entries = value.split(",") if isinstance(value, str) else value + identities: set[tuple[str, str]] = set() + for entry in entries: + text = str(entry).strip() + if not text: + continue + tenant_key, separator, open_id = text.partition(":") + tenant_key = tenant_key.strip() + open_id = open_id.strip() + if not separator or not tenant_key or not open_id: + raise ValueError(INVALID_ADMIN_IDENTITY) + identities.add((tenant_key, open_id)) + return frozenset(identities) diff --git a/app/modules/feishu_users/models.py b/app/modules/feishu_users/models.py new file mode 100644 index 0000000..7a8a9bb --- /dev/null +++ b/app/modules/feishu_users/models.py @@ -0,0 +1,63 @@ +from datetime import datetime, time + +from sqlalchemy import DateTime, Integer, String, Time, UniqueConstraint +from sqlalchemy.orm import Mapped, mapped_column + +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.feishu_users.constants import ( + DEFAULT_FEISHU_USER_TIMEZONE, + FeishuUserRole, + FeishuUserStatus, +) + + +class FeishuUser(Base): + __tablename__ = "feishu_users" + __table_args__ = ( + UniqueConstraint( + "tenant_key", + "open_id", + name="uq_feishu_user_tenant_open_id", + ), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column(String(64), unique=True, index=True) + tenant_key: Mapped[str] = mapped_column(String(128), index=True) + open_id: Mapped[str] = mapped_column(String(128), index=True) + union_id: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + user_id: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + role: Mapped[str] = mapped_column( + String(32), + default=FeishuUserRole.USER, + index=True, + ) + status: Mapped[str] = mapped_column( + String(32), + default=FeishuUserStatus.ACTIVE, + index=True, + ) + timezone: Mapped[str] = mapped_column( + String(64), + default=DEFAULT_FEISHU_USER_TIMEZONE, + ) + quiet_hours_start: Mapped[time | None] = mapped_column(Time, nullable=True) + quiet_hours_end: Mapped[time | None] = mapped_column(Time, nullable=True) + last_active_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) + + +class FeishuAdminBootstrapTombstone(Base): + """Irreversible marker preventing a deleted initial admin from re-bootstrap.""" + + __tablename__ = "feishu_admin_bootstrap_tombstones" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + identity_hash: Mapped[str] = mapped_column(String(64), unique=True, index=True) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) diff --git a/app/modules/feishu_users/principal.py b/app/modules/feishu_users/principal.py new file mode 100644 index 0000000..8cf5073 --- /dev/null +++ b/app/modules/feishu_users/principal.py @@ -0,0 +1,102 @@ +from dataclasses import dataclass +from datetime import time + +from fastapi import HTTPException, status + +from app.modules.feishu_users.constants import ( + FeishuCapability, + FeishuUserRole, + FeishuUserStatus, + capabilities_for_role, +) +from app.modules.feishu_users.models import FeishuUser + + +@dataclass(frozen=True, slots=True) +class FeishuMention: + """Structured mention identity supplied by a verified Feishu event.""" + + key: str | None = None + name: str | None = None + tenant_key: str | None = None + open_id: str | None = None + union_id: str | None = None + user_id: str | None = None + + +@dataclass(frozen=True, slots=True) +class FeishuPrincipal: + """Authenticated Feishu user plus the current chat context.""" + + owner_id: int + user_code: str + tenant_key: str + open_id: str + union_id: str | None + feishu_user_id: str | None + role: str + status: str + timezone: str + quiet_hours_start: time | None + quiet_hours_end: time | None + chat_id: str | None = None + chat_type: str | None = None + mentions: tuple[FeishuMention, ...] = () + + @property + def is_active(self) -> bool: + return self.status == FeishuUserStatus.ACTIVE + + @property + def is_admin(self) -> bool: + return self.role == FeishuUserRole.ADMIN + + def has_capability(self, capability: str | FeishuCapability) -> bool: + if not self.is_active: + return False + try: + required = FeishuCapability(capability) + return required in capabilities_for_role(self.role) + except ValueError: + return False + + def require_active(self) -> None: + if not self.is_active: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu user is disabled", + ) + + def require_capability(self, capability: str | FeishuCapability) -> None: + self.require_active() + if not self.has_capability(capability): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu user is not authorized for this capability", + ) + + @classmethod + def from_user( + cls, + user: FeishuUser, + *, + chat_id: str | None = None, + chat_type: str | None = None, + mentions: tuple[FeishuMention, ...] = (), + ) -> "FeishuPrincipal": + return cls( + owner_id=user.id, + user_code=user.code, + tenant_key=user.tenant_key, + open_id=user.open_id, + union_id=user.union_id, + feishu_user_id=user.user_id, + role=user.role, + status=user.status, + timezone=user.timezone, + quiet_hours_start=user.quiet_hours_start, + quiet_hours_end=user.quiet_hours_end, + chat_id=chat_id, + chat_type=chat_type, + mentions=mentions, + ) diff --git a/app/modules/feishu_users/routes.py b/app/modules/feishu_users/routes.py new file mode 100644 index 0000000..2953d8d --- /dev/null +++ b/app/modules/feishu_users/routes.py @@ -0,0 +1,80 @@ +from fastapi import APIRouter, Depends, Query +from sqlalchemy.orm import Session + +from app.application.feishu.personal_data import FeishuPersonalDataService +from app.core.database import get_db +from app.core.security import ApiPrincipal, require_api_key +from app.modules.feishu_users.constants import FeishuUserRole, FeishuUserStatus +from app.modules.feishu_users.schemas import ( + FeishuUserListRead, + FeishuUserRead, + FeishuUserUpdate, +) +from app.modules.feishu_users.services import FeishuUserManagementService +from app.modules.personalization.schemas import ErasureResult + +router = APIRouter(prefix="/users", dependencies=[Depends(require_api_key)]) + + +@router.get("", response_model=FeishuUserListRead) +def list_users( + role: FeishuUserRole | None = None, + status_filter: FeishuUserStatus | None = Query(default=None, alias="status"), + tenant_key: str | None = None, + limit: int = Query(default=100, ge=1, le=500), + offset: int = Query(default=0, ge=0), + db: Session = Depends(get_db), + principal: ApiPrincipal = Depends(require_api_key), +) -> dict: + items, total = FeishuUserManagementService(db).list_users( + role=role, + status_filter=status_filter, + tenant_key=tenant_key, + limit=limit, + offset=offset, + actor=principal.actor, + ) + return { + "items": items, + "total": total, + "limit": limit, + "offset": offset, + } + + +@router.get("/{code}", response_model=FeishuUserRead) +def get_user( + code: str, + db: Session = Depends(get_db), + principal: ApiPrincipal = Depends(require_api_key), +) -> FeishuUserRead: + return FeishuUserManagementService(db).get_user( + code, + actor=principal.actor, + ) + + +@router.patch("/{code}", response_model=FeishuUserRead) +def update_user( + code: str, + payload: FeishuUserUpdate, + db: Session = Depends(get_db), + principal: ApiPrincipal = Depends(require_api_key), +) -> FeishuUserRead: + return FeishuUserManagementService(db).update_user( + code, + changes=payload.model_dump(exclude_unset=True), + actor=principal.actor, + ) + + +@router.delete("/{code}/personal-data", response_model=ErasureResult) +def erase_user_personal_data( + code: str, + db: Session = Depends(get_db), + principal: ApiPrincipal = Depends(require_api_key), +) -> ErasureResult: + return FeishuPersonalDataService(db).erase_by_user_code( + code, + actor=principal.actor, + ) diff --git a/app/modules/feishu_users/schemas.py b/app/modules/feishu_users/schemas.py new file mode 100644 index 0000000..492c047 --- /dev/null +++ b/app/modules/feishu_users/schemas.py @@ -0,0 +1,40 @@ +from datetime import datetime, time + +from pydantic import BaseModel, ConfigDict, Field + +from app.modules.feishu_users.constants import FeishuUserRole, FeishuUserStatus + + +class FeishuUserRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + code: str + tenant_key: str + open_id: str + union_id: str | None + user_id: str | None + role: str + status: str + timezone: str + quiet_hours_start: time | None + quiet_hours_end: time | None + last_active_at: datetime + created_at: datetime + updated_at: datetime + + +class FeishuUserListRead(BaseModel): + items: list[FeishuUserRead] + total: int + limit: int + offset: int + + +class FeishuUserUpdate(BaseModel): + model_config = ConfigDict(extra="ignore") + + role: FeishuUserRole | None = None + status: FeishuUserStatus | None = None + timezone: str | None = Field(default=None, min_length=1, max_length=64) + quiet_hours_start: time | None = None + quiet_hours_end: time | None = None diff --git a/app/modules/feishu_users/services/__init__.py b/app/modules/feishu_users/services/__init__.py new file mode 100644 index 0000000..f9a1989 --- /dev/null +++ b/app/modules/feishu_users/services/__init__.py @@ -0,0 +1,9 @@ +from app.modules.feishu_users.services.identity import FeishuIdentityService +from app.modules.feishu_users.services.management import ( + FeishuUserManagementService, +) + +__all__ = [ + "FeishuIdentityService", + "FeishuUserManagementService", +] diff --git a/app/modules/feishu_users/services/identity.py b/app/modules/feishu_users/services/identity.py new file mode 100644 index 0000000..970dd1e --- /dev/null +++ b/app/modules/feishu_users/services/identity.py @@ -0,0 +1,161 @@ +from collections.abc import Iterable +from typing import Any +from uuid import uuid4 + +from fastapi import HTTPException, status +from sqlalchemy import inspect, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.core.config import get_settings +from app.core.constants import ActorValue +from app.core.utils.time import utc_now +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu_users.constants import ( + FEISHU_USER_CODE_PREFIX, + FEISHU_USER_TARGET_TYPE, + INVALID_FEISHU_IDENTITY, + FeishuUserAuditAction, + FeishuUserRole, + FeishuUserStatus, + parse_admin_identities, +) +from app.modules.feishu_users.bootstrap import admin_bootstrap_identity_hash +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) +from app.modules.feishu_users.principal import FeishuPrincipal + + +class FeishuIdentityService: + """Resolve or create identities only after the caller verifies the Feishu event.""" + + def __init__( + self, + db: Session, + admin_identities: str | Iterable[str] | None = None, + ): + self.db = db + self.audit = AuditService(db) + configured = ( + admin_identities + if admin_identities is not None + else getattr(get_settings(), "feishu_admin_identities", ()) + ) + self.admin_identities = parse_admin_identities(configured) + + def resolve_or_register( + self, + *, + tenant_key: str, + open_id: str, + union_id: str | None = None, + user_id: str | None = None, + actor: str = ActorValue.FEISHU, + ) -> FeishuPrincipal: + """Return a principal for a previously verified Feishu sender.""" + + tenant_key = _required_identity_part(tenant_key) + open_id = _required_identity_part(open_id) + union_id = _optional_identity_part(union_id) + user_id = _optional_identity_part(user_id) + record = self.get_by_identity(tenant_key=tenant_key, open_id=open_id) + created = False + if record is None: + candidate = FeishuUser( + code=f"{FEISHU_USER_CODE_PREFIX}-{uuid4().hex[:20].upper()}", + tenant_key=tenant_key, + open_id=open_id, + union_id=union_id, + user_id=user_id, + role=self._initial_role(tenant_key, open_id), + status=FeishuUserStatus.ACTIVE, + last_active_at=utc_now(), + ) + try: + with self.db.begin_nested(): + self.db.add(candidate) + self.db.flush() + record = candidate + created = True + except IntegrityError: + record = self.get_by_identity(tenant_key=tenant_key, open_id=open_id) + if record is None: + raise + + record.last_active_at = utc_now() + if union_id: + record.union_id = union_id + if user_id: + record.user_id = user_id + self.audit.record( + AuditLogCreate( + actor=actor, + source=AuditSource.FEISHU, + action=( + FeishuUserAuditAction.REGISTER + if created + else FeishuUserAuditAction.AUTHENTICATE + ), + target_type=FEISHU_USER_TARGET_TYPE, + target_id=record.code, + risk_level=AuditRiskLevel.LOW, + response_payload={ + "created": created, + "role": record.role, + "status": record.status, + }, + ) + ) + self.db.commit() + self.db.refresh(record) + if created and inspect(self.db.get_bind()).has_table("market_watchlists"): + # Import locally so the identity domain does not create a module cycle. + from app.modules.market.service import MarketService + + MarketService(self.db).claim_legacy_watchlist(record.id, record.open_id) + return FeishuPrincipal.from_user(record) + + def get_by_identity(self, *, tenant_key: str, open_id: str) -> FeishuUser | None: + return self.db.execute( + select(FeishuUser).where( + FeishuUser.tenant_key == tenant_key, + FeishuUser.open_id == open_id, + ) + ).scalar_one_or_none() + + def get_by_code(self, code: str) -> FeishuUser | None: + return self.db.execute( + select(FeishuUser).where(FeishuUser.code == code) + ).scalar_one_or_none() + + def _initial_role(self, tenant_key: str, open_id: str) -> str: + if (tenant_key, open_id) not in self.admin_identities: + return FeishuUserRole.USER + identity_hash = admin_bootstrap_identity_hash(tenant_key, open_id) + was_erased = self.db.scalar( + select(FeishuAdminBootstrapTombstone.id) + .where( + FeishuAdminBootstrapTombstone.identity_hash == identity_hash + ) + .limit(1) + ) + return FeishuUserRole.USER if was_erased is not None else FeishuUserRole.ADMIN + + +def _required_identity_part(value: Any) -> str: + text = str(value or "").strip() + if not text: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=INVALID_FEISHU_IDENTITY, + ) + return text + + +def _optional_identity_part(value: Any) -> str | None: + text = str(value or "").strip() + return text or None diff --git a/app/modules/feishu_users/services/management.py b/app/modules/feishu_users/services/management.py new file mode 100644 index 0000000..dea95be --- /dev/null +++ b/app/modules/feishu_users/services/management.py @@ -0,0 +1,290 @@ +from datetime import time +from typing import Any +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from fastapi import HTTPException, status +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.core.constants import ActorValue +from app.modules.audit.constants import AuditRiskLevel, AuditSource +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu_users.constants import ( + FEISHU_USER_NOT_FOUND, + FEISHU_USER_TARGET_TYPE, + INVALID_FEISHU_TIMEZONE, + INVALID_QUIET_HOURS, + LAST_ACTIVE_ADMIN_ERROR, + FeishuUserAuditAction, + FeishuUserRole, + FeishuUserStatus, +) +from app.modules.feishu_users.models import FeishuUser + +_UPDATABLE_FIELDS = frozenset( + { + "role", + "status", + "timezone", + "quiet_hours_start", + "quiet_hours_end", + } +) + + +class FeishuUserManagementService: + """Manage Feishu users while preserving an active administrator.""" + + def __init__(self, db: Session): + self.db = db + self.audit = AuditService(db) + + def list_users( + self, + *, + role: str | None = None, + status_filter: str | None = None, + tenant_key: str | None = None, + limit: int = 100, + offset: int = 0, + actor: str | None = None, + ) -> tuple[list[FeishuUser], int]: + filters = [] + if role: + filters.append(FeishuUser.role == FeishuUserRole(role)) + if status_filter: + filters.append(FeishuUser.status == FeishuUserStatus(status_filter)) + if tenant_key: + filters.append(FeishuUser.tenant_key == tenant_key) + total = int( + self.db.scalar( + select(func.count()).select_from(FeishuUser).where(*filters) + ) + or 0 + ) + items = list( + self.db.execute( + select(FeishuUser) + .where(*filters) + .order_by(FeishuUser.created_at.desc(), FeishuUser.id.desc()) + .offset(offset) + .limit(limit) + ).scalars() + ) + if actor: + self._audit_read( + actor=actor, + action=FeishuUserAuditAction.LIST, + response_payload={"count": len(items), "total": total}, + ) + return items, total + + def get_user(self, code: str, *, actor: str | None = None) -> FeishuUser: + record = self._find_user(code) + if actor: + self._audit_read( + actor=actor, + action=FeishuUserAuditAction.READ, + target_id=record.code, + ) + return record + + def update_user( + self, + code: str, + *, + changes: dict[str, Any], + actor: str = ActorValue.API, + ) -> FeishuUser: + unexpected = set(changes) - _UPDATABLE_FIELDS + if unexpected: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=f"Unsupported Feishu user fields: {', '.join(sorted(unexpected))}", + ) + active_admin_ids = ( + self._active_admin_ids() + if {"role", "status"} & changes.keys() + else [] + ) + record = self.db.execute( + select(FeishuUser) + .where(FeishuUser.code == code) + .with_for_update() + ).scalar_one_or_none() + if record is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=FEISHU_USER_NOT_FOUND, + ) + if not changes: + return record + + normalized = self._normalized_changes(record, changes) + proposed_role = normalized.get("role", record.role) + proposed_status = normalized.get("status", record.status) + removes_active_admin = ( + record.role == FeishuUserRole.ADMIN + and record.status == FeishuUserStatus.ACTIVE + and ( + proposed_role != FeishuUserRole.ADMIN + or proposed_status != FeishuUserStatus.ACTIVE + ) + ) + if removes_active_admin and active_admin_ids == [record.id]: + self.audit.record( + AuditLogCreate( + actor=actor, + source=AuditSource.API, + action=FeishuUserAuditAction.UPDATE_DENIED, + target_type=FEISHU_USER_TARGET_TYPE, + target_id=record.code, + risk_level=AuditRiskLevel.HIGH, + request_payload={ + "role": proposed_role, + "status": proposed_status, + }, + response_payload={ + "result": "denied", + "reason": "last_active_admin", + }, + status="denied", + ) + ) + self.db.commit() + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=LAST_ACTIVE_ADMIN_ERROR, + ) + + before = _auditable_state(record) + for field, value in normalized.items(): + setattr(record, field, value) + after = _auditable_state(record) + self.audit.record( + AuditLogCreate( + actor=actor, + source=AuditSource.API, + action=FeishuUserAuditAction.UPDATE, + target_type=FEISHU_USER_TARGET_TYPE, + target_id=record.code, + risk_level=AuditRiskLevel.HIGH, + request_payload={"before": before, "after": after}, + response_payload={"updated_fields": sorted(normalized)}, + ) + ) + self.db.commit() + self.db.refresh(record) + return record + + def _find_user(self, code: str) -> FeishuUser: + record = self.db.execute( + select(FeishuUser).where(FeishuUser.code == code) + ).scalar_one_or_none() + if record is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=FEISHU_USER_NOT_FOUND, + ) + return record + + def _normalized_changes( + self, + record: FeishuUser, + changes: dict[str, Any], + ) -> dict[str, Any]: + normalized = dict(changes) + if "role" in normalized: + if normalized["role"] is None: + raise _unprocessable("role cannot be null") + normalized["role"] = FeishuUserRole(normalized["role"]) + if "status" in normalized: + if normalized["status"] is None: + raise _unprocessable("status cannot be null") + normalized["status"] = FeishuUserStatus(normalized["status"]) + if "timezone" in normalized: + timezone = str(normalized["timezone"] or "").strip() + if not timezone: + raise _unprocessable(INVALID_FEISHU_TIMEZONE) + try: + ZoneInfo(timezone) + except ZoneInfoNotFoundError as exc: + raise _unprocessable(INVALID_FEISHU_TIMEZONE) from exc + normalized["timezone"] = timezone + for field in ("quiet_hours_start", "quiet_hours_end"): + if field in normalized: + normalized[field] = _optional_time(normalized[field]) + quiet_start = normalized.get("quiet_hours_start", record.quiet_hours_start) + quiet_end = normalized.get("quiet_hours_end", record.quiet_hours_end) + if (quiet_start is None) != (quiet_end is None): + raise _unprocessable(INVALID_QUIET_HOURS) + return normalized + + def _active_admin_ids(self) -> list[int]: + return list( + self.db.execute( + select(FeishuUser.id) + .where( + FeishuUser.role == FeishuUserRole.ADMIN, + FeishuUser.status == FeishuUserStatus.ACTIVE, + ) + .order_by(FeishuUser.id.asc()) + .with_for_update() + ).scalars() + ) + + def _audit_read( + self, + *, + actor: str, + action: str, + target_id: str | None = None, + response_payload: dict[str, Any] | None = None, + ) -> None: + self.audit.record( + AuditLogCreate( + actor=actor, + source=AuditSource.API, + action=action, + target_type=FEISHU_USER_TARGET_TYPE, + target_id=target_id, + risk_level=AuditRiskLevel.LOW, + response_payload=response_payload, + ) + ) + self.db.commit() + + +def _auditable_state(record: FeishuUser) -> dict[str, Any]: + return { + "role": record.role, + "status": record.status, + "timezone": record.timezone, + "quiet_hours_start": ( + record.quiet_hours_start.isoformat() + if record.quiet_hours_start is not None + else None + ), + "quiet_hours_end": ( + record.quiet_hours_end.isoformat() + if record.quiet_hours_end is not None + else None + ), + } + + +def _optional_time(value: Any) -> time | None: + if value is None or isinstance(value, time): + return value + try: + return time.fromisoformat(str(value)) + except ValueError as exc: + raise _unprocessable("Invalid quiet-hours time") from exc + + +def _unprocessable(detail: str) -> HTTPException: + return HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=detail, + ) diff --git a/app/modules/legacy_mysql/constants.py b/app/modules/legacy_mysql/constants.py index 147ea59..18f58f6 100644 --- a/app/modules/legacy_mysql/constants.py +++ b/app/modules/legacy_mysql/constants.py @@ -75,6 +75,8 @@ class LegacyQueryError(StrEnum): PROJECT_QUERY_NOT_CONFIGURED = "LEGACY_PROJECT_QUERY is not configured. Configure it first." TASK_QUERY_NOT_CONFIGURED = "LEGACY_TASK_QUERY is not configured. Configure it first." ONLY_SELECT_ALLOWED = "Only SELECT statements are allowed" + SQL_COMMENTS_NOT_ALLOWED = "SQL comments are not allowed in readonly queries" + SINGLE_STATEMENT_REQUIRED = "Only one SQL statement is allowed" FORBIDDEN_SQL_TOKEN = "Forbidden SQL token in readonly query" INVALID_LIMIT = "Invalid readonly query limit" APP_DB_UNAVAILABLE = "Application database session is not available" diff --git a/app/modules/legacy_mysql/routes.py b/app/modules/legacy_mysql/routes.py index a0ec768..d31a40d 100644 --- a/app/modules/legacy_mysql/routes.py +++ b/app/modules/legacy_mysql/routes.py @@ -4,7 +4,7 @@ from sqlalchemy.orm import Session from app.core.background.task_queue import enqueue_legacy_project_sync, enqueue_legacy_task_sync from app.core.database import get_db from app.core.http.masking import mask_configured -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.legacy_mysql.schemas import ( LegacyProjectSyncRequest, LegacyProjectSyncResult, @@ -60,7 +60,6 @@ def sync_projects( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() result = LegacyMySQLService(db).sync_projects( source_query=payload.source_query, source_query_name=payload.source_query_name, @@ -77,7 +76,6 @@ def enqueue_sync_projects( payload: LegacyProjectSyncRequest, principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return enqueue_legacy_project_sync( source_query=payload.source_query, source_query_name=payload.source_query_name, @@ -94,7 +92,6 @@ def sync_tasks( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() result = LegacyMySQLService(db).sync_tasks( source_query=payload.source_query, source_query_name=payload.source_query_name, @@ -111,7 +108,6 @@ def enqueue_sync_tasks( payload: LegacyTaskSyncRequest, principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return enqueue_legacy_task_sync( source_query=payload.source_query, source_query_name=payload.source_query_name, diff --git a/app/modules/legacy_mysql/services/common.py b/app/modules/legacy_mysql/services/common.py index 21abd6f..1514b46 100644 --- a/app/modules/legacy_mysql/services/common.py +++ b/app/modules/legacy_mysql/services/common.py @@ -7,6 +7,8 @@ from sqlalchemy.engine import RowMapping from app.modules.legacy_mysql.constants import LEGACY_SQL_TRAILING_TERMINATOR, LegacyQueryName FORBIDDEN_SQL_TOKENS = { + "benchmark", + "call", "insert", "update", "delete", @@ -14,9 +16,21 @@ FORBIDDEN_SQL_TOKENS = { "alter", "truncate", "create", + "do", + "dumpfile", + "execute", "replace", "grant", + "get_lock", + "handler", + "into", + "load_file", + "lock", + "outfile", + "release_lock", "revoke", + "set", + "sleep", } diff --git a/app/modules/legacy_mysql/services/project_sync.py b/app/modules/legacy_mysql/services/project_sync.py index 1d0eb91..d981174 100644 --- a/app/modules/legacy_mysql/services/project_sync.py +++ b/app/modules/legacy_mysql/services/project_sync.py @@ -4,7 +4,6 @@ from fastapi import HTTPException, status from sqlalchemy import select from app.core.constants import ActorValue -from app.core.security import ensure_business_mutations_enabled from app.core.utils.time import utc_now from app.modules.audit.constants import AuditAction, AuditRiskLevel, AuditSource, AuditStatus from app.modules.audit.schemas import AuditLogCreate @@ -44,7 +43,6 @@ class LegacyProjectSyncMixin: dry_run: bool = True, actor: str = ActorValue.API, ) -> dict[str, Any]: - ensure_business_mutations_enabled() if self.db is None: raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, diff --git a/app/modules/legacy_mysql/services/query.py b/app/modules/legacy_mysql/services/query.py index db3a844..57e58ba 100644 --- a/app/modules/legacy_mysql/services/query.py +++ b/app/modules/legacy_mysql/services/query.py @@ -1,3 +1,4 @@ +import re from typing import Any from fastapi import HTTPException, status @@ -23,6 +24,13 @@ from app.modules.legacy_mysql.constants import ( from app.modules.legacy_mysql.services.common import FORBIDDEN_SQL_TOKENS, _normalize_sql, _query_name_text, _row_to_dict +_SQL_COMMENT_MARKERS = ("--", "#", "/*", "*/") +_SQL_QUOTED_CONTENT_PATTERN = re.compile( + r"""'(?:''|\\.|[^'])*'|"(?:""|\\.|[^"])*"|`(?:``|[^`])*`""", + flags=re.DOTALL, +) +_SQL_WORD_PATTERN = re.compile(r"[a-z_]+") + class LegacyQueryMixin: @staticmethod @@ -36,13 +44,27 @@ class LegacyQueryMixin: @staticmethod def _ensure_readonly(sql: str) -> None: - stripped = sql.strip().lower() - if not stripped.startswith(LEGACY_SELECT_PREFIX): + stripped = sql.strip() + scrubbed = _SQL_QUOTED_CONTENT_PATTERN.sub(" ", stripped) + if any(marker in scrubbed for marker in _SQL_COMMENT_MARKERS): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=LegacyQueryError.SQL_COMMENTS_NOT_ALLOWED, + ) + statement = scrubbed.rstrip() + if statement.endswith(LEGACY_SQL_TRAILING_TERMINATOR): + statement = statement[:-1].rstrip() + if LEGACY_SQL_TRAILING_TERMINATOR in statement: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=LegacyQueryError.SINGLE_STATEMENT_REQUIRED, + ) + if not re.match(rf"^{LEGACY_SELECT_PREFIX}\b", statement, flags=re.IGNORECASE): raise HTTPException( status_code=status.HTTP_400_BAD_REQUEST, detail=LegacyQueryError.ONLY_SELECT_ALLOWED, ) - tokens = {token.strip(" ;,\n\t") for token in stripped.replace("(", " ").split()} + tokens = set(_SQL_WORD_PATTERN.findall(statement.lower())) if tokens & FORBIDDEN_SQL_TOKENS: raise HTTPException( status_code=status.HTTP_400_BAD_REQUEST, @@ -117,9 +139,10 @@ class LegacyQueryMixin: engine = self._ensure_engine() params = dict(params or {}) try: - params[LegacyResponseKey.LIMIT] = bounded_limit( + limit_value = bounded_limit( params.get(LegacyResponseKey.LIMIT, limit) ) + params[LegacyResponseKey.LIMIT] = limit_value except (TypeError, ValueError) as exc: raise HTTPException( status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, @@ -130,7 +153,10 @@ class LegacyQueryMixin: limited_sql = f"{sql.rstrip(LEGACY_SQL_TRAILING_TERMINATOR)}{LEGACY_LIMIT_CLAUSE}" with engine.connect() as conn: result = conn.execute(text(limited_sql), params) - rows = [_row_to_dict(row) for row in result.mappings().all()] + rows = [ + _row_to_dict(row) + for row in result.mappings().fetchmany(limit_value) + ] columns = list(rows[0].keys()) if rows else [] return { LegacyResponseKey.COLUMNS: columns, diff --git a/app/modules/legacy_mysql/services/task_sync.py b/app/modules/legacy_mysql/services/task_sync.py index 6581f5b..6daab10 100644 --- a/app/modules/legacy_mysql/services/task_sync.py +++ b/app/modules/legacy_mysql/services/task_sync.py @@ -4,7 +4,6 @@ from fastapi import HTTPException, status from sqlalchemy import select from app.core.constants import ActorValue -from app.core.security import ensure_business_mutations_enabled from app.core.utils.time import utc_now from app.modules.audit.constants import AuditAction, AuditRiskLevel, AuditSource, AuditStatus from app.modules.audit.schemas import AuditLogCreate @@ -44,7 +43,6 @@ class LegacyTaskSyncMixin: dry_run: bool = True, actor: str = ActorValue.API, ) -> dict[str, Any]: - ensure_business_mutations_enabled() if self.db is None: raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, diff --git a/app/modules/market/routes.py b/app/modules/market/routes.py index 3fccbcf..8429e12 100644 --- a/app/modules/market/routes.py +++ b/app/modules/market/routes.py @@ -7,7 +7,7 @@ from sqlalchemy.orm import Session from app.core.database import get_db from app.core.background.task_queue.market import enqueue_market_report -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.market.service import MarketService router = APIRouter(dependencies=[Depends(require_api_key)]) @@ -89,13 +89,11 @@ def announcements( @router.post("/sync/daily") def sync_daily(trade_date: date, db: Session = Depends(get_db)) -> dict: - require_operations_enabled() return MarketService(db).sync_daily(trade_date) @router.post("/sync/macro") def sync_macro(reference_date: date | None = None, db: Session = Depends(get_db)) -> dict: - require_operations_enabled() return MarketService(db).sync_macro(reference_date) @@ -103,13 +101,11 @@ def sync_macro(reference_date: date | None = None, db: Session = Depends(get_db) def sync_announcements( start_date: date, end_date: date, db: Session = Depends(get_db) ) -> dict: - require_operations_enabled() return {"processed": MarketService(db).sync_announcements(start_date, end_date)} @router.post("/reports/enqueue") def enqueue_report(payload: MarketReportRequest) -> dict: - require_operations_enabled() return enqueue_market_report(payload.report_type, payload.reference_date, payload.force) @@ -119,7 +115,6 @@ def add_watchlist( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return MarketService(db).add_watchlist(principal.actor, payload.symbol) diff --git a/app/modules/market/service.py b/app/modules/market/service.py index d47717c..65892d7 100644 --- a/app/modules/market/service.py +++ b/app/modules/market/service.py @@ -780,15 +780,31 @@ class MarketService: report["content"] = "\n".join(lines) return report - def add_watchlist(self, actor: str, symbol: str) -> dict[str, Any]: + def add_watchlist( + self, + actor: str, + symbol: str, + owner_id: int | None = None, + ) -> dict[str, Any]: code = normalize_symbol(symbol) + owner_clause = ( + MarketWatchlist.owner_id.is_(None) + if owner_id is None + else MarketWatchlist.owner_id == owner_id + ) record = self.db.execute( select(MarketWatchlist).where( - MarketWatchlist.actor == actor, MarketWatchlist.symbol == code + owner_clause, + MarketWatchlist.symbol == code, + *( + (MarketWatchlist.actor == actor,) + if owner_id is None + else () + ), ) ).scalar_one_or_none() if record is None: - record = MarketWatchlist(actor=actor, symbol=code) + record = MarketWatchlist(owner_id=owner_id, actor=actor, symbol=code) self.db.add(record) else: record.enabled = True @@ -804,16 +820,77 @@ class MarketService: response_payload={"enabled": True}, ) ) - return {"actor": actor, "symbol": code, "enabled": True} + return { + "actor": actor, + "owner_id": owner_id, + "symbol": code, + "enabled": True, + } - def watchlist(self, actor: str) -> list[dict[str, Any]]: + def watchlist( + self, + actor: str, + owner_id: int | None = None, + ) -> list[dict[str, Any]]: + owner_clause = ( + MarketWatchlist.owner_id.is_(None) + if owner_id is None + else MarketWatchlist.owner_id == owner_id + ) records = self.db.execute( select(MarketWatchlist).where( - MarketWatchlist.actor == actor, MarketWatchlist.enabled.is_(True) + owner_clause, + MarketWatchlist.enabled.is_(True), + *( + (MarketWatchlist.actor == actor,) + if owner_id is None + else () + ), ) ).scalars() return [{"symbol": r.symbol} for r in records] + def claim_legacy_watchlist(self, owner_id: int, open_id: str) -> int: + """Claim still-unowned rows created by the verified legacy Feishu actor.""" + + legacy_records = list( + self.db.execute( + select(MarketWatchlist).where( + MarketWatchlist.owner_id.is_(None), + MarketWatchlist.actor == open_id, + ) + ).scalars() + ) + claimed = 0 + for legacy in legacy_records: + existing = self.db.execute( + select(MarketWatchlist).where( + MarketWatchlist.owner_id == owner_id, + MarketWatchlist.symbol == legacy.symbol, + ) + ).scalar_one_or_none() + if existing is not None: + existing.enabled = existing.enabled or legacy.enabled + self.db.delete(legacy) + continue + legacy.owner_id = owner_id + claimed += 1 + self.db.commit() + return claimed + + def delete_owner_watchlist(self, owner_id: int) -> int: + """Stage deletion of all personal watchlist rows for an owner.""" + + records = list( + self.db.execute( + select(MarketWatchlist).where(MarketWatchlist.owner_id == owner_id) + ).scalars() + ) + for record in records: + self.db.delete(record) + self.db.flush() + return len(records) + def _ai(self, skill: AISkillId, report: dict[str, Any], actor: str) -> dict[str, Any]: try: result = AIService(self.db).run_skill( diff --git a/app/modules/observability/models.py b/app/modules/observability/models.py index d5c48e5..8592143 100644 --- a/app/modules/observability/models.py +++ b/app/modules/observability/models.py @@ -1,6 +1,6 @@ from datetime import datetime -from sqlalchemy import DateTime, Integer, String +from sqlalchemy import DateTime, Integer, String, UniqueConstraint from sqlalchemy.orm import Mapped, mapped_column from app.core.database import Base @@ -9,6 +9,9 @@ from app.core.utils.time import utc_now class SystemHeartbeat(Base): __tablename__ = "system_heartbeats" + __table_args__ = ( + UniqueConstraint("component", "instance_id", name="uq_system_heartbeat_component_instance"), + ) id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) component: Mapped[str] = mapped_column(String(128), index=True) diff --git a/app/modules/observability/routes.py b/app/modules/observability/routes.py index fe91fc3..88372d7 100644 --- a/app/modules/observability/routes.py +++ b/app/modules/observability/routes.py @@ -32,4 +32,6 @@ def ready( def metrics( db: Session = Depends(get_db), ) -> dict: - return ObservabilityService(db).metrics() + result = ObservabilityService(db).metrics() + db.commit() + return result diff --git a/app/modules/observability/service.py b/app/modules/observability/service.py index 8dff83a..f5b63db 100644 --- a/app/modules/observability/service.py +++ b/app/modules/observability/service.py @@ -1,7 +1,10 @@ +from collections.abc import Callable from datetime import timedelta from typing import Any -from sqlalchemy import select, text +from sqlalchemy import and_, func, or_, select, text +from sqlalchemy.dialects.postgresql import insert as postgresql_insert +from sqlalchemy.dialects.sqlite import insert as sqlite_insert from sqlalchemy.orm import Session from app.core.config import get_settings @@ -18,6 +21,10 @@ from app.modules.audit.schemas import AuditLogCreate from app.modules.audit.service import AuditService from app.modules.events.constants import EventStatus from app.modules.events.services import EventService +from app.modules.feishu.app_tickets import FeishuAppTicketService +from app.modules.feishu.constants import FeishuAppType +from app.modules.feishu_users.constants import FeishuUserStatus +from app.modules.feishu_users.models import FeishuUser from app.modules.observability.constants import ( HeartbeatStatus, ObservabilityKey, @@ -27,6 +34,11 @@ from app.modules.observability.constants import ( from app.modules.observability.models import SystemHeartbeat from app.modules.workflows.constants import WorkflowStatus from app.modules.workflows.service import WorkflowService +from app.modules.subscriptions.constants import ( + PushDeliveryStatus, + PushSubscriptionStatus, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription class ObservabilityService: @@ -40,31 +52,42 @@ class ObservabilityService: def ready(self) -> dict[str, Any]: checks = { - ObservabilityKey.DATABASE: self._database_check(), - ObservabilityKey.REDIS: self._redis_check(), - ObservabilityKey.EVENTS: self._events_check(), - ObservabilityKey.WORKFLOWS: self._workflows_check(), - ObservabilityKey.HEARTBEATS: self._heartbeats_check(), - } - degraded = any( - item[ObservabilityKey.STATUS] - in {ObservabilityStatus.DEGRADED, ObservabilityStatus.ERROR} - for item in checks.values() - ) - return { - ObservabilityKey.STATUS: ( - ObservabilityStatus.DEGRADED if degraded else ObservabilityStatus.OK + ObservabilityKey.DATABASE: self._safe_call(self._database_check), + ObservabilityKey.REDIS: self._safe_call(self._redis_check), + ObservabilityKey.EVENTS: self._safe_call(self._events_check), + ObservabilityKey.WORKFLOWS: self._safe_call(self._workflows_check), + ObservabilityKey.HEARTBEATS: self._safe_call(self._heartbeats_check), + "feishu_subscriptions": self._safe_call( + self._feishu_subscriptions_check ), + } + statuses = {item[ObservabilityKey.STATUS] for item in checks.values()} + if statuses & {ObservabilityStatus.ERROR, ObservabilityStatus.DEGRADED}: + overall_status = ObservabilityStatus.DEGRADED + else: + overall_status = ObservabilityStatus.OK + return { + ObservabilityKey.STATUS: overall_status, ObservabilityKey.CHECKS: checks, } def metrics(self) -> dict[str, Any]: return { ObservabilityKey.METRICS: { - ObservabilityKey.EVENTS: EventService(self.db).count_by_status(), - ObservabilityKey.WORKFLOWS: WorkflowService(self.db).count_by_status(), - ObservabilityKey.AI_MEMORY: AIMemoryService(self.db).count_by_status(), - ObservabilityKey.HEARTBEATS: self.heartbeat_summary(), + ObservabilityKey.EVENTS: self._safe_call( + lambda: EventService(self.db).count_by_status() + ), + ObservabilityKey.WORKFLOWS: self._safe_call( + lambda: WorkflowService(self.db).count_by_status() + ), + ObservabilityKey.AI_MEMORY: self._safe_call( + lambda: AIMemoryService(self.db).count_by_status() + ), + ObservabilityKey.HEARTBEATS: self._safe_call( + self.heartbeat_summary + ), + "feishu_users": self._safe_call(self._feishu_user_metrics), + "subscriptions": self._safe_call(self._subscription_metrics), } } @@ -76,24 +99,7 @@ class ObservabilityService: actor: str = ActorValue.SYSTEM, ) -> dict[str, Any]: now = utc_now() - record = self.db.execute( - select(SystemHeartbeat).where( - SystemHeartbeat.component == component, - SystemHeartbeat.instance_id == instance_id, - ) - ).scalar_one_or_none() - if record is None: - record = SystemHeartbeat( - component=component, - instance_id=instance_id, - status=status_value, - last_seen_at=now, - ) - self.db.add(record) - else: - record.status = status_value - record.last_seen_at = now - record.updated_at = now + record = self._upsert_heartbeat(component, instance_id, status_value, now) AuditService(self.db).record( AuditLogCreate( actor=actor, @@ -115,7 +121,13 @@ class ObservabilityService: return self._serialize_heartbeat(record) def heartbeat_summary(self) -> dict[str, Any]: - records = list(self.db.execute(select(SystemHeartbeat)).scalars()) + records = list( + self.db.execute( + select(SystemHeartbeat) + .where(SystemHeartbeat.last_seen_at >= self._heartbeat_retention_threshold()) + .order_by(SystemHeartbeat.component.asc(), SystemHeartbeat.instance_id.asc()) + ).scalars() + ) threshold = self._heartbeat_stale_threshold() stale = [item for item in records if item.last_seen_at < threshold] active = len(records) - len(stale) @@ -132,31 +144,75 @@ class ObservabilityService: ], } - def _database_check(self) -> dict[str, Any]: + def _safe_call(self, operation: Callable[[], Any]) -> Any: try: - self.db.execute(text("select 1")).scalar() - except Exception as exc: + return operation() + except Exception: + self.db.rollback() return { ObservabilityKey.STATUS: ObservabilityStatus.ERROR, - ObservabilityMetricKey.ERROR: str(exc), + ObservabilityMetricKey.ERROR: "unavailable", } + + def _database_check(self) -> dict[str, Any]: + self.db.execute(text("select 1")).scalar() return {ObservabilityKey.STATUS: ObservabilityStatus.OK} def _redis_check(self) -> dict[str, Any]: settings = get_settings() if not settings.task_queue_enabled: return {ObservabilityKey.STATUS: ObservabilityStatus.SKIPPED} - try: - from redis import Redis + from redis import Redis - Redis.from_url(settings.redis_url, socket_connect_timeout=1).ping() - except Exception as exc: - return { - ObservabilityKey.STATUS: ObservabilityStatus.DEGRADED, - ObservabilityMetricKey.ERROR: str(exc), - } + client = Redis.from_url( + settings.redis_url, + socket_connect_timeout=1, + socket_timeout=1, + ) + try: + client.ping() + finally: + client.close() return {ObservabilityKey.STATUS: ObservabilityStatus.OK} + def _upsert_heartbeat( + self, + component: str, + instance_id: str, + status_value: str, + now: Any, + ) -> SystemHeartbeat: + dialect_name = self.db.get_bind().dialect.name + insert_factory = { + "postgresql": postgresql_insert, + "sqlite": sqlite_insert, + }.get(dialect_name) + if insert_factory is None: + raise RuntimeError(f"Unsupported heartbeat database dialect: {dialect_name}") + statement = insert_factory(SystemHeartbeat).values( + component=component, + instance_id=instance_id, + status=status_value, + last_seen_at=now, + created_at=now, + updated_at=now, + ) + statement = statement.on_conflict_do_update( + index_elements=["component", "instance_id"], + set_={ + "status": status_value, + "last_seen_at": now, + "updated_at": now, + }, + ) + self.db.execute(statement) + return self.db.execute( + select(SystemHeartbeat).where( + SystemHeartbeat.component == component, + SystemHeartbeat.instance_id == instance_id, + ) + ).scalar_one() + def _events_check(self) -> dict[str, Any]: counts = EventService(self.db).count_by_status() failed = counts.get(EventStatus.FAILED, 0) @@ -196,6 +252,134 @@ class ObservabilityService: ], } + def _feishu_subscriptions_check(self) -> dict[str, Any]: + active = int( + self.db.scalar( + select(func.count()) + .select_from(PushSubscription) + .where(PushSubscription.status == PushSubscriptionStatus.ACTIVE) + ) + or 0 + ) + processable_delivery_filter = or_( + and_( + PushDelivery.status.in_( + [ + PushDeliveryStatus.PENDING, + PushDeliveryStatus.RETRY, + ] + ), + PushDelivery.next_attempt_at.is_not(None), + ), + and_( + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_until.is_not(None), + ), + ) + processable_deliveries = int( + self.db.scalar( + select(func.count()) + .select_from(PushDelivery) + .where(processable_delivery_filter) + ) + or 0 + ) + if not active and not processable_deliveries: + return { + ObservabilityKey.STATUS: ObservabilityStatus.SKIPPED, + "active": 0, + "processable_deliveries": 0, + } + settings = get_settings() + app_id = str(settings.feishu_app_id or "").strip() + credentials_configured = bool(app_id and settings.feishu_app_secret) + active_tenant_count = int( + self.db.scalar( + select(func.count(func.distinct(FeishuUser.tenant_key))) + .select_from(PushSubscription) + .join(FeishuUser, FeishuUser.id == PushSubscription.owner_id) + .outerjoin( + PushDelivery, + PushDelivery.subscription_id == PushSubscription.id, + ) + .where( + or_( + PushSubscription.status == PushSubscriptionStatus.ACTIVE, + processable_delivery_filter, + ) + ) + ) + or 0 + ) + ticket_configured = False + default_tenant_configured = bool( + str(settings.feishu_default_tenant_key or "").strip() + ) + reasons: list[str] = [] + if not credentials_configured: + reasons.append("credentials_missing") + if settings.feishu_app_type == FeishuAppType.STORE: + database_ticket = ( + FeishuAppTicketService(self.db).get_ticket(app_id) + if app_id + else None + ) + ticket_configured = bool( + str(database_ticket or settings.feishu_app_ticket or "").strip() + ) + if not ticket_configured: + reasons.append("app_ticket_missing") + if not default_tenant_configured: + reasons.append("default_tenant_missing") + elif active_tenant_count > 1: + reasons.append("self_app_multiple_tenants") + return { + ObservabilityKey.STATUS: ( + ObservabilityStatus.OK + if not reasons + else ObservabilityStatus.DEGRADED + ), + "active": active, + "processable_deliveries": processable_deliveries, + "app_type": settings.feishu_app_type, + "credentials_configured": credentials_configured, + "ticket_configured": ticket_configured, + "default_tenant_configured": default_tenant_configured, + "active_tenant_count": active_tenant_count, + "reasons": reasons, + } + + def _feishu_user_metrics(self) -> dict[str, int]: + active = int( + self.db.scalar( + select(func.count()) + .select_from(FeishuUser) + .where(FeishuUser.status == FeishuUserStatus.ACTIVE) + ) + or 0 + ) + return {"active": active} + + def _subscription_metrics(self) -> dict[str, int]: + active = int( + self.db.scalar( + select(func.count()) + .select_from(PushSubscription) + .where(PushSubscription.status == PushSubscriptionStatus.ACTIVE) + ) + or 0 + ) + delivery_rows = self.db.execute( + select(PushDelivery.status, func.count()).group_by(PushDelivery.status) + ).all() + deliveries = {str(status_value): int(count) for status_value, count in delivery_rows} + return { + "active": active, + "pending": deliveries.get(PushDeliveryStatus.PENDING, 0) + + deliveries.get(PushDeliveryStatus.RETRY, 0), + "failed": deliveries.get(PushDeliveryStatus.FAILED, 0), + } + @staticmethod def _serialize_heartbeat(record: SystemHeartbeat) -> dict[str, Any]: return { @@ -209,3 +393,12 @@ class ObservabilityService: def _heartbeat_stale_threshold() -> Any: settings = get_settings() return utc_now() - timedelta(seconds=settings.heartbeat_interval_seconds * 3) + + @staticmethod + def _heartbeat_retention_threshold() -> Any: + settings = get_settings() + retention_seconds = max( + settings.heartbeat_retention_seconds, + settings.heartbeat_interval_seconds * 3, + ) + return utc_now() - timedelta(seconds=retention_seconds) diff --git a/app/modules/personalization/__init__.py b/app/modules/personalization/__init__.py new file mode 100644 index 0000000..b71943e --- /dev/null +++ b/app/modules/personalization/__init__.py @@ -0,0 +1,23 @@ +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + UserPreference, +) +from app.modules.personalization.services import ( + ConversationService, + PersonalDataErasureService, + PersonalizationContextService, + PreferenceService, +) + +__all__ = [ + "AIConversation", + "AIConversationMessage", + "ConversationService", + "PersonalDataErasureRequest", + "PersonalDataErasureService", + "PersonalizationContextService", + "PreferenceService", + "UserPreference", +] diff --git a/app/modules/personalization/constants.py b/app/modules/personalization/constants.py new file mode 100644 index 0000000..1cd9a2b --- /dev/null +++ b/app/modules/personalization/constants.py @@ -0,0 +1,101 @@ +from enum import StrEnum + + +class PreferenceCategory(StrEnum): + LANGUAGE = "language" + TONE = "tone" + DETAIL = "detail" + TOPIC = "topic" + INTEREST = "interest" + + +class PreferenceSource(StrEnum): + EXPLICIT = "explicit" + AUTO = "auto" + + +class ConversationRole(StrEnum): + USER = "user" + ASSISTANT = "assistant" + + +class ConversationChatType(StrEnum): + PRIVATE = "private" + GROUP = "group" + + +class PersonalizationContextKey(StrEnum): + SYSTEM_CONSTRAINTS = "system_constraints" + COMPANY_RULES = "company_rules" + PERSONAL_RULES = "personal_rules" + CURRENT_REQUEST = "current_request" + PREFERENCES = "preferences" + INTERESTS = "interests" + PERSONAL_MEMORY = "personal_memory" + CONVERSATION_HISTORY = "conversation_history" + + +PREFERENCE_CODE_PREFIX = "PREF" +CONVERSATION_CODE_PREFIX = "CONV" +PREFERENCE_MAX_VALUE_LENGTH = 1000 +CONVERSATION_MAX_CONTENT_LENGTH = 20_000 +CONVERSATION_RETENTION_DAYS = 30 +CONVERSATION_MAX_TURNS = 20 +CONVERSATION_MAX_MESSAGES = CONVERSATION_MAX_TURNS * 2 +ERASURE_CONFIRMATION_TTL_MINUTES = 10 + +UNAVAILABLE_AI_PROVIDERS = frozenset({"", "noop"}) +PREFERENCE_SIGNAL_TERMS = ( + "以后", + "记住", + "偏好", + "喜欢", + "希望", + "请用", + "请保持", + "关注", + "感兴趣", + "prefer", + "preference", + "i like", + "interested in", +) + +# These terms identify categories that must never become an inferred personal profile. +# General topics such as public market news remain allowed; the financial terms below are +# intentionally limited to private account, compensation, and confidential-company facts. +SENSITIVE_PREFERENCE_TERMS = ( + "api key", + "api_key", + "access token", + "access_token", + "password", + "secret", + "token", + "密码", + "密钥", + "令牌", + "健康", + "病史", + "疾病", + "诊断", + "医疗记录", + "宗教", + "信仰", + "政治立场", + "党派", + "选举倾向", + "性取向", + "同性恋", + "异性恋", + "绩效", + "考核结果", + "银行账号", + "银行卡", + "工资", + "薪资", + "个人收入", + "财务秘密", + "未公开财务", + "保密预算", +) diff --git a/app/modules/personalization/models.py b/app/modules/personalization/models.py new file mode 100644 index 0000000..5e08bba --- /dev/null +++ b/app/modules/personalization/models.py @@ -0,0 +1,131 @@ +from datetime import datetime +from uuid import uuid4 + +from sqlalchemy import DateTime, ForeignKey, Integer, String, Text, UniqueConstraint +from sqlalchemy.orm import Mapped, mapped_column, relationship + +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.feishu_users.models import FeishuUser +from app.modules.personalization.constants import ( + CONVERSATION_CODE_PREFIX, + PREFERENCE_CODE_PREFIX, + PreferenceSource, +) + + +def _public_code(prefix: str) -> str: + return f"{prefix}-{uuid4().hex}" + + +class UserPreference(Base): + __tablename__ = "user_preferences" + __table_args__ = ( + UniqueConstraint( + "owner_id", + "category", + "normalized_value", + name="uq_user_preference_owner_category_value", + ), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column( + String(64), + default=lambda: _public_code(PREFERENCE_CODE_PREFIX), + unique=True, + index=True, + ) + owner_id: Mapped[int] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + index=True, + ) + owner: Mapped[FeishuUser] = relationship() + category: Mapped[str] = mapped_column(String(32), index=True) + value: Mapped[str] = mapped_column(Text) + normalized_value: Mapped[str] = mapped_column(String(1000)) + source: Mapped[str] = mapped_column( + String(32), + default=PreferenceSource.EXPLICIT, + index=True, + ) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) + + +class AIConversation(Base): + __tablename__ = "ai_conversations" + __table_args__ = ( + UniqueConstraint( + "owner_id", + "chat_type", + "chat_key", + name="uq_ai_conversation_owner_chat", + ), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column( + String(64), + default=lambda: _public_code(CONVERSATION_CODE_PREFIX), + unique=True, + index=True, + ) + owner_id: Mapped[int] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + index=True, + ) + owner: Mapped[FeishuUser] = relationship() + chat_type: Mapped[str] = mapped_column(String(32), index=True) + chat_key: Mapped[str] = mapped_column(String(256), index=True) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + index=True, + ) + messages: Mapped[list["AIConversationMessage"]] = relationship( + back_populates="conversation", + cascade="all, delete-orphan", + passive_deletes=True, + order_by="AIConversationMessage.id", + ) + + +class AIConversationMessage(Base): + __tablename__ = "ai_conversation_messages" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + conversation_id: Mapped[int] = mapped_column( + ForeignKey("ai_conversations.id", ondelete="CASCADE"), + index=True, + ) + role: Mapped[str] = mapped_column(String(32), index=True) + content: Mapped[str] = mapped_column(Text) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + conversation: Mapped[AIConversation] = relationship(back_populates="messages") + + +class PersonalDataErasureRequest(Base): + __tablename__ = "personal_data_erasure_requests" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + owner_id: Mapped[int] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + unique=True, + index=True, + ) + owner: Mapped[FeishuUser] = relationship() + token_hash: Mapped[str] = mapped_column(String(64)) + expires_at: Mapped[datetime] = mapped_column(DateTime, index=True) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) diff --git a/app/modules/personalization/schemas.py b/app/modules/personalization/schemas.py new file mode 100644 index 0000000..365e659 --- /dev/null +++ b/app/modules/personalization/schemas.py @@ -0,0 +1,65 @@ +from datetime import datetime +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from app.modules.personalization.constants import PreferenceCategory, PreferenceSource + + +class PreferenceCreate(BaseModel): + category: PreferenceCategory + value: str = Field(..., min_length=1, max_length=1000) + source: PreferenceSource = PreferenceSource.EXPLICIT + + +class PreferenceUpdate(BaseModel): + category: PreferenceCategory | None = None + value: str | None = Field(default=None, min_length=1, max_length=1000) + + +class PreferenceRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + code: str + category: str + value: str + source: str + created_at: datetime + updated_at: datetime + + +class ExtractedPreference(BaseModel): + category: PreferenceCategory + value: str = Field(..., min_length=1, max_length=1000) + + +class PreferenceExtractionPayload(BaseModel): + preferences: list[ExtractedPreference] = Field(default_factory=list, max_length=20) + + +class ConversationMessageRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + role: str + content: str + created_at: datetime + + +class ConversationRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + code: str + chat_type: str + chat_key: str + messages: list[ConversationMessageRead] = Field(default_factory=list) + + +class ErasureConfirmation(BaseModel): + confirmation_code: str + expires_at: datetime + + +class ErasureResult(BaseModel): + anonymous_id: str + deleted: dict[str, int] = Field(default_factory=dict) + extra: dict[str, Any] = Field(default_factory=dict) diff --git a/app/modules/personalization/services/__init__.py b/app/modules/personalization/services/__init__.py new file mode 100644 index 0000000..2b8c204 --- /dev/null +++ b/app/modules/personalization/services/__init__.py @@ -0,0 +1,15 @@ +from app.modules.personalization.services.context import ( + PersonalizationContext, + PersonalizationContextService, +) +from app.modules.personalization.services.conversations import ConversationService +from app.modules.personalization.services.erasure import PersonalDataErasureService +from app.modules.personalization.services.preferences import PreferenceService + +__all__ = [ + "ConversationService", + "PersonalDataErasureService", + "PersonalizationContext", + "PersonalizationContextService", + "PreferenceService", +] diff --git a/app/modules/personalization/services/context.py b/app/modules/personalization/services/context.py new file mode 100644 index 0000000..1f322ad --- /dev/null +++ b/app/modules/personalization/services/context.py @@ -0,0 +1,191 @@ +from dataclasses import dataclass, field +from typing import Any + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.core.constants import ActorValue +from app.modules.ai_memory.constants import AIMemoryScope +from app.modules.ai_memory.service import AIMemoryService +from app.modules.business.models import MarketWatchlist +from app.modules.personalization.constants import ( + PersonalizationContextKey, + PreferenceCategory, +) +from app.modules.personalization.services.conversations import ConversationService +from app.modules.personalization.services.preferences import PreferenceService + + +@dataclass(frozen=True) +class PersonalizationContext: + """Ordered, provider-neutral context sections for one AI request.""" + + system_constraints: str + company_rules: list[dict[str, Any]] + personal_rules: list[dict[str, Any]] + current_request: str + preferences: list[dict[str, Any]] + interests: list[dict[str, Any]] + personal_memory: list[dict[str, Any]] + conversation_history: list[dict[str, Any]] + provider_session_id: str | None = field(default=None) + + def as_ordered_dict(self) -> dict[str, Any]: + return { + PersonalizationContextKey.SYSTEM_CONSTRAINTS: self.system_constraints, + PersonalizationContextKey.COMPANY_RULES: self.company_rules, + PersonalizationContextKey.PERSONAL_RULES: self.personal_rules, + PersonalizationContextKey.CURRENT_REQUEST: self.current_request, + PersonalizationContextKey.PREFERENCES: self.preferences, + PersonalizationContextKey.INTERESTS: self.interests, + PersonalizationContextKey.PERSONAL_MEMORY: self.personal_memory, + PersonalizationContextKey.CONVERSATION_HISTORY: self.conversation_history, + } + + +class PersonalizationContextService: + """Load only the explicitly requested company and owner-scoped context layers.""" + + def __init__(self, db: Session): + self.db = db + self.memory = AIMemoryService(db) + self.preferences = PreferenceService(db) + self.conversations = ConversationService(db) + + def build( + self, + *, + owner_id: int | None, + request: str, + system_constraints: str, + chat_type: str | None = None, + chat_key: str | None = None, + scope: str = AIMemoryScope.GLOBAL, + subject: str | None = None, + actor: str = ActorValue.SYSTEM, + include_company_rules: bool = True, + include_personal_context: bool = True, + include_history: bool = True, + ) -> PersonalizationContext: + company_rules = ( + self.memory.active_rules( + scope=scope, + subject=subject, + owner_id=None, + ) + if include_company_rules + else [] + ) + personal_rules: list[dict[str, Any]] = [] + preference_items: list[dict[str, Any]] = [] + interests: list[dict[str, Any]] = [] + personal_memory: list[dict[str, Any]] = [] + history: list[dict[str, Any]] = [] + provider_session_id: str | None = None + + if owner_id is not None and include_personal_context: + personal_rules = self.memory.active_rules( + scope=scope, + subject=subject, + owner_id=owner_id, + ) + all_preferences = self.preferences.list_preferences(owner_id) + interest_categories = { + PreferenceCategory.TOPIC, + PreferenceCategory.INTEREST, + } + for preference in all_preferences: + if preference["category"] in interest_categories: + interests.append(preference) + else: + preference_items.append(preference) + interests.extend(self._watchlist_interests(owner_id)) + personal_memory = self.memory.recall( + query=request, + scope=scope, + subject=subject, + actor=actor, + owner_id=owner_id, + ) + if include_history and chat_type and chat_key: + history = self.conversations.history(owner_id, chat_type, chat_key) + provider_session_id = self.conversations.provider_session_id( + owner_id, + chat_type, + chat_key, + ) + + return PersonalizationContext( + system_constraints=system_constraints, + company_rules=company_rules, + personal_rules=personal_rules, + current_request=request, + preferences=preference_items, + interests=interests, + personal_memory=personal_memory, + conversation_history=history, + provider_session_id=provider_session_id, + ) + + def build_private_scheduled( + self, + *, + owner_id: int, + request: str, + system_constraints: str, + scope: str = AIMemoryScope.GLOBAL, + subject: str | None = None, + actor: str = ActorValue.SYSTEM, + ) -> PersonalizationContext: + """Build private scheduled context without company data or conversation history.""" + + return self.build( + owner_id=owner_id, + request=request, + system_constraints=system_constraints, + scope=scope, + subject=subject, + actor=actor, + include_company_rules=False, + include_personal_context=True, + include_history=False, + ) + + def build_group_scheduled( + self, + *, + request: str, + system_constraints: str, + scope: str = AIMemoryScope.GLOBAL, + subject: str | None = None, + ) -> PersonalizationContext: + """Build group scheduled context without any creator profile.""" + + return self.build( + owner_id=None, + request=request, + system_constraints=system_constraints, + scope=scope, + subject=subject, + include_company_rules=True, + include_personal_context=False, + include_history=False, + ) + + def _watchlist_interests(self, owner_id: int) -> list[dict[str, Any]]: + symbols = self.db.execute( + select(MarketWatchlist.symbol) + .where( + MarketWatchlist.owner_id == owner_id, + MarketWatchlist.enabled.is_(True), + ) + .order_by(MarketWatchlist.symbol.asc()) + ).scalars() + return [ + { + "category": "watchlist", + "value": symbol, + "source": "market_watchlist", + } + for symbol in symbols + ] diff --git a/app/modules/personalization/services/conversations.py b/app/modules/personalization/services/conversations.py new file mode 100644 index 0000000..1ac7b72 --- /dev/null +++ b/app/modules/personalization/services/conversations.py @@ -0,0 +1,323 @@ +from datetime import timedelta +from hashlib import sha256 +from typing import Any + +from fastapi import HTTPException, status +from sqlalchemy import delete, exists, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.core.utils.time import utc_now +from app.modules.personalization.constants import ( + CONVERSATION_MAX_CONTENT_LENGTH, + CONVERSATION_MAX_MESSAGES, + CONVERSATION_RETENTION_DAYS, + UNAVAILABLE_AI_PROVIDERS, + ConversationChatType, + ConversationRole, +) +from app.modules.personalization.models import AIConversation, AIConversationMessage + + +class ConversationService: + """Persist isolated Feishu conversations with bounded history.""" + + def __init__(self, db: Session): + self.db = db + + def history( + self, + owner_id: int, + chat_type: str | ConversationChatType, + chat_key: str, + ) -> list[dict[str, Any]]: + owner_id, chat_type_value, chat_key_value = _conversation_identity( + owner_id, + chat_type, + chat_key, + ) + self.cleanup_expired(owner_id=owner_id) + conversation = self._find(owner_id, chat_type_value, chat_key_value) + if conversation is None: + self.db.commit() + return [] + messages = list( + self.db.execute( + select(AIConversationMessage) + .where(AIConversationMessage.conversation_id == conversation.id) + .order_by( + AIConversationMessage.created_at.desc(), + AIConversationMessage.id.desc(), + ) + .limit(CONVERSATION_MAX_MESSAGES) + ).scalars() + ) + self.db.commit() + messages.reverse() + return [_serialize_message(message) for message in messages] + + def record_turn( + self, + owner_id: int, + chat_type: str | ConversationChatType, + chat_key: str, + *, + user_content: str, + assistant_content: str, + provider_name: str, + ai_available: bool = True, + ) -> bool: + """Record one complete turn only after a real AI answer succeeds.""" + + if not ai_available or provider_name.strip().lower() in UNAVAILABLE_AI_PROVIDERS: + return False + owner_id, chat_type_value, chat_key_value = _conversation_identity( + owner_id, + chat_type, + chat_key, + ) + user_text = _message_content(user_content) + assistant_text = _message_content(assistant_content) + self.cleanup_expired(owner_id=owner_id) + conversation = self._get_or_create( + owner_id, + chat_type_value, + chat_key_value, + ) + now = utc_now() + self.db.add_all( + [ + AIConversationMessage( + conversation_id=conversation.id, + role=ConversationRole.USER, + content=user_text, + created_at=now, + ), + AIConversationMessage( + conversation_id=conversation.id, + role=ConversationRole.ASSISTANT, + content=assistant_text, + created_at=now, + ), + ] + ) + conversation.updated_at = now + self.db.flush() + self._trim(conversation.id) + self.db.commit() + return True + + def reset( + self, + owner_id: int, + chat_type: str | ConversationChatType, + chat_key: str, + ) -> bool: + owner_id, chat_type_value, chat_key_value = _conversation_identity( + owner_id, + chat_type, + chat_key, + ) + conversation = self._find(owner_id, chat_type_value, chat_key_value) + if conversation is None: + return False + self.db.execute( + delete(AIConversationMessage).where( + AIConversationMessage.conversation_id == conversation.id + ) + ) + self.db.delete(conversation) + self.db.commit() + return True + + def cleanup_expired(self, owner_id: int | None = None) -> int: + """Stage retention cleanup for one owner or all owners. + + The surrounding operation owns the transaction. Interactive history and + write paths use this method before completing their own commit. + """ + + return self._cleanup_expired(owner_id)["conversation_messages"] + + def cleanup_expired_globally(self) -> dict[str, int]: + """Delete expired messages for every owner and commit the maintenance run.""" + + deleted = self._cleanup_expired(owner_id=None) + self.db.commit() + return deleted + + def _cleanup_expired(self, owner_id: int | None) -> dict[str, int]: + cutoff = utc_now() - timedelta(days=CONVERSATION_RETENTION_DAYS) + conversation_ids = select(AIConversation.id) + if owner_id is not None: + conversation_ids = conversation_ids.where(AIConversation.owner_id == owner_id) + message_result = self.db.execute( + delete(AIConversationMessage).where( + AIConversationMessage.conversation_id.in_(conversation_ids), + AIConversationMessage.created_at < cutoff, + ) + ) + empty_conversations = delete(AIConversation).where( + ~exists( + select(AIConversationMessage.id).where( + AIConversationMessage.conversation_id == AIConversation.id + ) + ) + ) + if owner_id is not None: + empty_conversations = empty_conversations.where( + AIConversation.owner_id == owner_id + ) + conversation_result = self.db.execute(empty_conversations) + return { + "conversation_messages": max(0, int(message_result.rowcount or 0)), + "conversations": max(0, int(conversation_result.rowcount or 0)), + } + + def delete_owner_conversations(self, owner_id: int) -> dict[str, int]: + conversation_ids = select(AIConversation.id).where( + AIConversation.owner_id == owner_id + ) + message_result = self.db.execute( + delete(AIConversationMessage).where( + AIConversationMessage.conversation_id.in_(conversation_ids) + ) + ) + conversation_result = self.db.execute( + delete(AIConversation).where(AIConversation.owner_id == owner_id) + ) + return { + "conversation_messages": max(0, int(message_result.rowcount or 0)), + "conversations": max(0, int(conversation_result.rowcount or 0)), + } + + @staticmethod + def provider_session_id( + owner_id: int, + chat_type: str | ConversationChatType, + chat_key: str, + ) -> str: + owner_id, chat_type_value, chat_key_value = _conversation_identity( + owner_id, + chat_type, + chat_key, + ) + digest = sha256( + f"{owner_id}\0{chat_type_value}\0{chat_key_value}".encode("utf-8") + ).hexdigest() + return f"feishu-{digest}" + + def _find( + self, + owner_id: int, + chat_type: str, + chat_key: str, + ) -> AIConversation | None: + return self.db.execute( + select(AIConversation).where( + AIConversation.owner_id == owner_id, + AIConversation.chat_type == chat_type, + AIConversation.chat_key == chat_key, + ) + ).scalar_one_or_none() + + def _get_or_create( + self, + owner_id: int, + chat_type: str, + chat_key: str, + ) -> AIConversation: + existing = self._find(owner_id, chat_type, chat_key) + if existing is not None: + return existing + conversation = AIConversation( + owner_id=owner_id, + chat_type=chat_type, + chat_key=chat_key, + ) + try: + with self.db.begin_nested(): + self.db.add(conversation) + self.db.flush() + except IntegrityError: + conversation = self._find(owner_id, chat_type, chat_key) + if conversation is None: + raise + return conversation + + def _trim(self, conversation_id: int) -> int: + stale_ids = list( + self.db.execute( + select(AIConversationMessage.id) + .where(AIConversationMessage.conversation_id == conversation_id) + .order_by( + AIConversationMessage.created_at.desc(), + AIConversationMessage.id.desc(), + ) + .offset(CONVERSATION_MAX_MESSAGES) + ).scalars() + ) + if not stale_ids: + return 0 + result = self.db.execute( + delete(AIConversationMessage).where( + AIConversationMessage.id.in_(stale_ids) + ) + ) + return max(0, int(result.rowcount or 0)) + + +def _conversation_identity( + owner_id: int, + chat_type: str | ConversationChatType, + chat_key: str, +) -> tuple[int, str, str]: + if owner_id <= 0: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Valid conversation owner is required", + ) + raw_chat_type = str(chat_type).strip().lower() + aliases = { + "p2p": ConversationChatType.PRIVATE, + "private": ConversationChatType.PRIVATE, + "group": ConversationChatType.GROUP, + "group_chat": ConversationChatType.GROUP, + } + try: + chat_type_value = str(aliases[raw_chat_type]) + except KeyError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Unsupported conversation chat type", + ) from exc + chat_key_value = str(chat_key).strip() + if not chat_key_value or len(chat_key_value) > 256: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Valid conversation chat key is required", + ) + return owner_id, chat_type_value, chat_key_value + + +def _message_content(value: str) -> str: + content = str(value).strip() + if not content: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Conversation message is required", + ) + if len(content) > CONVERSATION_MAX_CONTENT_LENGTH: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Conversation message is too long", + ) + return content + + +def _serialize_message(message: AIConversationMessage) -> dict[str, Any]: + return { + "role": message.role, + "content": message.content, + "created_at": message.created_at.isoformat(), + } diff --git a/app/modules/personalization/services/erasure.py b/app/modules/personalization/services/erasure.py new file mode 100644 index 0000000..106f772 --- /dev/null +++ b/app/modules/personalization/services/erasure.py @@ -0,0 +1,205 @@ +from collections.abc import Callable, Mapping, Sequence +from datetime import timedelta +from hashlib import sha256 +from hmac import compare_digest +from secrets import token_hex +from typing import Any +from uuid import uuid4 + +from fastapi import HTTPException, status +from sqlalchemy import delete, select +from sqlalchemy.orm import Session + +from app.core.utils.time import utc_now +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.business.models import MarketWatchlist +from app.modules.personalization.constants import ERASURE_CONFIRMATION_TTL_MINUTES +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + UserPreference, +) +from app.modules.personalization.schemas import ErasureConfirmation, ErasureResult + +ErasureHook = Callable[ + [Session, int, str], + int | Mapping[str, int] | None, +] + + +class PersonalDataErasureService: + """Issue one-time confirmations and erase owner data in one transaction.""" + + def __init__(self, db: Session): + self.db = db + + def request_confirmation( + self, + owner_id: int, + *, + ttl_minutes: int = ERASURE_CONFIRMATION_TTL_MINUTES, + ) -> ErasureConfirmation: + _validate_owner(owner_id) + if ttl_minutes <= 0 or ttl_minutes > 60: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Invalid erasure confirmation lifetime", + ) + confirmation_code = token_hex(4).upper() + now = utc_now() + expires_at = now + timedelta(minutes=ttl_minutes) + record = self.db.execute( + select(PersonalDataErasureRequest).where( + PersonalDataErasureRequest.owner_id == owner_id + ) + ).scalar_one_or_none() + if record is None: + record = PersonalDataErasureRequest( + owner_id=owner_id, + token_hash=_token_hash(owner_id, confirmation_code), + expires_at=expires_at, + ) + self.db.add(record) + else: + record.token_hash = _token_hash(owner_id, confirmation_code) + record.expires_at = expires_at + record.updated_at = now + self.db.commit() + return ErasureConfirmation( + confirmation_code=confirmation_code, + expires_at=expires_at, + ) + + def confirm_and_erase( + self, + owner_id: int, + confirmation_code: str, + *, + before_hooks: Sequence[ErasureHook] = (), + extra_hooks: Sequence[ErasureHook] = (), + ) -> ErasureResult: + """Erase core personal tables and run integration hooks before one commit.""" + + _validate_owner(owner_id) + request = self.db.execute( + select(PersonalDataErasureRequest) + .where(PersonalDataErasureRequest.owner_id == owner_id) + .with_for_update() + ).scalar_one_or_none() + if request is None or request.expires_at <= utc_now(): + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Erasure confirmation is invalid or expired", + ) + supplied_hash = _token_hash(owner_id, confirmation_code.strip().upper()) + if not compare_digest(supplied_hash, request.token_hash): + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Erasure confirmation is invalid or expired", + ) + + anonymous_id = f"anonymous-{uuid4().hex}" + deleted: dict[str, int] = {} + extra: dict[str, Any] = {} + try: + for index, hook in enumerate(before_hooks): + _merge_hook_result( + hook(self.db, owner_id, anonymous_id), + index=index, + prefix="before", + deleted=deleted, + extra=extra, + ) + conversation_ids = select(AIConversation.id).where( + AIConversation.owner_id == owner_id + ) + deleted["conversation_messages"] = _row_count( + self.db.execute( + delete(AIConversationMessage).where( + AIConversationMessage.conversation_id.in_(conversation_ids) + ) + ).rowcount + ) + deleted["conversations"] = _row_count( + self.db.execute( + delete(AIConversation).where(AIConversation.owner_id == owner_id) + ).rowcount + ) + deleted["preferences"] = _row_count( + self.db.execute( + delete(UserPreference).where(UserPreference.owner_id == owner_id) + ).rowcount + ) + deleted["ai_memory"] = _row_count( + self.db.execute( + delete(AIMemoryEntry).where(AIMemoryEntry.owner_id == owner_id) + ).rowcount + ) + deleted["watchlist"] = _row_count( + self.db.execute( + delete(MarketWatchlist).where(MarketWatchlist.owner_id == owner_id) + ).rowcount + ) + self.db.delete(request) + + for index, hook in enumerate(extra_hooks): + _merge_hook_result( + hook(self.db, owner_id, anonymous_id), + index=index, + prefix="extra", + deleted=deleted, + extra=extra, + ) + self.db.commit() + except Exception: + self.db.rollback() + raise + return ErasureResult( + anonymous_id=anonymous_id, + deleted=deleted, + extra=extra, + ) + + def purge_expired_confirmations(self) -> int: + result = self.db.execute( + delete(PersonalDataErasureRequest).where( + PersonalDataErasureRequest.expires_at <= utc_now() + ) + ) + count = _row_count(result.rowcount) + self.db.commit() + return count + + +def _token_hash(owner_id: int, confirmation_code: str) -> str: + return sha256(f"{owner_id}\0{confirmation_code}".encode("utf-8")).hexdigest() + + +def _validate_owner(owner_id: int) -> None: + if owner_id <= 0: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Valid erasure owner is required", + ) + + +def _row_count(value: int | None) -> int: + return max(0, int(value or 0)) + + +def _merge_hook_result( + hook_result: int | Mapping[str, int] | None, + *, + index: int, + prefix: str, + deleted: dict[str, int], + extra: dict[str, Any], +) -> None: + if isinstance(hook_result, Mapping): + for key, value in hook_result.items(): + deleted[str(key)] = int(value) + elif isinstance(hook_result, int): + deleted[f"{prefix}_{index}"] = hook_result + elif hook_result is not None: + extra[f"{prefix}_{index}"] = hook_result diff --git a/app/modules/personalization/services/preferences.py b/app/modules/personalization/services/preferences.py new file mode 100644 index 0000000..898285e --- /dev/null +++ b/app/modules/personalization/services/preferences.py @@ -0,0 +1,315 @@ +import json +import re +import unicodedata +from typing import Any + +from fastapi import HTTPException, status +from pydantic import ValidationError +from sqlalchemy import delete, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.modules.personalization.constants import ( + PREFERENCE_MAX_VALUE_LENGTH, + PREFERENCE_SIGNAL_TERMS, + SENSITIVE_PREFERENCE_TERMS, + UNAVAILABLE_AI_PROVIDERS, + PreferenceCategory, + PreferenceSource, +) +from app.modules.personalization.models import UserPreference +from app.modules.personalization.schemas import PreferenceExtractionPayload + + +class PreferenceService: + """Manage explicit and safely extracted preferences inside one owner boundary.""" + + def __init__(self, db: Session): + self.db = db + + def list_preferences( + self, + owner_id: int, + category: str | PreferenceCategory | None = None, + ) -> list[dict[str, Any]]: + _validate_owner(owner_id) + stmt = ( + select(UserPreference) + .where(UserPreference.owner_id == owner_id) + .order_by(UserPreference.category.asc(), UserPreference.id.asc()) + ) + if category is not None: + stmt = stmt.where(UserPreference.category == _category_value(category)) + return [_serialize(item) for item in self.db.execute(stmt).scalars()] + + def upsert( + self, + owner_id: int, + category: str | PreferenceCategory, + value: str, + source: str | PreferenceSource = PreferenceSource.EXPLICIT, + *, + commit: bool = True, + ) -> dict[str, Any]: + """Create one owner-scoped preference or reuse its normalized equivalent.""" + + _validate_owner(owner_id) + category_value, clean_value, normalized = validate_preference(category, value) + source_value = _source_value(source) + existing = self.db.execute( + select(UserPreference).where( + UserPreference.owner_id == owner_id, + UserPreference.category == category_value, + UserPreference.normalized_value == normalized, + ) + ).scalar_one_or_none() + if existing is not None: + existing.value = clean_value + if source_value == PreferenceSource.EXPLICIT: + existing.source = source_value + if commit: + self.db.commit() + self.db.refresh(existing) + return _serialize(existing) + + record = UserPreference( + owner_id=owner_id, + category=category_value, + value=clean_value, + normalized_value=normalized, + source=source_value, + ) + try: + with self.db.begin_nested(): + self.db.add(record) + self.db.flush() + except IntegrityError: + record = self.db.execute( + select(UserPreference).where( + UserPreference.owner_id == owner_id, + UserPreference.category == category_value, + UserPreference.normalized_value == normalized, + ) + ).scalar_one() + if commit: + self.db.commit() + self.db.refresh(record) + return _serialize(record) + + def update( + self, + owner_id: int, + code: str, + *, + category: str | PreferenceCategory | None = None, + value: str | None = None, + ) -> dict[str, Any]: + record = self._owned_record(owner_id, code) + next_category = category if category is not None else record.category + next_value = value if value is not None else record.value + category_value, clean_value, normalized = validate_preference( + next_category, + next_value, + ) + try: + with self.db.begin_nested(): + record.category = category_value + record.value = clean_value + record.normalized_value = normalized + self.db.flush() + except IntegrityError as exc: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail="Preference already exists", + ) from exc + self.db.commit() + self.db.refresh(record) + return _serialize(record) + + def delete(self, owner_id: int, code: str) -> None: + record = self._owned_record(owner_id, code) + self.db.delete(record) + self.db.commit() + + def delete_matching( + self, + owner_id: int, + *, + category: str | PreferenceCategory, + value: str, + ) -> bool: + category_value, _, normalized = validate_preference(category, value) + record = self.db.execute( + select(UserPreference).where( + UserPreference.owner_id == owner_id, + UserPreference.category == category_value, + UserPreference.normalized_value == normalized, + ) + ).scalar_one_or_none() + if record is None: + return False + self.db.delete(record) + self.db.commit() + return True + + def save_auto_extraction( + self, + owner_id: int, + *, + provider_name: str, + user_text: str, + structured_payload: str | dict[str, Any] | list[Any], + ) -> list[dict[str, Any]]: + """Persist only allowlisted, non-sensitive output from a real AI provider.""" + + if provider_name.strip().lower() in UNAVAILABLE_AI_PROVIDERS: + return [] + if not contains_preference_signal(user_text): + return [] + candidates = _parse_extraction_payload(structured_payload) + saved: list[dict[str, Any]] = [] + for candidate in candidates: + try: + saved.append( + self.upsert( + owner_id=owner_id, + category=candidate.category, + value=candidate.value, + source=PreferenceSource.AUTO, + commit=False, + ) + ) + except HTTPException: + # Automatic extraction is intentionally silent. Invalid or sensitive + # candidates are discarded without creating a rejected profile row. + continue + if saved: + self.db.commit() + return saved + + def delete_owner_preferences(self, owner_id: int) -> int: + """Stage deletion of every preference for an owner.""" + + result = self.db.execute( + delete(UserPreference).where(UserPreference.owner_id == owner_id) + ) + return max(0, int(result.rowcount or 0)) + + def _owned_record(self, owner_id: int, code: str) -> UserPreference: + _validate_owner(owner_id) + record = self.db.execute( + select(UserPreference).where( + UserPreference.owner_id == owner_id, + UserPreference.code == code, + ) + ).scalar_one_or_none() + if record is None: + # Deliberately indistinguishable from a nonexistent code. + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="Preference not found", + ) + return record + + +def contains_preference_signal(value: str) -> bool: + text = unicodedata.normalize("NFKC", value).casefold() + return any(term in text for term in PREFERENCE_SIGNAL_TERMS) + + +def validate_preference( + category: str | PreferenceCategory, + value: str, +) -> tuple[str, str, str]: + category_value = _category_value(category) + clean_value = _clean_value(value) + if is_sensitive_preference(clean_value): + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Sensitive preference content is not allowed", + ) + return category_value, clean_value, normalize_preference_value(clean_value) + + +def normalize_preference_value(value: str) -> str: + normalized = unicodedata.normalize("NFKC", value) + normalized = re.sub(r"\s+", " ", normalized).strip() + return normalized.casefold() + + +def is_sensitive_preference(value: str) -> bool: + normalized = unicodedata.normalize("NFKC", value).casefold() + return any(term.casefold() in normalized for term in SENSITIVE_PREFERENCE_TERMS) + + +def _parse_extraction_payload( + payload: str | dict[str, Any] | list[Any], +) -> list[Any]: + parsed: Any = payload + if isinstance(payload, str): + try: + parsed = json.loads(payload) + except json.JSONDecodeError: + return [] + if isinstance(parsed, list): + parsed = {"preferences": parsed} + elif isinstance(parsed, dict) and "preferences" not in parsed: + parsed = {"preferences": [parsed]} + try: + return PreferenceExtractionPayload.model_validate(parsed).preferences + except ValidationError: + return [] + + +def _category_value(category: str | PreferenceCategory) -> str: + try: + return str(PreferenceCategory(category)) + except ValueError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Unsupported preference category", + ) from exc + + +def _source_value(source: str | PreferenceSource) -> str: + try: + return str(PreferenceSource(source)) + except ValueError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Unsupported preference source", + ) from exc + + +def _clean_value(value: str) -> str: + clean_value = unicodedata.normalize("NFKC", str(value)).strip() + if not clean_value: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Preference value is required", + ) + if len(clean_value) > PREFERENCE_MAX_VALUE_LENGTH: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Preference value is too long", + ) + return clean_value + + +def _validate_owner(owner_id: int) -> None: + if owner_id <= 0: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="Valid preference owner is required", + ) + + +def _serialize(record: UserPreference) -> dict[str, Any]: + return { + "code": record.code, + "category": record.category, + "value": record.value, + "source": record.source, + "created_at": record.created_at.isoformat(), + "updated_at": record.updated_at.isoformat(), + } diff --git a/app/modules/reports/routes.py b/app/modules/reports/routes.py index 94a08dd..f1a0da4 100644 --- a/app/modules/reports/routes.py +++ b/app/modules/reports/routes.py @@ -14,7 +14,7 @@ from app.core.background.task_queue import ( enqueue_work_weekly_push, ) from app.core.database import get_db -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.reports.constants import ReportPushKey from app.modules.reports.schemas import ( LifecycleRunRequest, @@ -134,7 +134,6 @@ def enqueue_lifecycle( payload: LifecycleRunRequest, principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return enqueue_lifecycle_report( report_type=payload.report_type, receive_id=payload.receive_id, @@ -167,8 +166,6 @@ def generate_work_report( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - if payload.persist: - require_operations_enabled() return ReportService(db).generate_work_report( report_type=payload.report_type, reporter=payload.reporter, diff --git a/app/modules/reports/services/enterprise.py b/app/modules/reports/services/enterprise.py index 96983ff..a14a805 100644 --- a/app/modules/reports/services/enterprise.py +++ b/app/modules/reports/services/enterprise.py @@ -1,6 +1,10 @@ from datetime import date +from hashlib import sha256 +import json from typing import Any +from fastapi import HTTPException, status + from app.core.constants import ActorValue from app.modules.audit.constants import AuditAction, AuditSource, AuditTargetType @@ -26,7 +30,7 @@ from app.modules.reports.constants import ( ReportTitle, ) -from app.modules.reports.services.common import _json_safe, _money, _next_code, _rate +from app.modules.reports.services.common import _json_safe, _money, _rate class ReportEnterpriseAnalyticsMixin: @@ -39,8 +43,15 @@ class ReportEnterpriseAnalyticsMixin: actor: str = ActorValue.API, ) -> dict[str, Any]: """Build V3 read-only finance, procurement, performance, and operations analytics.""" + if period_start and period_end and period_start > period_end: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="period_start must be before or equal to period_end", + ) - code = _next_code("ANALYTICS") + include_global_metrics = not any( + value is not None for value in (project_code, owner, period_start, period_end) + ) lifecycle = self.project_lifecycle_report( project_code=project_code, owner=owner, @@ -61,9 +72,17 @@ class ReportEnterpriseAnalyticsMixin: MetricKey.BUDGET_TOTAL: projects[MetricKey.BUDGET_TOTAL], MetricKey.ACTUAL_TOTAL: projects[MetricKey.ACTUAL_TOTAL], MetricKey.BUDGET_USAGE_RATE: projects[MetricKey.BUDGET_USAGE_RATE], - MetricKey.CURRENT_BALANCE_TOTAL: funds[MetricKey.CURRENT_BALANCE_TOTAL], - MetricKey.NET_POSITION: funds[MetricKey.NET_POSITION], - MetricKey.RISK_ACCOUNTS: funds[MetricKey.RISK_ACCOUNTS], + MetricKey.CURRENT_BALANCE_TOTAL: ( + funds[MetricKey.CURRENT_BALANCE_TOTAL] + if include_global_metrics + else 0 + ), + MetricKey.NET_POSITION: ( + funds[MetricKey.NET_POSITION] if include_global_metrics else 0 + ), + MetricKey.RISK_ACCOUNTS: ( + funds[MetricKey.RISK_ACCOUNTS] if include_global_metrics else 0 + ), MetricKey.PAYMENT_EXPOSURE: ( procurements[MetricKey.ACTUAL_TOTAL] + expenses[MetricKey.AMOUNT_TOTAL] ), @@ -77,7 +96,7 @@ class ReportEnterpriseAnalyticsMixin: MetricKey.ACTUAL_TOTAL: procurements[MetricKey.ACTUAL_TOTAL], MetricKey.DELIVERY_RISK: procurements[MetricKey.PENDING_DELIVERY], } - performance = self._enterprise_performance_stats() + performance = self._enterprise_performance_stats(include_global_metrics) operations = { MetricKey.READINESS_SCORE: health[MetricKey.SCORE], MetricKey.LEVEL: health[MetricKey.LEVEL], @@ -97,9 +116,8 @@ class ReportEnterpriseAnalyticsMixin: operations, recommendations, ) - report = _json_safe( + snapshot = _json_safe( { - EnterpriseAnalyticsKey.CODE: code, EnterpriseAnalyticsKey.TITLE: ReportTitle.ENTERPRISE_ANALYTICS, EnterpriseAnalyticsKey.FILTERS: lifecycle[LifecycleResponseKey.FILTERS], EnterpriseAnalyticsKey.FINANCE: finance, @@ -111,6 +129,16 @@ class ReportEnterpriseAnalyticsMixin: EnterpriseAnalyticsKey.CONTENT: "\n".join(lines), } ) + canonical_snapshot = json.dumps( + snapshot, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ) + digest = sha256(canonical_snapshot.encode("utf-8")).hexdigest()[:24].upper() + code = f"ANALYTICS-{digest}" + report = {EnterpriseAnalyticsKey.CODE: code, **snapshot} AuditService(self.db).record( AuditLogCreate( actor=actor, @@ -136,7 +164,18 @@ class ReportEnterpriseAnalyticsMixin: self.db.commit() return report - def _enterprise_performance_stats(self) -> dict[str, Any]: + def _enterprise_performance_stats(self, include_global: bool) -> dict[str, Any]: + if not include_global: + return { + MetricKey.TOTAL: 0, + MetricKey.CONFIRMED: 0, + MetricKey.CONFIRMED_RATE: 0.0, + MetricKey.AVERAGE_AUTO_SCORE: 0.0, + MetricKey.AVERAGE_CONFIRMED_SCORE: 0.0, + MetricKey.WEIGHT_TOTAL: 0, + MetricKey.BY_STATUS: {}, + } + total = self._count(PerformanceMetric) confirmed = self._count(PerformanceMetric, PerformanceMetric.confirmed_score.is_not(None)) return { diff --git a/app/modules/reports/services/lifecycle/report.py b/app/modules/reports/services/lifecycle/report.py index c1e78da..aa6b0dd 100644 --- a/app/modules/reports/services/lifecycle/report.py +++ b/app/modules/reports/services/lifecycle/report.py @@ -45,7 +45,9 @@ class ReportLifecycleReportMixin: task_conditions, risk_conditions, ) - include_global_risk = not (project_code or owner) + include_global_risk = not any( + value is not None for value in (project_code, owner, period_start, period_end) + ) health = self._lifecycle_health( project_stats, task_stats, diff --git a/app/modules/reports/services/push_runs.py b/app/modules/reports/services/push_runs.py index 08fa8c6..b23d11c 100644 --- a/app/modules/reports/services/push_runs.py +++ b/app/modules/reports/services/push_runs.py @@ -26,6 +26,7 @@ class ReportPushRunMixin: receive_id_type: str, actor: str, status: str = ReportPushStatus.PENDING, + task_id: str | None = None, idempotency_key: str | None = None, ) -> ReportPushRun: if idempotency_key: @@ -43,6 +44,7 @@ class ReportPushRunMixin: receive_id=receive_id, receive_id_type=receive_id_type, status=status, + task_id=task_id, actor=actor, queued_at=utc_now(), idempotency_key=idempotency_key, diff --git a/app/modules/risk/routes.py b/app/modules/risk/routes.py index 365c308..0921aee 100644 --- a/app/modules/risk/routes.py +++ b/app/modules/risk/routes.py @@ -3,7 +3,7 @@ from sqlalchemy.orm import Session from app.core.background.task_queue import enqueue_risk_event_generation from app.core.database import get_db -from app.core.security import ApiPrincipal, require_api_key, require_operations_enabled +from app.core.security import ApiPrincipal, require_api_key from app.modules.risk.constants import RiskEventActionKey, RiskGenerationResultKey from app.modules.risk.schemas import ( RiskAssignRequest, @@ -94,7 +94,6 @@ def assign_risk_event( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).assign_event( event_id, assigned_to=payload.assigned_to, @@ -110,7 +109,6 @@ def comment_risk_event( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).comment_event( event_id, comment=payload.comment, @@ -126,7 +124,6 @@ def resolve_risk_event( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).resolve_event( event_id, comment=payload.comment, @@ -142,7 +139,6 @@ def close_risk_event( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).close_event( event_id, closed_reason=payload.closed_reason, @@ -158,7 +154,6 @@ def reopen_risk_event( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).reopen_event( event_id, comment=payload.comment, @@ -171,7 +166,6 @@ def generate_risk_events( db: Session = Depends(get_db), principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return RiskService(db).generate_events(actor=principal.actor) @@ -179,5 +173,4 @@ def generate_risk_events( def enqueue_risk_events( principal: ApiPrincipal = Depends(require_api_key), ) -> dict: - require_operations_enabled() return enqueue_risk_event_generation(actor=principal.actor) diff --git a/app/modules/risk/services/actions.py b/app/modules/risk/services/actions.py index f860ee2..d36f0f0 100644 --- a/app/modules/risk/services/actions.py +++ b/app/modules/risk/services/actions.py @@ -2,7 +2,6 @@ from typing import Any from app.core.constants import ActorValue -from app.core.security import ensure_business_mutations_enabled from app.core.utils.time import utc_now from app.modules.audit.constants import ( AuditAction, @@ -182,7 +181,6 @@ class RiskActionMixin: comment: str | None, payload: dict[str, Any], ) -> RiskEventAction: - ensure_business_mutations_enabled() action_record = RiskEventAction( code=f"RISK-ACTION-{utc_now():%Y%m%d%H%M%S%f}", risk_event_id=record.id, diff --git a/app/modules/risk/services/generation.py b/app/modules/risk/services/generation.py index c225a81..422a268 100644 --- a/app/modules/risk/services/generation.py +++ b/app/modules/risk/services/generation.py @@ -3,7 +3,6 @@ from typing import Any from sqlalchemy import select from app.core.constants import ActorValue -from app.core.security import ensure_business_mutations_enabled from app.modules.audit.constants import ( AuditAction, AuditRiskLevel, @@ -30,7 +29,6 @@ class RiskGenerationMixin: def generate_events(self, actor: str = ActorValue.API) -> dict[str, Any]: """Generate or refresh risk-event ledger entries from current signals.""" - ensure_business_mutations_enabled() payloads = self._build_event_payloads() created = 0 updated = 0 diff --git a/app/modules/subscriptions/__init__.py b/app/modules/subscriptions/__init__.py new file mode 100644 index 0000000..f48018c --- /dev/null +++ b/app/modules/subscriptions/__init__.py @@ -0,0 +1,22 @@ +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services import ( + DeliveryGenerationRequest, + DeliverySendRequest, + DeliveryService, + NormalizedSchedule, + SubscriptionManagementService, + SubscriptionScanner, + parse_schedule, +) + +__all__ = [ + "DeliveryGenerationRequest", + "DeliverySendRequest", + "DeliveryService", + "NormalizedSchedule", + "PushDelivery", + "PushSubscription", + "SubscriptionManagementService", + "SubscriptionScanner", + "parse_schedule", +] diff --git a/app/modules/subscriptions/constants.py b/app/modules/subscriptions/constants.py new file mode 100644 index 0000000..64e7b42 --- /dev/null +++ b/app/modules/subscriptions/constants.py @@ -0,0 +1,65 @@ +from enum import StrEnum + + +class SubscriptionTargetType(StrEnum): + USER = "user" + CHAT = "chat" + + +class SubscriptionScheduleType(StrEnum): + ONCE = "once" + DAILY = "daily" + WEEKDAY = "weekday" + WEEKLY = "weekly" + MONTHLY = "monthly" + INTERVAL = "interval" + + +class PushSubscriptionStatus(StrEnum): + ACTIVE = "active" + PAUSED = "paused" + CANCELLED = "cancelled" + COMPLETED = "completed" + + +class PushDeliveryStatus(StrEnum): + PENDING = "pending" + PROCESSING = "processing" + RETRY = "retry" + SENT = "sent" + FAILED = "failed" + SKIPPED = "skipped" + + +class SubscriptionAuditAction(StrEnum): + CREATE = "subscription.create" + PAUSE = "subscription.pause" + RESUME = "subscription.resume" + CANCEL = "subscription.cancel" + UPDATE_TIMEZONE = "subscription.update_timezone" + UPDATE_QUIET_HOURS = "subscription.update_quiet_hours" + DELIVERY_SENT = "subscription.delivery.sent" + DELIVERY_FAILED = "subscription.delivery.failed" + DELIVERY_SKIPPED = "subscription.delivery.skipped" + + +DEFAULT_SUBSCRIPTION_TIMEZONE = "Asia/Shanghai" +MAX_ACTIVE_SUBSCRIPTIONS = 50 +MAX_DAILY_DELIVERIES = 96 +MIN_INTERVAL_MINUTES = 15 +DELIVERY_RETRY_DELAYS_SECONDS = (60, 300, 900) +SUBSCRIPTION_LEASE_SECONDS = 120 +DELIVERY_LEASE_SECONDS = 300 + +SUBSCRIPTION_NOT_FOUND = "Subscription not found" +DELIVERY_NOT_FOUND = "Subscription delivery not found" +SUBSCRIPTION_LIMIT_REACHED = "A user may enable at most 50 subscriptions" +DAILY_DELIVERY_LIMIT_REACHED = "Daily delivery limit reached" +INVALID_SCHEDULE = "Unsupported or invalid schedule expression" +INVALID_TIMEZONE = "Invalid IANA timezone" +INVALID_QUIET_HOURS = "Quiet hours must use different HH:MM start and end values" +INVALID_PRIVATE_TARGET = "Private subscriptions must target the current user's open_id" +INVALID_GROUP_TARGET = "Group subscriptions must be created by an administrator in the current group" +INACTIVE_USER = "Feishu user is disabled" +EMPTY_PROMPT = "Subscription prompt is required" + diff --git a/app/modules/subscriptions/models.py b/app/modules/subscriptions/models.py new file mode 100644 index 0000000..d49a24b --- /dev/null +++ b/app/modules/subscriptions/models.py @@ -0,0 +1,122 @@ +from datetime import datetime +from typing import Any + +from sqlalchemy import ( + JSON, + DateTime, + ForeignKey, + Integer, + String, + Text, + UniqueConstraint, +) +from sqlalchemy.orm import Mapped, mapped_column, relationship + +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.subscriptions.constants import ( + PushDeliveryStatus, + PushSubscriptionStatus, +) + + +class PushSubscription(Base): + __tablename__ = "push_subscriptions" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column(String(64), unique=True, index=True) + owner_id: Mapped[int] = mapped_column( + ForeignKey("feishu_users.id", ondelete="CASCADE"), + index=True, + ) + target_type: Mapped[str] = mapped_column(String(16), index=True) + target_id: Mapped[str] = mapped_column(String(256)) + prompt: Mapped[str] = mapped_column(Text) + schedule_type: Mapped[str] = mapped_column(String(32), index=True) + schedule_config: Mapped[dict[str, Any]] = mapped_column(JSON) + timezone: Mapped[str] = mapped_column(String(64)) + next_run_at: Mapped[datetime | None] = mapped_column( + DateTime, + nullable=True, + index=True, + ) + status: Mapped[str] = mapped_column( + String(32), + default=PushSubscriptionStatus.ACTIVE, + index=True, + ) + consented_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now) + last_run_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) + locked_by: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + locked_until: Mapped[datetime | None] = mapped_column( + DateTime, + nullable=True, + index=True, + ) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) + + deliveries: Mapped[list["PushDelivery"]] = relationship( + back_populates="subscription", + cascade="all, delete-orphan", + passive_deletes=True, + ) + + +class PushDelivery(Base): + __tablename__ = "push_deliveries" + __table_args__ = ( + UniqueConstraint( + "subscription_id", + "scheduled_for", + name="uq_push_delivery_subscription_schedule", + ), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column(String(64), unique=True, index=True) + subscription_id: Mapped[int] = mapped_column( + ForeignKey("push_subscriptions.id", ondelete="CASCADE"), + index=True, + ) + scheduled_for: Mapped[datetime] = mapped_column(DateTime, index=True) + idempotency_key: Mapped[str] = mapped_column(String(64), unique=True, index=True) + message_uuid: Mapped[str] = mapped_column(String(36), unique=True, index=True) + status: Mapped[str] = mapped_column( + String(32), + default=PushDeliveryStatus.PENDING, + index=True, + ) + attempt_count: Mapped[int] = mapped_column(Integer, default=0) + next_attempt_at: Mapped[datetime | None] = mapped_column( + DateTime, + nullable=True, + index=True, + ) + rendered_content: Mapped[str | None] = mapped_column(Text, nullable=True) + provider_message_id: Mapped[str | None] = mapped_column( + String(256), + nullable=True, + index=True, + ) + last_error: Mapped[str | None] = mapped_column(Text, nullable=True) + locked_by: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + locked_until: Mapped[datetime | None] = mapped_column( + DateTime, + nullable=True, + index=True, + ) + sent_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True, index=True) + created_at: Mapped[datetime] = mapped_column(DateTime, default=utc_now, index=True) + updated_at: Mapped[datetime] = mapped_column( + DateTime, + default=utc_now, + onupdate=utc_now, + ) + + subscription: Mapped[PushSubscription] = relationship(back_populates="deliveries") + diff --git a/app/modules/subscriptions/routes.py b/app/modules/subscriptions/routes.py new file mode 100644 index 0000000..0117337 --- /dev/null +++ b/app/modules/subscriptions/routes.py @@ -0,0 +1,49 @@ +from fastapi import APIRouter, Depends +from sqlalchemy.orm import Session + +from app.core.database import get_db +from app.core.security import require_api_key +from app.modules.subscriptions.schemas import PushDeliveryRead, PushSubscriptionRead +from app.modules.subscriptions.services.management import SubscriptionManagementService + +router = APIRouter(dependencies=[Depends(require_api_key)]) + + +@router.get("") +def list_subscriptions( + status_filter: str | None = None, + owner_id: int | None = None, + limit: int = 100, + offset: int = 0, + db: Session = Depends(get_db), +) -> dict: + total, records = SubscriptionManagementService(db).list_all( + status_filter=status_filter, + owner_id=owner_id, + limit=limit, + offset=offset, + ) + return { + "total": total, + "items": [PushSubscriptionRead.model_validate(item) for item in records], + } + + +@router.get("/deliveries") +def list_deliveries( + status_filter: str | None = None, + subscription_code: str | None = None, + limit: int = 100, + offset: int = 0, + db: Session = Depends(get_db), +) -> dict: + total, records = SubscriptionManagementService(db).list_deliveries( + status_filter=status_filter, + subscription_code=subscription_code, + limit=limit, + offset=offset, + ) + return { + "total": total, + "items": [PushDeliveryRead.model_validate(item) for item in records], + } diff --git a/app/modules/subscriptions/schemas.py b/app/modules/subscriptions/schemas.py new file mode 100644 index 0000000..3c65303 --- /dev/null +++ b/app/modules/subscriptions/schemas.py @@ -0,0 +1,55 @@ +from datetime import datetime +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + + +class NormalizedScheduleRead(BaseModel): + schedule_type: str + schedule_config: dict[str, Any] + timezone: str + next_run_at: datetime + display: str + + +class SubscriptionCreate(BaseModel): + prompt: str = Field(min_length=1, max_length=8000) + schedule: str = Field(min_length=1, max_length=256) + + +class PushSubscriptionRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + code: str + owner_id: int + target_type: str + target_id: str + prompt: str + schedule_type: str + schedule_config: dict[str, Any] + timezone: str + next_run_at: datetime | None + status: str + consented_at: datetime + last_run_at: datetime | None + created_at: datetime + updated_at: datetime + + +class PushDeliveryRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + code: str + subscription_id: int + scheduled_for: datetime + idempotency_key: str + message_uuid: str + status: str + attempt_count: int + next_attempt_at: datetime | None + provider_message_id: str | None + last_error: str | None + sent_at: datetime | None + created_at: datetime + updated_at: datetime + diff --git a/app/modules/subscriptions/services/__init__.py b/app/modules/subscriptions/services/__init__.py new file mode 100644 index 0000000..5551f70 --- /dev/null +++ b/app/modules/subscriptions/services/__init__.py @@ -0,0 +1,39 @@ +from app.modules.subscriptions.services.delivery import ( + DeliveryGenerationRequest, + DeliveryGenerator, + DeliverySendRequest, + DeliverySender, + DeliveryService, + PermanentDeliveryError, + RetryableDeliveryError, +) +from app.modules.subscriptions.services.management import SubscriptionManagementService +from app.modules.subscriptions.services.scanner import SubscriptionScanner +from app.modules.subscriptions.services.schedule import ( + NormalizedSchedule, + ScheduleParseError, + is_in_quiet_hours, + next_occurrence, + next_quiet_end, + parse_schedule, + validate_timezone, +) + +__all__ = [ + "DeliveryGenerationRequest", + "DeliveryGenerator", + "DeliverySendRequest", + "DeliverySender", + "DeliveryService", + "NormalizedSchedule", + "PermanentDeliveryError", + "RetryableDeliveryError", + "ScheduleParseError", + "SubscriptionManagementService", + "SubscriptionScanner", + "is_in_quiet_hours", + "next_occurrence", + "next_quiet_end", + "parse_schedule", + "validate_timezone", +] diff --git a/app/modules/subscriptions/services/delivery.py b/app/modules/subscriptions/services/delivery.py new file mode 100644 index 0000000..a3d0eea --- /dev/null +++ b/app/modules/subscriptions/services/delivery.py @@ -0,0 +1,645 @@ +from dataclasses import dataclass +from datetime import UTC, datetime, time, timedelta +from typing import Any, Protocol +from uuid import uuid4 +from zoneinfo import ZoneInfo + +import httpx +from fastapi import HTTPException, status +from sqlalchemy import and_, func, or_, select, update +from sqlalchemy.orm import Session + +from app.core.http.pagination import bounded_limit +from app.core.utils.time import utc_now +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu.errors import FeishuAPIError +from app.modules.feishu_users.constants import FeishuUserRole, FeishuUserStatus +from app.modules.feishu_users.models import FeishuUser +from app.modules.subscriptions.constants import ( + DAILY_DELIVERY_LIMIT_REACHED, + DELIVERY_LEASE_SECONDS, + DELIVERY_NOT_FOUND, + DELIVERY_RETRY_DELAYS_SECONDS, + MAX_DAILY_DELIVERIES, + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionAuditAction, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services.schedule import ( + is_in_quiet_hours, + next_quiet_end, +) + + +@dataclass(frozen=True, slots=True) +class DeliveryGenerationRequest: + prompt: str + owner_id: int | None + use_personal_context: bool + use_company_rules: bool + allow_tools: bool = False + record_history: bool = False + infer_preferences: bool = False + write_memory: bool = False + + +@dataclass(frozen=True, slots=True) +class DeliverySendRequest: + receive_id: str + receive_id_type: str + tenant_key: str + text: str + uuid: str + + +class DeliveryGenerator(Protocol): + def generate(self, request: DeliveryGenerationRequest) -> str: + """Generate delivery text without side effects or business-data tools.""" + + +class DeliverySender(Protocol): + def send(self, request: DeliverySendRequest) -> dict[str, Any]: + """Send a message and return the provider response.""" + + +class RetryableDeliveryError(RuntimeError): + """A temporary generation or provider failure.""" + + +class PermanentDeliveryError(RuntimeError): + """A delivery failure that must not be retried.""" + + +class DeliveryService: + """Process durable deliveries with fencing and database-driven retry timing.""" + + def __init__( + self, + db: Session, + *, + generator: DeliveryGenerator, + sender: DeliverySender, + lease_seconds: int = DELIVERY_LEASE_SECONDS, + ): + self.db = db + self.generator = generator + self.sender = sender + self.lease_seconds = lease_seconds + + def process( + self, + delivery_code: str, + *, + now: datetime | None = None, + worker_id: str = "subscription-delivery", + ) -> PushDelivery: + current = _naive_utc(now or utc_now()) + existing = self._get(delivery_code) + if existing.status in { + PushDeliveryStatus.SENT, + PushDeliveryStatus.FAILED, + PushDeliveryStatus.SKIPPED, + }: + return existing + + lock_owner = f"{worker_id}:{uuid4().hex}" + if not self._claim(existing.id, lock_owner, current): + self.db.rollback() + return self._get(delivery_code) + delivery = self._get(delivery_code) + subscription, owner = self._load_context(delivery.subscription_id) + + skip_reason = self._skip_reason(subscription, owner) + if skip_reason is not None: + return self._finish_skipped(delivery, lock_owner, owner, skip_reason) + + if ( + owner is not None + and is_in_quiet_hours( + current, + owner.timezone, + owner.quiet_hours_start, + owner.quiet_hours_end, + ) + ): + quiet_end = next_quiet_end( + current, + owner.timezone, + owner.quiet_hours_start, + owner.quiet_hours_end, + ) + return self._defer_for_quiet_hours(delivery, lock_owner, quiet_end) + + if not self._start_attempt(delivery.id, lock_owner, current): + self.db.rollback() + return self._get(delivery_code) + delivery = self._get(delivery_code) + try: + content = delivery.rendered_content + if content is None: + content = str( + self.generator.generate( + self._generation_request(subscription) + ) + ).strip() + if not content: + raise PermanentDeliveryError("Delivery generator returned empty content") + self._save_content(delivery.id, lock_owner, content, current) + subscription, owner = self._lock_send_context(delivery.subscription_id) + skip_reason = self._skip_reason(subscription, owner) + if skip_reason is None and owner is not None: + if self._sent_today(owner, current) >= MAX_DAILY_DELIVERIES: + skip_reason = DAILY_DELIVERY_LIMIT_REACHED + if skip_reason is not None: + return self._finish_skipped( + delivery, + lock_owner, + owner, + skip_reason, + ) + response = self.sender.send( + DeliverySendRequest( + receive_id=( + owner.open_id + if subscription.target_type == SubscriptionTargetType.USER + else subscription.target_id + ), + receive_id_type=( + "open_id" + if subscription.target_type == SubscriptionTargetType.USER + else "chat_id" + ), + tenant_key=owner.tenant_key, + text=content, + uuid=delivery.message_uuid, + ) + ) + self._validate_provider_response(response) + except Exception as exc: + self.db.rollback() + return self._finish_failure(delivery_code, lock_owner, current, exc) + return self._finish_sent( + delivery_code, + lock_owner, + current, + owner, + response, + ) + + def process_due( + self, + *, + now: datetime | None = None, + limit: int = 100, + worker_id: str = "subscription-delivery", + ) -> list[PushDelivery]: + current = _naive_utc(now or utc_now()) + stmt = ( + select(PushDelivery.code) + .where( + or_( + and_( + PushDelivery.status.in_( + [ + PushDeliveryStatus.PENDING, + PushDeliveryStatus.RETRY, + ] + ), + PushDelivery.next_attempt_at.is_not(None), + PushDelivery.next_attempt_at <= current, + ), + and_( + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_until.is_not(None), + PushDelivery.locked_until <= current, + ), + ) + ) + .order_by(PushDelivery.next_attempt_at.asc(), PushDelivery.id.asc()) + .limit(bounded_limit(limit)) + ) + codes = list(self.db.execute(stmt).scalars()) + return [ + self.process( + code, + now=current, + worker_id=worker_id, + ) + for code in codes + ] + + def _claim(self, delivery_id: int, lock_owner: str, current: datetime) -> bool: + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.id == delivery_id, + or_( + and_( + PushDelivery.status.in_( + [ + PushDeliveryStatus.PENDING, + PushDeliveryStatus.RETRY, + ] + ), + PushDelivery.next_attempt_at.is_not(None), + PushDelivery.next_attempt_at <= current, + ), + and_( + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_until.is_not(None), + PushDelivery.locked_until <= current, + ), + ), + or_( + PushDelivery.locked_until.is_(None), + PushDelivery.locked_until <= current, + ), + ) + .values( + status=PushDeliveryStatus.PROCESSING, + locked_by=lock_owner, + locked_until=current + timedelta(seconds=self.lease_seconds), + ) + .execution_options(synchronize_session=False) + ) + self.db.commit() + return result.rowcount == 1 + + def _start_attempt( + self, + delivery_id: int, + lock_owner: str, + current: datetime, + ) -> bool: + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.id == delivery_id, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + attempt_count=PushDelivery.attempt_count + 1, + locked_until=current + timedelta(seconds=self.lease_seconds), + ) + .execution_options(synchronize_session=False) + ) + self.db.commit() + return result.rowcount == 1 + + def _save_content( + self, + delivery_id: int, + lock_owner: str, + content: str, + current: datetime, + ) -> None: + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.id == delivery_id, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + rendered_content=content, + locked_until=current + timedelta(seconds=self.lease_seconds), + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + raise RetryableDeliveryError("Delivery lease was lost") + self.db.commit() + + def _finish_sent( + self, + delivery_code: str, + lock_owner: str, + current: datetime, + owner: FeishuUser | None, + response: dict[str, Any], + ) -> PushDelivery: + provider_message_id = _provider_message_id(response) + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.code == delivery_code, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + status=PushDeliveryStatus.SENT, + next_attempt_at=None, + provider_message_id=provider_message_id, + last_error=None, + sent_at=current, + locked_by=None, + locked_until=None, + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + return self._get(delivery_code) + record = self._get(delivery_code) + self._audit( + owner, + record, + SubscriptionAuditAction.DELIVERY_SENT, + {"status": PushDeliveryStatus.SENT}, + ) + self.db.commit() + self.db.refresh(record) + return record + + def _finish_failure( + self, + delivery_code: str, + lock_owner: str, + current: datetime, + exc: Exception, + ) -> PushDelivery: + record = self._get(delivery_code) + retryable = _is_retryable(exc) + retry_index = record.attempt_count - 1 + will_retry = retryable and 0 <= retry_index < len( + DELIVERY_RETRY_DELAYS_SECONDS + ) + next_attempt_at = ( + current + timedelta(seconds=DELIVERY_RETRY_DELAYS_SECONDS[retry_index]) + if will_retry + else None + ) + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.code == delivery_code, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + status=( + PushDeliveryStatus.RETRY + if will_retry + else PushDeliveryStatus.FAILED + ), + next_attempt_at=next_attempt_at, + last_error=f"{type(exc).__name__}: {exc}"[:2000], + locked_by=None, + locked_until=None, + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + return self._get(delivery_code) + record = self._get(delivery_code) + if not will_retry: + _, owner = self._load_context(record.subscription_id) + self._audit( + owner, + record, + SubscriptionAuditAction.DELIVERY_FAILED, + { + "status": PushDeliveryStatus.FAILED, + "attempt_count": record.attempt_count, + }, + ) + self.db.commit() + self.db.refresh(record) + return record + + def _finish_skipped( + self, + delivery: PushDelivery, + lock_owner: str, + owner: FeishuUser | None, + reason: str, + ) -> PushDelivery: + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.id == delivery.id, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + status=PushDeliveryStatus.SKIPPED, + next_attempt_at=None, + last_error=reason, + locked_by=None, + locked_until=None, + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + return self._get(delivery.code) + record = self._get(delivery.code) + self._audit( + owner, + record, + SubscriptionAuditAction.DELIVERY_SKIPPED, + {"status": PushDeliveryStatus.SKIPPED, "reason": reason}, + ) + self.db.commit() + self.db.refresh(record) + return record + + def _defer_for_quiet_hours( + self, + delivery: PushDelivery, + lock_owner: str, + quiet_end: datetime, + ) -> PushDelivery: + status_value = ( + PushDeliveryStatus.PENDING + if delivery.attempt_count == 0 + else PushDeliveryStatus.RETRY + ) + result = self.db.execute( + update(PushDelivery) + .where( + PushDelivery.id == delivery.id, + PushDelivery.status == PushDeliveryStatus.PROCESSING, + PushDelivery.locked_by == lock_owner, + ) + .values( + status=status_value, + next_attempt_at=quiet_end, + locked_by=None, + locked_until=None, + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + self.db.rollback() + return self._get(delivery.code) + self.db.commit() + return self._get(delivery.code) + + def _generation_request( + self, + subscription: PushSubscription, + ) -> DeliveryGenerationRequest: + personal = subscription.target_type == SubscriptionTargetType.USER + return DeliveryGenerationRequest( + prompt=subscription.prompt, + owner_id=subscription.owner_id if personal else None, + use_personal_context=personal, + use_company_rules=not personal, + ) + + def _skip_reason( + self, + subscription: PushSubscription | None, + owner: FeishuUser | None, + ) -> str | None: + if subscription is None: + return "Subscription was removed" + if owner is None or owner.status != FeishuUserStatus.ACTIVE: + return "Feishu user is disabled" + if subscription.status not in { + PushSubscriptionStatus.ACTIVE, + PushSubscriptionStatus.COMPLETED, + }: + return "Subscription is not active" + if subscription.target_type == SubscriptionTargetType.USER: + if subscription.target_id != owner.open_id: + return "Private subscription target no longer matches its owner" + return None + if subscription.target_type == SubscriptionTargetType.CHAT: + if owner.role != FeishuUserRole.ADMIN or not subscription.target_id: + return "Group subscription is no longer authorized" + return None + return "Unsupported subscription target" + + def _load_context( + self, + subscription_id: int, + ) -> tuple[PushSubscription | None, FeishuUser | None]: + subscription = self.db.get(PushSubscription, subscription_id) + owner = ( + self.db.get(FeishuUser, subscription.owner_id) + if subscription is not None + else None + ) + return subscription, owner + + def _lock_send_context( + self, + subscription_id: int, + ) -> tuple[PushSubscription | None, FeishuUser | None]: + """Recheck authorization and serialize the final per-owner send decision.""" + + subscription = self.db.execute( + select(PushSubscription) + .where(PushSubscription.id == subscription_id) + .with_for_update() + .execution_options(populate_existing=True) + ).scalar_one_or_none() + owner = ( + self.db.execute( + select(FeishuUser) + .where(FeishuUser.id == subscription.owner_id) + .with_for_update() + .execution_options(populate_existing=True) + ).scalar_one_or_none() + if subscription is not None + else None + ) + return subscription, owner + + def _sent_today(self, owner: FeishuUser, current: datetime) -> int: + zone = ZoneInfo(owner.timezone) + local_now = current.replace(tzinfo=UTC).astimezone(zone) + local_start = datetime.combine(local_now.date(), time.min, tzinfo=zone) + local_end = local_start + timedelta(days=1) + start_utc = local_start.astimezone(UTC).replace(tzinfo=None) + end_utc = local_end.astimezone(UTC).replace(tzinfo=None) + return int( + self.db.scalar( + select(func.count()) + .select_from(PushDelivery) + .join(PushSubscription) + .where( + PushSubscription.owner_id == owner.id, + PushDelivery.status == PushDeliveryStatus.SENT, + PushDelivery.sent_at >= start_utc, + PushDelivery.sent_at < end_utc, + ) + ) + or 0 + ) + + def _get(self, delivery_code: str) -> PushDelivery: + record = self.db.execute( + select(PushDelivery).where(PushDelivery.code == delivery_code) + ).scalar_one_or_none() + if record is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=DELIVERY_NOT_FOUND, + ) + return record + + def _audit( + self, + owner: FeishuUser | None, + delivery: PushDelivery, + action: str, + response: dict[str, Any], + ) -> None: + AuditService(self.db).record( + AuditLogCreate( + actor=owner.code if owner is not None else "subscription-system", + source="subscriptions", + action=action, + target_type="push-delivery", + target_id=delivery.code, + response_payload=response, + ) + ) + + @staticmethod + def _validate_provider_response(response: dict[str, Any]) -> None: + if not isinstance(response, dict): + raise RetryableDeliveryError("Message provider returned an invalid response") + if "code" in response and response.get("code") != 0: + raise RetryableDeliveryError( + f"Message provider returned business code {response.get('code')}" + ) + + +def _provider_message_id(response: dict[str, Any]) -> str | None: + direct = response.get("message_id") + nested = response.get("data") + value = direct or (nested.get("message_id") if isinstance(nested, dict) else None) + return str(value) if value is not None else None + + +def _is_retryable(exc: Exception) -> bool: + if isinstance(exc, PermanentDeliveryError): + return False + if isinstance(exc, RetryableDeliveryError): + return True + if isinstance(exc, FeishuAPIError): + return exc.retryable + if isinstance(exc, httpx.HTTPStatusError): + code = exc.response.status_code + return code == 429 or code >= 500 + if isinstance(exc, (httpx.TimeoutException, httpx.NetworkError)): + return True + if isinstance(exc, HTTPException): + return exc.status_code == 429 or exc.status_code >= 500 + return True + + +def _naive_utc(value: datetime) -> datetime: + if value.tzinfo is None: + return value + return value.astimezone(UTC).replace(tzinfo=None) diff --git a/app/modules/subscriptions/services/management.py b/app/modules/subscriptions/services/management.py new file mode 100644 index 0000000..0c9c34f --- /dev/null +++ b/app/modules/subscriptions/services/management.py @@ -0,0 +1,498 @@ +from datetime import UTC, datetime +from typing import Any +from uuid import uuid4 + +from fastapi import HTTPException, status +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.core.http.pagination import bounded_limit, bounded_offset +from app.core.utils.time import utc_now +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu_users.constants import ( + FeishuCapability, + FeishuUserRole, + FeishuUserStatus, +) +from app.modules.feishu_users.models import FeishuUser +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.subscriptions.constants import ( + EMPTY_PROMPT, + INVALID_GROUP_TARGET, + INVALID_QUIET_HOURS, + MAX_ACTIVE_SUBSCRIPTIONS, + PushSubscriptionStatus, + SUBSCRIPTION_LIMIT_REACHED, + SUBSCRIPTION_NOT_FOUND, + SubscriptionAuditAction, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services.schedule import ( + NormalizedSchedule, + ScheduleParseError, + next_occurrence, + parse_quiet_clock, + parse_schedule, + validate_timezone, +) + + +class SubscriptionManagementService: + """Manage subscriptions only through authenticated Feishu principals.""" + + def __init__(self, db: Session): + self.db = db + + def create_private( + self, + principal: FeishuPrincipal, + schedule_expression: str, + prompt: str, + *, + now: datetime | None = None, + ) -> tuple[PushSubscription, NormalizedSchedule]: + principal.require_capability(FeishuCapability.PRIVATE_SUBSCRIPTION) + owner = self._active_owner(principal, for_update=True) + return self._create( + owner=owner, + target_type=SubscriptionTargetType.USER, + target_id=owner.open_id, + schedule_expression=schedule_expression, + prompt=prompt, + now=now, + ) + + def create_group( + self, + principal: FeishuPrincipal, + schedule_expression: str, + prompt: str, + *, + now: datetime | None = None, + ) -> tuple[PushSubscription, NormalizedSchedule]: + principal.require_capability(FeishuCapability.GROUP_SUBSCRIPTION) + owner = self._active_owner(principal, for_update=True) + if ( + owner.role != FeishuUserRole.ADMIN + or not principal.chat_id + or principal.chat_type not in {"group", "group_chat"} + ): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail=INVALID_GROUP_TARGET, + ) + return self._create( + owner=owner, + target_type=SubscriptionTargetType.CHAT, + target_id=principal.chat_id, + schedule_expression=schedule_expression, + prompt=prompt, + now=now, + ) + + def list_for_owner(self, principal: FeishuPrincipal) -> list[PushSubscription]: + principal.require_active() + return list( + self.db.execute( + select(PushSubscription) + .where(PushSubscription.owner_id == principal.owner_id) + .order_by(PushSubscription.id.desc()) + ).scalars() + ) + + def latest_deliveries_for_owner( + self, + principal: FeishuPrincipal, + ) -> dict[int, PushDelivery]: + """Return at most one latest delivery per owner-scoped subscription.""" + + principal.require_active() + latest_ids = ( + select(func.max(PushDelivery.id)) + .join(PushSubscription) + .where(PushSubscription.owner_id == principal.owner_id) + .group_by(PushDelivery.subscription_id) + ) + records = self.db.execute( + select(PushDelivery).where(PushDelivery.id.in_(latest_ids)) + ).scalars() + return {record.subscription_id: record for record in records} + + def pause(self, principal: FeishuPrincipal, code: str) -> PushSubscription: + principal.require_capability(FeishuCapability.PRIVATE_SUBSCRIPTION) + record = self._owned_subscription(principal.owner_id, code, for_update=True) + if record.status == PushSubscriptionStatus.ACTIVE: + record.status = PushSubscriptionStatus.PAUSED + self._audit(principal, SubscriptionAuditAction.PAUSE, record) + self.db.commit() + self.db.refresh(record) + return record + + def resume( + self, + principal: FeishuPrincipal, + code: str, + *, + now: datetime | None = None, + ) -> PushSubscription: + principal.require_capability(FeishuCapability.PRIVATE_SUBSCRIPTION) + self._active_owner(principal, for_update=True) + record = self._owned_subscription(principal.owner_id, code, for_update=True) + if record.status != PushSubscriptionStatus.PAUSED: + return record + self._ensure_active_capacity(principal.owner_id) + current = _naive_utc(now or utc_now()) + if record.next_run_at is None or record.next_run_at <= current: + next_run = next_occurrence( + record.schedule_type, + record.schedule_config, + record.timezone, + after=current, + ) + if next_run is None: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail="Expired one-time subscriptions cannot be resumed", + ) + record.next_run_at = next_run + record.status = PushSubscriptionStatus.ACTIVE + self._audit(principal, SubscriptionAuditAction.RESUME, record) + self.db.commit() + self.db.refresh(record) + return record + + def cancel(self, principal: FeishuPrincipal, code: str) -> PushSubscription: + principal.require_capability(FeishuCapability.PRIVATE_SUBSCRIPTION) + record = self._owned_subscription(principal.owner_id, code, for_update=True) + if record.status != PushSubscriptionStatus.CANCELLED: + record.status = PushSubscriptionStatus.CANCELLED + record.next_run_at = None + self._audit(principal, SubscriptionAuditAction.CANCEL, record) + self.db.commit() + self.db.refresh(record) + return record + + def set_timezone( + self, + principal: FeishuPrincipal, + timezone_name: str, + *, + now: datetime | None = None, + ) -> FeishuUser: + principal.require_active() + try: + validate_timezone(timezone_name) + except ScheduleParseError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=str(exc), + ) from exc + owner = self._active_owner(principal, for_update=True) + owner.timezone = timezone_name + current = _naive_utc(now or utc_now()) + subscriptions = list( + self.db.execute( + select(PushSubscription).where( + PushSubscription.owner_id == owner.id, + PushSubscription.status.in_( + [ + PushSubscriptionStatus.ACTIVE, + PushSubscriptionStatus.PAUSED, + ] + ), + ) + ).scalars() + ) + for subscription in subscriptions: + subscription.timezone = timezone_name + if ( + subscription.status == PushSubscriptionStatus.ACTIVE + and subscription.schedule_type + not in { + SubscriptionScheduleType.ONCE, + SubscriptionScheduleType.INTERVAL, + } + ): + subscription.next_run_at = next_occurrence( + subscription.schedule_type, + subscription.schedule_config, + timezone_name, + after=current, + ) + self._audit_user( + principal, + SubscriptionAuditAction.UPDATE_TIMEZONE, + {"timezone": timezone_name}, + ) + self.db.commit() + self.db.refresh(owner) + return owner + + def set_quiet_hours( + self, + principal: FeishuPrincipal, + start: str, + end: str, + ) -> FeishuUser: + principal.require_active() + try: + quiet_start = parse_quiet_clock(start) + quiet_end = parse_quiet_clock(end) + except ScheduleParseError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=str(exc), + ) from exc + if quiet_start == quiet_end: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=INVALID_QUIET_HOURS, + ) + owner = self._active_owner(principal, for_update=True) + owner.quiet_hours_start = quiet_start + owner.quiet_hours_end = quiet_end + self._audit_user( + principal, + SubscriptionAuditAction.UPDATE_QUIET_HOURS, + {"enabled": True}, + ) + self.db.commit() + self.db.refresh(owner) + return owner + + def clear_quiet_hours(self, principal: FeishuPrincipal) -> FeishuUser: + principal.require_active() + owner = self._active_owner(principal, for_update=True) + owner.quiet_hours_start = None + owner.quiet_hours_end = None + self._audit_user( + principal, + SubscriptionAuditAction.UPDATE_QUIET_HOURS, + {"enabled": False}, + ) + self.db.commit() + self.db.refresh(owner) + return owner + + def list_all( + self, + *, + status_filter: str | None = None, + owner_id: int | None = None, + limit: int = 100, + offset: int = 0, + ) -> tuple[int, list[PushSubscription]]: + stmt = select(PushSubscription) + count_stmt = select(func.count()).select_from(PushSubscription) + if status_filter: + stmt = stmt.where(PushSubscription.status == status_filter) + count_stmt = count_stmt.where(PushSubscription.status == status_filter) + if owner_id is not None: + stmt = stmt.where(PushSubscription.owner_id == owner_id) + count_stmt = count_stmt.where(PushSubscription.owner_id == owner_id) + stmt = ( + stmt.order_by(PushSubscription.id.desc()) + .limit(bounded_limit(limit)) + .offset(bounded_offset(offset)) + ) + total = int(self.db.scalar(count_stmt) or 0) + return total, list(self.db.execute(stmt).scalars()) + + def list_deliveries( + self, + *, + status_filter: str | None = None, + subscription_code: str | None = None, + limit: int = 100, + offset: int = 0, + ) -> tuple[int, list[PushDelivery]]: + stmt = select(PushDelivery).join(PushSubscription) + count_stmt = ( + select(func.count()) + .select_from(PushDelivery) + .join(PushSubscription) + ) + if status_filter: + stmt = stmt.where(PushDelivery.status == status_filter) + count_stmt = count_stmt.where(PushDelivery.status == status_filter) + if subscription_code: + stmt = stmt.where(PushSubscription.code == subscription_code) + count_stmt = count_stmt.where(PushSubscription.code == subscription_code) + stmt = ( + stmt.order_by(PushDelivery.id.desc()) + .limit(bounded_limit(limit)) + .offset(bounded_offset(offset)) + ) + total = int(self.db.scalar(count_stmt) or 0) + return total, list(self.db.execute(stmt).scalars()) + + def _create( + self, + *, + owner: FeishuUser, + target_type: str, + target_id: str, + schedule_expression: str, + prompt: str, + now: datetime | None, + ) -> tuple[PushSubscription, NormalizedSchedule]: + clean_prompt = str(prompt or "").strip() + if not clean_prompt: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=EMPTY_PROMPT, + ) + self._ensure_active_capacity(owner.id) + try: + schedule = parse_schedule( + schedule_expression, + owner.timezone, + now=now, + ) + except ScheduleParseError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=str(exc), + ) from exc + record = PushSubscription( + code=f"SUB-{uuid4().hex.upper()}", + owner_id=owner.id, + target_type=target_type, + target_id=target_id, + prompt=clean_prompt, + schedule_type=schedule.schedule_type, + schedule_config=schedule.schedule_config, + timezone=schedule.timezone, + next_run_at=schedule.next_run_at, + status=PushSubscriptionStatus.ACTIVE, + consented_at=_naive_utc(now or utc_now()), + ) + self.db.add(record) + self.db.flush() + self._audit_values( + actor=owner.code, + action=SubscriptionAuditAction.CREATE, + target_id=record.code, + response={ + "target_type": target_type, + "schedule_type": schedule.schedule_type, + }, + ) + self.db.commit() + self.db.refresh(record) + return record, schedule + + def _ensure_active_capacity(self, owner_id: int) -> None: + count = int( + self.db.scalar( + select(func.count()) + .select_from(PushSubscription) + .where( + PushSubscription.owner_id == owner_id, + PushSubscription.status == PushSubscriptionStatus.ACTIVE, + ) + ) + or 0 + ) + if count >= MAX_ACTIVE_SUBSCRIPTIONS: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=SUBSCRIPTION_LIMIT_REACHED, + ) + + def _active_owner( + self, + principal: FeishuPrincipal, + *, + for_update: bool = False, + ) -> FeishuUser: + stmt = select(FeishuUser).where(FeishuUser.id == principal.owner_id) + if for_update: + stmt = stmt.with_for_update() + owner = self.db.execute(stmt).scalar_one_or_none() + if owner is None or owner.status != FeishuUserStatus.ACTIVE: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu user is disabled", + ) + if owner.tenant_key != principal.tenant_key or owner.open_id != principal.open_id: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Feishu identity mismatch", + ) + return owner + + def _owned_subscription( + self, + owner_id: int, + code: str, + *, + for_update: bool = False, + ) -> PushSubscription: + stmt = select(PushSubscription).where( + PushSubscription.owner_id == owner_id, + PushSubscription.code == code, + ) + if for_update: + stmt = stmt.with_for_update() + record = self.db.execute(stmt).scalar_one_or_none() + if record is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=SUBSCRIPTION_NOT_FOUND, + ) + return record + + def _audit( + self, + principal: FeishuPrincipal, + action: str, + record: PushSubscription, + ) -> None: + self._audit_values( + actor=principal.user_code, + action=action, + target_id=record.code, + response={"status": record.status}, + ) + + def _audit_user( + self, + principal: FeishuPrincipal, + action: str, + response: dict[str, Any], + ) -> None: + self._audit_values( + actor=principal.user_code, + action=action, + target_id=principal.user_code, + response=response, + ) + + def _audit_values( + self, + *, + actor: str, + action: str, + target_id: str, + response: dict[str, Any], + ) -> None: + AuditService(self.db).record( + AuditLogCreate( + actor=actor, + source="subscriptions", + action=action, + target_type="subscription", + target_id=target_id, + response_payload=response, + ) + ) + + +def _naive_utc(value: datetime) -> datetime: + if value.tzinfo is None: + return value + return value.astimezone(UTC).replace(tzinfo=None) diff --git a/app/modules/subscriptions/services/scanner.py b/app/modules/subscriptions/services/scanner.py new file mode 100644 index 0000000..7d04ceb --- /dev/null +++ b/app/modules/subscriptions/services/scanner.py @@ -0,0 +1,309 @@ +from datetime import UTC, datetime, time, timedelta +from hashlib import sha256 +from uuid import NAMESPACE_URL, uuid4, uuid5 + +from sqlalchemy import func, or_, select, update +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.core.http.pagination import bounded_limit +from app.core.utils.time import utc_now +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.feishu_users.constants import FeishuUserRole, FeishuUserStatus +from app.modules.feishu_users.models import FeishuUser +from app.modules.subscriptions.constants import ( + DAILY_DELIVERY_LIMIT_REACHED, + MAX_DAILY_DELIVERIES, + PushDeliveryStatus, + PushSubscriptionStatus, + SUBSCRIPTION_LEASE_SECONDS, + SubscriptionAuditAction, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services.schedule import ( + is_in_quiet_hours, + next_occurrence, + next_quiet_end, + validate_timezone, +) + + +class SubscriptionScanner: + """Claim due plans and materialize one durable delivery per schedule window.""" + + def __init__(self, db: Session, *, lease_seconds: int = SUBSCRIPTION_LEASE_SECONDS): + self.db = db + self.lease_seconds = lease_seconds + + def scan_due( + self, + *, + now: datetime | None = None, + limit: int = 100, + worker_id: str = "subscription-scanner", + ) -> list[PushDelivery]: + current = _naive_utc(now or utc_now()) + claims = self._claim_due_subscriptions( + current=current, + limit=limit, + worker_id=worker_id, + ) + deliveries: list[PushDelivery] = [] + for subscription_id, lock_owner in claims: + try: + delivery = self._materialize_delivery( + subscription_id=subscription_id, + lock_owner=lock_owner, + current=current, + ) + except Exception: + self.db.rollback() + self._release_claim(subscription_id, lock_owner) + raise + if delivery is not None: + deliveries.append(delivery) + return deliveries + + def _claim_due_subscriptions( + self, + *, + current: datetime, + limit: int, + worker_id: str, + ) -> list[tuple[int, str]]: + stmt = ( + select(PushSubscription.id) + .where( + PushSubscription.status == PushSubscriptionStatus.ACTIVE, + PushSubscription.next_run_at.is_not(None), + PushSubscription.next_run_at <= current, + or_( + PushSubscription.locked_until.is_(None), + PushSubscription.locked_until <= current, + ), + ) + .order_by(PushSubscription.next_run_at.asc(), PushSubscription.id.asc()) + .limit(bounded_limit(limit)) + ) + if self.db.get_bind().dialect.name == "postgresql": + stmt = stmt.with_for_update(skip_locked=True) + candidate_ids = list(self.db.execute(stmt).scalars()) + claims: list[tuple[int, str]] = [] + locked_until = current + timedelta(seconds=self.lease_seconds) + for subscription_id in candidate_ids: + lock_owner = f"{worker_id}:{uuid4().hex}" + result = self.db.execute( + update(PushSubscription) + .where( + PushSubscription.id == subscription_id, + PushSubscription.status == PushSubscriptionStatus.ACTIVE, + PushSubscription.next_run_at.is_not(None), + PushSubscription.next_run_at <= current, + or_( + PushSubscription.locked_until.is_(None), + PushSubscription.locked_until <= current, + ), + ) + .values(locked_by=lock_owner, locked_until=locked_until) + .execution_options(synchronize_session=False) + ) + if result.rowcount == 1: + claims.append((subscription_id, lock_owner)) + self.db.commit() + return claims + + def _materialize_delivery( + self, + *, + subscription_id: int, + lock_owner: str, + current: datetime, + ) -> PushDelivery | None: + subscription = self.db.execute( + select(PushSubscription) + .where( + PushSubscription.id == subscription_id, + PushSubscription.locked_by == lock_owner, + ) + .with_for_update() + ).scalar_one_or_none() + if subscription is None: + self.db.rollback() + return None + if ( + subscription.status != PushSubscriptionStatus.ACTIVE + or subscription.next_run_at is None + ): + subscription.locked_by = None + subscription.locked_until = None + self.db.commit() + return None + + owner = self.db.execute( + select(FeishuUser) + .where(FeishuUser.id == subscription.owner_id) + .with_for_update() + ).scalar_one_or_none() + scheduled_for = subscription.next_run_at + skip_reason = self._delivery_skip_reason(subscription, owner, current) + next_attempt_at = current + if ( + skip_reason is None + and owner is not None + and is_in_quiet_hours( + current, + owner.timezone, + owner.quiet_hours_start, + owner.quiet_hours_end, + ) + ): + next_attempt_at = next_quiet_end( + current, + owner.timezone, + owner.quiet_hours_start, + owner.quiet_hours_end, + ) + + idempotency_key = _delivery_key(subscription.id, scheduled_for) + message_uuid = str(uuid5(NAMESPACE_URL, f"company-ai-platform:{idempotency_key}")) + delivery = PushDelivery( + code=f"DEL-{uuid4().hex.upper()}", + subscription_id=subscription.id, + scheduled_for=scheduled_for, + idempotency_key=idempotency_key, + message_uuid=message_uuid, + status=( + PushDeliveryStatus.SKIPPED + if skip_reason is not None + else PushDeliveryStatus.PENDING + ), + next_attempt_at=None if skip_reason is not None else next_attempt_at, + last_error=skip_reason, + created_at=current, + updated_at=current, + ) + try: + with self.db.begin_nested(): + self.db.add(delivery) + self.db.flush() + except IntegrityError: + delivery = self.db.execute( + select(PushDelivery).where( + PushDelivery.idempotency_key == idempotency_key + ) + ).scalar_one() + + subscription.last_run_at = scheduled_for + if subscription.schedule_type == SubscriptionScheduleType.ONCE: + subscription.status = PushSubscriptionStatus.COMPLETED + subscription.next_run_at = None + else: + subscription.next_run_at = next_occurrence( + subscription.schedule_type, + subscription.schedule_config, + subscription.timezone, + after=current, + ) + subscription.locked_by = None + subscription.locked_until = None + if skip_reason is not None: + self._audit_skipped(owner, delivery, skip_reason) + self.db.commit() + self.db.refresh(delivery) + return delivery + + def _delivery_skip_reason( + self, + subscription: PushSubscription, + owner: FeishuUser | None, + current: datetime, + ) -> str | None: + if owner is None or owner.status != FeishuUserStatus.ACTIVE: + return "Feishu user is disabled" + if ( + subscription.target_type == SubscriptionTargetType.USER + and subscription.target_id != owner.open_id + ): + return "Private subscription target no longer matches its owner" + if subscription.target_type == SubscriptionTargetType.CHAT and ( + owner.role != FeishuUserRole.ADMIN or not subscription.target_id + ): + return "Group subscription owner is no longer an administrator" + if subscription.target_type not in { + SubscriptionTargetType.USER, + SubscriptionTargetType.CHAT, + }: + return "Unsupported subscription target" + if self._daily_delivery_count(owner, current) >= MAX_DAILY_DELIVERIES: + return DAILY_DELIVERY_LIMIT_REACHED + return None + + def _daily_delivery_count(self, owner: FeishuUser, current: datetime) -> int: + zone = validate_timezone(owner.timezone) + local_now = current.replace(tzinfo=UTC).astimezone(zone) + local_start = datetime.combine(local_now.date(), time.min, tzinfo=zone) + local_end = local_start + timedelta(days=1) + start_utc = local_start.astimezone(UTC).replace(tzinfo=None) + end_utc = local_end.astimezone(UTC).replace(tzinfo=None) + return int( + self.db.scalar( + select(func.count()) + .select_from(PushDelivery) + .join(PushSubscription) + .where( + PushSubscription.owner_id == owner.id, + PushDelivery.created_at >= start_utc, + PushDelivery.created_at < end_utc, + PushDelivery.status.not_in( + [ + PushDeliveryStatus.FAILED, + PushDeliveryStatus.SKIPPED, + ] + ), + ) + ) + or 0 + ) + + def _audit_skipped( + self, + owner: FeishuUser | None, + delivery: PushDelivery, + reason: str, + ) -> None: + AuditService(self.db).record( + AuditLogCreate( + actor=owner.code if owner is not None else "subscription-system", + source="subscriptions", + action=SubscriptionAuditAction.DELIVERY_SKIPPED, + target_type="push-delivery", + target_id=delivery.code, + response_payload={"status": PushDeliveryStatus.SKIPPED, "reason": reason}, + ) + ) + + def _release_claim(self, subscription_id: int, lock_owner: str) -> None: + self.db.execute( + update(PushSubscription) + .where( + PushSubscription.id == subscription_id, + PushSubscription.locked_by == lock_owner, + ) + .values(locked_by=None, locked_until=None) + .execution_options(synchronize_session=False) + ) + self.db.commit() + + +def _delivery_key(subscription_id: int, scheduled_for: datetime) -> str: + material = f"{subscription_id}:{_naive_utc(scheduled_for).isoformat(timespec='microseconds')}" + return sha256(material.encode("utf-8")).hexdigest() + + +def _naive_utc(value: datetime) -> datetime: + if value.tzinfo is None: + return value + return value.astimezone(UTC).replace(tzinfo=None) diff --git a/app/modules/subscriptions/services/schedule.py b/app/modules/subscriptions/services/schedule.py new file mode 100644 index 0000000..6cc3f0c --- /dev/null +++ b/app/modules/subscriptions/services/schedule.py @@ -0,0 +1,462 @@ +import re +from calendar import monthrange +from dataclasses import dataclass +from datetime import UTC, date, datetime, time, timedelta +from typing import Any +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from app.modules.subscriptions.constants import ( + INVALID_SCHEDULE, + INVALID_TIMEZONE, + MIN_INTERVAL_MINUTES, + SubscriptionScheduleType, +) + +_WEEKDAYS = { + "一": 0, + "二": 1, + "三": 2, + "四": 3, + "五": 4, + "六": 5, + "日": 6, + "天": 6, +} +_WEEKDAY_NAMES = ("一", "二", "三", "四", "五", "六", "日") +_INTERVAL_PATTERN = re.compile(r"每隔\s*(?P\d+)\s*(?P分钟|小时)") +_DAILY_PATTERN = re.compile(r"每天\s*(?P.+)") +_WEEKDAY_PATTERN = re.compile(r"(?:每个)?工作日\s*(?P.+)") +_WEEKLY_PATTERN = re.compile(r"每周(?P[一二三四五六日天])\s*(?P.+)") +_MONTHLY_PATTERN = re.compile( + r"每月\s*(?P\d{1,2})\s*(?:号|日)\s*(?P.+)" +) +_RELATIVE_PATTERN = re.compile(r"(?P今天|明天)\s*(?P.+)") +_ISO_DATE_PATTERN = re.compile( + r"(?P\d{4})[-/](?P\d{1,2})[-/](?P\d{1,2})" + r"\s+(?P.+)" +) +_CHINESE_DATE_PATTERN = re.compile( + r"(?P\d{4})年(?P\d{1,2})月(?P\d{1,2})[日号]" + r"\s*(?P.+)" +) +_COLON_CLOCK_PATTERN = re.compile(r"(?P\d{1,2}):(?P\d{1,2})") +_CHINESE_CLOCK_PATTERN = re.compile( + r"(?P\d{1,2})点(?:(?P半)|(?P\d{1,2})分?)?" +) + + +class ScheduleParseError(ValueError): + """Raised when a controlled schedule expression cannot be normalized.""" + + +@dataclass(frozen=True, slots=True) +class NormalizedSchedule: + schedule_type: str + schedule_config: dict[str, Any] + timezone: str + next_run_at: datetime + display: str + + +def parse_schedule( + expression: str, + timezone_name: str, + *, + now: datetime | None = None, +) -> NormalizedSchedule: + """Parse the supported Chinese schedule grammar into a UTC plan.""" + + text = _normalize_expression(expression) + zone = validate_timezone(timezone_name) + now_utc = _as_utc(now) + local_now = now_utc.astimezone(zone) + + match = _INTERVAL_PATTERN.fullmatch(text) + if match: + value = int(match.group("value")) + minutes = value * (60 if match.group("unit") == "小时" else 1) + if minutes < MIN_INTERVAL_MINUTES: + raise ScheduleParseError(f"订阅间隔不得短于 {MIN_INTERVAL_MINUTES} 分钟") + try: + next_run = now_utc + timedelta(minutes=minutes) + except OverflowError as exc: + raise ScheduleParseError("订阅间隔过大") from exc + config = { + "minutes": minutes, + "anchor_at": _to_naive_utc(next_run).isoformat(), + } + display_value = ( + f"每隔 {value} 小时" if match.group("unit") == "小时" else f"每隔 {value} 分钟" + ) + return NormalizedSchedule( + schedule_type=SubscriptionScheduleType.INTERVAL, + schedule_config=config, + timezone=timezone_name, + next_run_at=_to_naive_utc(next_run), + display=display_value, + ) + + match = _DAILY_PATTERN.fullmatch(text) + if match: + hour, minute = _parse_clock(match.group("clock")) + config = {"hour": hour, "minute": minute} + return _recurring_schedule( + SubscriptionScheduleType.DAILY, + config, + timezone_name, + now_utc, + f"每天 {hour:02d}:{minute:02d}", + ) + + match = _WEEKDAY_PATTERN.fullmatch(text) + if match: + hour, minute = _parse_clock(match.group("clock")) + config = {"hour": hour, "minute": minute} + return _recurring_schedule( + SubscriptionScheduleType.WEEKDAY, + config, + timezone_name, + now_utc, + f"工作日 {hour:02d}:{minute:02d}", + ) + + match = _WEEKLY_PATTERN.fullmatch(text) + if match: + hour, minute = _parse_clock(match.group("clock")) + weekday = _WEEKDAYS[match.group("weekday")] + config = {"weekday": weekday, "hour": hour, "minute": minute} + return _recurring_schedule( + SubscriptionScheduleType.WEEKLY, + config, + timezone_name, + now_utc, + f"每周{_WEEKDAY_NAMES[weekday]} {hour:02d}:{minute:02d}", + ) + + match = _MONTHLY_PATTERN.fullmatch(text) + if match: + day = int(match.group("day")) + if not 1 <= day <= 31: + raise ScheduleParseError("每月日期必须在 1 到 31 之间") + hour, minute = _parse_clock(match.group("clock")) + config = {"day": day, "hour": hour, "minute": minute} + return _recurring_schedule( + SubscriptionScheduleType.MONTHLY, + config, + timezone_name, + now_utc, + f"每月 {day} 号 {hour:02d}:{minute:02d}", + ) + + match = _RELATIVE_PATTERN.fullmatch(text) + if match: + hour, minute = _parse_clock(match.group("clock")) + offset = 1 if match.group("day") == "明天" else 0 + target_date = local_now.date() + timedelta(days=offset) + return _once_schedule( + target_date, + hour, + minute, + timezone_name, + now_utc, + f"{match.group('day')} {hour:02d}:{minute:02d}", + ) + + match = _ISO_DATE_PATTERN.fullmatch(text) or _CHINESE_DATE_PATTERN.fullmatch(text) + if match: + try: + target_date = date( + int(match.group("year")), + int(match.group("month")), + int(match.group("day")), + ) + except ValueError as exc: + raise ScheduleParseError("日期不存在") from exc + hour, minute = _parse_clock(match.group("clock")) + return _once_schedule( + target_date, + hour, + minute, + timezone_name, + now_utc, + f"{target_date.isoformat()} {hour:02d}:{minute:02d}", + ) + + raise ScheduleParseError( + f"{INVALID_SCHEDULE}。示例：每天 09:00、每周一 18:00、每隔 30 分钟" + ) + + +def next_occurrence( + schedule_type: str, + schedule_config: dict[str, Any], + timezone_name: str, + *, + after: datetime, +) -> datetime | None: + """Return the first UTC occurrence strictly after ``after``.""" + + zone = validate_timezone(timezone_name) + after_utc = _as_utc(after) + plan_type = SubscriptionScheduleType(schedule_type) + + if plan_type == SubscriptionScheduleType.ONCE: + run_at = _parse_stored_utc(schedule_config["run_at"]) + return _to_naive_utc(run_at) if run_at > after_utc else None + + if plan_type == SubscriptionScheduleType.INTERVAL: + interval = timedelta(minutes=int(schedule_config["minutes"])) + anchor = _parse_stored_utc(schedule_config["anchor_at"]) + if anchor > after_utc: + return _to_naive_utc(anchor) + elapsed = after_utc - anchor + steps = elapsed // interval + 1 + return _to_naive_utc(anchor + interval * steps) + + hour = int(schedule_config["hour"]) + minute = int(schedule_config["minute"]) + local_after = after_utc.astimezone(zone) + + if plan_type == SubscriptionScheduleType.DAILY: + return _next_daily(local_after, hour, minute, zone) + if plan_type == SubscriptionScheduleType.WEEKDAY: + return _next_weekday(local_after, hour, minute, zone) + if plan_type == SubscriptionScheduleType.WEEKLY: + weekday = int(schedule_config["weekday"]) + return _next_weekly(local_after, weekday, hour, minute, zone) + if plan_type == SubscriptionScheduleType.MONTHLY: + day = int(schedule_config["day"]) + return _next_monthly(local_after, day, hour, minute, zone) + raise ScheduleParseError(INVALID_SCHEDULE) + + +def is_in_quiet_hours( + current: datetime, + timezone_name: str, + quiet_start: time | str | None, + quiet_end: time | str | None, +) -> bool: + if quiet_start is None or quiet_end is None: + return False + start = _coerce_time(quiet_start) + end = _coerce_time(quiet_end) + if start == end: + return False + local_time = _as_utc(current).astimezone(validate_timezone(timezone_name)).time() + local_time = local_time.replace(tzinfo=None) + if start < end: + return start <= local_time < end + return local_time >= start or local_time < end + + +def next_quiet_end( + current: datetime, + timezone_name: str, + quiet_start: time | str, + quiet_end: time | str, +) -> datetime: + """Return quiet-window end as a naive UTC timestamp.""" + + zone = validate_timezone(timezone_name) + now_local = _as_utc(current).astimezone(zone) + start = _coerce_time(quiet_start) + end = _coerce_time(quiet_end) + end_date = now_local.date() + if start > end and now_local.time().replace(tzinfo=None) >= start: + end_date += timedelta(days=1) + candidate = _local_candidate(end_date, end.hour, end.minute, zone) + if candidate is None: + candidate = _first_valid_local_after(end_date, end.hour, end.minute, zone) + return _to_naive_utc(candidate) + + +def validate_timezone(timezone_name: str) -> ZoneInfo: + try: + return ZoneInfo(timezone_name) + except (ZoneInfoNotFoundError, ValueError, TypeError) as exc: + raise ScheduleParseError(INVALID_TIMEZONE) from exc + + +def parse_quiet_clock(value: str) -> time: + hour, minute = _parse_clock(_normalize_expression(value)) + return time(hour=hour, minute=minute) + + +def _recurring_schedule( + schedule_type: str, + config: dict[str, Any], + timezone_name: str, + now_utc: datetime, + display: str, +) -> NormalizedSchedule: + next_run = next_occurrence( + schedule_type, + config, + timezone_name, + after=now_utc, + ) + if next_run is None: + raise ScheduleParseError(INVALID_SCHEDULE) + return NormalizedSchedule( + schedule_type=schedule_type, + schedule_config=config, + timezone=timezone_name, + next_run_at=next_run, + display=display, + ) + + +def _once_schedule( + target_date: date, + hour: int, + minute: int, + timezone_name: str, + now_utc: datetime, + display: str, +) -> NormalizedSchedule: + zone = validate_timezone(timezone_name) + target = _local_candidate(target_date, hour, minute, zone) + if target is None: + raise ScheduleParseError("该本地时间不存在") + if target <= now_utc: + raise ScheduleParseError("执行时间必须晚于当前时间") + run_at = _to_naive_utc(target) + return NormalizedSchedule( + schedule_type=SubscriptionScheduleType.ONCE, + schedule_config={"run_at": run_at.isoformat()}, + timezone=timezone_name, + next_run_at=run_at, + display=display, + ) + + +def _next_daily(local_after: datetime, hour: int, minute: int, zone: ZoneInfo) -> datetime: + for offset in range(0, 370): + candidate = _local_candidate(local_after.date() + timedelta(days=offset), hour, minute, zone) + if candidate is not None and candidate > local_after.astimezone(UTC): + return _to_naive_utc(candidate) + raise ScheduleParseError(INVALID_SCHEDULE) + + +def _next_weekday(local_after: datetime, hour: int, minute: int, zone: ZoneInfo) -> datetime: + for offset in range(0, 14): + target_date = local_after.date() + timedelta(days=offset) + if target_date.weekday() >= 5: + continue + candidate = _local_candidate(target_date, hour, minute, zone) + if candidate is not None and candidate > local_after.astimezone(UTC): + return _to_naive_utc(candidate) + raise ScheduleParseError(INVALID_SCHEDULE) + + +def _next_weekly( + local_after: datetime, + weekday: int, + hour: int, + minute: int, + zone: ZoneInfo, +) -> datetime: + offset = (weekday - local_after.weekday()) % 7 + for weeks in range(0, 3): + target_date = local_after.date() + timedelta(days=offset + weeks * 7) + candidate = _local_candidate(target_date, hour, minute, zone) + if candidate is not None and candidate > local_after.astimezone(UTC): + return _to_naive_utc(candidate) + raise ScheduleParseError(INVALID_SCHEDULE) + + +def _next_monthly( + local_after: datetime, + day: int, + hour: int, + minute: int, + zone: ZoneInfo, +) -> datetime: + year = local_after.year + month = local_after.month + for _ in range(0, 240): + if day <= monthrange(year, month)[1]: + candidate = _local_candidate(date(year, month, day), hour, minute, zone) + if candidate is not None and candidate > local_after.astimezone(UTC): + return _to_naive_utc(candidate) + month += 1 + if month == 13: + year += 1 + month = 1 + raise ScheduleParseError(INVALID_SCHEDULE) + + +def _normalize_expression(expression: str) -> str: + text = re.sub(r"\s+", " ", str(expression or "").strip()).replace("：", ":") + if not text: + raise ScheduleParseError(INVALID_SCHEDULE) + return text + + +def _parse_clock(value: str) -> tuple[int, int]: + text = value.strip().replace("：", ":") + match = _COLON_CLOCK_PATTERN.fullmatch(text) + if match: + hour = int(match.group("hour")) + minute = int(match.group("minute")) + else: + match = _CHINESE_CLOCK_PATTERN.fullmatch(text) + if not match: + raise ScheduleParseError("时间必须使用 HH:MM 或 H点M分") + hour = int(match.group("hour")) + minute = 30 if match.group("half") else int(match.group("minute") or 0) + if not 0 <= hour <= 23 or not 0 <= minute <= 59: + raise ScheduleParseError("时间超出有效范围") + return hour, minute + + +def _local_candidate( + target_date: date, + hour: int, + minute: int, + zone: ZoneInfo, +) -> datetime | None: + naive = datetime.combine(target_date, time(hour=hour, minute=minute)) + aware = naive.replace(tzinfo=zone) + roundtrip = aware.astimezone(UTC).astimezone(zone).replace(tzinfo=None) + if roundtrip != naive: + return None + return aware.astimezone(UTC) + + +def _first_valid_local_after( + target_date: date, + hour: int, + minute: int, + zone: ZoneInfo, +) -> datetime: + base = datetime.combine(target_date, time(hour=hour, minute=minute)) + for offset in range(0, 181): + candidate = base + timedelta(minutes=offset) + aware = _local_candidate(candidate.date(), candidate.hour, candidate.minute, zone) + if aware is not None: + return aware + raise ScheduleParseError("安静时段结束时间无效") + + +def _coerce_time(value: time | str) -> time: + if isinstance(value, time): + return value.replace(tzinfo=None, second=0, microsecond=0) + return parse_quiet_clock(value) + + +def _as_utc(value: datetime | None) -> datetime: + if value is None: + return datetime.now(UTC) + if value.tzinfo is None: + return value.replace(tzinfo=UTC) + return value.astimezone(UTC) + + +def _to_naive_utc(value: datetime) -> datetime: + return _as_utc(value).replace(tzinfo=None) + + +def _parse_stored_utc(value: str | datetime) -> datetime: + parsed = value if isinstance(value, datetime) else datetime.fromisoformat(str(value)) + return _as_utc(parsed) diff --git a/app/modules/workflows/models.py b/app/modules/workflows/models.py index 8d36712..360e35a 100644 --- a/app/modules/workflows/models.py +++ b/app/modules/workflows/models.py @@ -47,6 +47,16 @@ class WorkflowAction(Base): ForeignKey("workflow_instances.code", ondelete="RESTRICT"), index=True, ) + source_event_id: Mapped[str | None] = mapped_column( + ForeignKey( + "domain_events.event_id", + name="fk_workflow_actions_source_event_id", + ondelete="RESTRICT", + ), + nullable=True, + unique=True, + index=True, + ) action: Mapped[str] = mapped_column(String(128), index=True) actor: Mapped[str] = mapped_column(String(128), default=ActorValue.SYSTEM, index=True) from_status: Mapped[str | None] = mapped_column(String(32), nullable=True) diff --git a/app/modules/workflows/service.py b/app/modules/workflows/service.py index a7ff682..554c90d 100644 --- a/app/modules/workflows/service.py +++ b/app/modules/workflows/service.py @@ -3,6 +3,7 @@ from uuid import uuid4 from fastapi import HTTPException, status from sqlalchemy import func, select +from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import Session from app.core.constants import ActorValue @@ -34,20 +35,30 @@ class WorkflowService: actor: str = ActorValue.SYSTEM, payload: dict[str, Any] | None = None, commit: bool = True, + source_event_id: str | None = None, ) -> WorkflowInstance: + existing_workflow = self._workflow_for_source_event(source_event_id) + if existing_workflow is not None: + return existing_workflow + aggregate_id_text = str(aggregate_id) if aggregate_id is not None else None - record = self.db.execute( + workflow_query = ( select(WorkflowInstance).where( WorkflowInstance.workflow_type == workflow_type, WorkflowInstance.aggregate_type == aggregate_type, WorkflowInstance.aggregate_id == aggregate_id_text, ) - ).scalar_one_or_none() + .with_for_update() + ) + record = self.db.execute(workflow_query).scalar_one_or_none() previous_status = None now = utc_now() if record is None: - record = WorkflowInstance( - code=f"{WORKFLOW_CODE_PREFIX}-{now:%Y%m%d%H%M%S%f}", + candidate = WorkflowInstance( + code=( + f"{WORKFLOW_CODE_PREFIX}-{now:%Y%m%d%H%M%S%f}-" + f"{uuid4().hex[:8]}" + ), workflow_type=workflow_type, aggregate_type=aggregate_type, aggregate_id=aggregate_id_text, @@ -56,10 +67,26 @@ class WorkflowService: current_step=action, payload=payload or {}, ) - self.db.add(record) - self.db.flush() + try: + with self.db.begin_nested(): + self.db.add(candidate) + self.db.flush() + record = candidate + except IntegrityError: + record = self.db.execute(workflow_query).scalar_one() + previous_status = record.status else: previous_status = record.status + + existing_workflow = self._workflow_for_source_event(source_event_id) + if existing_workflow is not None: + if existing_workflow.code != record.code: + raise ValueError( + "Source event is already attached to a different workflow" + ) + return existing_workflow + + if previous_status is not None: record.status = status_value record.actor = actor record.current_step = action @@ -86,6 +113,7 @@ class WorkflowService: f"{uuid4().hex[:8]}" ), workflow_code=record.code, + source_event_id=source_event_id, action=action, actor=actor, from_status=previous_status, @@ -100,6 +128,23 @@ class WorkflowService: self.db.flush() return record + def _workflow_for_source_event( + self, + source_event_id: str | None, + ) -> WorkflowInstance | None: + if not source_event_id: + return None + action = self.db.execute( + select(WorkflowAction).where( + WorkflowAction.source_event_id == source_event_id + ) + ).scalar_one_or_none() + if action is None: + return None + return self.db.execute( + select(WorkflowInstance).where(WorkflowInstance.code == action.workflow_code) + ).scalar_one() + def list_workflows( self, status_filter: str | None = None, diff --git a/app/tasks/__init__.py b/app/tasks/__init__.py index 83455f7..13cc82d 100644 --- a/app/tasks/__init__.py +++ b/app/tasks/__init__.py @@ -5,6 +5,7 @@ from app.tasks import lifecycle as _lifecycle # noqa: F401 from app.tasks import market as _market # noqa: F401 from app.tasks import reports as _reports # noqa: F401 from app.tasks import risk as _risk # noqa: F401 +from app.tasks import subscriptions as _subscriptions # noqa: F401 __all__ = ["celery_app"] diff --git a/app/tasks/app.py b/app/tasks/app.py index ca5a9d6..cb4013d 100644 --- a/app/tasks/app.py +++ b/app/tasks/app.py @@ -9,4 +9,16 @@ celery_app = Celery( broker=settings.redis_url, backend=settings.celery_result_backend_url or settings.redis_url, ) -celery_app.conf.task_always_eager = settings.task_queue_always_eager +celery_app.conf.update( + accept_content=["json"], + broker_connection_retry_on_startup=True, + enable_utc=True, + result_serializer="json", + task_acks_late=True, + task_always_eager=settings.task_queue_always_eager, + task_eager_propagates=True, + task_reject_on_worker_lost=True, + task_serializer="json", + timezone="Asia/Shanghai", + worker_prefetch_multiplier=1, +) diff --git a/app/tasks/constants.py b/app/tasks/constants.py index 6f74431..e64237e 100644 --- a/app/tasks/constants.py +++ b/app/tasks/constants.py @@ -11,3 +11,4 @@ TASK_DISPATCH_PENDING_EVENTS = "events.dispatch_pending" TASK_RUN_LIFECYCLE = "reports.run_lifecycle" TASK_RUN_MARKET_REPORT = "market.report.run" TASK_RUN_MARKET_CLOSE = TASK_RUN_MARKET_REPORT +TASK_RUN_SUBSCRIPTION_CYCLE = "subscriptions.run_cycle" diff --git a/app/tasks/reports.py b/app/tasks/reports.py index bc4934c..2531012 100644 --- a/app/tasks/reports.py +++ b/app/tasks/reports.py @@ -12,6 +12,7 @@ from app.tasks.constants import ( from app.core.constants import ActorValue from app.core.database import SessionLocal from app.modules.feishu.constants import FeishuReceiveIdType +from app.modules.reports.constants import ReportPushStatus from app.tasks.app import celery_app ReportBuilder = Callable[[Any, str], dict[str, Any]] @@ -96,6 +97,10 @@ def _push_report( db = SessionLocal() try: service = ReportService(db) + if push_run_code: + push_run = service._get_push_run(push_run_code) + if push_run.status == ReportPushStatus.SUCCESS: + return dict(push_run.provider_response or {}) report = build_report(service, actor) return ReportDeliveryService(db).push_report( report, @@ -104,6 +109,18 @@ def _push_report( actor, push_run_code=push_run_code, ) + except Exception as exc: + if push_run_code: + db.rollback() + try: + ReportService(db).update_push_run( + push_run_code, + ReportPushStatus.FAILED, + error_message=str(exc), + ) + except Exception: + db.rollback() + raise finally: db.close() diff --git a/app/tasks/subscriptions.py b/app/tasks/subscriptions.py new file mode 100644 index 0000000..8aecc7a --- /dev/null +++ b/app/tasks/subscriptions.py @@ -0,0 +1,11 @@ +from app.application.delivery.subscriptions import run_subscription_cycle +from app.core.constants import ActorValue +from app.tasks.app import celery_app +from app.tasks.constants import TASK_RUN_SUBSCRIPTION_CYCLE + + +@celery_app.task(name=TASK_RUN_SUBSCRIPTION_CYCLE) +def run_subscription_cycle_task( + actor: str = ActorValue.SCHEDULER, +) -> dict: + return run_subscription_cycle(actor=actor) diff --git a/app/tools/init_db.py b/app/tools/init_db.py index ed50c55..5d6cb68 100644 --- a/app/tools/init_db.py +++ b/app/tools/init_db.py @@ -1,4 +1,8 @@ -from app.core.database import Base, engine +from pathlib import Path + +from alembic import command +from alembic.config import Config + from app.modules.ai_memory.models import AIMemoryEntry from app.modules.audit.models import AuditLog from app.modules.business.models import ( @@ -31,14 +35,27 @@ from app.modules.business.models import ( WorkTask, ) from app.modules.feishu.models import FeishuEventReceipt +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) from app.modules.events.models import DomainEvent from app.modules.observability.models import SystemHeartbeat +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + UserPreference, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription from app.modules.workflows.models import WorkflowAction, WorkflowInstance _MODELS = [ AuditLog, Employee, FeishuEventReceipt, + FeishuUser, + FeishuAdminBootstrapTombstone, Project, ProjectCashFlow, ProjectContract, @@ -68,14 +85,23 @@ _MODELS = [ WorkflowInstance, WorkflowAction, AIMemoryEntry, + UserPreference, + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + PushSubscription, + PushDelivery, SystemHeartbeat, SourceSyncCursor, ] def main() -> None: - Base.metadata.create_all(bind=engine) - print("Database schema initialized.") + project_root = Path(__file__).resolve().parents[2] + config = Config(str(project_root / "alembic.ini")) + config.set_main_option("script_location", str(project_root / "alembic")) + command.upgrade(config, "head") + print("Database schema migrated to Alembic head.") if __name__ == "__main__": diff --git a/scripts/sample_requests.http b/scripts/sample_requests.http index eb70445..1ee4470 100644 --- a/scripts/sample_requests.http +++ b/scripts/sample_requests.http @@ -4,54 +4,10 @@ GET http://127.0.0.1:8010/api/v1/health X-API-Key: {{apiKey}} -### Create project -POST http://127.0.0.1:8010/api/v1/business/projects -Content-Type: application/json -X-API-Key: {{apiKey}} - -{ - "actor": "demo", - "data": { - "code": "P-2026-001", - "name": "公司AI管理系统一期", - "owner": "负责人A", - "status": "执行中", - "budget_amount": 100000, - "actual_amount": 25000, - "progress_percent": 30, - "risk_level": "medium" - } -} - ### Daily brief GET http://127.0.0.1:8010/api/v1/reports/daily-brief X-API-Key: {{apiKey}} -### Sync legacy tasks -POST http://127.0.0.1:8010/api/v1/integrations/mysql/tasks/sync -Content-Type: application/json -X-API-Key: {{apiKey}} - -{ - "dry_run": true, - "limit": 50, - "field_map": { - "title": "task_name", - "owner": "task_assignee", - "due_date": "task_end_time" - } -} - -### Risk assign -POST http://127.0.0.1:8010/api/v1/risks/events/1/assign -Content-Type: application/json -X-API-Key: {{apiKey}} - -{ - "assigned_to": "risk-owner", - "comment": "请跟进处理" -} - ### Report push runs GET http://127.0.0.1:8010/api/v1/reports/push-runs X-API-Key: {{apiKey}} diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..da24f26 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,60 @@ +import os +import sys +import tempfile +from pathlib import Path + + +_database_file = tempfile.NamedTemporaryFile( + prefix="company-ai-platform-tests-", + suffix=".db", + delete=False, +) +_database_file.close() +_database_path = Path(_database_file.name) + +# This module is loaded before test modules are imported. Keep every suite run isolated +# from a developer's local .env, database, queues, and external AI/Feishu credentials. +os.environ.update( + { + "COMPANY_AI_DISABLE_DOTENV": "true", + "APP_ENV": "test", + "DATABASE_URL": f"sqlite:///{_database_path.as_posix()}", + "LEGACY_DATABASE_URL": "", + "LEGACY_ALLOWED_QUERIES": "{}", + "LEGACY_PROJECT_QUERY": "", + "LEGACY_TASK_QUERY": "", + "API_KEY": "test-key", + "API_KEYS": "[]", + "AUDIT_API_KEY": "audit-key", + "AUDIT_API_KEYS": "[]", + "AUDIT_API_ACTOR": "audit-manager", + "MODEL_PROVIDER": "noop", + "DIRECT_LLM_API_KEY": "", + "HERMES_API_KEY": "", + "OPENCLAW_API_KEY": "", + "OPENCLAW_GATEWAY_TOKEN": "", + "FEISHU_APP_ID": "", + "FEISHU_APP_SECRET": "", + "FEISHU_ENCRYPT_KEY": "", + "FEISHU_DEFAULT_CHAT_ID": "", + "FEISHU_VERIFICATION_TOKEN": "test-feishu-token", + "FEISHU_ADMIN_IDENTITIES": "", + "FEISHU_USER_FEATURES_ENABLED": "false", + "MARKET_DATA_TOKEN": "", + "SCHEDULER_ENABLED": "false", + "TASK_QUEUE_ENABLED": "false", + "TASK_QUEUE_ALWAYS_EAGER": "false", + "LEGACY_SYNC_ENABLED": "false", + } +) + + +def pytest_sessionfinish() -> None: + """Remove the suite database after every test module has finished.""" + + database_module = sys.modules.get("app.core.database.session") + if database_module is not None: + database_module.engine.dispose() + if database_module.legacy_engine is not None: + database_module.legacy_engine.dispose() + _database_path.unlink(missing_ok=True) diff --git a/tests/test_architecture_hardening.py b/tests/test_architecture_hardening.py index 01fd9af..5d643e5 100644 --- a/tests/test_architecture_hardening.py +++ b/tests/test_architecture_hardening.py @@ -9,7 +9,6 @@ from sqlalchemy.orm import Session, sessionmaker from app.core.config import get_settings from app.core.config import Settings from app.core.database import Base -from app.core.security import OperationsDisabledError from app.modules.audit.models import AuditLog from app.modules.audit.schemas import AuditLogCreate from app.modules.audit.service import AuditService @@ -39,13 +38,19 @@ def test_audit_and_outbox_rollback_with_business_transaction() -> None: assert db.scalar(select(func.count()).select_from(DomainEvent)) == 0 -def test_business_service_enforces_read_only_policy(monkeypatch: pytest.MonkeyPatch) -> None: +def test_read_only_mode_allows_local_risk_state(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) try: - with pytest.raises(OperationsDisabledError): - RiskService(Session()).generate_events(actor="pytest") + with Session(engine) as db: + result = RiskService(db).generate_events(actor="pytest") + + assert result["created"] == 0 + assert db.scalar(select(func.count()).select_from(AuditLog)) == 1 finally: + engine.dispose() get_settings.cache_clear() diff --git a/tests/test_event_dispatch_fencing.py b/tests/test_event_dispatch_fencing.py new file mode 100644 index 0000000..2919a85 --- /dev/null +++ b/tests/test_event_dispatch_fencing.py @@ -0,0 +1,101 @@ +from datetime import timedelta +from pathlib import Path + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, select, update +from sqlalchemy.orm import sessionmaker + +from app.application.events import EventDispatchService +from app.core.utils.time import utc_now +from app.modules.audit.constants import AuditAction +from app.modules.audit.models import AuditLog +from app.modules.events.constants import EventStatus +from app.modules.events.models import DomainEvent +from app.modules.events.services import EventService + + +def test_stale_worker_is_fenced_after_expired_lease_is_reclaimed( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine = create_engine(f"sqlite:///{tmp_path / 'event-fencing.db'}") + with engine.begin() as connection: + connection.exec_driver_sql("PRAGMA journal_mode=WAL") + DomainEvent.__table__.create(engine) + AuditLog.__table__.create(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + + try: + with factory() as db: + event = EventService(db).emit( + event_type="test.fencing", + source="pytest", + aggregate_type="test", + aggregate_id="fencing", + idempotency_key="test-event-fencing", + ) + event_id = event.event_id + + second_result: dict[str, DomainEvent] = {} + with factory() as first_db: + first_worker = EventDispatchService(first_db) + + def fail_after_lease_is_reclaimed(record: DomainEvent) -> None: + first_db.add( + AuditLog( + actor="worker-one", + action="stale-worker-side-effect", + target_id=record.event_id, + ) + ) + with factory() as second_db: + second_db.execute( + update(DomainEvent) + .where(DomainEvent.event_id == record.event_id) + .values(locked_until=utc_now() - timedelta(seconds=1)) + ) + second_db.commit() + + second_worker = EventDispatchService(second_db) + monkeypatch.setattr( + second_worker, + "_handle_event", + lambda claimed: None, + ) + second_result["event"] = second_worker.dispatch_event( + record.event_id, + worker_id="worker-two", + ) + raise RuntimeError("worker one failed after losing its lease") + + monkeypatch.setattr( + first_worker, + "_handle_event", + fail_after_lease_is_reclaimed, + ) + with pytest.raises(HTTPException) as exc_info: + first_worker.dispatch_event(event_id, worker_id="worker-one") + + assert exc_info.value.status_code == 409 + + assert second_result["event"].status == EventStatus.PROCESSED + with factory() as db: + stored = db.execute( + select(DomainEvent).where(DomainEvent.event_id == event_id) + ).scalar_one() + audit_actions = list( + db.execute( + select(AuditLog.action).order_by(AuditLog.id.asc()) + ).scalars() + ) + + assert stored.status == EventStatus.PROCESSED + assert stored.attempts == 2 + assert stored.last_error is None + assert stored.locked_by is None + assert stored.locked_until is None + assert audit_actions == [AuditAction.EVENT_DISPATCH] + assert "stale-worker-side-effect" not in audit_actions + finally: + engine.dispose() diff --git a/tests/test_feishu_app_ticket.py b/tests/test_feishu_app_ticket.py new file mode 100644 index 0000000..8c142a3 --- /dev/null +++ b/tests/test_feishu_app_ticket.py @@ -0,0 +1,269 @@ +import json +import sys +from collections.abc import Iterator +from types import SimpleNamespace +from typing import Any + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.application.feishu.events import FeishuEventService +from app.core.config import get_settings +from app.core.database import Base +from app.modules.audit.models import AuditLog +from app.modules.feishu import long_connection +from app.modules.feishu.app_tickets import FeishuAppTicketService +from app.modules.feishu.constants import FeishuEventSource +from app.modules.feishu.models import FeishuAppTicket, FeishuEventReceipt + + +@pytest.fixture +def session_factory( + monkeypatch: pytest.MonkeyPatch, +) -> Iterator[sessionmaker[Session]]: + monkeypatch.setenv("FEISHU_APP_ID", "cli-ticket-app") + monkeypatch.setenv("FEISHU_APP_SECRET", "ticket-app-secret") + monkeypatch.setenv("FEISHU_VERIFICATION_TOKEN", "ticket-token") + monkeypatch.setenv("FEISHU_USER_FEATURES_ENABLED", "false") + get_settings.cache_clear() + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all( + engine, + tables=[ + AuditLog.__table__, + FeishuEventReceipt.__table__, + FeishuAppTicket.__table__, + ], + ) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + Base.metadata.drop_all( + engine, + tables=[ + FeishuAppTicket.__table__, + FeishuEventReceipt.__table__, + AuditLog.__table__, + ], + ) + engine.dispose() + get_settings.cache_clear() + + +def _v2_ticket_event( + event_id: str, + ticket: str, + *, + app_id: str = "cli-ticket-app", +) -> dict[str, Any]: + return { + "schema": "2.0", + "header": { + "event_id": event_id, + "event_type": "app_ticket", + "token": "ticket-token", + "app_id": app_id, + }, + "event": {"app_ticket": ticket}, + } + + +def _v1_ticket_event( + event_id: str, + ticket: str, + *, + app_id: str = "cli-ticket-app", +) -> dict[str, Any]: + return { + "ts": "1785081600.000", + "uuid": event_id, + "token": "ticket-token", + "type": "app_ticket", + "event": { + "app_id": app_id, + "app_ticket": ticket, + }, + } + + +def test_verified_ticket_is_deduplicated_rotated_and_never_leaked( + session_factory: sessionmaker[Session], +) -> None: + first_ticket = "ticket-secret-first" + rotated_ticket = "ticket-secret-rotated" + with session_factory() as db: + service = FeishuEventService(db) + first = service._handle_verified_event( + _v2_ticket_event("ticket-event-1", first_ticket), + source=FeishuEventSource.WEBHOOK, + ) + stored = db.scalar(select(FeishuAppTicket)) + assert stored is not None + stored_id = stored.id + first_received_at = stored.received_at + assert FeishuAppTicketService(db).get_ticket("cli-ticket-app") == first_ticket + assert first_ticket not in json.dumps(first, ensure_ascii=False) + + duplicate_payload = _v2_ticket_event( + "ticket-event-1", + "ticket-secret-duplicate-must-not-win", + ) + duplicate = service._handle_verified_event( + duplicate_payload, + source=FeishuEventSource.LONG_CONNECTION, + ) + assert duplicate["duplicate"] is True + assert FeishuAppTicketService(db).get_ticket("cli-ticket-app") == first_ticket + + rotated = service._handle_verified_event( + _v2_ticket_event("ticket-event-2", rotated_ticket), + source=FeishuEventSource.LONG_CONNECTION, + ) + db.expire_all() + current = db.scalar(select(FeishuAppTicket)) + assert current is not None + assert current.id == stored_id + assert current.app_ticket == rotated_ticket + assert current.received_at >= first_received_at + assert db.scalar(select(func.count()).select_from(FeishuAppTicket)) == 1 + assert db.scalar(select(func.count()).select_from(FeishuEventReceipt)) == 2 + assert rotated_ticket not in json.dumps(rotated, ensure_ascii=False) + + audits = list(db.execute(select(AuditLog)).scalars()) + assert len(audits) == 2 + serialized_audits = json.dumps( + [ + { + "request": item.request_payload, + "response": item.response_payload, + "target": item.target_id, + } + for item in audits + ], + ensure_ascii=False, + ) + assert first_ticket not in serialized_audits + assert rotated_ticket not in serialized_audits + assert "ticket-secret-duplicate-must-not-win" not in serialized_audits + + +def test_only_verified_matching_app_ticket_events_can_write( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + service = FeishuEventService(db) + unverified = _v2_ticket_event("unverified-ticket", "unverified-secret") + unverified["header"]["token"] = "invalid-token" + with pytest.raises(HTTPException) as unverified_error: + service.handle_event(unverified, source=FeishuEventSource.WEBHOOK) + assert unverified_error.value.status_code == 401 + + with pytest.raises(HTTPException) as mismatch_error: + service._handle_verified_event( + _v2_ticket_event( + "wrong-app-ticket", + "wrong-app-secret", + app_id="cli-other-app", + ), + source=FeishuEventSource.WEBHOOK, + ) + assert mismatch_error.value.status_code == 401 + + missing_ticket = _v2_ticket_event("missing-ticket", "") + with pytest.raises(HTTPException) as missing_error: + service._handle_verified_event( + missing_ticket, + source=FeishuEventSource.WEBHOOK, + ) + assert missing_error.value.status_code == 400 + + not_ticket_event = _v2_ticket_event("ordinary-event", "must-not-store") + not_ticket_event["header"]["event_type"] = "im.message.receive_v1" + result = service._handle_verified_event( + not_ticket_event, + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + assert result["handled"] is False + assert db.scalar(select(FeishuAppTicket)) is None + + +def test_v1_app_ticket_payload_uses_uuid_receipt( + session_factory: sessionmaker[Session], +) -> None: + ticket = "v1-ticket-secret" + with session_factory() as db: + result = FeishuEventService(db).handle_event( + _v1_ticket_event("v1-ticket-uuid", ticket), + source=FeishuEventSource.WEBHOOK, + ) + + receipt = db.scalar(select(FeishuEventReceipt)) + assert result == {"ok": True, "handled": True} + assert receipt is not None + assert receipt.event_id == "v1-ticket-uuid" + assert receipt.event_key == "cli-ticket-app:app_ticket:v1-ticket-uuid" + assert FeishuAppTicketService(db).get_ticket("cli-ticket-app") == ticket + assert ticket not in json.dumps(result, ensure_ascii=False) + + +def test_long_connection_registers_custom_app_ticket_handler( + monkeypatch: pytest.MonkeyPatch, +) -> None: + registrations: dict[str, Any] = {} + + class FakeBuilder: + def register_p2_im_message_receive_v1(self, handler: Any) -> "FakeBuilder": + registrations["message"] = handler + return self + + def register_p1_customized_event( + self, + event_type: str, + handler: Any, + ) -> "FakeBuilder": + registrations[event_type] = handler + return self + + def build(self) -> "FakeBuilder": + return self + + class FakeDispatcherHandler: + @staticmethod + def builder(encrypt_key: str, verification_token: str) -> FakeBuilder: + registrations["builder_args"] = (encrypt_key, verification_token) + return FakeBuilder() + + class FakeClient: + def __init__(self, **kwargs: Any): + registrations["client_kwargs"] = kwargs + + def start(self) -> None: + registrations["started"] = True + + fake_lark = SimpleNamespace( + EventDispatcherHandler=FakeDispatcherHandler, + LogLevel=SimpleNamespace(WARNING="warning"), + ws=SimpleNamespace(Client=FakeClient), + ) + monkeypatch.setitem(sys.modules, "lark_oapi", fake_lark) + monkeypatch.setenv("FEISHU_APP_ID", "cli-ticket-app") + monkeypatch.setenv("FEISHU_APP_SECRET", "ticket-app-secret") + monkeypatch.setenv("FEISHU_VERIFICATION_TOKEN", "ticket-token") + get_settings.cache_clear() + try: + long_connection.run_long_connection() + finally: + get_settings.cache_clear() + + assert registrations["app_ticket"] is long_connection._handle_app_ticket_event + assert registrations["message"] is long_connection._handle_message_event + assert registrations["started"] is True diff --git a/tests/test_feishu_event_identity.py b/tests/test_feishu_event_identity.py new file mode 100644 index 0000000..94718b4 --- /dev/null +++ b/tests/test_feishu_event_identity.py @@ -0,0 +1,487 @@ +import json +from collections.abc import Iterator +from dataclasses import replace +from typing import Any + +import pytest +from fastapi import FastAPI, HTTPException +from fastapi.testclient import TestClient +from sqlalchemy import create_engine, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.application.feishu.commands import FeishuCommandService +from app.application.feishu.events import FeishuEventService +from app.core.config import get_settings +from app.core.database import Base, get_db +from app.modules.audit.models import AuditLog +from app.modules.feishu.constants import ( + FeishuCommandName, + FeishuEventSource, +) +from app.modules.feishu.models import FeishuEventReceipt +from app.modules.feishu.routes import router as feishu_router +from app.modules.feishu_users.constants import ( + FeishuUserAuditAction, + FeishuUserRole, + FeishuUserStatus, +) +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) +from app.modules.feishu_users.principal import FeishuMention +from app.modules.feishu_users.services import ( + FeishuIdentityService, + FeishuUserManagementService, +) + + +@pytest.fixture +def session_factory( + monkeypatch: pytest.MonkeyPatch, +) -> Iterator[sessionmaker[Session]]: + monkeypatch.setenv("FEISHU_USER_FEATURES_ENABLED", "true") + monkeypatch.setenv("FEISHU_ADMIN_IDENTITIES", "tenant-a:ou-admin") + monkeypatch.setenv("FEISHU_APP_ID", "") + monkeypatch.setenv("FEISHU_APP_SECRET", "") + monkeypatch.setenv("FEISHU_VERIFICATION_TOKEN", "verified-token") + get_settings.cache_clear() + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all( + engine, + tables=[ + AuditLog.__table__, + FeishuEventReceipt.__table__, + FeishuAdminBootstrapTombstone.__table__, + FeishuUser.__table__, + ], + ) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + Base.metadata.drop_all( + engine, + tables=[ + FeishuAdminBootstrapTombstone.__table__, + FeishuUser.__table__, + FeishuEventReceipt.__table__, + AuditLog.__table__, + ], + ) + engine.dispose() + get_settings.cache_clear() + + +def _message_event( + *, + event_id: str, + text: str, + tenant_key: str | None = "tenant-a", + open_id: str | None = "ou-user", + chat_id: str = "oc-chat", + chat_type: str = "p2p", + mentions: list[dict[str, Any]] | None = None, +) -> dict[str, Any]: + header: dict[str, Any] = { + "event_id": event_id, + "event_type": "im.message.receive_v1", + "token": "verified-token", + } + if tenant_key is not None: + header["tenant_key"] = tenant_key + sender_id: dict[str, str] = {"user_id": "legacy-user-id"} + if open_id is not None: + sender_id["open_id"] = open_id + message: dict[str, Any] = { + "chat_id": chat_id, + "chat_type": chat_type, + "message_id": f"om-{event_id}", + "message_type": "text", + "content": json.dumps({"text": text}, ensure_ascii=False), + } + if mentions is not None: + message["mentions"] = mentions + return { + "schema": "2.0", + "header": header, + "event": { + "sender": {"sender_id": sender_id}, + "message": message, + }, + } + + +def _mention( + key: str, + open_id: str, + *, + name: str, + tenant_key: str = "tenant-a", +) -> dict[str, Any]: + return { + "key": key, + "name": name, + "tenant_key": tenant_key, + "id": { + "open_id": open_id, + "union_id": f"on-{open_id}", + "user_id": f"u-{open_id}", + }, + } + + +def test_verified_event_requires_tenant_and_open_id_without_registering( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + missing_tenant = FeishuEventService(db)._handle_verified_event( + _message_event( + event_id="missing-tenant", + text="risk", + tenant_key=None, + ), + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + missing_open_id = FeishuEventService(db)._handle_verified_event( + _message_event( + event_id="missing-open", + text="risk", + open_id=None, + ), + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + + assert missing_tenant["result"]["command"] == "permission_denied" + assert missing_open_id["result"]["command"] == "permission_denied" + assert db.scalar(select(FeishuUser)) is None + denial_logs = list( + db.execute( + select(AuditLog).where( + AuditLog.action == FeishuUserAuditAction.PERMISSION_DENIED + ) + ).scalars() + ) + assert len(denial_logs) == 2 + assert all(item.actor == "feishu" for item in denial_logs) + + +def test_unverified_event_cannot_create_identity( + session_factory: sessionmaker[Session], +) -> None: + payload = _message_event( + event_id="invalid-token", + text="问 你好", + open_id="ou-unverified", + ) + payload["header"]["token"] = "invalid-token" + with session_factory() as db: + with pytest.raises(HTTPException) as exc_info: + FeishuEventService(db).handle_event( + payload, + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + + assert exc_info.value.status_code == 401 + assert db.scalar(select(FeishuUser)) is None + + +def test_group_event_uses_sender_principal_and_structured_context( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: dict[str, Any] = {} + + def fake_handle_text( + self: FeishuCommandService, + text: str, + chat_id: str | None = None, + actor: str = "feishu", + auto_reply: bool = True, + principal: Any = None, + ) -> dict[str, Any]: + captured.update( + { + "text": text, + "chat_id": chat_id, + "actor": actor, + "principal": principal, + } + ) + return { + "command": "ai_ask", + "reply_type": "text", + "title": "test", + "content": "ok", + "provider_response": None, + } + + monkeypatch.setattr(FeishuCommandService, "handle_text", fake_handle_text) + payload = _message_event( + event_id="group-principal", + text="@_bot 问 项目状态", + open_id="ou-sender", + chat_id="oc-group", + chat_type="group", + mentions=[_mention("@_target", "ou-target", name="目标用户")], + ) + with session_factory() as db: + result = FeishuEventService(db)._handle_verified_event( + payload, + source=FeishuEventSource.LONG_CONNECTION, + auto_reply=False, + ) + principal = captured["principal"] + + assert result["handled"] is True + assert principal.open_id == "ou-sender" + assert principal.chat_id == "oc-group" + assert principal.chat_type == "group" + assert principal.mentions[0].open_id == "ou-target" + assert captured["actor"] == principal.user_code + assert captured["chat_id"] == "oc-group" + + +@pytest.mark.parametrize( + "command", + [ + "日报", + "项目资金 P-001", + "风险", + "学习公司规则：所有人共享", + "设为管理员 @_target", + ], +) +def test_ordinary_user_cannot_run_company_commands( + session_factory: sessionmaker[Session], + command: str, +) -> None: + with session_factory() as db: + principal = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id=f"ou-{abs(hash(command))}", + ) + result = FeishuCommandService(db).handle_text( + command, + principal=principal, + auto_reply=False, + ) + + assert result["command"] == FeishuCommandName.PERMISSION_DENIED + assert "公司级功能" in result["content"] + + +def test_disabled_user_is_rejected_before_command_execution( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + principal = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-disabled", + ) + FeishuUserManagementService(db).update_user( + principal.user_code, + changes={"status": FeishuUserStatus.DISABLED}, + actor="service-admin", + ) + disabled = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-disabled", + ) + result = FeishuCommandService(db).handle_text( + "问 你好", + principal=disabled, + auto_reply=False, + ) + + assert result["command"] == FeishuCommandName.PERMISSION_DENIED + assert "已停用" in result["content"] + + +def test_admin_command_uses_target_mention_and_ignores_bot_mention( + session_factory: sessionmaker[Session], +) -> None: + payload = _message_event( + event_id="admin-promote", + text="@_bot 设为管理员 @_target", + open_id="ou-admin", + chat_id="oc-group", + chat_type="group", + mentions=[ + _mention("@_bot", "ou-bot", name="机器人"), + _mention("@_target", "ou-target", name="张三"), + ], + ) + with session_factory() as db: + result = FeishuEventService(db)._handle_verified_event( + payload, + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + target = FeishuIdentityService(db).get_by_identity( + tenant_key="tenant-a", + open_id="ou-target", + ) + + assert result["result"]["command"] == FeishuCommandName.USER_SET_ADMIN + assert "张三 已设为管理员" in result["result"]["content"] + assert target is not None + assert target.role == FeishuUserRole.ADMIN + assert target.union_id == "on-ou-target" + + +def test_admin_command_rejects_text_identity_cross_tenant_and_last_admin( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + admin = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-admin", + ) + admin = replace(admin, chat_id="oc-group", chat_type="group") + service = FeishuCommandService(db) + + forged = service.handle_text( + "设为管理员 @ou-forged", + principal=admin, + auto_reply=False, + ) + assert forged["command"] == FeishuCommandName.USER_SET_ADMIN + assert "真实的飞书 @用户" in forged["content"] + assert ( + FeishuIdentityService(db).get_by_identity( + tenant_key="tenant-a", + open_id="ou-forged", + ) + is None + ) + + cross_tenant = service.handle_text( + "设为管理员 @_target", + principal=replace( + admin, + mentions=( + FeishuMention( + key="@_target", + name="其他租户用户", + tenant_key="tenant-b", + open_id="ou-other", + ), + ), + ), + auto_reply=False, + ) + assert "当前租户" in cross_tenant["content"] + assert ( + FeishuIdentityService(db).get_by_identity( + tenant_key="tenant-b", + open_id="ou-other", + ) + is None + ) + + self_demote = service.handle_text( + "设为普通用户 @_self", + principal=replace( + admin, + mentions=( + FeishuMention( + key="@_self", + name="管理员", + tenant_key="tenant-a", + open_id="ou-admin", + ), + ), + ), + auto_reply=False, + ) + assert "最后一个有效管理员" in self_demote["content"] + stored_admin = FeishuIdentityService(db).get_by_identity( + tenant_key="tenant-a", + open_id="ou-admin", + ) + assert stored_admin is not None + assert stored_admin.role == FeishuUserRole.ADMIN + + +def test_preview_cannot_forge_feishu_principal_when_features_enabled( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("API_KEY", "service-key") + monkeypatch.setenv("API_ACTOR", "service-principal") + get_settings.cache_clear() + app = FastAPI() + app.include_router(feishu_router, prefix="/api/v1/integrations/feishu") + + def override_db() -> Iterator[Session]: + with session_factory() as db: + yield db + + app.dependency_overrides[get_db] = override_db + response = TestClient(app).post( + "/api/v1/integrations/feishu/commands/preview", + headers={"X-API-Key": "service-key"}, + json={ + "text": "日报", + "actor": "ou-admin", + "auto_reply": False, + }, + ) + + assert response.status_code == 200 + assert response.json()["command"] == "permission_denied" + with session_factory() as db: + assert db.scalar(select(FeishuUser)) is None + + +def test_disabled_feature_flag_preserves_legacy_actor_flow( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_USER_FEATURES_ENABLED", "false") + get_settings.cache_clear() + captured: dict[str, Any] = {} + + def fake_handle_text( + self: FeishuCommandService, + text: str, + chat_id: str | None = None, + actor: str = "feishu", + auto_reply: bool = True, + principal: Any = None, + ) -> dict[str, Any]: + captured["actor"] = actor + captured["principal"] = principal + return { + "command": "fallback_ai", + "reply_type": "text", + "title": "test", + "content": "ok", + "provider_response": None, + } + + monkeypatch.setattr(FeishuCommandService, "handle_text", fake_handle_text) + with session_factory() as db: + FeishuEventService(db)._handle_verified_event( + _message_event( + event_id="legacy-flow", + text="hello", + tenant_key=None, + open_id=None, + ), + source=FeishuEventSource.WEBHOOK, + auto_reply=False, + ) + + assert captured["actor"] == "legacy-user-id" + assert captured["principal"] is None + assert db.scalar(select(FeishuUser)) is None diff --git a/tests/test_feishu_message_delivery.py b/tests/test_feishu_message_delivery.py new file mode 100644 index 0000000..f5ec8f0 --- /dev/null +++ b/tests/test_feishu_message_delivery.py @@ -0,0 +1,77 @@ +import time +from typing import Any + +import pytest + +from app.core.config import get_settings +from app.modules.feishu.client import FeishuClient +from app.modules.feishu.errors import FeishuAPIError + + +class _Response: + def __init__(self, payload: dict[str, Any], status_code: int = 200): + self.payload = payload + self.status_code = status_code + + def json(self) -> dict[str, Any]: + return self.payload + + +class _Client: + response = _Response({"code": 0, "data": {"message_id": "om-ok"}}) + request: dict[str, Any] | None = None + + def __init__(self, **_: Any): + pass + + def __enter__(self) -> "_Client": + return self + + def __exit__(self, *_: Any) -> None: + return None + + def post(self, url: str, **kwargs: Any) -> _Response: + _Client.request = {"url": url, **kwargs} + return self.response + + +def _configured_client(monkeypatch: pytest.MonkeyPatch) -> FeishuClient: + monkeypatch.setenv("FEISHU_APP_ID", "test-app") + monkeypatch.setenv("FEISHU_APP_SECRET", "test-secret") + get_settings.cache_clear() + client = FeishuClient() + client._tenant_access_token = "tenant-token" + client._token_expires_at = time.time() + 60 + monkeypatch.setattr("app.modules.feishu.client.httpx.Client", _Client) + return client + + +def test_message_uuid_is_forwarded_to_feishu(monkeypatch: pytest.MonkeyPatch) -> None: + client = _configured_client(monkeypatch) + try: + result = client.send_text( + "hello", + receive_id="ou-user", + receive_id_type="open_id", + uuid="stable-delivery-uuid", + ) + assert result["code"] == 0 + assert _Client.request is not None + assert _Client.request["json"]["uuid"] == "stable-delivery-uuid" + finally: + get_settings.cache_clear() + + +def test_nonzero_feishu_business_code_is_not_success( + monkeypatch: pytest.MonkeyPatch, +) -> None: + client = _configured_client(monkeypatch) + _Client.response = _Response({"code": 99991400, "msg": "rate limited"}) + try: + with pytest.raises(FeishuAPIError) as exc_info: + client.send_text("hello", receive_id="ou-user", receive_id_type="open_id") + assert exc_info.value.provider_code == 99991400 + assert exc_info.value.retryable is True + finally: + _Client.response = _Response({"code": 0, "data": {"message_id": "om-ok"}}) + get_settings.cache_clear() diff --git a/tests/test_feishu_multitenant_auth.py b/tests/test_feishu_multitenant_auth.py new file mode 100644 index 0000000..156ca04 --- /dev/null +++ b/tests/test_feishu_multitenant_auth.py @@ -0,0 +1,267 @@ +import sys +from types import ModuleType +from typing import Any + +import pytest +from fastapi import HTTPException + +from app.core.config import get_settings +from app.modules.feishu.client import FeishuClient +from app.modules.feishu.errors import FeishuAPIError + + +class _Response: + def __init__(self, payload: Any, status_code: int = 200): + self.payload = payload + self.status_code = status_code + + def json(self) -> Any: + if isinstance(self.payload, Exception): + raise self.payload + return self.payload + + +class _HTTPClient: + responses: list[_Response] = [] + requests: list[dict[str, Any]] = [] + + def __init__(self, *, timeout: int): + self.timeout = timeout + + def __enter__(self) -> "_HTTPClient": + return self + + def __exit__(self, *_: Any) -> None: + return None + + def post(self, url: str, **kwargs: Any) -> _Response: + self.requests.append({"url": url, "timeout": self.timeout, **kwargs}) + return self.responses.pop(0) + + +@pytest.fixture(autouse=True) +def _reset_settings_and_http(monkeypatch: pytest.MonkeyPatch) -> None: + _HTTPClient.responses = [] + _HTTPClient.requests = [] + monkeypatch.setattr("app.modules.feishu.client.httpx.Client", _HTTPClient) + get_settings.cache_clear() + yield + get_settings.cache_clear() + + +def _configure( + monkeypatch: pytest.MonkeyPatch, + *, + app_type: str, + app_ticket: str = "", +) -> None: + monkeypatch.setenv("FEISHU_BASE_URL", "https://open.feishu.test/open-apis") + monkeypatch.setenv("FEISHU_APP_ID", "cli-test") + monkeypatch.setenv("FEISHU_APP_SECRET", "app-secret-value") + monkeypatch.setenv("FEISHU_APP_TYPE", app_type) + monkeypatch.setenv("FEISHU_APP_TICKET", app_ticket) + monkeypatch.delenv("FEISHU_DEFAULT_TENANT_KEY", raising=False) + get_settings.cache_clear() + + +def test_self_app_uses_internal_tenant_token_request( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="self") + _HTTPClient.responses = [ + _Response( + { + "code": 0, + "tenant_access_token": "self-tenant-token", + "expire": 7200, + } + ) + ] + + token = FeishuClient()._get_tenant_access_token("ignored-tenant") + + assert token == "self-tenant-token" + assert len(_HTTPClient.requests) == 1 + request = _HTTPClient.requests[0] + assert request["url"].endswith("/auth/v3/tenant_access_token/internal") + assert request["json"] == { + "app_id": "cli-test", + "app_secret": "app-secret-value", + } + + +def test_store_app_caches_app_token_and_isolates_tenant_tokens( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="latest-ticket") + _HTTPClient.responses = [ + _Response({"code": 0, "app_access_token": "app-token", "expire": 7200}), + _Response( + {"code": 0, "tenant_access_token": "tenant-a-token", "expire": 7200} + ), + _Response( + {"code": 0, "tenant_access_token": "tenant-b-token", "expire": 7200} + ), + ] + client = FeishuClient() + + tenant_a = client._get_tenant_access_token("tenant-a") + tenant_b = client._get_tenant_access_token("tenant-b") + tenant_a_again = client._get_tenant_access_token("tenant-a") + + assert (tenant_a, tenant_b, tenant_a_again) == ( + "tenant-a-token", + "tenant-b-token", + "tenant-a-token", + ) + assert len(_HTTPClient.requests) == 3 + app_request, tenant_a_request, tenant_b_request = _HTTPClient.requests + assert app_request["url"].endswith("/auth/v3/app_access_token") + assert app_request["json"] == { + "app_id": "cli-test", + "app_secret": "app-secret-value", + "app_ticket": "latest-ticket", + } + assert tenant_a_request["url"].endswith("/auth/v3/tenant_access_token") + assert tenant_a_request["json"] == { + "app_access_token": "app-token", + "tenant_key": "tenant-a", + } + assert tenant_b_request["json"] == { + "app_access_token": "app-token", + "tenant_key": "tenant-b", + } + + +def test_store_app_prefers_persisted_ticket( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="environment-ticket") + + class _TicketService: + def __init__(self, db: object): + assert db is database + + def get_ticket(self, app_id: str) -> str: + assert app_id == "cli-test" + return "persisted-ticket" + + database = object() + ticket_module = ModuleType("app.modules.feishu.app_tickets") + ticket_module.FeishuAppTicketService = _TicketService + monkeypatch.setitem(sys.modules, ticket_module.__name__, ticket_module) + _HTTPClient.responses = [ + _Response({"code": 0, "app_access_token": "app-token", "expire": 7200}), + _Response( + {"code": 0, "tenant_access_token": "tenant-token", "expire": 7200} + ), + ] + + FeishuClient(database)._get_tenant_access_token("tenant-a") + + assert _HTTPClient.requests[0]["json"]["app_ticket"] == "persisted-ticket" + + +def test_store_app_requires_tenant_key_before_request( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="latest-ticket") + + with pytest.raises(HTTPException) as exc_info: + FeishuClient()._get_tenant_access_token() + + assert exc_info.value.status_code == 400 + assert exc_info.value.detail == "tenant_key is required for Feishu store apps" + assert _HTTPClient.requests == [] + + +def test_store_app_requires_ticket_before_request( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="store") + + with pytest.raises(HTTPException) as exc_info: + FeishuClient()._get_tenant_access_token("tenant-a") + + assert exc_info.value.status_code == 503 + assert exc_info.value.detail == "Feishu store app ticket is not available" + assert _HTTPClient.requests == [] + + +@pytest.mark.parametrize( + ("status_code", "retryable"), + [(401, False), (429, True), (500, True)], +) +def test_token_http_errors_are_classified_without_exposing_secrets( + monkeypatch: pytest.MonkeyPatch, + status_code: int, + retryable: bool, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="secret-ticket-value") + _HTTPClient.responses = [ + _Response( + { + "code": 1, + "app_ticket": "secret-ticket-value", + "tenant_access_token": "secret-token-value", + }, + status_code=status_code, + ) + ] + + with pytest.raises(FeishuAPIError) as exc_info: + FeishuClient()._get_tenant_access_token("tenant-a") + + error = exc_info.value + assert error.retryable is retryable + serialized = f"{error.detail} {error.provider_response}" + assert "secret-ticket-value" not in serialized + assert "secret-token-value" not in serialized + + +@pytest.mark.parametrize( + ("payload", "retryable"), + [ + ({"code": 99991400, "msg": "rate limited"}, True), + ({"code": 10003, "msg": "invalid app credentials"}, False), + ], +) +def test_token_business_errors_are_classified( + monkeypatch: pytest.MonkeyPatch, + payload: dict[str, Any], + retryable: bool, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="latest-ticket") + _HTTPClient.responses = [_Response(payload)] + + with pytest.raises(FeishuAPIError) as exc_info: + FeishuClient()._get_tenant_access_token("tenant-a") + + assert exc_info.value.provider_code == payload["code"] + assert exc_info.value.retryable is retryable + + +def test_send_text_uses_the_requested_tenant_token( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _configure(monkeypatch, app_type="store", app_ticket="latest-ticket") + _HTTPClient.responses = [ + _Response({"code": 0, "app_access_token": "app-token", "expire": 7200}), + _Response( + {"code": 0, "tenant_access_token": "tenant-a-token", "expire": 7200} + ), + _Response({"code": 0, "data": {"message_id": "om-message"}}), + ] + + result = FeishuClient().send_text( + "hello", + receive_id="ou-user", + receive_id_type="open_id", + uuid="stable-uuid", + tenant_key="tenant-a", + ) + + assert result["code"] == 0 + message_request = _HTTPClient.requests[2] + assert message_request["headers"]["Authorization"] == "Bearer tenant-a-token" + assert message_request["json"]["uuid"] == "stable-uuid" diff --git a/tests/test_feishu_personalization_commands.py b/tests/test_feishu_personalization_commands.py new file mode 100644 index 0000000..5d0a196 --- /dev/null +++ b/tests/test_feishu_personalization_commands.py @@ -0,0 +1,486 @@ +from collections.abc import Iterator +from typing import Any + +import pytest +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.application.feishu.commands import FeishuCommandService +from app.application.feishu.handlers import personal_data as personal_data_handler +from app.application.feishu.personal_data import PERSONAL_DATA_ERASURE_ACTION +from app.core.config import get_settings +from app.core.database import Base +from app.modules.ai_agent.constants import AIResponseKey +from app.modules.ai_agent.service import AIService +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.audit.constants import AuditAction +from app.modules.audit.models import AuditLog +from app.modules.feishu.constants import FeishuCommandName +from app.modules.feishu_users.constants import FeishuUserRole +from app.modules.feishu_users.models import FeishuUser +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.feishu_users.services import FeishuIdentityService +from app.modules.personalization.models import UserPreference +from app.modules.personalization.services import ConversationService +from app.modules.subscriptions.models import PushSubscription + + +@pytest.fixture +def session_factory( + monkeypatch: pytest.MonkeyPatch, +) -> Iterator[sessionmaker[Session]]: + monkeypatch.setenv("FEISHU_USER_FEATURES_ENABLED", "true") + monkeypatch.setenv("FEISHU_APP_ID", "") + monkeypatch.setenv("FEISHU_APP_SECRET", "") + get_settings.cache_clear() + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + engine.dispose() + get_settings.cache_clear() + + +def _principal( + db: Session, + suffix: str, + *, + role: str = FeishuUserRole.USER, + chat_id: str | None = None, + chat_type: str = "p2p", +) -> FeishuPrincipal: + user = FeishuUser( + code=f"FSU-COMMAND-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + role=role, + ) + db.add(user) + db.commit() + db.refresh(user) + return FeishuPrincipal.from_user( + user, + chat_id=chat_id or f"chat-{suffix}", + chat_type=chat_type, + ) + + +def test_personal_rules_are_owner_scoped_and_company_rules_require_admin( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + owner = _principal(db, "owner") + other = _principal(db, "other") + admin = _principal(db, "admin", role=FeishuUserRole.ADMIN) + + created = FeishuCommandService(db).handle_text( + "学习规则 80：回答尽量简洁", + principal=owner, + auto_reply=False, + ) + denied = FeishuCommandService(db).handle_text( + "学习公司规则：所有人使用中文", + principal=other, + auto_reply=False, + ) + company = FeishuCommandService(db).handle_text( + "学习公司规则：所有人使用中文", + principal=admin, + auto_reply=False, + ) + + records = list( + db.execute( + select(AIMemoryEntry).order_by(AIMemoryEntry.id.asc()) + ).scalars() + ) + assert created["command"] == FeishuCommandName.RULE_CREATE + assert denied["command"] == FeishuCommandName.PERMISSION_DENIED + assert company["command"] == FeishuCommandName.RULE_CREATE + assert len(records) == 2 + assert records[0].owner_id == owner.owner_id + assert records[1].owner_id is None + + owner_list = FeishuCommandService(db).handle_text( + "查看规则", + principal=owner, + auto_reply=False, + ) + other_list = FeishuCommandService(db).handle_text( + "查看规则", + principal=other, + auto_reply=False, + ) + assert "回答尽量简洁" in owner_list["content"] + assert "回答尽量简洁" not in other_list["content"] + assert "所有人使用中文" not in owner_list["content"] + + +def test_preferences_and_topics_are_isolated_and_sensitive_content_is_rejected( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + owner = _principal(db, "preference-owner") + other = _principal(db, "preference-other") + service = FeishuCommandService(db) + + preference = service.handle_text( + "记住偏好 语气：简洁", + principal=owner, + auto_reply=False, + ) + topic = service.handle_text( + "关注主题：人工智能", + principal=owner, + auto_reply=False, + ) + code = str(preference["content"]).split("编号：", 1)[1].splitlines()[0] + cross_owner_delete = service.handle_text( + f"删除偏好 {code}", + principal=other, + auto_reply=False, + ) + sensitive = service.handle_text( + "记住偏好 兴趣：我的银行卡是 123456", + principal=owner, + auto_reply=False, + ) + + assert preference["command"] == FeishuCommandName.PREFERENCE_SET + assert topic["command"] == FeishuCommandName.PREFERENCE_SET + assert "没有找到" in cross_owner_delete["content"] + assert "敏感" in sensitive["content"] + assert db.scalar( + select(func.count()) + .select_from(UserPreference) + .where(UserPreference.owner_id == owner.owner_id) + ) == 2 + assert db.scalar( + select(func.count()) + .select_from(UserPreference) + .where(UserPreference.owner_id == other.owner_id) + ) == 0 + + +def test_conversation_reset_only_clears_current_user_and_chat( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + owner = _principal( + db, + "conversation-owner", + chat_id="group-current", + chat_type="group", + ) + other = _principal( + db, + "conversation-other", + chat_id="group-current", + chat_type="group", + ) + conversations = ConversationService(db) + conversations.record_turn( + owner.owner_id, + "group", + "group-current", + user_content="owner question", + assistant_content="owner answer", + provider_name="direct_llm", + ) + conversations.record_turn( + owner.owner_id, + "group", + "group-other", + user_content="other chat question", + assistant_content="other chat answer", + provider_name="direct_llm", + ) + conversations.record_turn( + other.owner_id, + "group", + "group-current", + user_content="other user question", + assistant_content="other user answer", + provider_name="direct_llm", + ) + + result = FeishuCommandService(db).handle_text( + "重置对话", + principal=owner, + auto_reply=False, + ) + + assert result["command"] == FeishuCommandName.CONVERSATION_RESET + assert conversations.history(owner.owner_id, "group", "group-current") == [] + assert conversations.history(owner.owner_id, "group", "group-other") + assert conversations.history(other.owner_id, "group", "group-current") + + +def test_ai_and_subscription_commands_are_wired_to_verified_principal( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: dict[str, Any] = {} + + def fake_personalized( + self: AIService, + owner_id: int, + chat_type: str, + chat_key: str, + prompt: str, + **_: Any, + ) -> dict[str, Any]: + captured.update( + { + "owner_id": owner_id, + "chat_type": chat_type, + "chat_key": chat_key, + "prompt": prompt, + } + ) + return { + AIResponseKey.OK: True, + AIResponseKey.ANSWER: "账号隔离回答", + AIResponseKey.PROVIDER: "test", + AIResponseKey.RAW: {}, + } + + monkeypatch.setattr(AIService, "ask_personalized", fake_personalized) + with session_factory() as db: + principal = _principal( + db, + "wiring", + chat_id="verified-private-chat", + ) + service = FeishuCommandService(db) + + answer = service.handle_text( + "问 你好", + principal=principal, + auto_reply=False, + ) + subscription = service.handle_text( + "订阅 每天 09:00：给我一个问候", + principal=principal, + auto_reply=False, + ) + + assert answer["content"] == "账号隔离回答" + assert captured == { + "owner_id": principal.owner_id, + "chat_type": "p2p", + "chat_key": "verified-private-chat", + "prompt": "你好", + } + assert subscription["command"] == FeishuCommandName.SUBSCRIPTION_CREATE + stored = db.scalar(select(PushSubscription)) + assert stored is not None + assert stored.owner_id == principal.owner_id + assert stored.target_id == principal.open_id + + +def test_help_lists_only_role_allowed_company_commands( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + user = _principal(db, "help-user") + admin = _principal(db, "help-admin", role=FeishuUserRole.ADMIN) + + user_help = FeishuCommandService(db).handle_text( + "帮助", + principal=user, + auto_reply=False, + ) + admin_help = FeishuCommandService(db).handle_text( + "帮助", + principal=admin, + auto_reply=False, + ) + + assert "管理员还可以使用" not in user_help["content"] + assert "管理员还可以使用" in admin_help["content"] + + +def test_my_data_summary_and_two_step_erasure_use_current_identity_only( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + principal = _principal(db, "personal-data") + other = _principal(db, "personal-data-other") + service = FeishuCommandService(db) + service.handle_text( + "学习规则：只属于我的规则", + principal=principal, + auto_reply=False, + ) + service.handle_text( + "记住偏好 语言：中文", + principal=principal, + auto_reply=False, + ) + service.handle_text( + "关注主题：低空经济", + principal=principal, + auto_reply=False, + ) + service.handle_text( + "记住偏好 语气：不应出现在另一用户摘要", + principal=other, + auto_reply=False, + ) + service.handle_text( + "订阅 每天 09:00：个人提醒", + principal=principal, + auto_reply=False, + ) + + summary = service.handle_text( + "我的数据", + principal=principal, + auto_reply=False, + ) + request = service.handle_text( + "忘记我", + principal=principal, + auto_reply=False, + ) + confirmation_code = ( + str(request["content"]).split("确认码：", 1)[1].splitlines()[0] + ) + erased = service.handle_text( + f"确认忘记我 {confirmation_code}", + principal=principal, + auto_reply=False, + ) + + assert summary["command"] == FeishuCommandName.PERSONAL_DATA_SUMMARY + assert "个人规则：1 条" in summary["content"] + assert "language=中文" in summary["content"] + assert "低空经济" in summary["content"] + assert "不应出现在另一用户摘要" not in summary["content"] + assert request["command"] == ( + FeishuCommandName.PERSONAL_DATA_ERASURE_REQUEST + ) + assert erased["command"] == ( + FeishuCommandName.PERSONAL_DATA_ERASURE_CONFIRM + ) + assert db.get(FeishuUser, principal.owner_id) is None + assert db.get(FeishuUser, other.owner_id) is not None + + recreated = FeishuIdentityService(db).resolve_or_register( + tenant_key=principal.tenant_key, + open_id=principal.open_id, + ) + assert recreated.owner_id != principal.owner_id + assert recreated.role == FeishuUserRole.USER + + +def test_erasure_confirmation_reply_does_not_create_identifying_send_audit( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: list[dict[str, Any]] = [] + + def fake_send_text( + _feishu: Any, + chat_id: str | None, + text: str, + actor: str, + *, + record_audit: bool = True, + ) -> dict[str, Any]: + captured.append( + { + "chat_id": chat_id, + "text": text, + "actor": actor, + "record_audit": record_audit, + } + ) + return {"code": 0} + + monkeypatch.setattr( + personal_data_handler, + "send_text_if_configured", + fake_send_text, + ) + with session_factory() as db: + principal = _principal(db, "erasure-reply") + service = FeishuCommandService(db) + request = service.handle_text( + "忘记我", + principal=principal, + auto_reply=False, + ) + confirmation_code = ( + str(request["content"]).split("确认码：", 1)[1].splitlines()[0] + ) + + service.handle_text( + f"确认忘记我 {confirmation_code}", + principal=principal, + auto_reply=True, + ) + + assert len(captured) == 1 + assert captured[0]["actor"].startswith("anonymous-") + assert captured[0]["record_audit"] is False + anonymous_logs = list( + db.execute( + select(AuditLog).where( + AuditLog.actor == captured[0]["actor"] + ) + ).scalars() + ) + assert anonymous_logs + assert any( + item.action == PERSONAL_DATA_ERASURE_ACTION + for item in anonymous_logs + ) + assert all( + item.action != AuditAction.FEISHU_SEND_TEXT + for item in anonymous_logs + ) + assert all( + item.target_type is None + and item.target_id is None + and item.request_payload is None + and item.response_payload is None + and item.request_id is None + for item in anonymous_logs + ) + + +def test_group_chat_does_not_disclose_summary_or_erasure_code( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + principal = _principal( + db, + "group-personal-data", + chat_id="group-personal-data", + chat_type="group", + ) + + summary = FeishuCommandService(db).handle_text( + "我的数据", + principal=principal, + auto_reply=False, + ) + erasure = FeishuCommandService(db).handle_text( + "忘记我", + principal=principal, + auto_reply=False, + ) + + assert "请私聊" in summary["content"] + assert "请私聊" in erasure["content"] + assert "确认码" not in erasure["content"] + assert db.get(FeishuUser, principal.owner_id) is not None diff --git a/tests/test_feishu_personalization_migration.py b/tests/test_feishu_personalization_migration.py new file mode 100644 index 0000000..00f018b --- /dev/null +++ b/tests/test_feishu_personalization_migration.py @@ -0,0 +1,193 @@ +from datetime import datetime +from pathlib import Path + +import pytest +from alembic import command +from alembic.config import Config +from sqlalchemy import create_engine, inspect, text + +from app.core.config import get_settings + + +def test_personalization_migration_preserves_and_classifies_legacy_data( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + database_path = tmp_path / "personalization-migration.db" + database_url = f"sqlite:///{database_path.as_posix()}" + monkeypatch.setenv("DATABASE_URL", database_url) + get_settings.cache_clear() + config = Config("alembic.ini") + engine = create_engine(database_url) + timestamp = datetime(2026, 7, 26, 8, 0) + + try: + command.upgrade(config, "202607260002") + with engine.begin() as connection: + connection.execute( + text( + """ + INSERT INTO ai_memory_entries ( + code, + fingerprint, + scope, + subject, + content, + source, + importance, + status, + actor, + created_at, + updated_at + ) + VALUES ( + :code, + :fingerprint, + 'global', + :subject, + :content, + :source, + 1, + 'active', + 'legacy-user', + :created_at, + :updated_at + ) + """ + ), + [ + { + "code": "legacy-rule", + "fingerprint": "legacy-rule-fingerprint", + "subject": "rule", + "content": "Use concise Chinese.", + "source": "user_rule", + "created_at": timestamp, + "updated_at": timestamp, + }, + { + "code": "legacy-auto-memory", + "fingerprint": "legacy-auto-fingerprint", + "subject": "preference", + "content": "Prefers market summaries.", + "source": "auto", + "created_at": timestamp, + "updated_at": timestamp, + }, + { + "code": "legacy-hermes-memory", + "fingerprint": "legacy-hermes-fingerprint", + "subject": "preference", + "content": "Prefers short answers.", + "source": "hermes", + "created_at": timestamp, + "updated_at": timestamp, + }, + ], + ) + connection.execute( + text( + """ + INSERT INTO market_watchlists ( + actor, + symbol, + enabled, + created_at, + updated_at + ) + VALUES ( + 'legacy-user', + '600000.SH', + 1, + :created_at, + :updated_at + ) + """ + ), + {"created_at": timestamp, "updated_at": timestamp}, + ) + + command.upgrade(config, "202607260005") + + inspector = inspect(engine) + expected_tables = { + "feishu_app_tickets", + "feishu_admin_bootstrap_tombstones", + "feishu_users", + "user_preferences", + "ai_conversations", + "ai_conversation_messages", + "personal_data_erasure_requests", + "push_subscriptions", + "push_deliveries", + } + assert expected_tables <= set(inspector.get_table_names()) + tombstone_columns = { + column["name"] + for column in inspector.get_columns( + "feishu_admin_bootstrap_tombstones" + ) + } + assert tombstone_columns == {"id", "identity_hash", "created_at"} + + memory_columns = { + column["name"] for column in inspector.get_columns("ai_memory_entries") + } + assert {"owner_id", "kind"} <= memory_columns + memory_indexes = { + index["name"]: index for index in inspector.get_indexes("ai_memory_entries") + } + assert memory_indexes["ix_ai_memory_entries_fingerprint"]["unique"] == 0 + assert { + "ix_ai_memory_entries_kind", + "ix_ai_memory_entries_owner_id", + } <= set(memory_indexes) + memory_constraints = { + constraint["name"] + for constraint in inspector.get_unique_constraints("ai_memory_entries") + } + assert "uq_ai_memory_owner_fingerprint" in memory_constraints + memory_foreign_keys = { + foreign_key["name"] + for foreign_key in inspector.get_foreign_keys("ai_memory_entries") + } + assert "fk_ai_memory_entries_owner_id" in memory_foreign_keys + + with engine.connect() as connection: + memories = { + row.code: row + for row in connection.execute( + text( + """ + SELECT code, owner_id, kind, source, status + FROM ai_memory_entries + ORDER BY code + """ + ) + ) + } + watchlist_owner_id = connection.scalar( + text( + """ + SELECT owner_id + FROM market_watchlists + WHERE actor = 'legacy-user' AND symbol = '600000.SH' + """ + ) + ) + + legacy_rule = memories["legacy-rule"] + assert legacy_rule.owner_id is None + assert legacy_rule.kind == "company_rule" + assert legacy_rule.source == "legacy_company" + assert legacy_rule.status == "active" + for code in ("legacy-auto-memory", "legacy-hermes-memory"): + assert memories[code].owner_id is None + assert memories[code].kind == "memory" + assert memories[code].status == "archived" + assert watchlist_owner_id is None + + command.check(config) + finally: + engine.dispose() + get_settings.cache_clear() diff --git a/tests/test_feishu_subscription_commands.py b/tests/test_feishu_subscription_commands.py new file mode 100644 index 0000000..7dc9325 --- /dev/null +++ b/tests/test_feishu_subscription_commands.py @@ -0,0 +1,322 @@ +from datetime import datetime +from uuid import uuid4 + +import pytest +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session + +from app.application.feishu.handlers.subscriptions import handle_subscription_command +from app.core.database import Base +from app.modules.feishu.constants import FeishuCommandName +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.constants import FeishuUserRole +from app.modules.feishu_users.models import FeishuUser +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.subscriptions.constants import ( + DAILY_DELIVERY_LIMIT_REACHED, + MAX_ACTIVE_SUBSCRIPTIONS, + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription + + +def _user( + db: Session, + *, + suffix: str, + role: str = FeishuUserRole.USER, +) -> FeishuUser: + record = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + role=role, + timezone="Asia/Shanghai", + ) + db.add(record) + db.commit() + db.refresh(record) + return record + + +def _handle( + db: Session, + text: str, + principal: FeishuPrincipal, +) -> dict: + result = handle_subscription_command( + db, + FeishuService(db), + text, + principal, + False, + ) + assert result is not None + return result + + +def test_create_private_subscription_replies_with_normalized_plan() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="private-command") + principal = FeishuPrincipal.from_user( + user, + chat_id="private-chat", + chat_type="p2p", + ) + + result = _handle( + db, + "订阅 每天 09:00：请提醒我：喝水", + principal, + ) + + subscription = db.scalar(select(PushSubscription)) + assert subscription is not None + assert result["command"] == FeishuCommandName.SUBSCRIPTION_CREATE + assert "订阅已启用" in result["content"] + assert "计划：每天 09:00" in result["content"] + assert "下次执行：" in result["content"] + assert f"暂停订阅 {subscription.code}" in result["content"] + assert subscription.owner_id == user.id + assert subscription.target_type == SubscriptionTargetType.USER + assert subscription.target_id == user.open_id + assert subscription.prompt == "请提醒我：喝水" + finally: + engine.dispose() + + +def test_group_subscription_uses_current_verified_chat_and_requires_admin() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + ordinary = _user(db, suffix="ordinary-group") + denied = _handle( + db, + "订阅 每周一 09:00：群提醒", + FeishuPrincipal.from_user( + ordinary, + chat_id="verified-group", + chat_type="group", + ), + ) + assert "无权" in denied["content"] + assert db.scalar(select(func.count()).select_from(PushSubscription)) == 0 + + admin = _user(db, suffix="admin-group", role=FeishuUserRole.ADMIN) + accepted = _handle( + db, + "订阅 每周一 09:00：群提醒", + FeishuPrincipal.from_user( + admin, + chat_id="verified-group", + chat_type="group", + ), + ) + subscription = db.scalar(select(PushSubscription)) + assert accepted["command"] == FeishuCommandName.SUBSCRIPTION_CREATE + assert subscription is not None + assert subscription.target_type == SubscriptionTargetType.CHAT + assert subscription.target_id == "verified-group" + finally: + engine.dispose() + + +def test_list_pause_resume_and_cancel_subscription_commands() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="lifecycle-command") + principal = FeishuPrincipal.from_user(user, chat_type="p2p") + _handle(db, "订阅 每天 10:00：生命周期测试", principal) + subscription = db.scalar(select(PushSubscription)) + assert subscription is not None + + listed = _handle(db, "我的订阅", principal) + assert listed["command"] == FeishuCommandName.SUBSCRIPTION_LIST + assert subscription.code in listed["content"] + assert "每天 10:00" in listed["content"] + + paused = _handle(db, f"暂停订阅 {subscription.code}", principal) + db.refresh(subscription) + assert paused["command"] == FeishuCommandName.SUBSCRIPTION_PAUSE + assert subscription.status == PushSubscriptionStatus.PAUSED + + resumed = _handle(db, f"恢复订阅 {subscription.code}", principal) + db.refresh(subscription) + assert resumed["command"] == FeishuCommandName.SUBSCRIPTION_RESUME + assert subscription.status == PushSubscriptionStatus.ACTIVE + assert "下次执行：" in resumed["content"] + + cancelled = _handle(db, f"退订 {subscription.code}", principal) + db.refresh(subscription) + assert cancelled["command"] == FeishuCommandName.SUBSCRIPTION_CANCEL + assert subscription.status == PushSubscriptionStatus.CANCELLED + assert subscription.next_run_at is None + finally: + engine.dispose() + + +def test_timezone_and_quiet_hour_commands_update_current_user() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="settings-command") + principal = FeishuPrincipal.from_user(user, chat_type="p2p") + + timezone_result = _handle(db, "设置时区 Asia/Tokyo", principal) + db.refresh(user) + assert timezone_result["command"] == FeishuCommandName.SUBSCRIPTION_TIMEZONE + assert user.timezone == "Asia/Tokyo" + + quiet_result = _handle(db, "设置安静时段 22:00-07:00", principal) + db.refresh(user) + assert quiet_result["command"] == FeishuCommandName.SUBSCRIPTION_QUIET_HOURS + assert user.quiet_hours_start.strftime("%H:%M") == "22:00" + assert user.quiet_hours_end.strftime("%H:%M") == "07:00" + + closed = _handle(db, "关闭安静时段", principal) + db.refresh(user) + assert closed["command"] == FeishuCommandName.SUBSCRIPTION_QUIET_HOURS + assert user.quiet_hours_start is None + assert user.quiet_hours_end is None + finally: + engine.dispose() + + +def test_invalid_schedule_and_capacity_error_create_no_partial_record() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="invalid-command") + principal = FeishuPrincipal.from_user(user, chat_type="p2p") + + invalid = _handle( + db, + "订阅 每隔10分钟：过于频繁", + principal, + ) + assert "示例" not in invalid["content"] + assert "订阅 每天 09:00" in invalid["content"] + assert db.scalar(select(func.count()).select_from(PushSubscription)) == 0 + + for index in range(MAX_ACTIVE_SUBSCRIPTIONS): + db.add( + PushSubscription( + code=f"SUB-CAPACITY-{index}", + owner_id=user.id, + target_type=SubscriptionTargetType.USER, + target_id=user.open_id, + prompt=f"已有订阅 {index}", + schedule_type=SubscriptionScheduleType.DAILY, + schedule_config={"hour": 9, "minute": 0}, + timezone=user.timezone, + next_run_at=datetime(2026, 7, 27, 1, 0), + status=PushSubscriptionStatus.ACTIVE, + consented_at=datetime(2026, 7, 26, 0, 0), + ) + ) + db.commit() + + limited = _handle( + db, + "订阅 每天 11:00：第 51 条", + principal, + ) + assert "最多 50 个" in limited["content"] + assert ( + db.scalar(select(func.count()).select_from(PushSubscription)) + == MAX_ACTIVE_SUBSCRIPTIONS + ) + finally: + engine.dispose() + + +def test_unrelated_text_is_not_claimed_by_subscription_handler() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="unrelated-command") + + assert ( + handle_subscription_command( + db, + FeishuService(db), + "今天怎么样", + FeishuPrincipal.from_user(user), + False, + ) + is None + ) + finally: + engine.dispose() + + +def test_list_explains_daily_delivery_limit_skip() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="delivery-note") + principal = FeishuPrincipal.from_user(user, chat_type="p2p") + _handle(db, "订阅 每天 10:00：投递说明", principal) + subscription = db.scalar(select(PushSubscription)) + assert subscription is not None + now = datetime(2026, 7, 26, 1, 0) + db.add( + PushDelivery( + code="DEL-DAILY-LIMIT", + subscription_id=subscription.id, + scheduled_for=now, + idempotency_key=uuid4().hex + uuid4().hex, + message_uuid=str(uuid4()), + status=PushDeliveryStatus.SKIPPED, + next_attempt_at=None, + last_error=DAILY_DELIVERY_LIMIT_REACHED, + created_at=now, + updated_at=now, + ) + ) + db.commit() + + listed = _handle(db, "我的订阅", principal) + + assert "每日最多 96 条限制" in listed["content"] + finally: + engine.dispose() + + +@pytest.mark.parametrize( + "command", + [ + "设置时区 Invalid/Timezone", + "设置安静时段 25:00-07:00", + "订阅 每隔999999999999999999小时：不会创建", + ], +) +def test_invalid_settings_and_oversized_interval_return_command_error( + command: str, +) -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix=f"invalid-{abs(hash(command))}") + principal = FeishuPrincipal.from_user(user, chat_type="p2p") + + result = _handle(db, command, principal) + + assert "未执行" in result["content"] or "无法识别" in result["content"] + assert db.scalar(select(func.count()).select_from(PushSubscription)) == 0 + finally: + engine.dispose() diff --git a/tests/test_feishu_subscription_readiness.py b/tests/test_feishu_subscription_readiness.py new file mode 100644 index 0000000..2d803b2 --- /dev/null +++ b/tests/test_feishu_subscription_readiness.py @@ -0,0 +1,327 @@ +import json +from datetime import datetime, timedelta +from uuid import uuid4 + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import Session + +from app.core.config import get_settings +from app.core.database import Base +from app.modules.feishu.app_tickets import FeishuAppTicketService +from app.modules.feishu_users.models import FeishuUser +from app.modules.observability.constants import ( + ObservabilityKey, + ObservabilityStatus, +) +from app.modules.observability.service import ObservabilityService +from app.modules.subscriptions.constants import ( + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription + + +@pytest.fixture(autouse=True) +def _reset_settings() -> None: + get_settings.cache_clear() + yield + get_settings.cache_clear() + + +def _add_active_subscription( + db: Session, + *, + tenant_key: str, + suffix: str, +) -> PushSubscription: + owner = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=tenant_key, + open_id=f"open-{suffix}", + ) + db.add(owner) + db.flush() + subscription = PushSubscription( + code=f"SUB-{suffix}", + owner_id=owner.id, + target_type=SubscriptionTargetType.USER, + target_id=owner.open_id, + prompt="发送个人提醒", + schedule_type=SubscriptionScheduleType.DAILY, + schedule_config={"hour": 9, "minute": 0}, + timezone="Asia/Shanghai", + next_run_at=datetime(2026, 7, 27, 1, 0), + status=PushSubscriptionStatus.ACTIVE, + consented_at=datetime(2026, 7, 26, 1, 0), + ) + db.add(subscription) + db.commit() + db.refresh(subscription) + return subscription + + +def _add_delivery( + db: Session, + subscription: PushSubscription, + *, + suffix: str, + status_value: str, + scheduled_for: datetime, +) -> None: + future = datetime(2030, 1, 1, 0, 0) + db.add( + PushDelivery( + code=f"DEL-{suffix}", + subscription_id=subscription.id, + scheduled_for=scheduled_for, + idempotency_key=uuid4().hex + uuid4().hex, + message_uuid=str(uuid4()), + status=status_value, + next_attempt_at=( + future + if status_value + in { + PushDeliveryStatus.PENDING, + PushDeliveryStatus.RETRY, + } + else None + ), + locked_by=( + "readiness-worker" + if status_value == PushDeliveryStatus.PROCESSING + else None + ), + locked_until=( + future + if status_value == PushDeliveryStatus.PROCESSING + else None + ), + ) + ) + db.commit() + + +def test_subscription_readiness_requires_basic_credentials( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "self") + monkeypatch.setenv("FEISHU_APP_ID", "") + monkeypatch.setenv("FEISHU_APP_SECRET", "") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + _add_active_subscription(db, tenant_key="tenant-a", suffix="missing") + + result = ObservabilityService(db)._feishu_subscriptions_check() + + assert result[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert result["credentials_configured"] is False + assert result["reasons"] == ["credentials_missing"] + finally: + engine.dispose() + + +def test_self_app_readiness_rejects_multiple_active_tenants( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "self") + monkeypatch.setenv("FEISHU_APP_ID", "cli-self") + monkeypatch.setenv("FEISHU_APP_SECRET", "secret") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + _add_active_subscription(db, tenant_key="tenant-a", suffix="self-a") + single = ObservabilityService(db)._feishu_subscriptions_check() + assert single[ObservabilityKey.STATUS] == ObservabilityStatus.OK + assert single["active_tenant_count"] == 1 + + _add_active_subscription(db, tenant_key="tenant-b", suffix="self-b") + multiple = ObservabilityService(db)._feishu_subscriptions_check() + assert multiple[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert multiple["active_tenant_count"] == 2 + assert multiple["reasons"] == ["self_app_multiple_tenants"] + finally: + engine.dispose() + + +def test_store_app_readiness_accepts_persisted_ticket_without_exposing_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + ticket = "readiness-ticket-secret" + monkeypatch.setenv("FEISHU_APP_TYPE", "store") + monkeypatch.setenv("FEISHU_APP_ID", "cli-store") + monkeypatch.setenv("FEISHU_APP_SECRET", "secret") + monkeypatch.setenv("FEISHU_APP_TICKET", "") + monkeypatch.setenv("FEISHU_DEFAULT_TENANT_KEY", "tenant-a") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + _add_active_subscription(db, tenant_key="tenant-a", suffix="store") + missing = ObservabilityService(db)._feishu_subscriptions_check() + assert missing[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert missing["reasons"] == ["app_ticket_missing"] + + FeishuAppTicketService(db).store_verified("cli-store", ticket) + configured = ObservabilityService(db)._feishu_subscriptions_check() + + assert configured[ObservabilityKey.STATUS] == ObservabilityStatus.OK + assert configured["ticket_configured"] is True + assert configured["default_tenant_configured"] is True + assert configured["reasons"] == [] + assert ticket not in json.dumps(configured) + finally: + engine.dispose() + + +def test_store_app_readiness_requires_default_tenant( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "store") + monkeypatch.setenv("FEISHU_APP_ID", "cli-store") + monkeypatch.setenv("FEISHU_APP_SECRET", "secret") + monkeypatch.setenv("FEISHU_APP_TICKET", "environment-ticket") + monkeypatch.setenv("FEISHU_DEFAULT_TENANT_KEY", "") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + _add_active_subscription(db, tenant_key="tenant-a", suffix="default") + + result = ObservabilityService(db)._feishu_subscriptions_check() + + assert result[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert result["ticket_configured"] is True + assert result["default_tenant_configured"] is False + assert result["reasons"] == ["default_tenant_missing"] + finally: + engine.dispose() + + +def test_readiness_checks_credentials_for_processable_deliveries_without_active_plan( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "self") + monkeypatch.setenv("FEISHU_APP_ID", "") + monkeypatch.setenv("FEISHU_APP_SECRET", "") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + subscription = _add_active_subscription( + db, + tenant_key="tenant-a", + suffix="durable", + ) + subscription.status = PushSubscriptionStatus.COMPLETED + subscription.next_run_at = None + db.commit() + scheduled_for = datetime(2026, 7, 27, 1, 0) + for index, status_value in enumerate( + [ + PushDeliveryStatus.PENDING, + PushDeliveryStatus.RETRY, + PushDeliveryStatus.PROCESSING, + ] + ): + _add_delivery( + db, + subscription, + suffix=f"durable-{index}", + status_value=status_value, + scheduled_for=scheduled_for + timedelta(minutes=index), + ) + + result = ObservabilityService(db)._feishu_subscriptions_check() + + assert result[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert result["active"] == 0 + assert result["processable_deliveries"] == 3 + assert result["active_tenant_count"] == 1 + assert result["reasons"] == ["credentials_missing"] + finally: + engine.dispose() + + +def test_readiness_ignores_terminal_deliveries_without_active_plan() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + subscription = _add_active_subscription( + db, + tenant_key="tenant-a", + suffix="terminal", + ) + subscription.status = PushSubscriptionStatus.COMPLETED + subscription.next_run_at = None + db.commit() + scheduled_for = datetime(2026, 7, 27, 1, 0) + for index, status_value in enumerate( + [ + PushDeliveryStatus.SENT, + PushDeliveryStatus.FAILED, + PushDeliveryStatus.SKIPPED, + ] + ): + _add_delivery( + db, + subscription, + suffix=f"terminal-{index}", + status_value=status_value, + scheduled_for=scheduled_for + timedelta(minutes=index), + ) + + result = ObservabilityService(db)._feishu_subscriptions_check() + + assert result == { + ObservabilityKey.STATUS: ObservabilityStatus.SKIPPED, + "active": 0, + "processable_deliveries": 0, + } + finally: + engine.dispose() + + +def test_self_app_counts_tenants_from_processable_deliveries( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "self") + monkeypatch.setenv("FEISHU_APP_ID", "cli-self") + monkeypatch.setenv("FEISHU_APP_SECRET", "secret") + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + scheduled_for = datetime(2026, 7, 27, 1, 0) + for index, tenant_key in enumerate(["tenant-a", "tenant-b"]): + subscription = _add_active_subscription( + db, + tenant_key=tenant_key, + suffix=f"delivery-tenant-{index}", + ) + subscription.status = PushSubscriptionStatus.COMPLETED + subscription.next_run_at = None + db.commit() + _add_delivery( + db, + subscription, + suffix=f"delivery-tenant-{index}", + status_value=PushDeliveryStatus.PENDING, + scheduled_for=scheduled_for + timedelta(minutes=index), + ) + + result = ObservabilityService(db)._feishu_subscriptions_check() + + assert result[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert result["active"] == 0 + assert result["processable_deliveries"] == 2 + assert result["active_tenant_count"] == 2 + assert result["reasons"] == ["self_app_multiple_tenants"] + finally: + engine.dispose() diff --git a/tests/test_feishu_tenant_routing.py b/tests/test_feishu_tenant_routing.py new file mode 100644 index 0000000..dfcfe0d --- /dev/null +++ b/tests/test_feishu_tenant_routing.py @@ -0,0 +1,157 @@ +from collections.abc import Iterator +from typing import Any + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import Session + +from app.application.feishu.commands import FeishuCommandService +from app.core.config import get_settings +from app.core.database import Base +from app.modules.audit.models import AuditLog +from app.modules.feishu.service import FeishuService +from app.modules.feishu_users.constants import FeishuUserRole, FeishuUserStatus +from app.modules.feishu_users.principal import FeishuPrincipal + + +class RecordingFeishuClient: + def __init__(self) -> None: + self.calls: list[dict[str, Any]] = [] + + def send_text( + self, + text: str, + receive_id: str | None, + receive_id_type: str, + uuid: str | None, + tenant_key: str | None = None, + ) -> dict[str, Any]: + self.calls.append( + { + "operation": "text", + "receive_id": receive_id, + "tenant_key": tenant_key, + "uuid": uuid, + } + ) + return {"code": 0, "data": {"message_id": "om-text"}} + + def send_card( + self, + card: dict[str, Any], + receive_id: str | None, + receive_id_type: str, + uuid: str | None, + tenant_key: str | None = None, + ) -> dict[str, Any]: + self.calls.append( + { + "operation": "card", + "receive_id": receive_id, + "tenant_key": tenant_key, + "uuid": uuid, + } + ) + return {"code": 0, "data": {"message_id": "om-card"}} + + def upload_image( + self, + image: bytes, + filename: str = "lifecycle-report.png", + tenant_key: str | None = None, + ) -> dict[str, Any]: + self.calls.append( + { + "operation": "image", + "filename": filename, + "tenant_key": tenant_key, + } + ) + return {"code": 0, "data": {"image_key": "img-key"}} + + +@pytest.fixture +def database( + monkeypatch: pytest.MonkeyPatch, +) -> Iterator[Session]: + monkeypatch.setenv("FEISHU_APP_ID", "cli-routing") + monkeypatch.setenv("FEISHU_APP_SECRET", "routing-secret") + monkeypatch.setenv("FEISHU_DEFAULT_TENANT_KEY", "tenant-default") + monkeypatch.setenv("FEISHU_USER_FEATURES_ENABLED", "true") + get_settings.cache_clear() + engine = create_engine("sqlite://") + Base.metadata.create_all(engine, tables=[AuditLog.__table__]) + try: + with Session(engine) as db: + yield db + finally: + engine.dispose() + get_settings.cache_clear() + + +def test_service_routes_text_card_and_image_through_selected_tenant( + database: Session, +) -> None: + service = FeishuService(database) + client = RecordingFeishuClient() + service.client = client + + service.send_text("默认消息", receive_id="oc-default") + service.set_tenant_key("tenant-a") + service.send_card({"elements": []}, receive_id="oc-a") + service.upload_image(b"png") + service.send_text( + "显式覆盖", + receive_id="oc-b", + tenant_key="tenant-b", + ) + + assert [ + (item["operation"], item["tenant_key"]) + for item in client.calls + ] == [ + ("text", "tenant-default"), + ("card", "tenant-a"), + ("image", "tenant-a"), + ("text", "tenant-b"), + ] + + +def test_command_reply_uses_verified_principal_tenant( + database: Session, +) -> None: + commands = FeishuCommandService(database) + client = RecordingFeishuClient() + commands.feishu.client = client + principal = FeishuPrincipal( + owner_id=1, + user_code="FSU-routing", + tenant_key="tenant-principal", + open_id="ou-routing", + union_id=None, + feishu_user_id=None, + role=FeishuUserRole.USER, + status=FeishuUserStatus.ACTIVE, + timezone="Asia/Shanghai", + quiet_hours_start=None, + quiet_hours_end=None, + chat_id="oc-routing", + chat_type="p2p", + ) + + result = commands.handle_text( + "帮助", + auto_reply=True, + principal=principal, + ) + + assert result["command"] == "help" + assert commands.feishu.tenant_key == "tenant-principal" + assert client.calls == [ + { + "operation": "text", + "receive_id": "oc-routing", + "tenant_key": "tenant-principal", + "uuid": None, + } + ] diff --git a/tests/test_feishu_users.py b/tests/test_feishu_users.py new file mode 100644 index 0000000..7dc5553 --- /dev/null +++ b/tests/test_feishu_users.py @@ -0,0 +1,418 @@ +from collections.abc import Iterator + +import pytest +from fastapi import FastAPI, HTTPException +from fastapi.testclient import TestClient +from sqlalchemy import create_engine, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.core.config import get_settings +from app.core.database import Base, get_db +from app.modules.audit.models import AuditLog +from app.modules.business.models import MarketWatchlist +from app.modules.feishu_users.constants import ( + FeishuCapability, + FeishuUserAuditAction, + FeishuUserRole, + FeishuUserStatus, + parse_admin_identities, +) +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) +from app.modules.feishu_users.routes import router +from app.modules.feishu_users.services import ( + FeishuIdentityService, + FeishuUserManagementService, +) + + +@pytest.fixture +def session_factory() -> Iterator[sessionmaker[Session]]: + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all( + engine, + tables=[ + AuditLog.__table__, + FeishuAdminBootstrapTombstone.__table__, + FeishuUser.__table__, + MarketWatchlist.__table__, + ], + ) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + Base.metadata.drop_all( + engine, + tables=[ + MarketWatchlist.__table__, + FeishuAdminBootstrapTombstone.__table__, + FeishuUser.__table__, + AuditLog.__table__, + ], + ) + engine.dispose() + + +def test_identity_registration_is_tenant_scoped_and_bootstraps_admin( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + service = FeishuIdentityService( + db, + admin_identities="tenant-a:ou-admin", + ) + admin = service.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-admin", + union_id="on-admin", + user_id="u-admin", + ) + ordinary = service.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + other_tenant = service.resolve_or_register( + tenant_key="tenant-b", + open_id="ou-user", + ) + + assert admin.role == FeishuUserRole.ADMIN + assert admin.is_admin is True + assert admin.has_capability(FeishuCapability.USER_ADMINISTRATION) is True + assert ordinary.role == FeishuUserRole.USER + assert ordinary.has_capability(FeishuCapability.PERSONAL_AI) is True + assert ordinary.has_capability(FeishuCapability.COMPANY_REPORTS) is False + assert ordinary.owner_id != other_tenant.owner_id + assert db.scalar(select(FeishuUser).where(FeishuUser.id == admin.owner_id)).union_id == ( + "on-admin" + ) + + registrations = list( + db.execute( + select(AuditLog).where( + AuditLog.action == FeishuUserAuditAction.REGISTER + ) + ).scalars() + ) + assert len(registrations) == 3 + assert all("ou-" not in (item.request_payload or "") for item in registrations) + + +def test_each_configured_admin_identity_can_bootstrap_once( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + service = FeishuIdentityService( + db, + admin_identities="tenant-a:ou-admin-a,tenant-b:ou-admin-b", + ) + + first = service.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-admin-a", + ) + second = service.resolve_or_register( + tenant_key="tenant-b", + open_id="ou-admin-b", + ) + + assert first.role == FeishuUserRole.ADMIN + assert second.role == FeishuUserRole.ADMIN + + +def test_identity_refreshes_existing_sender_without_reapplying_admin_bootstrap( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + principal = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-existing", + ) + refreshed = FeishuIdentityService( + db, + admin_identities="tenant-a:ou-existing", + ).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-existing", + union_id="on-existing", + ) + + assert refreshed.owner_id == principal.owner_id + assert refreshed.role == FeishuUserRole.USER + assert refreshed.union_id == "on-existing" + assert db.scalar( + select(AuditLog).where( + AuditLog.action == FeishuUserAuditAction.AUTHENTICATE + ) + ) + + +def test_first_registration_claims_only_matching_legacy_watchlist( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + db.add_all( + [ + MarketWatchlist( + actor="ou-claim", + symbol="600000.SH", + enabled=True, + ), + MarketWatchlist( + actor="ou-other", + symbol="000001.SZ", + enabled=True, + ), + ] + ) + db.commit() + + principal = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-claim", + open_id="ou-claim", + ) + + records = list( + db.execute( + select(MarketWatchlist).order_by(MarketWatchlist.id.asc()) + ).scalars() + ) + assert records[0].owner_id == principal.owner_id + assert records[1].owner_id is None + + +def test_identity_rejects_missing_parts_and_invalid_admin_config( + session_factory: sessionmaker[Session], +) -> None: + assert parse_admin_identities("tenant-a:ou-a, tenant-b:ou-b") == { + ("tenant-a", "ou-a"), + ("tenant-b", "ou-b"), + } + with pytest.raises(ValueError): + parse_admin_identities("missing-separator") + + with session_factory() as db: + with pytest.raises(HTTPException) as exc_info: + FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id=" ", + ) + assert exc_info.value.status_code == 422 + assert db.scalar(select(FeishuUser)) is None + + +def test_management_protects_last_active_admin_and_audits_denial( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + identity = FeishuIdentityService( + db, + admin_identities="tenant-a:ou-admin", + ) + admin = identity.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-admin", + ) + user = identity.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + management = FeishuUserManagementService(db) + + with pytest.raises(HTTPException) as demote_error: + management.update_user( + admin.user_code, + changes={"role": FeishuUserRole.USER}, + actor="service-admin", + ) + assert demote_error.value.status_code == 409 + + with pytest.raises(HTTPException) as disable_error: + management.update_user( + admin.user_code, + changes={"status": FeishuUserStatus.DISABLED}, + actor="service-admin", + ) + assert disable_error.value.status_code == 409 + + promoted = management.update_user( + user.user_code, + changes={"role": FeishuUserRole.ADMIN}, + actor="service-admin", + ) + assert promoted.role == FeishuUserRole.ADMIN + demoted = management.update_user( + admin.user_code, + changes={"role": FeishuUserRole.USER}, + actor="service-admin", + ) + assert demoted.role == FeishuUserRole.USER + + denied = list( + db.execute( + select(AuditLog).where( + AuditLog.action == FeishuUserAuditAction.UPDATE_DENIED + ) + ).scalars() + ) + assert len(denied) == 2 + assert all(item.actor == "service-admin" for item in denied) + + +def test_management_validates_timezone_and_quiet_hours( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + user = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + management = FeishuUserManagementService(db) + updated = management.update_user( + user.user_code, + changes={ + "timezone": "Europe/London", + "quiet_hours_start": "22:30", + "quiet_hours_end": "07:15", + }, + actor="service-admin", + ) + assert updated.timezone == "Europe/London" + assert updated.quiet_hours_start.isoformat() == "22:30:00" + assert updated.quiet_hours_end.isoformat() == "07:15:00" + + with pytest.raises(HTTPException) as timezone_error: + management.update_user( + user.user_code, + changes={"timezone": "Invalid/Nowhere"}, + actor="service-admin", + ) + assert timezone_error.value.status_code == 422 + + with pytest.raises(HTTPException) as quiet_error: + management.update_user( + user.user_code, + changes={"quiet_hours_end": None}, + actor="service-admin", + ) + assert quiet_error.value.status_code == 422 + + cleared = management.update_user( + user.user_code, + changes={ + "quiet_hours_start": None, + "quiet_hours_end": None, + }, + actor="service-admin", + ) + assert cleared.quiet_hours_start is None + assert cleared.quiet_hours_end is None + + +def test_disabled_principal_has_no_capabilities( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + principal = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + FeishuUserManagementService(db).update_user( + principal.user_code, + changes={"status": FeishuUserStatus.DISABLED}, + actor="service-admin", + ) + refreshed = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + assert refreshed.is_active is False + assert refreshed.has_capability(FeishuCapability.PERSONAL_AI) is False + with pytest.raises(HTTPException) as exc_info: + refreshed.require_capability(FeishuCapability.PERSONAL_AI) + assert exc_info.value.status_code == 403 + + +def test_internal_user_routes_use_api_principal_and_ignore_body_identity( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("API_KEY", "service-key") + monkeypatch.setenv("API_ACTOR", "service-admin") + monkeypatch.setenv("API_KEYS", "[]") + get_settings.cache_clear() + + with session_factory() as db: + identity = FeishuIdentityService( + db, + admin_identities="tenant-a:ou-admin", + ) + identity.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-admin", + ) + user = identity.resolve_or_register( + tenant_key="tenant-a", + open_id="ou-user", + ) + user_code = user.user_code + + app = FastAPI() + app.include_router(router, prefix="/api/v1/integrations/feishu") + + def override_db() -> Iterator[Session]: + with session_factory() as db: + yield db + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + try: + assert ( + client.get("/api/v1/integrations/feishu/users").status_code + == 401 + ) + response = client.patch( + f"/api/v1/integrations/feishu/users/{user_code}", + headers={"X-API-Key": "service-key"}, + json={ + "role": "admin", + "actor": "forged-user", + "open_id": "ou-forged", + }, + ) + assert response.status_code == 200 + assert response.json()["role"] == "admin" + assert response.json()["open_id"] == "ou-user" + + listed = client.get( + "/api/v1/integrations/feishu/users?role=admin", + headers={"X-API-Key": "service-key"}, + ) + assert listed.status_code == 200 + assert listed.json()["total"] == 2 + detail = client.get( + f"/api/v1/integrations/feishu/users/{user_code}", + headers={"X-API-Key": "service-key"}, + ) + assert detail.status_code == 200 + + with session_factory() as db: + update_log = db.execute( + select(AuditLog) + .where(AuditLog.action == FeishuUserAuditAction.UPDATE) + .order_by(AuditLog.id.desc()) + ).scalars().first() + assert update_log is not None + assert update_log.actor == "service-admin" + assert "forged-user" not in (update_log.request_payload or "") + assert "ou-forged" not in (update_log.request_payload or "") + finally: + get_settings.cache_clear() diff --git a/tests/test_feishu_webhook_verification.py b/tests/test_feishu_webhook_verification.py new file mode 100644 index 0000000..179c8bd --- /dev/null +++ b/tests/test_feishu_webhook_verification.py @@ -0,0 +1,84 @@ +import base64 +import json +import time +from hashlib import sha256 + +import pytest +from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes +from cryptography.hazmat.primitives.padding import PKCS7 +from fastapi import HTTPException + +from app.core.config import get_settings +from app.modules.feishu.event_verification import FeishuWebhookVerifier + + +def _encrypt_event(payload: dict, encrypt_key: str) -> str: + key = sha256(encrypt_key.encode("utf-8")).digest() + padder = PKCS7(algorithms.AES.block_size).padder() + cleartext = json.dumps(payload, ensure_ascii=False).encode("utf-8") + padded = padder.update(cleartext) + padder.finalize() + encryptor = Cipher(algorithms.AES(key), modes.CBC(key[:16])).encryptor() + return base64.b64encode(encryptor.update(padded) + encryptor.finalize()).decode() + + +def test_plain_webhook_requires_the_configured_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("FEISHU_VERIFICATION_TOKEN", "verification-token") + monkeypatch.delenv("FEISHU_ENCRYPT_KEY", raising=False) + get_settings.cache_clear() + try: + raw = json.dumps( + {"header": {"token": "verification-token"}, "event": {}} + ).encode() + assert FeishuWebhookVerifier().verify(raw, {})["event"] == {} + + invalid = json.dumps( + {"header": {"token": "wrong-token"}, "event": {}} + ).encode() + with pytest.raises(HTTPException) as exc_info: + FeishuWebhookVerifier().verify(invalid, {}) + assert exc_info.value.status_code == 401 + finally: + get_settings.cache_clear() + + +def test_encrypted_webhook_requires_valid_signature_and_decrypts( + monkeypatch: pytest.MonkeyPatch, +) -> None: + encrypt_key = "test-encrypt-key" + monkeypatch.setenv("FEISHU_VERIFICATION_TOKEN", "verification-token") + monkeypatch.setenv("FEISHU_ENCRYPT_KEY", encrypt_key) + get_settings.cache_clear() + try: + event = { + "header": { + "token": "verification-token", + "tenant_key": "tenant-a", + }, + "event": {"sender": {"sender_id": {"open_id": "ou-a"}}}, + } + raw = json.dumps( + {"encrypt": _encrypt_event(event, encrypt_key)}, + separators=(",", ":"), + ).encode() + timestamp = str(int(time.time())) + nonce = "nonce" + signature = sha256( + timestamp.encode() + + nonce.encode() + + encrypt_key.encode() + + raw + ).hexdigest() + headers = { + "X-Lark-Request-Timestamp": timestamp, + "X-Lark-Request-Nonce": nonce, + "X-Lark-Signature": signature, + } + + assert FeishuWebhookVerifier().verify(raw, headers) == event + + headers["X-Lark-Signature"] = "invalid" + with pytest.raises(HTTPException) as exc_info: + FeishuWebhookVerifier().verify(raw, headers) + assert exc_info.value.status_code == 401 + finally: + get_settings.cache_clear() diff --git a/tests/test_lifecycle_queue_failure.py b/tests/test_lifecycle_queue_failure.py new file mode 100644 index 0000000..8aca6a0 --- /dev/null +++ b/tests/test_lifecycle_queue_failure.py @@ -0,0 +1,69 @@ +from types import SimpleNamespace + +import pytest +from sqlalchemy import create_engine, select +from sqlalchemy.orm import sessionmaker + +from app.core.background.task_queue import lifecycle as lifecycle_queue +from app.core.database import Base +from app.modules.reports.constants import ReportType +from app.modules.workflows.constants import WorkflowStatus, WorkflowType +from app.modules.workflows.models import WorkflowAction, WorkflowInstance +from app.tasks import celery_app + + +def test_lifecycle_enqueue_failure_marks_workflow_failed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + + class BrokenSignature: + def apply_async(self) -> None: + raise RuntimeError("broker unavailable") + + monkeypatch.setattr(lifecycle_queue, "SessionLocal", factory) + monkeypatch.setattr( + lifecycle_queue, + "get_settings", + lambda: SimpleNamespace(task_queue_enabled=True), + ) + monkeypatch.setattr( + celery_app, + "signature", + lambda *args, **kwargs: BrokenSignature(), + ) + try: + with pytest.raises(RuntimeError, match="broker unavailable"): + lifecycle_queue.enqueue_lifecycle_report( + report_type=ReportType.WEEKLY, + actor="pytest", + ) + + with factory() as db: + workflow = db.execute( + select(WorkflowInstance).where( + WorkflowInstance.workflow_type + == WorkflowType.LIFECYCLE_REPORT + ) + ).scalar_one() + actions = list( + db.execute( + select(WorkflowAction) + .where(WorkflowAction.workflow_code == workflow.code) + .order_by(WorkflowAction.id.asc()) + ).scalars() + ) + + assert workflow.status == WorkflowStatus.FAILED + assert workflow.current_step == "enqueue_failed" + assert workflow.completed_at is not None + assert workflow.payload == {"error": "broker unavailable"} + assert [action.action for action in actions] == [ + "queued", + "enqueue_failed", + ] + assert actions[-1].to_status == WorkflowStatus.FAILED + finally: + engine.dispose() diff --git a/tests/test_memory_retention_boundaries.py b/tests/test_memory_retention_boundaries.py new file mode 100644 index 0000000..9f62a08 --- /dev/null +++ b/tests/test_memory_retention_boundaries.py @@ -0,0 +1,95 @@ +from datetime import timedelta + +from sqlalchemy import create_engine, select +from sqlalchemy.orm import sessionmaker + +from app.core.database import Base +from app.core.security import ApiPrincipal +from app.core.utils.time import utc_now +from app.modules.ai_memory.constants import AIMemoryResponseKey, AIMemoryStatus +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.ai_memory.routes import list_memory +from app.modules.dashboard.routes import dashboard_summary +from app.modules.observability.routes import metrics as observability_metrics + + +def _session_factory(): + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + return engine, sessionmaker(bind=engine, expire_on_commit=False) + + +def _seed_expired_memory(factory: sessionmaker, code: str, subject: str) -> None: + with factory() as db: + db.add( + AIMemoryEntry( + code=code, + scope="project", + subject=subject, + content="expired", + status=AIMemoryStatus.ACTIVE, + expires_at=utc_now() - timedelta(seconds=1), + ) + ) + db.commit() + + +def _assert_archived(factory: sessionmaker, code: str) -> None: + with factory() as db: + stored = db.execute( + select(AIMemoryEntry).where(AIMemoryEntry.code == code) + ).scalar_one() + assert stored.status == AIMemoryStatus.ARCHIVED + + +def test_memory_list_route_persists_expiration_at_request_boundary() -> None: + engine, factory = _session_factory() + try: + code = "MEM-LIST-EXPIRED" + _seed_expired_memory(factory, code, "P-LIST") + + with factory() as db: + result = list_memory( + scope="project", + subject="P-LIST", + status=AIMemoryStatus.ACTIVE, + limit=100, + db=db, + ) + assert result[AIMemoryResponseKey.ITEMS] == [] + + _assert_archived(factory, code) + finally: + engine.dispose() + + +def test_observability_metrics_route_persists_memory_expiration() -> None: + engine, factory = _session_factory() + try: + code = "MEM-METRICS-EXPIRED" + _seed_expired_memory(factory, code, "P-METRICS") + + with factory() as db: + observability_metrics(db=db) + + _assert_archived(factory, code) + finally: + engine.dispose() + + +def test_dashboard_route_excludes_and_archives_expired_memory() -> None: + engine, factory = _session_factory() + try: + code = "MEM-DASHBOARD-EXPIRED" + _seed_expired_memory(factory, code, "P-DASHBOARD") + + with factory() as db: + result = dashboard_summary( + db=db, + principal=ApiPrincipal(actor="pytest"), + ) + assert result["metrics"]["active_ai_memory"] == 0 + + _assert_archived(factory, code) + finally: + engine.dispose() diff --git a/tests/test_personal_data_erasure.py b/tests/test_personal_data_erasure.py new file mode 100644 index 0000000..a34640e --- /dev/null +++ b/tests/test_personal_data_erasure.py @@ -0,0 +1,469 @@ +import json +from collections.abc import Iterator +from datetime import timedelta +from uuid import uuid4 + +import pytest +from fastapi import FastAPI, HTTPException +from fastapi.testclient import TestClient +from sqlalchemy import create_engine, event, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.application.feishu.personal_data import ( + PERSONAL_DATA_ERASURE_ACTION, + FeishuPersonalDataService, +) +from app.core.config import get_settings +from app.core.database import Base, get_db +from app.core.utils.time import utc_now +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.audit.models import AuditLog +from app.modules.business.models import MarketWatchlist +from app.modules.feishu_users.constants import FeishuUserRole +from app.modules.feishu_users.models import ( + FeishuAdminBootstrapTombstone, + FeishuUser, +) +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.feishu_users.routes import router +from app.modules.feishu_users.services import FeishuIdentityService +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + UserPreference, +) +from app.modules.personalization.schemas import ErasureResult +from app.modules.subscriptions.models import PushDelivery, PushSubscription + + +@pytest.fixture +def session_factory() -> Iterator[sessionmaker[Session]]: + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + + @event.listens_for(engine, "connect") + def enable_foreign_keys(dbapi_connection, _connection_record) -> None: + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.close() + + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + Base.metadata.drop_all(engine) + engine.dispose() + + +def _create_user( + db: Session, + suffix: str, + *, + role: str = FeishuUserRole.USER, +) -> FeishuUser: + user = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"ou-open-{suffix}", + union_id=f"on-union-{suffix}", + user_id=f"cli-user-{suffix}", + role=role, + timezone="Asia/Shanghai", + ) + db.add(user) + db.flush() + return user + + +def _seed_personal_data(db: Session, user: FeishuUser) -> None: + db.add( + UserPreference( + owner_id=user.id, + category="tone", + value="简洁", + normalized_value="简洁", + source="explicit", + ) + ) + conversation = AIConversation( + owner_id=user.id, + chat_type="private", + chat_key=user.open_id, + ) + db.add(conversation) + db.flush() + db.add( + AIConversationMessage( + conversation_id=conversation.id, + role="user", + content="仅属于我的历史", + ) + ) + db.add( + AIMemoryEntry( + code=f"MEM-{uuid4().hex}", + owner_id=user.id, + kind="memory", + scope="user", + subject="preference", + content="个人记忆", + source="explicit", + actor=user.open_id, + ) + ) + db.add( + MarketWatchlist( + owner_id=user.id, + actor=user.open_id, + symbol="600000.SH", + enabled=True, + ) + ) + subscription = PushSubscription( + code=f"SUB-{uuid4().hex}", + owner_id=user.id, + target_type="user", + target_id=user.open_id, + prompt="个人提醒", + schedule_type="daily", + schedule_config={"hour": 9, "minute": 0}, + timezone="Asia/Shanghai", + next_run_at=utc_now() + timedelta(days=1), + status="active", + consented_at=utc_now(), + ) + db.add(subscription) + db.flush() + db.add( + PushDelivery( + code=f"DEL-{uuid4().hex}", + subscription_id=subscription.id, + scheduled_for=utc_now(), + idempotency_key=uuid4().hex, + message_uuid=str(uuid4()), + status="pending", + next_attempt_at=utc_now(), + ) + ) + db.add( + AuditLog( + actor=f"feishu:{user.open_id}", + source="feishu", + action="test.personal.action", + target_type="feishu-user", + target_id=user.code, + request_payload=json.dumps( + { + "open_id": user.open_id, + "union_id": user.union_id, + "private_note": "仅属于该用户的请求内容", + } + ), + response_payload=json.dumps( + { + "user_id": user.user_id, + "code": user.code, + "private_answer": "仅属于该用户的响应内容", + } + ), + request_id=f"request-{user.code}", + ) + ) + db.commit() + + +def test_confirmation_is_one_user_only_and_expires( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + first = _create_user(db, "first") + second = _create_user(db, "second") + db.commit() + first_principal = FeishuPrincipal.from_user(first) + second_principal = FeishuPrincipal.from_user(second) + service = FeishuPersonalDataService(db) + + confirmation = service.request_confirmation(first_principal) + with pytest.raises(HTTPException) as wrong_code: + service.confirm(first_principal, "WRONG") + assert wrong_code.value.status_code == 422 + + with pytest.raises(HTTPException) as cross_user: + service.confirm(second_principal, confirmation.confirmation_code) + assert cross_user.value.status_code == 422 + + request = db.execute( + select(PersonalDataErasureRequest).where( + PersonalDataErasureRequest.owner_id == first.id + ) + ).scalar_one() + request.expires_at = utc_now() - timedelta(seconds=1) + db.commit() + with pytest.raises(HTTPException) as expired: + service.confirm(first_principal, confirmation.confirmation_code) + assert expired.value.status_code == 422 + assert db.get(FeishuUser, first.id) is not None + assert db.get(FeishuUser, second.id) is not None + + +def test_confirm_erases_all_personal_data_and_anonymizes_audit( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + user = _create_user(db, "erase") + other = _create_user(db, "other") + _seed_personal_data(db, user) + _seed_personal_data(db, other) + owner_id = user.id + tenant_key = user.tenant_key + open_id = user.open_id + identifiers = (user.code, user.open_id, user.union_id, user.user_id) + principal = FeishuPrincipal.from_user(user) + service = FeishuPersonalDataService(db) + + confirmation = service.request_confirmation(principal) + result = service.confirm(principal, confirmation.confirmation_code) + + assert result.deleted["deliveries"] == 1 + assert result.deleted["subscriptions"] == 1 + assert result.deleted["preferences"] == 1 + assert result.deleted["conversation_messages"] == 1 + assert result.deleted["conversations"] == 1 + assert result.deleted["ai_memory"] == 1 + assert result.deleted["watchlist"] == 1 + assert result.deleted["identity"] == 1 + assert db.get(FeishuUser, owner_id) is None + assert db.get(FeishuUser, other.id) is not None + assert db.scalar( + select(PushSubscription).where(PushSubscription.owner_id == owner_id) + ) is None + assert db.scalar( + select(UserPreference).where(UserPreference.owner_id == owner_id) + ) is None + assert db.scalar( + select(AIMemoryEntry).where(AIMemoryEntry.owner_id == owner_id) + ) is None + assert db.scalar( + select(MarketWatchlist).where(MarketWatchlist.owner_id == owner_id) + ) is None + assert db.scalar( + select(AIConversation).where(AIConversation.owner_id == owner_id) + ) is None + + logs = list(db.execute(select(AuditLog)).scalars()) + serialized_logs = "\n".join( + " ".join( + str(value or "") + for value in ( + log.actor, + log.target_id, + log.request_payload, + log.response_payload, + ) + ) + for log in logs + ) + assert all(identifier not in serialized_logs for identifier in identifiers) + final_log = db.execute( + select(AuditLog).where(AuditLog.action == PERSONAL_DATA_ERASURE_ACTION) + ).scalar_one() + assert final_log.actor == result.anonymous_id + assert final_log.target_type is None + assert final_log.target_id is None + assert final_log.request_payload is None + assert final_log.response_payload is None + assert final_log.request_id is None + assert final_log.status == "success" + + anonymized_history = db.execute( + select(AuditLog).where( + AuditLog.action == "test.personal.action", + AuditLog.actor == result.anonymous_id, + ) + ).scalar_one() + assert anonymized_history.target_type is None + assert anonymized_history.target_id is None + assert anonymized_history.request_payload is None + assert anonymized_history.response_payload is None + assert anonymized_history.request_id is None + assert anonymized_history.status == "success" + + recreated = FeishuIdentityService(db).resolve_or_register( + tenant_key=tenant_key, + open_id=open_id, + ) + assert recreated.owner_id != owner_id + assert recreated.user_code not in identifiers + assert recreated.role == FeishuUserRole.USER + + +def test_last_active_admin_is_protected_until_another_admin_exists( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + admin = _create_user(db, "admin", role=FeishuUserRole.ADMIN) + db.commit() + principal = FeishuPrincipal.from_user(admin) + service = FeishuPersonalDataService(db) + confirmation = service.request_confirmation(principal) + + with pytest.raises(HTTPException) as protected: + service.confirm(principal, confirmation.confirmation_code) + assert protected.value.status_code == 409 + assert db.get(FeishuUser, admin.id) is not None + + second_admin = _create_user( + db, + "second-admin", + role=FeishuUserRole.ADMIN, + ) + db.commit() + result = service.confirm(principal, confirmation.confirmation_code) + assert result.deleted["identity"] == 1 + assert db.get(FeishuUser, admin.id) is None + assert db.get(FeishuUser, second_admin.id) is not None + + +def test_erased_configured_admin_is_not_bootstrapped_again( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv( + "FEISHU_ADMIN_IDENTITIES", + "tenant-bootstrap:ou-bootstrap", + ) + get_settings.cache_clear() + try: + with session_factory() as db: + identity = FeishuIdentityService(db) + initial = identity.resolve_or_register( + tenant_key="tenant-bootstrap", + open_id="ou-bootstrap", + ) + assert initial.role == FeishuUserRole.ADMIN + _create_user(db, "replacement-admin", role=FeishuUserRole.ADMIN) + db.commit() + + service = FeishuPersonalDataService(db) + confirmation = service.request_confirmation(initial) + result = service.confirm(initial, confirmation.confirmation_code) + + tombstone = db.scalar(select(FeishuAdminBootstrapTombstone)) + assert tombstone is not None + assert len(tombstone.identity_hash) == 64 + assert "tenant-bootstrap" not in tombstone.identity_hash + assert "ou-bootstrap" not in tombstone.identity_hash + + recreated = FeishuIdentityService(db).resolve_or_register( + tenant_key="tenant-bootstrap", + open_id="ou-bootstrap", + ) + assert recreated.owner_id != initial.owner_id + assert recreated.role == FeishuUserRole.USER + assert result.deleted["identity"] == 1 + finally: + get_settings.cache_clear() + + +def test_extra_hook_failure_rolls_back_the_deletion_transaction( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + user = _create_user(db, "rollback") + _seed_personal_data(db, user) + owner_id = user.id + principal = FeishuPrincipal.from_user(user) + confirmation = FeishuPersonalDataService(db).request_confirmation(principal) + + def fail_hook(_db: Session, _owner_id: int, _anonymous_id: str) -> None: + raise RuntimeError("integration erasure failed") + + with pytest.raises(RuntimeError, match="integration erasure failed"): + FeishuPersonalDataService( + db, + extra_hooks=(fail_hook,), + ).confirm(principal, confirmation.confirmation_code) + + assert db.get(FeishuUser, owner_id) is not None + assert db.scalar( + select(UserPreference).where(UserPreference.owner_id == owner_id) + ) is not None + assert db.scalar( + select(AIMemoryEntry).where(AIMemoryEntry.owner_id == owner_id) + ) is not None + assert db.scalar( + select(PushSubscription).where(PushSubscription.owner_id == owner_id) + ) is not None + assert db.scalar( + select(PersonalDataErasureRequest).where( + PersonalDataErasureRequest.owner_id == owner_id + ) + ) is not None + + +def test_delete_api_uses_service_principal_actor_and_ignores_body_identity( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("API_KEY", "service-key") + monkeypatch.setenv("API_ACTOR", "service-admin") + monkeypatch.setenv("API_KEYS", "[]") + get_settings.cache_clear() + seen: dict[str, str] = {} + + def fake_erase( + _service: FeishuPersonalDataService, + code: str, + *, + actor: str, + ) -> ErasureResult: + seen.update(code=code, actor=actor) + return ErasureResult( + anonymous_id="anonymous-test", + deleted={"identity": 1}, + ) + + monkeypatch.setattr( + FeishuPersonalDataService, + "erase_by_user_code", + fake_erase, + ) + app = FastAPI() + app.include_router(router, prefix="/api/v1/integrations/feishu") + + def override_db() -> Iterator[Session]: + with session_factory() as db: + yield db + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + try: + path = "/api/v1/integrations/feishu/users/FSU-target/personal-data" + assert client.delete(path).status_code == 401 + assert ( + client.delete(path, headers={"X-API-Key": "wrong-key"}).status_code + == 401 + ) + response = client.request( + "DELETE", + path, + headers={"X-API-Key": "service-key"}, + json={ + "actor": "forged-user", + "open_id": "ou-forged", + }, + ) + assert response.status_code == 200 + assert response.json()["anonymous_id"] == "anonymous-test" + assert seen == { + "code": "FSU-target", + "actor": "service-admin", + } + finally: + get_settings.cache_clear() diff --git a/tests/test_personalization.py b/tests/test_personalization.py new file mode 100644 index 0000000..e5fa245 --- /dev/null +++ b/tests/test_personalization.py @@ -0,0 +1,365 @@ +from datetime import timedelta + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, select, update +from sqlalchemy.orm import Session, sessionmaker + +from app.core.config import get_settings +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.ai_memory.constants import AIMemoryScope +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.ai_memory.service import AIMemoryService +from app.modules.business.models import MarketWatchlist +from app.modules.feishu_users.models import FeishuUser +from app.modules.market.service import MarketService +from app.modules.personalization.constants import ( + CONVERSATION_MAX_MESSAGES, + PersonalizationContextKey, +) +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + PersonalDataErasureRequest, + UserPreference, +) +from app.modules.personalization.services import ( + ConversationService, + PersonalDataErasureService, + PersonalizationContextService, + PreferenceService, +) + + +def _session_factory(): + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + return engine, sessionmaker(bind=engine, expire_on_commit=False) + + +def _user(db: Session, suffix: str, *, open_id: str | None = None) -> FeishuUser: + record = FeishuUser( + code=f"USR-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=open_id or f"open-{suffix}", + ) + db.add(record) + db.commit() + db.refresh(record) + return record + + +def test_rules_memory_and_watchlists_are_owner_scoped(monkeypatch) -> None: + monkeypatch.setenv("AI_MEMORY_ENABLED", "true") + monkeypatch.setenv("AI_MEMORY_AUTO_WRITE_ENABLED", "true") + get_settings.cache_clear() + engine, factory = _session_factory() + try: + with factory() as db: + first = _user(db, "first", open_id="same-open-id") + second = _user(db, "second", open_id="same-open-id") + memory = AIMemoryService(db) + company_rule = memory.create_rule( + content="公司规则", + scope=AIMemoryScope.GLOBAL, + subject="company", + priority=90, + tags=[], + actor="service", + ) + first_rule = memory.create_rule( + content="相同个人规则", + scope=AIMemoryScope.USER, + subject="profile", + priority=50, + tags=[], + actor="first", + owner_id=first.id, + ) + second_rule = memory.create_rule( + content="相同个人规则", + scope=AIMemoryScope.USER, + subject="profile", + priority=50, + tags=[], + actor="second", + owner_id=second.id, + ) + + assert [item["code"] for item in memory.list_rules()] == [company_rule["code"]] + assert [item["code"] for item in memory.list_rules(owner_id=first.id)] == [ + first_rule["code"] + ] + assert [item["code"] for item in memory.list_rules(owner_id=second.id)] == [ + second_rule["code"] + ] + with pytest.raises(HTTPException) as exc_info: + memory.update_rule( + code=second_rule["code"], + content="越权修改", + priority=None, + tags=None, + enabled=None, + actor="first", + owner_id=first.id, + ) + assert exc_info.value.status_code == 404 + + first_memory = memory.auto_write( + prompt="Remember concise project risk summaries", + context={"scope": "user", "subject": "profile"}, + answer="Use concise project risk summaries.", + actor="first", + owner_id=first.id, + ) + second_memory = memory.auto_write( + prompt="Remember concise project risk summaries", + context={"scope": "user", "subject": "profile"}, + answer="Use concise project risk summaries.", + actor="second", + owner_id=second.id, + ) + assert first_memory is not None + assert second_memory is not None + assert first_memory.code != second_memory.code + assert first_memory.fingerprint != second_memory.fingerprint + + market = MarketService(db) + market.add_watchlist("same-open-id", "600000", owner_id=first.id) + market.add_watchlist("same-open-id", "600000", owner_id=second.id) + assert market.watchlist("ignored", owner_id=first.id) == [{"symbol": "600000.SH"}] + assert market.watchlist("ignored", owner_id=second.id) == [{"symbol": "600000.SH"}] + + market.add_watchlist("legacy-open", "000001") + assert market.watchlist("legacy-open") == [{"symbol": "000001.SZ"}] + assert market.claim_legacy_watchlist(first.id, "legacy-open") == 1 + assert market.watchlist("legacy-open") == [] + assert market.watchlist("ignored", owner_id=first.id) == [ + {"symbol": "600000.SH"}, + {"symbol": "000001.SZ"}, + ] + finally: + get_settings.cache_clear() + engine.dispose() + + +def test_preferences_are_allowlisted_sensitive_safe_and_owner_scoped() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + first = _user(db, "pref-first") + second = _user(db, "pref-second") + service = PreferenceService(db) + first_pref = service.upsert(first.id, "tone", "简洁直接") + second_pref = service.upsert(second.id, "tone", "简洁直接") + + assert first_pref["code"] != second_pref["code"] + assert service.list_preferences(first.id) == [first_pref] + with pytest.raises(HTTPException) as exc_info: + service.delete(first.id, second_pref["code"]) + assert exc_info.value.status_code == 404 + with pytest.raises(HTTPException) as exc_info: + service.upsert(first.id, "topic", "记住我的银行账号 123456") + assert exc_info.value.status_code == 422 + + assert ( + service.save_auto_extraction( + first.id, + provider_name="noop", + user_text="以后请用中文", + structured_payload={ + "preferences": [{"category": "language", "value": "中文"}] + }, + ) + == [] + ) + assert ( + service.save_auto_extraction( + first.id, + provider_name="direct_llm", + user_text="今天怎么样", + structured_payload={ + "preferences": [{"category": "language", "value": "中文"}] + }, + ) + == [] + ) + saved = service.save_auto_extraction( + first.id, + provider_name="direct_llm", + user_text="以后请用中文，并记住我的健康诊断", + structured_payload={ + "preferences": [ + {"category": "language", "value": "中文"}, + {"category": "topic", "value": "我的健康诊断"}, + ] + }, + ) + assert [(item["category"], item["value"]) for item in saved] == [ + ("language", "中文") + ] + finally: + engine.dispose() + + +def test_conversations_keep_twenty_turns_and_isolate_sessions() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + first = _user(db, "conversation-first") + second = _user(db, "conversation-second") + service = ConversationService(db) + assert ( + service.record_turn( + second.id, + "private", + "chat-1", + user_content="placeholder", + assistant_content="placeholder", + provider_name="noop", + ) + is False + ) + for index in range(21): + assert service.record_turn( + first.id, + "private", + "chat-1", + user_content=f"user-{index}", + assistant_content=f"assistant-{index}", + provider_name="direct_llm", + ) + + history = service.history(first.id, "private", "chat-1") + assert len(history) == CONVERSATION_MAX_MESSAGES + assert history[0]["content"] == "user-1" + assert history[-1]["content"] == "assistant-20" + assert service.history(second.id, "private", "chat-1") == [] + assert service.provider_session_id(first.id, "private", "chat-1") == ( + service.provider_session_id(first.id, "p2p", "chat-1") + ) + assert service.provider_session_id(first.id, "group", "chat-1") != ( + service.provider_session_id(second.id, "group", "chat-1") + ) + + conversation = db.execute( + select(AIConversation).where(AIConversation.owner_id == first.id) + ).scalar_one() + db.execute( + update(AIConversationMessage) + .where(AIConversationMessage.conversation_id == conversation.id) + .values(created_at=utc_now() - timedelta(days=31)) + ) + db.commit() + assert service.history(first.id, "private", "chat-1") == [] + finally: + engine.dispose() + + +def test_context_order_and_erasure_core_hooks() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + owner = _user(db, "context-owner") + other = _user(db, "context-other") + memory = AIMemoryService(db) + memory.create_rule( + content="公司规则优先", + scope="global", + subject="company", + priority=100, + tags=[], + actor="service", + ) + memory.create_rule( + content="个人规则", + scope="global", + subject="profile", + priority=80, + tags=[], + actor="owner", + owner_id=owner.id, + ) + PreferenceService(db).upsert(owner.id, "tone", "简洁") + PreferenceService(db).upsert(owner.id, "interest", "人工智能") + MarketService(db).add_watchlist("owner-open", "600000", owner_id=owner.id) + ConversationService(db).record_turn( + owner.id, + "private", + "chat-context", + user_content="上一问", + assistant_content="上一答", + provider_name="direct_llm", + ) + + context = PersonalizationContextService(db).build( + owner_id=owner.id, + request="当前请求", + system_constraints="只读安全规则", + chat_type="private", + chat_key="chat-context", + subject="profile", + actor="owner", + ) + assert [str(key) for key in context.as_ordered_dict()] == [ + PersonalizationContextKey.SYSTEM_CONSTRAINTS, + PersonalizationContextKey.COMPANY_RULES, + PersonalizationContextKey.PERSONAL_RULES, + PersonalizationContextKey.CURRENT_REQUEST, + PersonalizationContextKey.PREFERENCES, + PersonalizationContextKey.INTERESTS, + PersonalizationContextKey.PERSONAL_MEMORY, + PersonalizationContextKey.CONVERSATION_HISTORY, + ] + assert context.company_rules[0]["rule"] == "公司规则优先" + assert context.personal_rules[0]["rule"] == "个人规则" + assert {item["value"] for item in context.interests} == { + "人工智能", + "600000.SH", + } + assert context.conversation_history[-1]["content"] == "上一答" + + confirmation = PersonalDataErasureService(db).request_confirmation(owner.id) + with pytest.raises(HTTPException): + PersonalDataErasureService(db).confirm_and_erase(owner.id, "BAD-CODE") + + seen: dict[str, str | int] = {} + + def integration_hook( + _db: Session, + owner_id: int, + anonymous_id: str, + ) -> dict[str, int]: + seen.update(owner_id=owner_id, anonymous_id=anonymous_id) + return {"subscriptions": 0} + + erased = PersonalDataErasureService(db).confirm_and_erase( + owner.id, + confirmation.confirmation_code, + extra_hooks=(integration_hook,), + ) + assert erased.deleted["preferences"] == 2 + assert erased.deleted["conversations"] == 1 + assert erased.deleted["ai_memory"] == 1 + assert erased.deleted["watchlist"] == 1 + assert seen["owner_id"] == owner.id + assert seen["anonymous_id"] == erased.anonymous_id + assert db.scalar( + select(UserPreference).where(UserPreference.owner_id == owner.id) + ) is None + assert db.scalar( + select(AIMemoryEntry).where(AIMemoryEntry.owner_id == owner.id) + ) is None + assert db.scalar( + select(MarketWatchlist).where(MarketWatchlist.owner_id == owner.id) + ) is None + assert db.scalar( + select(PersonalDataErasureRequest).where( + PersonalDataErasureRequest.owner_id == owner.id + ) + ) is None + assert db.get(FeishuUser, other.id) is not None + assert memory.active_rules()[0]["rule"] == "公司规则优先" + finally: + engine.dispose() diff --git a/tests/test_personalization_cleanup.py b/tests/test_personalization_cleanup.py new file mode 100644 index 0000000..3bb479c --- /dev/null +++ b/tests/test_personalization_cleanup.py @@ -0,0 +1,167 @@ +from collections.abc import Iterator +from datetime import timedelta + +import pytest +from fastapi import FastAPI +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.application.scheduling import create_scheduler +from app.core.config import get_settings +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.feishu_users.models import FeishuUser +from app.modules.personalization.models import AIConversation, AIConversationMessage +from app.modules.personalization.services import ConversationService + + +@pytest.fixture(autouse=True) +def _reset_settings() -> Iterator[None]: + get_settings.cache_clear() + yield + get_settings.cache_clear() + + +@pytest.fixture +def session_factory() -> Iterator[sessionmaker[Session]]: + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + engine.dispose() + + +def test_global_cleanup_removes_inactive_user_history( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + expired = _conversation(db, "expired") + mixed = _conversation(db, "mixed") + fresh = _conversation(db, "fresh") + old_time = utc_now() - timedelta(days=31) + fresh_time = utc_now() - timedelta(days=29) + db.add_all( + [ + AIConversationMessage( + conversation_id=expired.id, + role="user", + content="expired question", + created_at=old_time, + ), + AIConversationMessage( + conversation_id=expired.id, + role="assistant", + content="expired answer", + created_at=old_time, + ), + AIConversationMessage( + conversation_id=mixed.id, + role="user", + content="old mixed question", + created_at=old_time, + ), + AIConversationMessage( + conversation_id=mixed.id, + role="assistant", + content="fresh mixed answer", + created_at=fresh_time, + ), + AIConversationMessage( + conversation_id=fresh.id, + role="user", + content="fresh question", + created_at=fresh_time, + ), + ] + ) + db.commit() + + deleted = ConversationService(db).cleanup_expired_globally() + + assert deleted == { + "conversation_messages": 3, + "conversations": 1, + } + assert db.get(AIConversation, expired.id) is None + assert db.get(AIConversation, mixed.id) is not None + assert db.get(AIConversation, fresh.id) is not None + assert db.scalar( + select(func.count()).select_from(AIConversationMessage) + ) == 2 + + with session_factory() as verification_db: + assert verification_db.get(AIConversation, expired.id) is None + + +@pytest.mark.parametrize( + ("features_enabled", "job_expected"), + [(False, False), (True, True)], +) +def test_scheduler_wires_one_global_retention_job( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, + features_enabled: bool, + job_expected: bool, +) -> None: + monkeypatch.setenv( + "FEISHU_USER_FEATURES_ENABLED", + str(features_enabled).lower(), + ) + monkeypatch.setattr("app.core.database.SessionLocal", session_factory) + get_settings.cache_clear() + app = FastAPI() + scheduler = create_scheduler(app) + + job = scheduler.get_job("personalization_retention_cleanup") + + assert (job is not None) is job_expected + if job is None: + return + assert job.trigger.interval == timedelta(minutes=1) + + with session_factory() as db: + expired = _conversation(db, "scheduled") + db.add( + AIConversationMessage( + conversation_id=expired.id, + role="user", + content="expired scheduled message", + created_at=utc_now() - timedelta(days=31), + ) + ) + db.commit() + expired_id = expired.id + + job.func() + + assert app.state.last_personalization_retention_cleanup == { + "conversation_messages": 1, + "conversations": 1, + } + with session_factory() as db: + assert db.get(AIConversation, expired_id) is None + + +def _conversation(db: Session, suffix: str) -> AIConversation: + owner = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + ) + db.add(owner) + db.flush() + conversation = AIConversation( + owner_id=owner.id, + chat_type="private", + chat_key=f"chat-{suffix}", + ) + db.add(conversation) + db.flush() + return conversation diff --git a/tests/test_personalized_ai.py b/tests/test_personalized_ai.py new file mode 100644 index 0000000..a66e9f3 --- /dev/null +++ b/tests/test_personalized_ai.py @@ -0,0 +1,361 @@ +from typing import Any + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session, sessionmaker + +from app.core.config import get_settings +from app.core.database import Base +from app.modules.ai_agent.adapters.common import _ordered_context +from app.modules.ai_agent.constants import ( + PREFERENCE_EXTRACTION_INSTRUCTIONS, + AIContextKey, + AIExecutionMode, + AIResponseKey, +) +from app.modules.ai_agent.service import AIService +from app.modules.ai_memory.constants import AIMemoryKind +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.ai_memory.service import AIMemoryService +from app.modules.business.models import MarketWatchlist +from app.modules.feishu_users.models import FeishuUser +from app.modules.market.service import MarketService +from app.modules.personalization.models import ( + AIConversation, + AIConversationMessage, + UserPreference, +) +from app.modules.personalization.services import ConversationService, PreferenceService + + +class CapturingAdapter: + provider_name = "direct_llm" + + def __init__(self) -> None: + self.calls: list[tuple[str, dict[str, Any]]] = [] + + def ask( + self, + prompt: str, + context: dict[str, Any] | None = None, + ) -> dict[str, Any]: + self.calls.append((prompt, dict(context or {}))) + if prompt == PREFERENCE_EXTRACTION_INSTRUCTIONS: + return { + AIResponseKey.ANSWER: ( + '{"preferences":[{"category":"language","value":"中文"}]}' + ), + AIResponseKey.RAW: {}, + } + return { + AIResponseKey.ANSWER: "personalized answer", + AIResponseKey.RAW: {"request": len(self.calls)}, + } + + +class FailingNoopAdapter: + provider_name = "noop" + + def ask( + self, + prompt: str, + context: dict[str, Any] | None = None, + ) -> dict[str, Any]: + raise AssertionError("noop must short-circuit before adapter.ask") + + +def _session_factory(): + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + return engine, sessionmaker(bind=engine, expire_on_commit=False) + + +def _user(db: Session, suffix: str) -> FeishuUser: + record = FeishuUser( + code=f"USR-AI-{suffix}", + tenant_key=f"tenant-ai-{suffix}", + open_id=f"open-ai-{suffix}", + ) + db.add(record) + db.commit() + db.refresh(record) + return record + + +def _rule( + db: Session, + content: str, + *, + owner_id: int | None = None, +) -> dict[str, Any]: + return AIMemoryService(db).create_rule( + content=content, + scope="global", + subject="profile", + priority=80, + tags=[], + actor="pytest", + owner_id=owner_id, + ) + + +def test_personalized_ai_uses_ordered_owner_context_and_persists_after_success( + monkeypatch, +) -> None: + monkeypatch.setenv("AI_MEMORY_ENABLED", "true") + monkeypatch.setenv("AI_MEMORY_AUTO_WRITE_ENABLED", "true") + get_settings.cache_clear() + engine, factory = _session_factory() + adapter = CapturingAdapter() + monkeypatch.setattr("app.modules.ai_agent.service.get_adapter", lambda: adapter) + try: + with factory() as db: + owner = _user(db, "owner") + other = _user(db, "other") + _rule(db, "company rule") + _rule(db, "owner personal rule", owner_id=owner.id) + _rule(db, "other personal rule", owner_id=other.id) + PreferenceService(db).upsert(owner.id, "tone", "简洁") + PreferenceService(db).upsert(owner.id, "interest", "风险管理") + MarketService(db).add_watchlist("owner", "600000", owner_id=owner.id) + AIMemoryService(db).auto_write( + prompt="risk preference", + context={"scope": "user", "subject": f"owner:{owner.id}"}, + answer="remember owner risk preference", + owner_id=owner.id, + actor="owner", + ) + ConversationService(db).record_turn( + owner.id, + "private", + "chat-personal", + user_content="previous question", + assistant_content="previous answer", + provider_name="direct_llm", + ) + + response = AIService(db).ask_personalized( + owner.id, + "private", + "chat-personal", + "risk preference 以后请用中文", + actor="owner", + ) + + assert response[AIResponseKey.OK] is True + assert response[AIResponseKey.ANSWER] == "personalized answer" + assert len(adapter.calls) == 2 + prompt, context = adapter.calls[0] + assert prompt == "risk preference 以后请用中文" + assert context[AIContextKey.COMPANY_RULES][0]["rule"] == "company rule" + assert context[AIContextKey.PERSONAL_RULES][0]["rule"] == ( + "owner personal rule" + ) + assert "other personal rule" not in str(context) + assert context[AIContextKey.PREFERENCES][0]["value"] == "简洁" + assert {item["value"] for item in context[AIContextKey.INTERESTS]} == { + "风险管理", + "600000.SH", + } + assert context[AIContextKey.LOCAL_MEMORY] + assert len(context[AIContextKey.CONVERSATION_HISTORY]) == 2 + assert context[AIContextKey.PROVIDER_SESSION_ID] == ( + ConversationService.provider_session_id( + owner.id, + "private", + "chat-personal", + ) + ) + assert context[AIContextKey.ALLOW_PROVIDER_MEMORY] is False + assert context[AIContextKey.EXECUTION_MODE] == AIExecutionMode.PERSONALIZED + assert AIContextKey.OPENCLAW_TOOL not in context + + serialized = _ordered_context(prompt, context) + headings = [ + "公司规则:", + "个人规则:", + "当前请求:", + "个人偏好与兴趣:", + "个人相关记忆:", + "当前会话历史:", + ] + positions = [serialized.index(heading) for heading in headings] + assert positions == sorted(positions) + + extraction_context = adapter.calls[1][1] + assert extraction_context[AIContextKey.ALLOW_PROVIDER_MEMORY] is False + assert extraction_context[AIContextKey.EXECUTION_MODE] == ( + AIExecutionMode.PREFERENCE_EXTRACTION + ) + assert db.scalar( + select(func.count()) + .select_from(AIConversationMessage) + .join( + AIConversation, + AIConversation.id == AIConversationMessage.conversation_id, + ) + .where(AIConversation.owner_id == owner.id) + ) == 4 + owner_memory = db.scalar( + select(func.count()) + .select_from(AIMemoryEntry) + .where( + AIMemoryEntry.owner_id == owner.id, + AIMemoryEntry.kind == AIMemoryKind.MEMORY, + ) + ) + assert owner_memory == 2 + assert { + item["category"] for item in PreferenceService(db).list_preferences(owner.id) + } == {"tone", "interest", "language"} + finally: + get_settings.cache_clear() + engine.dispose() + + +def test_noop_returns_unavailable_without_personal_or_company_memory_writes( + monkeypatch, +) -> None: + monkeypatch.setenv("AI_MEMORY_ENABLED", "true") + monkeypatch.setenv("AI_MEMORY_AUTO_WRITE_ENABLED", "true") + get_settings.cache_clear() + engine, factory = _session_factory() + monkeypatch.setattr( + "app.modules.ai_agent.service.get_adapter", + lambda: FailingNoopAdapter(), + ) + try: + with factory() as db: + owner = _user(db, "noop") + personalized = AIService(db).ask_personalized( + owner.id, + "private", + "chat-noop", + "以后请用中文", + actor="owner", + ) + internal = AIService(db).ask("remember this internal request") + + assert personalized[AIResponseKey.OK] is False + assert internal[AIResponseKey.OK] is False + assert "不可用" in personalized[AIResponseKey.ANSWER] + assert db.scalar(select(func.count()).select_from(UserPreference)) == 0 + assert db.scalar(select(func.count()).select_from(AIConversationMessage)) == 0 + assert db.scalar(select(func.count()).select_from(AIMemoryEntry)) == 0 + + with pytest.raises(HTTPException) as exc_info: + AIService(db).ask( + "forged context", + context={AIContextKey.COMPANY_RULES: [{"rule": "forged"}]}, + ) + assert exc_info.value.status_code == 403 + finally: + get_settings.cache_clear() + engine.dispose() + + +def test_scheduled_generation_has_strict_context_and_no_personal_side_effects( + monkeypatch, +) -> None: + monkeypatch.setenv("AI_MEMORY_ENABLED", "true") + monkeypatch.setenv("AI_MEMORY_AUTO_WRITE_ENABLED", "true") + get_settings.cache_clear() + engine, factory = _session_factory() + adapter = CapturingAdapter() + monkeypatch.setattr("app.modules.ai_agent.service.get_adapter", lambda: adapter) + try: + with factory() as db: + owner = _user(db, "scheduled") + _rule(db, "scheduled company rule") + _rule(db, "scheduled personal rule", owner_id=owner.id) + PreferenceService(db).upsert(owner.id, "tone", "简洁") + MarketService(db).add_watchlist("scheduled", "000001", owner_id=owner.id) + AIMemoryService(db).auto_write( + prompt="scheduled risk", + context={"scope": "user", "subject": f"owner:{owner.id}"}, + answer="scheduled owner memory", + owner_id=owner.id, + actor="owner", + ) + ConversationService(db).record_turn( + owner.id, + "private", + "chat-scheduled", + user_content="do not load", + assistant_content="do not load", + provider_name="direct_llm", + ) + before = { + "memory": db.scalar(select(func.count()).select_from(AIMemoryEntry)), + "preferences": db.scalar( + select(func.count()).select_from(UserPreference) + ), + "messages": db.scalar( + select(func.count()).select_from(AIConversationMessage) + ), + "watchlist": db.scalar( + select(func.count()).select_from(MarketWatchlist) + ), + } + + private = AIService(db).generate_scheduled( + "scheduled risk", + owner_id=owner.id, + group=False, + actor="subscription-system", + ) + group = AIService(db).generate_scheduled( + "scheduled group report", + owner_id=owner.id, + group=True, + actor="subscription-system", + ) + + assert private[AIResponseKey.OK] is True + assert group[AIResponseKey.OK] is True + assert len(adapter.calls) == 2 + private_context = adapter.calls[0][1] + assert private_context[AIContextKey.COMPANY_RULES] == [] + assert private_context[AIContextKey.PERSONAL_RULES][0]["rule"] == ( + "scheduled personal rule" + ) + assert private_context[AIContextKey.PREFERENCES] + assert private_context[AIContextKey.INTERESTS] + assert private_context[AIContextKey.LOCAL_MEMORY] + assert private_context[AIContextKey.CONVERSATION_HISTORY] == [] + assert AIContextKey.PROVIDER_SESSION_ID not in private_context + assert private_context[AIContextKey.ALLOW_PROVIDER_MEMORY] is False + assert private_context[AIContextKey.EXECUTION_MODE] == ( + AIExecutionMode.SCHEDULED_PRIVATE + ) + + group_context = adapter.calls[1][1] + assert group_context[AIContextKey.COMPANY_RULES][0]["rule"] == ( + "scheduled company rule" + ) + assert group_context[AIContextKey.PERSONAL_RULES] == [] + assert group_context[AIContextKey.PREFERENCES] == [] + assert group_context[AIContextKey.INTERESTS] == [] + assert group_context[AIContextKey.LOCAL_MEMORY] == [] + assert group_context[AIContextKey.CONVERSATION_HISTORY] == [] + assert group_context[AIContextKey.ALLOW_PROVIDER_MEMORY] is False + assert group_context[AIContextKey.EXECUTION_MODE] == ( + AIExecutionMode.SCHEDULED_GROUP + ) + after = { + "memory": db.scalar(select(func.count()).select_from(AIMemoryEntry)), + "preferences": db.scalar( + select(func.count()).select_from(UserPreference) + ), + "messages": db.scalar( + select(func.count()).select_from(AIConversationMessage) + ), + "watchlist": db.scalar( + select(func.count()).select_from(MarketWatchlist) + ), + } + assert after == before + finally: + get_settings.cache_clear() + engine.dispose() diff --git a/tests/test_smoke.py b/tests/test_smoke.py index 4e8df19..62b898a 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -1,8 +1,5 @@ import json -import os -import tempfile from datetime import date, datetime, timedelta -from pathlib import Path import pytest from fastapi import HTTPException @@ -11,22 +8,6 @@ from sqlalchemy import select from app.modules.ai_agent.constants import AIProviderName, AIResponseKey from app.modules.business.constants import BusinessResponseKey, StatusValue -_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db") -_db.close() - -os.environ["DATABASE_URL"] = "sqlite:///" + _db.name.replace("\\", "/") -os.environ["API_KEY"] = "test-key" -os.environ["AUDIT_API_KEY"] = "audit-key" -os.environ["AUDIT_API_ACTOR"] = "audit-manager" -os.environ["FEISHU_APP_ID"] = "" -os.environ["FEISHU_APP_SECRET"] = "" -os.environ["FEISHU_VERIFICATION_TOKEN"] = "test-feishu-token" -os.environ["LEGACY_ALLOWED_QUERIES"] = "{}" -os.environ["LEGACY_DATABASE_URL"] = "" -os.environ["LEGACY_PROJECT_QUERY"] = "" -os.environ["MODEL_PROVIDER"] = AIProviderName.NOOP -os.environ["SCHEDULER_ENABLED"] = "false" - from fastapi.testclient import TestClient from app.core.config import Settings, get_settings @@ -153,13 +134,6 @@ def create_business_record(domain: str, data: dict, actor: str = "pytest") -> Se db.close() -def teardown_module() -> None: - engine.dispose() - path = Path(_db.name) - if path.exists(): - path.unlink() - - def test_project_report_and_feishu_command_preview() -> None: response = create_business_record( "projects", @@ -449,7 +423,7 @@ def test_v3_event_retry_and_dispatch_pending_route() -> None: assert dispatched[0]["status"] == EventStatus.PROCESSED -def test_v3_ai_memory_recall_and_auto_write() -> None: +def test_v3_ai_noop_does_not_auto_write_memory() -> None: response = client.post( "/api/v1/ai/ask", headers=headers, @@ -464,45 +438,33 @@ def test_v3_ai_memory_recall_and_auto_write() -> None: assert response.status_code == 200 data = response.json() assert data[AIResponseKey.PROVIDER] == AIProviderName.NOOP - assert data[AIResponseKey.RAW][AIResponseKey.MEMORY_WRITE][AIMemoryPayloadKey.STATUS] == ( - AIMemoryStatus.ACTIVE - ) + assert data[AIResponseKey.OK] is False + assert AIResponseKey.MEMORY_WRITE not in data[AIResponseKey.RAW] list_response = client.get( "/api/v1/ai/memory?scope=project&subject=P-MEM-SMOKE", headers=headers, ) assert list_response.status_code == 200 - assert list_response.json()[AIMemoryResponseKey.ITEMS] - - recall_response = client.post( - "/api/v1/ai/memory/recall", - headers=headers, - json={ - "query": "concise bullet summaries", - "scope": "project", - "subject": "P-MEM-SMOKE", - }, - ) - assert recall_response.status_code == 200 - assert recall_response.json()[AIMemoryResponseKey.ITEMS] + assert list_response.json()[AIMemoryResponseKey.ITEMS] == [] def test_ai_memory_rejects_financial_facts_and_applies_retention() -> None: - response = client.post( - "/api/v1/ai/ask", - headers=headers, - json={ - "prompt": "Remember the project cash flow and budget details for next quarter", - "context": { + db = SessionLocal() + try: + memory = AIMemoryService(db).auto_write( + prompt="Remember the project cash flow and budget details for next quarter", + context={ AIMemoryPayloadKey.SCOPE: "project", AIMemoryPayloadKey.SUBJECT: "P-MEM-FINANCIAL", }, - }, - ) - assert response.status_code == 200 - memory_write = response.json()[AIResponseKey.RAW][AIResponseKey.MEMORY_WRITE] - assert memory_write[AIMemoryPayloadKey.STATUS] == AIMemoryStatus.REJECTED + answer="Retain the confidential cash flow and budget details.", + actor="pytest", + ) + assert memory is not None + assert memory.status == AIMemoryStatus.REJECTED + finally: + db.close() rejected = client.get( "/api/v1/ai/memory", @@ -531,7 +493,9 @@ def test_default_json_response_masks_sensitive_fields() -> None: assert payload["nested"]["status"] == "ok" -def test_v3_risk_action_routes_are_disabled_in_read_only_mode() -> None: +def test_v3_risk_action_routes_write_local_state_in_read_only_mode(monkeypatch) -> None: + monkeypatch.setenv("READ_ONLY_MODE", "true") + get_settings.cache_clear() response = create_business_record( "risk-events", { @@ -549,10 +513,14 @@ def test_v3_risk_action_routes_are_disabled_in_read_only_mode() -> None: headers=headers, json={"assigned_to": "risk-owner", "comment": "route to owner"}, ) - assert response.status_code == 405 + assert response.status_code == 200 + assert response.json()["risk_event"]["assigned_to"] == "risk-owner" -def test_writeback_and_approval_routes_are_removed() -> None: +@pytest.mark.parametrize("read_only_mode", ["true", "false"]) +def test_writeback_and_approval_routes_are_removed(monkeypatch, read_only_mode: str) -> None: + monkeypatch.setenv("READ_ONLY_MODE", read_only_mode) + get_settings.cache_clear() response = client.post( "/api/v1/writebacks", headers=headers, @@ -577,6 +545,7 @@ def test_writeback_and_approval_routes_are_removed() -> None: }, ) assert response.status_code == 404 + get_settings.cache_clear() def test_feishu_webhook_challenge_uses_event_service_verification() -> None: @@ -687,6 +656,7 @@ def test_dashboard_and_response_masking() -> None: dashboard_response = client.get("/api/v1/dashboard/summary", headers=headers) assert dashboard_response.status_code == 200 assert "metrics" in dashboard_response.json() + assert "latest_audit_logs" not in dashboard_response.json() def test_configured_domain_response_masking(monkeypatch) -> None: @@ -714,7 +684,10 @@ def test_configured_domain_response_masking(monkeypatch) -> None: get_settings.cache_clear() -def test_business_write_routes_are_disabled_in_read_only_mode() -> None: +@pytest.mark.parametrize("read_only_mode", ["true", "false"]) +def test_business_write_routes_are_permanently_disabled(monkeypatch, read_only_mode: str) -> None: + monkeypatch.setenv("READ_ONLY_MODE", read_only_mode) + get_settings.cache_clear() create_response = client.post( "/api/v1/business/projects", headers=headers, @@ -737,9 +710,15 @@ def test_business_write_routes_are_disabled_in_read_only_mode() -> None: }, ) assert update_response.status_code == 405 + get_settings.cache_clear() -def test_approval_and_feishu_approval_card_routes_are_removed() -> None: +@pytest.mark.parametrize("read_only_mode", ["true", "false"]) +def test_approval_and_feishu_approval_card_routes_are_removed( + monkeypatch, read_only_mode: str +) -> None: + monkeypatch.setenv("READ_ONLY_MODE", read_only_mode) + get_settings.cache_clear() approval_response = client.post( "/api/v1/approvals", headers=headers, @@ -768,9 +747,30 @@ def test_approval_and_feishu_approval_card_routes_are_removed() -> None: }, ) assert callback_response.status_code == 404 + openclaw_response = client.post( + "/api/v1/ai/openclaw/tools/invoke", + headers=headers, + json={ + "tool": "sessions_list", + "action": "json", + "args": {}, + "session_key": "main", + }, + ) + assert openclaw_response.status_code == 404 + assert "/api/v1/approvals" not in app.openapi()["paths"] + assert "/api/v1/integrations/feishu/approval-card-action" not in app.openapi()["paths"] + assert "/api/v1/ai/openclaw/tools/invoke" not in app.openapi()["paths"] + get_settings.cache_clear() -def test_new_ledgers_reports_and_risk_events() -> None: +def test_new_ledgers_reports_and_local_risk_state(monkeypatch) -> None: + monkeypatch.setenv("READ_ONLY_MODE", "true") + get_settings.cache_clear() + monkeypatch.setattr( + "app.modules.risk.routes.enqueue_risk_event_generation", + lambda actor: {"queued": True, "actor": actor}, + ) domains_response = client.get("/api/v1/business/domains", headers=headers) assert domains_response.status_code == 200 domains = domains_response.json()["domains"] @@ -810,20 +810,26 @@ def test_new_ledgers_reports_and_risk_events() -> None: report_response = client.post( "/api/v1/reports/work-reports/generate", headers=headers, - json={"report_type": ReportType.DAILY, "reporter": "pytest", "actor": "pytest"}, + json={ + "report_type": ReportType.DAILY, + "reporter": "pytest", + "actor": "pytest", + "persist": True, + }, ) assert report_response.status_code == 200 - assert report_response.json()["data"] is None + assert report_response.json()["data"]["reporter"] == "pytest" assert report_response.json()["report"]["report_type"] == ReportType.DAILY risk_response = client.post( "/api/v1/risks/events/generate?actor=pytest", headers=headers, ) - assert risk_response.status_code == 405 + assert risk_response.status_code == 200 enqueue_response = client.post("/api/v1/risks/events/enqueue", headers=headers) - assert enqueue_response.status_code == 405 + assert enqueue_response.status_code == 200 + assert enqueue_response.json()["queued"] is True overdue_response = client.get("/api/v1/risks/overdue-tasks", headers=headers) assert overdue_response.status_code == 200 @@ -1004,8 +1010,9 @@ def test_project_lifecycle_report_summarizes_progress_cost_and_risk() -> None: ) assert ai_response.status_code == 200 ai_data = ai_response.json() - assert ai_data[LifecycleResponseKey.AI_ANALYSIS][AIResponseKey.OK] is True + assert ai_data[LifecycleResponseKey.AI_ANALYSIS][AIResponseKey.OK] is False assert ai_data[LifecycleResponseKey.AI_ANALYSIS][AIResponseKey.PROVIDER] == AIProviderName.NOOP + assert "不可用" in ai_data[LifecycleResponseKey.AI_ANALYSIS][AIResponseKey.ANSWER] def test_v3_enterprise_analytics_returns_read_only_sections() -> None: @@ -1101,7 +1108,9 @@ def test_work_report_counts_pending_approval_backlog_outside_period() -> None: assert metrics["expenses_pending"] == 1 -def test_legacy_task_read_query_allowed_but_sync_disabled(monkeypatch) -> None: +def test_legacy_read_query_and_local_import_allowed_in_read_only_mode(monkeypatch) -> None: + monkeypatch.setenv("READ_ONLY_MODE", "true") + get_settings.cache_clear() rows = [ { "id": 9001, @@ -1138,10 +1147,15 @@ def test_legacy_task_read_query_allowed_but_sync_disabled(monkeypatch) -> None: "field_map": {"title": "task_name"}, }, ) - assert response.status_code == 405 + assert response.status_code == 200 + assert response.json()["dry_run"] is False + assert response.json()["items"][0]["task"]["source_system"] == "legacy_mysql" + get_settings.cache_clear() -def test_risk_event_action_routes_are_disabled_in_read_only_mode() -> None: +def test_risk_event_actions_write_local_state_in_read_only_mode(monkeypatch) -> None: + monkeypatch.setenv("READ_ONLY_MODE", "true") + get_settings.cache_clear() create_response = create_business_record( "risk-events", { @@ -1162,21 +1176,21 @@ def test_risk_event_action_routes_are_disabled_in_read_only_mode() -> None: headers=headers, json={"assigned_to": "risk-owner", "comment": "please handle"}, ) - assert assign_response.status_code == 405 + assert assign_response.status_code == 200 comment_response = client.post( f"/api/v1/risks/events/{event_id}/comment", headers=headers, json={"comment": "working on it", "payload": {"step": 1}}, ) - assert comment_response.status_code == 405 + assert comment_response.status_code == 200 resolve_response = client.post( f"/api/v1/risks/events/{event_id}/resolve", headers=headers, json={"comment": "resolved"}, ) - assert resolve_response.status_code == 405 + assert resolve_response.status_code == 200 close_response = client.post( f"/api/v1/risks/events/{event_id}/close", @@ -1186,21 +1200,22 @@ def test_risk_event_action_routes_are_disabled_in_read_only_mode() -> None: "review_summary": "handled", }, ) - assert close_response.status_code == 405 + assert close_response.status_code == 200 reopen_response = client.post( f"/api/v1/risks/events/{event_id}/reopen", headers=headers, json={"comment": "recheck"}, ) - assert reopen_response.status_code == 405 + assert reopen_response.status_code == 200 actions_response = client.get( f"/api/v1/risks/events/{event_id}/actions", headers=headers, ) assert actions_response.status_code == 200 - assert actions_response.json()["items"] == [] + assert len(actions_response.json()["items"]) == 5 + get_settings.cache_clear() def test_report_push_failure_is_recorded() -> None: @@ -1278,7 +1293,8 @@ def test_legacy_readonly_query_clamps_param_limit(monkeypatch) -> None: def mappings(self) -> "FakeResult": return self - def all(self) -> list: + def fetchmany(self, size: int) -> list: + _ = size return [] class FakeConnection: @@ -1486,7 +1502,7 @@ def test_personnel_lifecycle_does_not_treat_missing_ding_mapping_as_absence() -> def test_lifecycle_pipeline_is_idempotent(monkeypatch) -> None: - monkeypatch.setenv("READ_ONLY_MODE", "false") + monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() monkeypatch.setattr( IntasectSyncService, @@ -1524,16 +1540,21 @@ def test_lifecycle_pipeline_is_idempotent(monkeypatch) -> None: get_settings.cache_clear() -def test_lifecycle_enqueue_respects_read_only_guard(monkeypatch) -> None: +def test_lifecycle_enqueue_allows_local_work_in_read_only_mode(monkeypatch) -> None: monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() + monkeypatch.setattr( + "app.modules.reports.routes.enqueue_lifecycle_report", + lambda **kwargs: {"status": "queued", "report_type": kwargs["report_type"]}, + ) try: response = client.post( "/api/v1/reports/lifecycle/enqueue", headers=headers, json={"report_type": "daily"}, ) - assert response.status_code == 405 + assert response.status_code == 200 + assert response.json()["status"] == "queued" finally: monkeypatch.delenv("READ_ONLY_MODE", raising=False) get_settings.cache_clear() @@ -1685,7 +1706,7 @@ def test_lifecycle_chart_is_uploaded_and_embedded_in_feishu_card(monkeypatch) -> def test_user_rule_api_creates_and_disables_rule(monkeypatch) -> None: - monkeypatch.setenv("READ_ONLY_MODE", "false") + monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() try: created = client.post( @@ -2260,7 +2281,7 @@ def test_market_closed_day_skips_quote_collection() -> None: def test_market_scheduler_registers_close_and_weekly_jobs(monkeypatch) -> None: monkeypatch.setenv("MARKET_ANALYSIS_ENABLED", "true") - monkeypatch.setenv("READ_ONLY_MODE", "false") + monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() try: scheduler = create_scheduler() @@ -2399,7 +2420,7 @@ def test_market_pipeline_is_idempotent_and_requires_complete_ai_chart_delivery( monkeypatch.setenv("FEISHU_APP_ID", "test-app") monkeypatch.setenv("FEISHU_APP_SECRET", "test-secret") monkeypatch.setenv("FEISHU_DEFAULT_CHAT_ID", "oc_market") - monkeypatch.setenv("READ_ONLY_MODE", "false") + monkeypatch.setenv("READ_ONLY_MODE", "true") get_settings.cache_clear() monkeypatch.setattr( "app.modules.feishu.service.FeishuService.upload_image", @@ -2518,31 +2539,25 @@ def test_feishu_market_rules_are_scoped_to_market(monkeypatch) -> None: get_settings.cache_clear() -def test_read_only_mode_blocks_feishu_mutations_and_market_pipeline(monkeypatch) -> None: - class MustNotRunMarket: - def sync_daily(self, target): - pytest.fail("read-only mode must block market synchronization") - +def test_read_only_mode_allows_local_rules_and_watchlist(monkeypatch) -> None: monkeypatch.setenv("READ_ONLY_MODE", "true") monkeypatch.setenv("MARKET_ANALYSIS_ENABLED", "true") get_settings.cache_clear() db = SessionLocal() try: rules = FeishuCommandService(db).handle_text( - "学习市场规则：这条规则不应写入", actor="ou_read_only", auto_reply=False + "学习市场规则：这条规则应写入本地状态", + actor="ou_read_only", + auto_reply=False, ) watchlist = FeishuCommandService(db).handle_text( "加入自选 600105", actor="ou_read_only", auto_reply=False ) - pipeline = MarketPipelineService(db, MustNotRunMarket()).run( - "close", date(2032, 7, 12), actor="pytest" - ) - assert "只读模式" in rules["content"] - assert "只读模式" in watchlist["content"] - assert pipeline["status"] == "operations_disabled" + assert "规则已学习" in rules["content"] + assert "已加入自选：600105.SH" in watchlist["content"] assert db.execute( - select(AIMemoryEntry).where(AIMemoryEntry.content == "这条规则不应写入") - ).scalar_one_or_none() is None + select(AIMemoryEntry).where(AIMemoryEntry.content == "这条规则应写入本地状态") + ).scalar_one_or_none() is not None finally: db.close() monkeypatch.delenv("READ_ONLY_MODE", raising=False) diff --git a/tests/test_subscription_delivery.py b/tests/test_subscription_delivery.py new file mode 100644 index 0000000..35c6b34 --- /dev/null +++ b/tests/test_subscription_delivery.py @@ -0,0 +1,375 @@ +from datetime import datetime, timedelta +from typing import Any +from uuid import uuid4 + +from sqlalchemy import create_engine +from sqlalchemy.orm import Session + +from app.application.delivery.subscriptions import FeishuSubscriptionSender +from app.core.database import Base +from app.modules.feishu.errors import FeishuAPIError +from app.modules.feishu_users.constants import FeishuUserRole +from app.modules.feishu_users.models import FeishuUser +from app.modules.subscriptions.constants import ( + MAX_DAILY_DELIVERIES, + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services import ( + DeliveryGenerationRequest, + DeliverySendRequest, + DeliveryService, +) + + +class StubGenerator: + def __init__(self, content: str = "生成后的提醒") -> None: + self.content = content + self.requests: list[DeliveryGenerationRequest] = [] + + def generate(self, request: DeliveryGenerationRequest) -> str: + self.requests.append(request) + return self.content + + +class StubSender: + def __init__(self, outcomes: list[dict[str, Any] | Exception]) -> None: + self.outcomes = outcomes + self.requests: list[DeliverySendRequest] = [] + + def send(self, request: DeliverySendRequest) -> dict[str, Any]: + self.requests.append(request) + outcome = self.outcomes.pop(0) + if isinstance(outcome, Exception): + raise outcome + return outcome + + +class StubFeishuService: + def __init__(self) -> None: + self.calls: list[dict[str, Any]] = [] + + def send_text(self, text: str, **kwargs: Any) -> dict[str, Any]: + self.calls.append({"text": text, **kwargs}) + return {"code": 0} + + +def _delivery_fixture( + db: Session, + *, + suffix: str, + current: datetime, + target_type: str = SubscriptionTargetType.USER, +) -> tuple[FeishuUser, PushSubscription, PushDelivery]: + owner = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + role=( + FeishuUserRole.ADMIN + if target_type == SubscriptionTargetType.CHAT + else FeishuUserRole.USER + ), + timezone="Asia/Shanghai", + ) + db.add(owner) + db.flush() + subscription = PushSubscription( + code=f"SUB-{suffix}", + owner_id=owner.id, + target_type=target_type, + target_id=( + f"chat-{suffix}" + if target_type == SubscriptionTargetType.CHAT + else owner.open_id + ), + prompt="按我的偏好生成提醒", + schedule_type=SubscriptionScheduleType.DAILY, + schedule_config={"hour": 9, "minute": 0}, + timezone=owner.timezone, + next_run_at=current + timedelta(days=1), + status=PushSubscriptionStatus.ACTIVE, + consented_at=current - timedelta(days=1), + ) + db.add(subscription) + db.flush() + delivery = PushDelivery( + code=f"DEL-{suffix}", + subscription_id=subscription.id, + scheduled_for=current, + idempotency_key=uuid4().hex + uuid4().hex, + message_uuid=str(uuid4()), + status=PushDeliveryStatus.PENDING, + next_attempt_at=current, + created_at=current, + updated_at=current, + ) + db.add(delivery) + db.commit() + db.refresh(owner) + db.refresh(subscription) + db.refresh(delivery) + return owner, subscription, delivery + + +def test_private_delivery_uses_personal_context_open_id_and_is_idempotent() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + owner, _, delivery = _delivery_fixture( + db, + suffix="private", + current=current, + ) + generator = StubGenerator() + sender = StubSender( + [{"code": 0, "data": {"message_id": "provider-message"}}] + ) + service = DeliveryService(db, generator=generator, sender=sender) + + sent = service.process(delivery.code, now=current) + duplicate = service.process(delivery.code, now=current) + + assert sent.status == PushDeliveryStatus.SENT + assert sent.attempt_count == 1 + assert sent.provider_message_id == "provider-message" + assert duplicate.status == PushDeliveryStatus.SENT + assert len(generator.requests) == 1 + assert generator.requests[0] == DeliveryGenerationRequest( + prompt="按我的偏好生成提醒", + owner_id=owner.id, + use_personal_context=True, + use_company_rules=False, + ) + assert len(sender.requests) == 1 + assert sender.requests[0].receive_id == owner.open_id + assert sender.requests[0].receive_id_type == "open_id" + assert sender.requests[0].tenant_key == owner.tenant_key + assert sender.requests[0].uuid == delivery.message_uuid + finally: + engine.dispose() + + +def test_feishu_sender_forwards_delivery_tenant_key() -> None: + feishu = StubFeishuService() + sender = FeishuSubscriptionSender(feishu) # type: ignore[arg-type] + request = DeliverySendRequest( + receive_id="open-user", + receive_id_type="open_id", + tenant_key="tenant-a", + text="提醒内容", + uuid="stable-delivery-uuid", + ) + + result = sender.send(request) + + assert result == {"code": 0} + assert feishu.calls == [ + { + "text": "提醒内容", + "receive_id": "open-user", + "receive_id_type": "open_id", + "actor": "scheduler", + "uuid": "stable-delivery-uuid", + "tenant_key": "tenant-a", + } + ] + + +def test_group_delivery_never_exposes_creator_personal_context() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + owner, subscription, delivery = _delivery_fixture( + db, + suffix="group", + current=current, + target_type=SubscriptionTargetType.CHAT, + ) + generator = StubGenerator() + sender = StubSender([{"code": 0, "data": {"message_id": "group-message"}}]) + + result = DeliveryService( + db, + generator=generator, + sender=sender, + ).process(delivery.code, now=current) + + assert result.status == PushDeliveryStatus.SENT + assert generator.requests[0].owner_id is None + assert generator.requests[0].use_personal_context is False + assert generator.requests[0].use_company_rules is True + assert generator.requests[0].allow_tools is False + assert generator.requests[0].record_history is False + assert sender.requests[0].receive_id == subscription.target_id + assert sender.requests[0].receive_id_type == "chat_id" + assert sender.requests[0].tenant_key == owner.tenant_key + finally: + engine.dispose() + + +def test_business_failure_retries_after_one_five_and_fifteen_minutes() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + first_at = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + _, _, delivery = _delivery_fixture( + db, + suffix="retry", + current=first_at, + ) + generator = StubGenerator() + sender = StubSender([{"code": 999}] * 4) + service = DeliveryService(db, generator=generator, sender=sender) + + first = service.process(delivery.code, now=first_at) + assert first.status == PushDeliveryStatus.RETRY + assert first.attempt_count == 1 + assert first.next_attempt_at == first_at + timedelta(minutes=1) + + too_early = service.process( + delivery.code, + now=first_at + timedelta(seconds=30), + ) + assert too_early.attempt_count == 1 + assert len(sender.requests) == 1 + + second_at = first_at + timedelta(minutes=1) + second = service.process(delivery.code, now=second_at) + assert second.status == PushDeliveryStatus.RETRY + assert second.attempt_count == 2 + assert second.next_attempt_at == second_at + timedelta(minutes=5) + + third_at = second_at + timedelta(minutes=5) + third = service.process(delivery.code, now=third_at) + assert third.status == PushDeliveryStatus.RETRY + assert third.attempt_count == 3 + assert third.next_attempt_at == third_at + timedelta(minutes=15) + + fourth_at = third_at + timedelta(minutes=15) + fourth = service.process(delivery.code, now=fourth_at) + assert fourth.status == PushDeliveryStatus.FAILED + assert fourth.attempt_count == 4 + assert fourth.next_attempt_at is None + assert len(generator.requests) == 1 + assert len({request.uuid for request in sender.requests}) == 1 + finally: + engine.dispose() + + +def test_paused_subscription_is_skipped_without_generation_or_send() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + _, subscription, delivery = _delivery_fixture( + db, + suffix="paused", + current=current, + ) + subscription.status = PushSubscriptionStatus.PAUSED + db.commit() + generator = StubGenerator() + sender = StubSender([{"code": 0}]) + + result = DeliveryService( + db, + generator=generator, + sender=sender, + ).process(delivery.code, now=current) + + assert result.status == PushDeliveryStatus.SKIPPED + assert result.attempt_count == 0 + assert generator.requests == [] + assert sender.requests == [] + finally: + engine.dispose() + + +def test_non_retryable_feishu_client_error_fails_without_retry() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + _, _, delivery = _delivery_fixture( + db, + suffix="permanent-feishu", + current=current, + ) + generator = StubGenerator() + sender = StubSender( + [ + FeishuAPIError( + "invalid receive target", + retryable=False, + http_status=400, + ) + ] + ) + + result = DeliveryService( + db, + generator=generator, + sender=sender, + ).process(delivery.code, now=current) + + assert result.status == PushDeliveryStatus.FAILED + assert result.attempt_count == 1 + assert result.next_attempt_at is None + assert len(sender.requests) == 1 + finally: + engine.dispose() + + +def test_send_time_daily_limit_counts_deferred_deliveries_by_sent_at() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + _, subscription, delivery = _delivery_fixture( + db, + suffix="send-limit", + current=current, + ) + for index in range(MAX_DAILY_DELIVERIES): + db.add( + PushDelivery( + code=f"DEL-SENT-{index}", + subscription_id=subscription.id, + scheduled_for=current - timedelta(days=1, minutes=index + 1), + idempotency_key=uuid4().hex + uuid4().hex, + message_uuid=str(uuid4()), + status=PushDeliveryStatus.SENT, + attempt_count=1, + sent_at=current - timedelta(seconds=index + 1), + created_at=current - timedelta(days=1), + updated_at=current, + ) + ) + db.commit() + generator = StubGenerator() + sender = StubSender([{"code": 0}]) + + result = DeliveryService( + db, + generator=generator, + sender=sender, + ).process(delivery.code, now=current) + + assert result.status == PushDeliveryStatus.SKIPPED + assert result.attempt_count == 1 + assert "Daily delivery limit" in str(result.last_error) + assert sender.requests == [] + finally: + engine.dispose() diff --git a/tests/test_subscription_dispatch.py b/tests/test_subscription_dispatch.py new file mode 100644 index 0000000..6971ab2 --- /dev/null +++ b/tests/test_subscription_dispatch.py @@ -0,0 +1,300 @@ +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, time, timedelta +from pathlib import Path +from threading import Barrier +from uuid import uuid4 + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import Session, sessionmaker + +from app.core.database import Base +from app.modules.feishu_users.constants import FeishuUserRole +from app.modules.feishu_users.models import FeishuUser +from app.modules.feishu_users.principal import FeishuPrincipal +from app.modules.subscriptions.constants import ( + DAILY_DELIVERY_LIMIT_REACHED, + MAX_ACTIVE_SUBSCRIPTIONS, + MAX_DAILY_DELIVERIES, + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.services import ( + SubscriptionManagementService, + SubscriptionScanner, +) + + +def _user( + db: Session, + *, + suffix: str, + role: str = FeishuUserRole.USER, + quiet_start: time | None = None, + quiet_end: time | None = None, +) -> FeishuUser: + record = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + role=role, + timezone="Asia/Shanghai", + quiet_hours_start=quiet_start, + quiet_hours_end=quiet_end, + ) + db.add(record) + db.commit() + db.refresh(record) + return record + + +def _subscription( + db: Session, + owner: FeishuUser, + *, + code: str, + next_run_at: datetime, + target_type: str = SubscriptionTargetType.USER, + target_id: str | None = None, +) -> PushSubscription: + record = PushSubscription( + code=code, + owner_id=owner.id, + target_type=target_type, + target_id=target_id or owner.open_id, + prompt="给我一条简短提醒", + schedule_type=SubscriptionScheduleType.DAILY, + schedule_config={"hour": 9, "minute": 0}, + timezone=owner.timezone, + next_run_at=next_run_at, + status=PushSubscriptionStatus.ACTIVE, + consented_at=next_run_at - timedelta(days=1), + ) + db.add(record) + db.commit() + db.refresh(record) + return record + + +def test_management_binds_private_and_group_targets() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="user") + private_principal = FeishuPrincipal.from_user( + user, + chat_id="private-chat", + chat_type="p2p", + ) + private, _ = SubscriptionManagementService(db).create_private( + private_principal, + "每天 09:00", + "给我一条提醒", + now=datetime(2026, 7, 26, 0, 0), + ) + + assert private.target_type == SubscriptionTargetType.USER + assert private.target_id == user.open_id + + with pytest.raises(HTTPException) as ordinary_group: + SubscriptionManagementService(db).create_group( + FeishuPrincipal.from_user( + user, + chat_id="group-chat", + chat_type="group", + ), + "每天 10:00", + "群提醒", + now=datetime(2026, 7, 26, 0, 0), + ) + assert ordinary_group.value.status_code == 403 + + admin = _user(db, suffix="admin", role=FeishuUserRole.ADMIN) + with pytest.raises(HTTPException, match="current group"): + SubscriptionManagementService(db).create_group( + FeishuPrincipal.from_user( + admin, + chat_id="manually-supplied-chat", + chat_type="p2p", + ), + "每天 10:00", + "群提醒", + now=datetime(2026, 7, 26, 0, 0), + ) + + group, _ = SubscriptionManagementService(db).create_group( + FeishuPrincipal.from_user( + admin, + chat_id="verified-current-group", + chat_type="group", + ), + "每天 10:00", + "群提醒", + now=datetime(2026, 7, 26, 0, 0), + ) + assert group.target_type == SubscriptionTargetType.CHAT + assert group.target_id == "verified-current-group" + finally: + engine.dispose() + + +def test_management_rejects_more_than_fifty_active_subscriptions() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + try: + with Session(engine) as db: + user = _user(db, suffix="limit") + for index in range(MAX_ACTIVE_SUBSCRIPTIONS): + _subscription( + db, + user, + code=f"SUB-LIMIT-{index}", + next_run_at=datetime(2026, 7, 27, 1, 0), + ) + with pytest.raises(HTTPException, match="at most 50"): + SubscriptionManagementService(db).create_private( + FeishuPrincipal.from_user(user), + "每天 09:00", + "第 51 条", + now=datetime(2026, 7, 26, 0, 0), + ) + finally: + engine.dispose() + + +def test_scanner_materializes_one_window_and_advances_the_plan() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 1, 0) + try: + with Session(engine) as db: + user = _user(db, suffix="scan") + subscription = _subscription( + db, + user, + code="SUB-SCAN", + next_run_at=current, + ) + + first = SubscriptionScanner(db).scan_due(now=current) + second = SubscriptionScanner(db).scan_due(now=current) + + assert len(first) == 1 + assert second == [] + assert first[0].scheduled_for == current + assert first[0].status == PushDeliveryStatus.PENDING + assert first[0].next_attempt_at == current + db.refresh(subscription) + assert subscription.next_run_at == datetime(2026, 7, 27, 1, 0) + assert db.scalar(select(func.count()).select_from(PushDelivery)) == 1 + finally: + engine.dispose() + + +def test_scanner_defers_quiet_hours_and_skips_daily_limit() -> None: + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + current = datetime(2026, 7, 26, 15, 30) + try: + with Session(engine) as db: + quiet_user = _user( + db, + suffix="quiet", + quiet_start=time(22, 0), + quiet_end=time(7, 0), + ) + _subscription( + db, + quiet_user, + code="SUB-QUIET", + next_run_at=current, + ) + quiet_delivery = SubscriptionScanner(db).scan_due(now=current)[0] + assert quiet_delivery.status == PushDeliveryStatus.PENDING + assert quiet_delivery.next_attempt_at == datetime(2026, 7, 26, 23, 0) + + limited_user = _user(db, suffix="daily-limit") + history = _subscription( + db, + limited_user, + code="SUB-HISTORY", + next_run_at=current + timedelta(days=1), + ) + for index in range(MAX_DAILY_DELIVERIES): + db.add( + PushDelivery( + code=f"DEL-HISTORY-{index}", + subscription_id=history.id, + scheduled_for=current - timedelta(minutes=index), + idempotency_key=f"{index:064x}", + message_uuid=str(uuid4()), + status=PushDeliveryStatus.SENT, + attempt_count=1, + sent_at=current, + created_at=current, + updated_at=current, + ) + ) + _subscription( + db, + limited_user, + code="SUB-OVER-LIMIT", + next_run_at=current, + ) + db.commit() + + limited_delivery = SubscriptionScanner(db).scan_due(now=current)[0] + assert limited_delivery.status == PushDeliveryStatus.SKIPPED + assert limited_delivery.last_error == DAILY_DELIVERY_LIMIT_REACHED + finally: + engine.dispose() + + +def test_two_sqlite_scanners_claim_only_one_delivery(tmp_path: Path) -> None: + database_path = tmp_path / "subscription-scanner.db" + engine = create_engine( + f"sqlite:///{database_path}", + connect_args={"check_same_thread": False, "timeout": 10}, + ) + with engine.begin() as connection: + connection.exec_driver_sql("PRAGMA journal_mode=WAL") + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + current = datetime(2026, 7, 26, 1, 0) + try: + with factory() as db: + user = _user(db, suffix="concurrent") + _subscription( + db, + user, + code="SUB-CONCURRENT", + next_run_at=current, + ) + + barrier = Barrier(2) + + def scan(worker_id: str) -> list[str]: + with factory() as db: + barrier.wait() + return [ + delivery.code + for delivery in SubscriptionScanner(db).scan_due( + now=current, + worker_id=worker_id, + ) + ] + + with ThreadPoolExecutor(max_workers=2) as executor: + results = list(executor.map(scan, ("scanner-one", "scanner-two"))) + + assert sum(len(items) for items in results) == 1 + with factory() as db: + assert db.scalar(select(func.count()).select_from(PushDelivery)) == 1 + finally: + engine.dispose() + diff --git a/tests/test_subscription_runtime_wiring.py b/tests/test_subscription_runtime_wiring.py new file mode 100644 index 0000000..c59f424 --- /dev/null +++ b/tests/test_subscription_runtime_wiring.py @@ -0,0 +1,360 @@ +from collections.abc import Iterator +from datetime import datetime, timedelta +from types import SimpleNamespace +from uuid import uuid4 + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from sqlalchemy import create_engine +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +import app.core.background.task_queue.subscriptions as subscription_queue +from app.application.scheduling import create_scheduler +from app.core.config import get_settings +from app.core.database import Base, get_db +from app.modules.feishu_users.models import FeishuUser +from app.modules.observability.constants import ( + ObservabilityKey, + ObservabilityStatus, +) +from app.modules.observability.routes import router as observability_router +from app.modules.subscriptions.constants import ( + PushDeliveryStatus, + PushSubscriptionStatus, + SubscriptionScheduleType, + SubscriptionTargetType, +) +from app.modules.subscriptions.models import PushDelivery, PushSubscription +from app.modules.subscriptions.routes import router as subscriptions_router + + +@pytest.fixture(autouse=True) +def _reset_settings() -> Iterator[None]: + get_settings.cache_clear() + yield + get_settings.cache_clear() + + +@pytest.fixture +def session_factory( + monkeypatch: pytest.MonkeyPatch, +) -> Iterator[sessionmaker[Session]]: + monkeypatch.setenv("API_KEY", "subscription-service-key") + monkeypatch.setenv("API_KEYS", "[]") + get_settings.cache_clear() + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + yield factory + finally: + engine.dispose() + + +def _seed_subscription( + db: Session, + *, + suffix: str, + status_value: str = PushSubscriptionStatus.ACTIVE, +) -> PushSubscription: + owner = FeishuUser( + code=f"FSU-{suffix}", + tenant_key=f"tenant-{suffix}", + open_id=f"open-{suffix}", + ) + db.add(owner) + db.flush() + subscription = PushSubscription( + code=f"SUB-{suffix}", + owner_id=owner.id, + target_type=SubscriptionTargetType.USER, + target_id=owner.open_id, + prompt="生成一条个人提醒", + schedule_type=SubscriptionScheduleType.DAILY, + schedule_config={"hour": 9, "minute": 0}, + timezone="Asia/Shanghai", + next_run_at=( + datetime(2030, 1, 1, 1, 0) + if status_value == PushSubscriptionStatus.ACTIVE + else None + ), + status=status_value, + consented_at=datetime(2026, 7, 26, 1, 0), + ) + db.add(subscription) + db.commit() + db.refresh(subscription) + return subscription + + +def _seed_delivery( + db: Session, + subscription: PushSubscription, + *, + suffix: str, + status_value: str, +) -> PushDelivery: + delivery = PushDelivery( + code=f"DEL-{suffix}", + subscription_id=subscription.id, + scheduled_for=datetime(2026, 7, 27, 1, 0), + idempotency_key=uuid4().hex + uuid4().hex, + message_uuid=str(uuid4()), + status=status_value, + attempt_count=2, + next_attempt_at=datetime(2030, 1, 1, 1, 0), + last_error="temporary provider failure", + ) + db.add(delivery) + db.commit() + db.refresh(delivery) + return delivery + + +def _test_app( + session_factory: sessionmaker[Session], + *, + include_subscriptions: bool = False, + include_observability: bool = False, +) -> FastAPI: + app = FastAPI() + if include_subscriptions: + app.include_router( + subscriptions_router, + prefix="/api/v1/subscriptions", + ) + if include_observability: + app.include_router(observability_router, prefix="/api/v1") + + def override_db() -> Iterator[Session]: + with session_factory() as db: + yield db + + app.dependency_overrides[get_db] = override_db + return app + + +def test_subscription_internal_apis_require_key_and_serialize_records( + session_factory: sessionmaker[Session], +) -> None: + with session_factory() as db: + subscription = _seed_subscription(db, suffix="api") + delivery = _seed_delivery( + db, + subscription, + suffix="api", + status_value=PushDeliveryStatus.RETRY, + ) + + client = TestClient( + _test_app(session_factory, include_subscriptions=True) + ) + assert client.get("/api/v1/subscriptions").status_code == 401 + assert client.get("/api/v1/subscriptions/deliveries").status_code == 401 + assert ( + client.get( + "/api/v1/subscriptions", + headers={"X-API-Key": "invalid-key"}, + ).status_code + == 401 + ) + + headers = {"X-API-Key": "subscription-service-key"} + subscriptions_response = client.get( + "/api/v1/subscriptions", + headers=headers, + ) + deliveries_response = client.get( + "/api/v1/subscriptions/deliveries", + headers=headers, + ) + + assert subscriptions_response.status_code == 200 + subscriptions_body = subscriptions_response.json() + assert subscriptions_body["total"] == 1 + subscription_item = subscriptions_body["items"][0] + assert set(subscription_item) == { + "code", + "owner_id", + "target_type", + "target_id", + "prompt", + "schedule_type", + "schedule_config", + "timezone", + "next_run_at", + "status", + "consented_at", + "last_run_at", + "created_at", + "updated_at", + } + assert subscription_item["code"] == subscription.code + assert subscription_item["schedule_config"] == {"hour": 9, "minute": 0} + assert subscription_item["next_run_at"] is not None + + assert deliveries_response.status_code == 200 + deliveries_body = deliveries_response.json() + assert deliveries_body["total"] == 1 + delivery_item = deliveries_body["items"][0] + assert set(delivery_item) == { + "code", + "subscription_id", + "scheduled_for", + "idempotency_key", + "message_uuid", + "status", + "attempt_count", + "next_attempt_at", + "provider_message_id", + "last_error", + "sent_at", + "created_at", + "updated_at", + } + assert delivery_item["code"] == delivery.code + assert delivery_item["message_uuid"] == delivery.message_uuid + assert delivery_item["attempt_count"] == 2 + assert "rendered_content" not in delivery_item + + +@pytest.mark.parametrize( + ("features_enabled", "job_expected"), + [(False, False), (True, True)], +) +def test_subscription_scheduler_job_follows_feature_flag( + monkeypatch: pytest.MonkeyPatch, + features_enabled: bool, + job_expected: bool, +) -> None: + monkeypatch.setenv( + "FEISHU_USER_FEATURES_ENABLED", + str(features_enabled).lower(), + ) + get_settings.cache_clear() + + scheduler = create_scheduler() + job = scheduler.get_job("subscription_delivery_cycle") + + assert scheduler.running is False + assert (job is not None) is job_expected + if job is not None: + assert job.trigger.interval == timedelta(minutes=1) + assert scheduler._job_defaults["coalesce"] is True + assert scheduler._job_defaults["max_instances"] == 1 + + +def test_subscription_queue_runs_inline_without_celery( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("TASK_QUEUE_ENABLED", "false") + get_settings.cache_clear() + calls: list[str] = [] + + def fake_cycle(*, actor: str) -> dict[str, list[str]]: + calls.append(actor) + return {"created": [], "processed": []} + + monkeypatch.setattr( + subscription_queue, + "run_subscription_cycle", + fake_cycle, + ) + + result = subscription_queue.enqueue_subscription_cycle(actor="runtime-test") + + assert result == { + "queued": False, + "mode": "inline", + "task_name": "subscriptions.run_cycle", + "result": {"created": [], "processed": []}, + } + assert calls == ["runtime-test"] + + +def test_subscription_queue_uses_celery_when_enabled( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.tasks import celery_app + + monkeypatch.setenv("TASK_QUEUE_ENABLED", "true") + get_settings.cache_clear() + calls: dict[str, object] = {} + + class FakeSignature: + def apply_async(self) -> SimpleNamespace: + calls["applied"] = True + return SimpleNamespace(id="queued-task-id") + + def fake_signature( + task_name: str, + *, + kwargs: dict[str, str], + ) -> FakeSignature: + calls["task_name"] = task_name + calls["kwargs"] = kwargs + return FakeSignature() + + monkeypatch.setattr(celery_app, "signature", fake_signature) + monkeypatch.setattr( + subscription_queue, + "run_subscription_cycle", + lambda **_: pytest.fail("Celery dispatch must not run inline"), + ) + + result = subscription_queue.enqueue_subscription_cycle(actor="runtime-test") + + assert result == { + "queued": True, + "mode": "celery", + "task_name": "subscriptions.run_cycle", + "task_id": "queued-task-id", + } + assert calls == { + "task_name": "subscriptions.run_cycle", + "kwargs": {"actor": "runtime-test"}, + "applied": True, + } + + +def test_ready_route_returns_503_for_processable_delivery_without_credentials( + session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FEISHU_APP_TYPE", "self") + monkeypatch.setenv("FEISHU_APP_ID", "") + monkeypatch.setenv("FEISHU_APP_SECRET", "") + get_settings.cache_clear() + with session_factory() as db: + subscription = _seed_subscription( + db, + suffix="ready", + status_value=PushSubscriptionStatus.COMPLETED, + ) + _seed_delivery( + db, + subscription, + suffix="ready", + status_value=PushDeliveryStatus.PENDING, + ) + + client = TestClient( + _test_app(session_factory, include_observability=True) + ) + response = client.get("/api/v1/health/ready") + + assert response.status_code == 503 + body = response.json() + assert body[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + check = body[ObservabilityKey.CHECKS]["feishu_subscriptions"] + assert check[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert check["active"] == 0 + assert check["processable_deliveries"] == 1 + assert check["reasons"] == ["credentials_missing"] + assert "生成一条个人提醒" not in response.text diff --git a/tests/test_subscription_schedule.py b/tests/test_subscription_schedule.py new file mode 100644 index 0000000..c4af4a4 --- /dev/null +++ b/tests/test_subscription_schedule.py @@ -0,0 +1,105 @@ +from datetime import UTC, datetime, time + +import pytest + +from app.modules.subscriptions.constants import SubscriptionScheduleType +from app.modules.subscriptions.services.schedule import ( + ScheduleParseError, + is_in_quiet_hours, + next_occurrence, + next_quiet_end, + parse_schedule, +) + + +NOW = datetime(2026, 7, 26, 0, 0, tzinfo=UTC) + + +@pytest.mark.parametrize( + ("expression", "schedule_type", "next_run_at"), + [ + ("今天 09:00", SubscriptionScheduleType.ONCE, datetime(2026, 7, 26, 1, 0)), + ("明天 9点半", SubscriptionScheduleType.ONCE, datetime(2026, 7, 27, 1, 30)), + ( + "2026-08-01 10:15", + SubscriptionScheduleType.ONCE, + datetime(2026, 8, 1, 2, 15), + ), + ("每天 09:00", SubscriptionScheduleType.DAILY, datetime(2026, 7, 26, 1, 0)), + ("工作日 09:00", SubscriptionScheduleType.WEEKDAY, datetime(2026, 7, 27, 1, 0)), + ("每周一 09:00", SubscriptionScheduleType.WEEKLY, datetime(2026, 7, 27, 1, 0)), + ("每月26号 09:00", SubscriptionScheduleType.MONTHLY, datetime(2026, 7, 26, 1, 0)), + ( + "每隔 2 小时", + SubscriptionScheduleType.INTERVAL, + datetime(2026, 7, 26, 2, 0), + ), + ], +) +def test_parse_supported_chinese_schedules( + expression: str, + schedule_type: str, + next_run_at: datetime, +) -> None: + result = parse_schedule(expression, "Asia/Shanghai", now=NOW) + + assert result.schedule_type == schedule_type + assert result.next_run_at == next_run_at + assert result.timezone == "Asia/Shanghai" + + +def test_monthly_schedule_skips_months_without_the_requested_day() -> None: + result = parse_schedule( + "每月31号 09:00", + "Asia/Shanghai", + now=datetime(2026, 4, 30, 2, 0, tzinfo=UTC), + ) + + assert result.next_run_at == datetime(2026, 5, 31, 1, 0) + assert ( + next_occurrence( + result.schedule_type, + result.schedule_config, + result.timezone, + after=datetime(2026, 5, 31, 1, 0), + ) + == datetime(2026, 7, 31, 1, 0) + ) + + +@pytest.mark.parametrize( + "expression", + [ + "每隔 14 分钟", + "有空时提醒我", + "今天 07:59", + "2026-02-30 09:00", + "每天 25:00", + ], +) +def test_invalid_or_ambiguous_schedule_is_rejected(expression: str) -> None: + with pytest.raises(ScheduleParseError): + parse_schedule(expression, "Asia/Shanghai", now=NOW) + + +def test_invalid_iana_timezone_is_rejected() -> None: + with pytest.raises(ScheduleParseError, match="Invalid IANA timezone"): + parse_schedule("每天 09:00", "Mars/Olympus", now=NOW) + + +def test_quiet_hours_support_cross_midnight_windows() -> None: + current = datetime(2026, 7, 26, 15, 30, tzinfo=UTC) + + assert is_in_quiet_hours( + current, + "Asia/Shanghai", + time(22, 0), + time(7, 0), + ) + assert next_quiet_end( + current, + "Asia/Shanghai", + time(22, 0), + time(7, 0), + ) == datetime(2026, 7, 26, 23, 0) + diff --git a/tests/test_v2_hardening.py b/tests/test_v2_hardening.py new file mode 100644 index 0000000..6544f10 --- /dev/null +++ b/tests/test_v2_hardening.py @@ -0,0 +1,317 @@ +import json +from pathlib import Path + +import pytest +from sqlalchemy import create_engine, select +from sqlalchemy.orm import Session, sessionmaker + +from app.core.background.task_queue.reports import _queue_celery_report_push +from app.core.config import Settings +from app.core.http.masking import MASKED_VALUE, mask_sensitive +from app.core.utils.time import utc_now +from app.modules.ai_memory.constants import AIMemoryStatus +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.ai_memory.service import AIMemoryService +from app.modules.audit.constants import AUDIT_REDACTED_VALUE +from app.modules.audit.models import AuditLog +from app.modules.audit.schemas import AuditLogCreate +from app.modules.audit.service import AuditService +from app.modules.business.models import Project, ReportPushRun +from app.modules.events.models import DomainEvent +from app.modules.feishu.service import FeishuService +from app.modules.reports.constants import ReportPushStatus +from app.tasks import celery_app +from app.tasks.reports import _push_report +from app.tools import init_db + + +def _production_settings(**overrides: object) -> Settings: + values: dict[str, object] = { + "app_env": "production", + "database_url": "postgresql+psycopg://app:secret@db/app", + "api_key": "service-key", + "audit_api_key": "audit-key", + "cors_origins": ["https://internal.example.com"], + "debug": False, + "mask_sensitive_responses": True, + "read_only_mode": True, + } + values.update(overrides) + return Settings(_env_file=None, **values) + + +def test_production_keys_must_be_enabled_isolated_and_safe() -> None: + settings = _production_settings() + assert settings.api_key == "service-key" + + with pytest.raises(ValueError, match="API_KEY or API_KEYS"): + _production_settings( + api_key=None, + api_keys=[{"key": "disabled-key", "enabled": False}], + ) + + with pytest.raises(ValueError, match="cannot overlap"): + _production_settings(audit_api_key="service-key") + + with pytest.raises(ValueError, match="MASK_SENSITIVE_RESPONSES"): + _production_settings(mask_sensitive_responses=False) + + with pytest.raises(ValueError, match="READ_ONLY_MODE"): + _production_settings(read_only_mode=False) + + +def test_audit_json_strings_and_responses_mask_sensitive_fields() -> None: + engine = create_engine("sqlite://") + AuditLog.__table__.create(engine) + try: + with Session(engine) as db: + service = AuditService(db) + dict_record = service.log( + AuditLogCreate( + action="security.redaction.dict", + request_payload={ + "accessToken": "access-secret", + "nested": { + "clientSecret": "client-secret", + "safe": "visible", + }, + }, + ) + ) + json_record = service.log( + AuditLogCreate( + action="security.redaction.json", + request_payload=json.dumps( + { + "authorization": "Bearer secret", + "nested": { + "refreshToken": "refresh-secret", + "safe": "visible", + }, + } + ), + ) + ) + dict_stored = json.loads(dict_record.request_payload or "{}") + json_stored = json.loads(json_record.request_payload or "{}") + + assert dict_stored["accessToken"] == AUDIT_REDACTED_VALUE + assert dict_stored["nested"]["clientSecret"] == AUDIT_REDACTED_VALUE + assert dict_stored["nested"]["safe"] == "visible" + assert json_stored["authorization"] == AUDIT_REDACTED_VALUE + assert json_stored["nested"]["refreshToken"] == AUDIT_REDACTED_VALUE + assert json_stored["nested"]["safe"] == "visible" + + masked = mask_sensitive( + { + "accessToken": "access-secret", + "nested": { + "clientSecret": "client-secret", + "privateKey": "private-secret", + "safe": "visible", + }, + } + ) + assert masked["accessToken"] == MASKED_VALUE + assert masked["nested"]["clientSecret"] == MASKED_VALUE + assert masked["nested"]["privateKey"] == MASKED_VALUE + assert masked["nested"]["safe"] == "visible" + finally: + engine.dispose() + + +def test_report_queue_failure_is_persisted( + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine = create_engine("sqlite://") + ReportPushRun.__table__.create(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + + class BrokenSignature: + def apply_async(self, task_id: str) -> None: + _ = task_id + raise RuntimeError("broker unavailable") + + monkeypatch.setattr("app.core.database.SessionLocal", factory) + monkeypatch.setattr( + celery_app, + "signature", + lambda *args, **kwargs: BrokenSignature(), + ) + try: + with pytest.raises(RuntimeError, match="broker unavailable"): + _queue_celery_report_push( + task_name="reports.test", + report_type="test", + title="Test report", + receive_id=None, + receive_id_type="chat_id", + actor="pytest", + ) + + with factory() as db: + run = db.execute(select(ReportPushRun)).scalar_one() + assert run.status == ReportPushStatus.FAILED + assert run.task_id + assert run.error_message == "broker unavailable" + finally: + engine.dispose() + + +def test_eager_report_terminal_state_is_not_overwritten( + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine = create_engine("sqlite://") + ReportPushRun.__table__.create(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + + class EagerSignature: + def __init__(self, push_run_code: str): + self.push_run_code = push_run_code + + def apply_async(self, task_id: str) -> None: + with factory() as db: + run = db.execute( + select(ReportPushRun).where( + ReportPushRun.code == self.push_run_code + ) + ).scalar_one() + assert run.task_id == task_id + run.status = ReportPushStatus.SUCCESS + db.commit() + + def eager_signature(*args: object, **kwargs: object) -> EagerSignature: + _ = args + task_kwargs = kwargs["kwargs"] + assert isinstance(task_kwargs, dict) + return EagerSignature(str(task_kwargs["push_run_code"])) + + monkeypatch.setattr("app.core.database.SessionLocal", factory) + monkeypatch.setattr(celery_app, "signature", eager_signature) + try: + queued = _queue_celery_report_push( + task_name="reports.test", + report_type="test", + title="Test report", + receive_id=None, + receive_id_type="chat_id", + actor="pytest", + ) + + with factory() as db: + run = db.execute(select(ReportPushRun)).scalar_one() + assert run.status == ReportPushStatus.SUCCESS + assert run.task_id == queued["task_id"] + finally: + engine.dispose() + + +def test_successful_report_task_redelivery_does_not_send_twice( + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine = create_engine("sqlite://") + for table in (AuditLog.__table__, DomainEvent.__table__, ReportPushRun.__table__): + table.create(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + calls = {"build": 0, "send": 0} + + def build_report(service: object, actor: str) -> dict[str, object]: + _ = service, actor + calls["build"] += 1 + return { + "title": "Idempotent report", + "report_type": "test", + "lines": ["ok"], + "content": "ok", + } + + def fake_send_card( + service: object, + card: dict, + receive_id: str | None, + receive_id_type: str, + actor: str, + ) -> dict[str, object]: + _ = service, card, receive_id, receive_id_type, actor + calls["send"] += 1 + return {"code": 0, "message_id": "om-idempotent"} + + monkeypatch.setattr("app.tasks.reports.SessionLocal", factory) + monkeypatch.setattr(FeishuService, "send_card", fake_send_card) + try: + with factory() as db: + run = ReportPushRun( + code="PUSH-IDEMPOTENT", + report_type="test", + title="Idempotent report", + receive_id="oc-idempotent", + receive_id_type="chat_id", + status=ReportPushStatus.QUEUED, + actor="pytest", + ) + db.add(run) + db.commit() + + first = _push_report(build_report, "oc-idempotent", "chat_id", "pytest", run.code) + second = _push_report(build_report, "oc-idempotent", "chat_id", "pytest", run.code) + + assert first == second == {"code": 0, "message_id": "om-idempotent"} + assert calls == {"build": 1, "send": 1} + with factory() as db: + stored = db.execute(select(ReportPushRun)).scalar_one() + assert stored.status == ReportPushStatus.SUCCESS + assert stored.sent_at is not None + finally: + engine.dispose() + + +def test_init_db_upgrades_to_alembic_head( + monkeypatch: pytest.MonkeyPatch, +) -> None: + called: dict[str, object] = {} + + def fake_upgrade(config: object, revision: str) -> None: + called["config"] = config + called["revision"] = revision + + monkeypatch.setattr(init_db.command, "upgrade", fake_upgrade) + init_db.main() + + config = called["config"] + assert called["revision"] == "head" + assert Path(config.config_file_name).name == "alembic.ini" + assert Path(config.get_main_option("script_location")).name == "alembic" + + +def test_memory_retention_does_not_commit_caller_transaction() -> None: + engine = create_engine("sqlite://") + AIMemoryEntry.__table__.create(engine) + Project.__table__.create(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + try: + with factory() as db: + expired = AIMemoryEntry( + code="MEM-EXPIRED", + scope="project", + subject="P-ROLLBACK", + content="expired", + status=AIMemoryStatus.ACTIVE, + expires_at=utc_now(), + ) + db.add(expired) + db.commit() + + db.add(Project(code="P-ROLLBACK", name="Must roll back")) + assert AIMemoryService(db).list_entries(scope="project") == [] + db.rollback() + + with factory() as db: + assert db.execute( + select(Project).where(Project.code == "P-ROLLBACK") + ).scalar_one_or_none() is None + stored = db.execute( + select(AIMemoryEntry).where(AIMemoryEntry.code == "MEM-EXPIRED") + ).scalar_one() + assert stored.status == AIMemoryStatus.ACTIVE + finally: + engine.dispose() diff --git a/tests/test_v2_v3_completion.py b/tests/test_v2_v3_completion.py new file mode 100644 index 0000000..529d264 --- /dev/null +++ b/tests/test_v2_v3_completion.py @@ -0,0 +1,243 @@ +from datetime import timedelta +from pathlib import Path + +import pytest +from fastapi import HTTPException +from sqlalchemy import UniqueConstraint, create_engine, func, select, text +from sqlalchemy.orm import Session, sessionmaker + +from app.application.events import EventDispatchService +from app.application.scheduling import create_scheduler +from app.core.config import get_settings +from app.core.database import Base +from app.core.utils.time import utc_now +from app.main import app +from app.modules.ai_agent.constants import AIContextKey +from app.modules.ai_agent.service import AIService +from app.modules.audit.constants import AuditSource +from app.modules.events.constants import EventStatus +from app.modules.events.models import DomainEvent +from app.modules.events.services import EventService +from app.modules.legacy_mysql.services import LegacyMySQLService +from app.modules.observability.constants import ( + HeartbeatComponent, + ObservabilityKey, + ObservabilityStatus, +) +from app.modules.observability.models import SystemHeartbeat +from app.modules.observability.service import ObservabilityService +from app.tasks import celery_app + + +def _session_factory(): + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + return engine, sessionmaker(bind=engine, expire_on_commit=False) + + +def test_readiness_and_metrics_degrade_when_schema_is_missing() -> None: + engine = create_engine("sqlite://") + with Session(engine) as db: + readiness = ObservabilityService(db).ready() + metrics = ObservabilityService(db).metrics() + + assert readiness[ObservabilityKey.STATUS] == ObservabilityStatus.DEGRADED + assert readiness[ObservabilityKey.CHECKS][ObservabilityKey.DATABASE][ + ObservabilityKey.STATUS + ] == ObservabilityStatus.OK + assert readiness[ObservabilityKey.CHECKS][ObservabilityKey.EVENTS][ + ObservabilityKey.STATUS + ] == ObservabilityStatus.ERROR + assert metrics[ObservabilityKey.METRICS][ObservabilityKey.EVENTS][ + ObservabilityKey.STATUS + ] == ObservabilityStatus.ERROR + + +@pytest.mark.parametrize( + "sql", + [ + "SELECT 1; DELETE FROM projects", + "SELECT * FROM projects /* hidden operation */", + "SELECT * FROM projects -- hidden operation", + "SELECT * FROM projects FOR UPDATE", + "SELECT * INTO OUTFILE '/tmp/data' FROM projects", + ], +) +def test_legacy_query_rejects_unsafe_allowlist_sql(sql: str) -> None: + with pytest.raises(HTTPException) as exc_info: + LegacyMySQLService(None)._ensure_readonly(sql) + + assert exc_info.value.status_code == 400 + + +def test_legacy_query_accepts_parameterized_select() -> None: + LegacyMySQLService(None)._ensure_readonly( + "SELECT code, name FROM projects WHERE owner = :owner LIMIT :limit" + ) + + +def test_celery_and_scheduler_have_safe_runtime_defaults() -> None: + assert celery_app.conf.task_serializer == "json" + assert celery_app.conf.result_serializer == "json" + assert tuple(celery_app.conf.accept_content) == ("json",) + assert celery_app.conf.task_acks_late is True + assert celery_app.conf.worker_prefetch_multiplier == 1 + + scheduler = create_scheduler() + assert scheduler._job_defaults["coalesce"] is True + assert scheduler._job_defaults["max_instances"] == 1 + + +def test_unknown_event_type_fails_instead_of_being_discarded() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + event = EventService(db).emit( + event_type="unknown.event", + source="pytest", + aggregate_type="test", + aggregate_id="unknown", + idempotency_key="unknown-event", + ) + event.max_attempts = 1 + db.commit() + + result = EventDispatchService(db).dispatch_event( + event.event_id, + worker_id="pytest", + ) + + assert result.status == EventStatus.FAILED + assert "Unsupported domain event type" in (result.last_error or "") + finally: + engine.dispose() + + +def test_event_failure_rolls_back_handler_transaction( + monkeypatch: pytest.MonkeyPatch, +) -> None: + engine, factory = _session_factory() + try: + with factory() as db: + event = EventService(db).emit( + event_type="test.database_failure", + source="pytest", + aggregate_type="test", + aggregate_id="database-failure", + idempotency_key="database-failure", + ) + event.max_attempts = 1 + db.commit() + dispatcher = EventDispatchService(db) + + def fail_with_database_error(record: DomainEvent) -> None: + _ = record + db.execute(text("SELECT * FROM missing_handler_table")).all() + + monkeypatch.setattr(dispatcher, "_handle_event", fail_with_database_error) + result = dispatcher.dispatch_event(event.event_id, worker_id="pytest") + + assert result.status == EventStatus.FAILED + assert result.last_error + finally: + engine.dispose() + + +def test_retry_rejects_an_active_event_lease() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + event = EventService(db).emit( + event_type="test.locked", + source="pytest", + aggregate_type="test", + aggregate_id="locked", + idempotency_key="locked-event", + ) + event.locked_by = "active-worker" + event.locked_until = utc_now() + timedelta(minutes=5) + db.commit() + + with pytest.raises(HTTPException) as exc_info: + EventDispatchService(db).retry_event(event.event_id) + + assert exc_info.value.status_code == 409 + finally: + engine.dispose() + + +def test_heartbeat_identity_is_unique_and_expired_instances_are_ignored( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("HEARTBEAT_RETENTION_SECONDS", "3600") + get_settings.cache_clear() + engine, factory = _session_factory() + try: + constraints = SystemHeartbeat.__table__.constraints + assert any( + isinstance(item, UniqueConstraint) + and {column.name for column in item.columns} == {"component", "instance_id"} + for item in constraints + ) + + with factory() as db: + service = ObservabilityService(db) + service.record_heartbeat( + component=HeartbeatComponent.WORKER, + instance_id="worker-current", + ) + service.record_heartbeat( + component=HeartbeatComponent.WORKER, + instance_id="worker-current", + ) + db.add( + SystemHeartbeat( + component=HeartbeatComponent.WORKER, + instance_id="worker-expired", + status="ok", + last_seen_at=utc_now() - timedelta(hours=2), + ) + ) + db.commit() + + count = db.scalar(select(func.count()).select_from(SystemHeartbeat)) + summary = service.heartbeat_summary() + + assert count == 2 + assert summary["total"] == 1 + assert summary["stale"] == 0 + finally: + get_settings.cache_clear() + engine.dispose() + + +@pytest.mark.parametrize( + ("field", "value"), + [ + (AIContextKey.OPENCLAW_TOOL, "sessions_list"), + (AIContextKey.OPENCLAW_ACTION, "json"), + (AIContextKey.OPENCLAW_ARGS, {"limit": 1}), + (AIContextKey.OPENCLAW_SESSION_KEY, "main"), + ], +) +def test_external_ai_cannot_invoke_openclaw_tools( + field: AIContextKey, + value: object, +) -> None: + with pytest.raises(HTTPException) as exc_info: + AIService(Session()).ask( + "run a tool", + context={field: value}, + source=AuditSource.API, + ) + + assert exc_info.value.status_code == 403 + assert "/api/v1/ai/openclaw/tools/invoke" not in app.openapi()["paths"] + + +def test_sample_requests_are_read_only() -> None: + sample = Path("scripts/sample_requests.http").read_text(encoding="utf-8") + + assert "POST http://127.0.0.1:8010/api/v1/business/" not in sample + assert "/sync" not in sample + assert "/assign" not in sample diff --git a/tests/test_v3_remaining.py b/tests/test_v3_remaining.py new file mode 100644 index 0000000..6543059 --- /dev/null +++ b/tests/test_v3_remaining.py @@ -0,0 +1,199 @@ +from datetime import date, timedelta + +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine, func, select +from sqlalchemy.orm import sessionmaker + +from app.application.events import EventDispatchService +from app.core.database import Base +from app.core.utils.time import utc_now +from app.modules.ai_memory.constants import ( + AIMemoryPayloadKey, + AIMemoryStatus, +) +from app.modules.ai_memory.models import AIMemoryEntry +from app.modules.ai_memory.service import AIMemoryService +from app.modules.business.constants import StatusValue +from app.modules.business.models import FundAccount, PerformanceMetric, Project +from app.modules.events.constants import ( + EventAggregateType, + EventSource, + EventStatus, + EventType, +) +from app.modules.events.models import DomainEvent +from app.modules.events.services import EventService +from app.modules.reports.constants import EnterpriseAnalyticsKey, MetricKey +from app.modules.reports.services import ReportService +from app.modules.workflows.constants import WorkflowStatus, WorkflowType +from app.modules.workflows.models import WorkflowAction, WorkflowInstance + + +def _session_factory(): + engine = create_engine("sqlite://") + Base.metadata.create_all(engine) + return engine, sessionmaker(bind=engine, expire_on_commit=False) + + +def test_workflow_actions_are_idempotent_per_source_event() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + event = EventService(db).emit( + event_type=EventType.ENTERPRISE_ANALYTICS_GENERATED, + source=EventSource.ANALYTICS, + aggregate_type=EventAggregateType.ENTERPRISE_ANALYTICS, + aggregate_id="snapshot-1", + idempotency_key="analytics:snapshot-1", + ) + + first = EventDispatchService(db).dispatch_event( + event.event_id, + worker_id="pytest-first", + ) + assert first.status == EventStatus.PROCESSED + + first.status = EventStatus.PENDING + first.processed_at = None + first.next_attempt_at = utc_now() + db.commit() + second = EventDispatchService(db).dispatch_event( + event.event_id, + worker_id="pytest-second", + ) + + assert second.status == EventStatus.PROCESSED + assert db.scalar(select(func.count()).select_from(WorkflowInstance)) == 1 + assert db.scalar(select(func.count()).select_from(WorkflowAction)) == 1 + action = db.execute(select(WorkflowAction)).scalar_one() + assert action.source_event_id == event.event_id + finally: + engine.dispose() + + +def test_ai_memory_has_no_unrelated_fallback_and_archives_expired_entries() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + service = AIMemoryService(db) + first = service.auto_write( + prompt="Remember concise operational summaries", + context={ + AIMemoryPayloadKey.SCOPE: "project", + AIMemoryPayloadKey.SUBJECT: "P-MEM-IDEMPOTENT", + }, + answer="Use concise bullet summaries for project updates.", + ) + duplicate = service.auto_write( + prompt="Remember concise operational summaries", + context={ + AIMemoryPayloadKey.SCOPE: "project", + AIMemoryPayloadKey.SUBJECT: "P-MEM-IDEMPOTENT", + }, + answer="Use concise bullet summaries for project updates.", + ) + + assert first is not None + assert duplicate is not None + assert duplicate.code == first.code + assert db.scalar(select(func.count()).select_from(AIMemoryEntry)) == 1 + assert ( + service.recall( + query="no-such-memory-token", + scope="project", + subject="P-MEM-IDEMPOTENT", + ) + == [] + ) + + first.expires_at = utc_now() - timedelta(seconds=1) + db.commit() + assert ( + service.list_entries( + scope="project", + subject="P-MEM-IDEMPOTENT", + status_filter=AIMemoryStatus.ACTIVE, + ) + == [] + ) + db.refresh(first) + assert first.status == AIMemoryStatus.ARCHIVED + finally: + engine.dispose() + + +def test_enterprise_analytics_validates_scope_and_reuses_identical_snapshot() -> None: + engine, factory = _session_factory() + try: + with factory() as db: + db.add_all( + [ + Project( + code="P-SCOPED", + name="Scoped project", + owner="owner-a", + status=StatusValue.RUNNING_CN, + ), + PerformanceMetric( + code="PERF-GLOBAL", + name="Global metric", + weight=10, + auto_score=80, + confirmed_score=75, + status=StatusValue.REVIEWED, + ), + FundAccount( + code="FUND-GLOBAL", + name="Global account", + current_balance=1000, + ), + ] + ) + db.commit() + service = ReportService(db) + + with pytest.raises(HTTPException) as exc_info: + service.enterprise_analytics( + period_start=date(2030, 2, 2), + period_end=date(2030, 2, 1), + ) + assert exc_info.value.status_code == 422 + + first = service.enterprise_analytics(project_code="P-SCOPED") + second = service.enterprise_analytics(project_code="P-SCOPED") + + assert first[EnterpriseAnalyticsKey.CODE] == second[EnterpriseAnalyticsKey.CODE] + assert first[EnterpriseAnalyticsKey.PERFORMANCE][MetricKey.TOTAL] == 0 + assert ( + first[EnterpriseAnalyticsKey.FINANCE][ + MetricKey.CURRENT_BALANCE_TOTAL + ] + == 0 + ) + events = list( + db.execute( + select(DomainEvent).where( + DomainEvent.event_type + == EventType.ENTERPRISE_ANALYTICS_GENERATED + ) + ).scalars() + ) + assert len(events) == 1 + assert events[0].aggregate_id == first[EnterpriseAnalyticsKey.CODE] + + dispatched = EventDispatchService(db).dispatch_event( + events[0].event_id, + worker_id="pytest-enterprise", + ) + assert dispatched.status == EventStatus.PROCESSED + workflow = db.execute( + select(WorkflowInstance).where( + WorkflowInstance.workflow_type + == WorkflowType.ENTERPRISE_ANALYTICS + ) + ).scalar_one() + assert workflow.status == WorkflowStatus.COMPLETED + assert workflow.aggregate_id == first[EnterpriseAnalyticsKey.CODE] + finally: + engine.dispose()