文档目录
使用置信度进行分类
用一道 Choice（选择题）将 SEC 年报归入 75 个行业组，再根据答案自带的置信度，决定是报出该行业组，还是报出其上一级的更宽泛的行业门类（division）。
非官方中文翻译 · 内容以 英文原文 ↗ 为准 · 同步于 2026-10-15
每一家向 SEC 提交年报的公司都会在文件中描述自身业务。我们按照标准行业分类（Standard Industrial Classification）对这些描述进行归类：75 个行业组，每份文件对应一道 Choice 问题。
大部分申报材料都很容易归类。区域性银行就是区域性银行。但有些则不然：一家公司刚刚出售了两个业务部门中的一个，或者一家初创公司描述的是它计划进入的业务，而非它正在经营的业务。模型无论如何都必须选出一个组，而困难案例的答案与简单案例的答案看起来毫无差别。区分这两类案例通常正是产生成本的环节：第二个模型、额外的调用、人工复核。
Choice 已经自带这些信息。除了胜出的选项之外，它还会返回 confidence（置信度）：当几乎所有概率都集中到同一个选项上时置信度较高，而当概率分散在多个选项上时置信度较低。仅凭这一个数字，就能把可以信任的答案与不能信任的答案区分开。
如何处理不可信的答案，取决于你的标签体系。SIC 标签构成一个层级：行业组向上汇总为更宽泛的行业门类。这让一次响应几乎不需要任何额外开销。当模型对行业组没有把握时，就报出它所属的行业门类。这个更宽泛的标签由较窄的标签自然得出，因此无需第二次调用。
在 60 份申报材料中，置信度阈值为 0.9 将它们对半分开。置信度高的一半正确率为 90%；另一半则为 40%。如果向上汇报一级，那 40% 就会变成 70%。我们最终得到一个 classify() 函数，它返回一个标签及其具体程度，每个文件只需一次请求。
flowchart LR
doc["Item 1 'Business'<br/>from one 10-K"]
subgraph request["one request"]
q["Choice<br/>75 industry groups"]
end
sure{"confidence<br/>≥ 0.9?"}
grp["report the industry group<br/><i>e.g. 28</i>"]
div["report its division<br/><i>e.g. manufacturing</i>"]
doc --> request --> sure
%% both branches leave the test, so they share a rank and stack on their own
sure -- "yes" --> grp
sure -- "no" --> div环境配置
pip install ipython matplotlib 'cooksafe>=0.2.0,<0.3.0'接下来设置 TYPESAFE_API_KEY。每次 API 调用都会缓存到 json_cache.json，该文件随本示例一同发布，因此重新渲染时可以重放已发布的数字而无需调用 API。删除该文件即可重新实时运行所有内容。
下面的数字来自 jev-1.12，于 2026-08-12 跑出。
import json
from collections import defaultdict
from pathlib import Path
import matplotlib
import matplotlib.pyplot as plt
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, TypeSafeClient
matplotlib.use("Agg") # headless render
import os # noqa: E402
TYPESAFE_MODEL = "jev-1.12"
CONFIDENT = 0.9 # above this the group is reported; below it, the division
client = TypeSafeClient(
api_key=os.environ.get(
"TYPESAFE_API_KEY", "cache-only"
), # keyless kernels replay the cache
base_url=os.environ.get("TYPESAFE_ENDPOINT"),
timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))构建分类法的两个层级
sic_codes.tsv 是 SEC 向申报人发布的行业清单，用于让他们选择自身代码，获取时间为 2026-08-10：共 444 个四位代码，每个都带有一个行业名称。这些数字本身构成一个层级。前两位是 大组（major group）（这里共 75 个，从 01 农业生产到 99 不可分类），固定范围的大组又组成十个 行业门类（division），即 SIC 中最宽泛的划分。
两个层级都来自这一个文件，不涉及任何模型：按前两位数字对代码分组，再将这些数字映射到行业门类。
DIVISIONS = [
(1, 9, "agriculture, forestry and fishing"),
(10, 14, "mining"),
(15, 17, "construction"),
(20, 39, "manufacturing"),
(40, 49, "transportation, communications and utilities"),
(50, 51, "wholesale trade"),
(52, 59, "retail trade"),
(60, 67, "finance, insurance and real estate"),
(70, 89, "services"),
(91, 99, "public administration"),
]
INDUSTRIES: dict[str, str] = {}
for line in Path("sic_codes.tsv").read_text().splitlines()[1:]:
code, _office, title = line.split("\t")
INDUSTRIES[code] = title.lower()
GROUPS: dict[str, list[str]] = defaultdict(list)
for code in sorted(INDUSTRIES):
GROUPS[code[:2]].append(code)
def division(group: str) -> str:
number = int(group)
return next(name for low, high, name in DIVISIONS if low <= number <= high)
print(
f"{len(INDUSTRIES)} industries -> {len(GROUPS)} major groups -> {len(DIVISIONS)} divisions"
)
print(
f" group 35 = {division('35')} / {', '.join(INDUSTRIES[c] for c in GROUPS['35'][:3])} ..."
)444 industries -> 75 major groups -> 10 divisions
group 35 = manufacturing / engines & turbines, farm machinery & equipment, lawn & garden tractors & home lawn & gardens equip ...一道 Choice 问题需要对每个选项进行描述，而组自身的名称并不总是存在：75 个组中有 42 个在 SEC 的清单中带有总括名称，其余则没有。因此，每个组通过其内部所包含的行业来描述，而这正是阅读申报材料的人会去匹配的内容。
MAX_NAMED = (
8 # industries listed per group; enough to characterise it without a wall of text
)
def describe(group: str) -> str:
umbrella = INDUSTRIES.get(f"{group}00")
inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
listed = "; ".join(inside)
return (
f"{umbrella} — includes: {listed}"
if umbrella and listed
else (umbrella or listed)
)
print(f"group 20: {describe('20')[:150]}")
print(f"\ngroup 65: {describe('65')[:150]}")group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product
group 65: real estate — includes: real estate operators (no developers) & lessors; operators of nonresidential buildings; operators of apartment buildings; less申报材料
filings.jsonl 包含 60 份年报（10-K），每份都裁剪到第 1 项「Business」（业务描述），即公司描述自身业务的部分——这也是行业代码唯一相关的部分。它们横跨 1993–2024 年，字数从 700 到 2,200 不等。每份都附带申报人自己选择的 SIC 代码，以及在 EDGAR 上查询它的归档号。
在任何准确率数字之前，标签的来源至关重要。它是自我报告的：无论是谁准备的文件，都曾经选过一次；而当公司出售了代码所指的业务并保留该代码时，它就会过时。这 60 份经过筛选，只保留自身文本能够支持其代码的文件，因此这里的数字衡量的是这个方法本身，而非 EDGAR 元数据的状态。
FILINGS = [json.loads(line) for line in Path("filings.jsonl").read_text().splitlines()]
example = FILINGS[7]
print(
f"{len(FILINGS)} filings, {sum(f['words'] for f in FILINGS) // len(FILINGS)} words on average"
)
print(f"\n{example['id']} (filed {example['year']}, accession {example['accession']}):")
print(f" {example['text'][:230]}...")
print(f" filer's code: {example['sic']} {INDUSTRIES[example['sic']]}")60 filings, 1438 words on average
1389870_2008 (filed 2008, accession 0001079974-09-000155):
Item 1. DESCRIPTION OF BUSINESS. NARRATIVE DESCRIPTION OF THE BUSINESS Across America Financial Services, Inc. is a corporation which was formed under the laws of the State of Colorado on December 1, 2005. Until March 23, 2007, we...
filer's code: 6163 loan brokers提出一道 Choice 问题，并读取置信度
一道 Choice 问题，其选项是 75 个组。整个分类法可以放进一次请求中：Choice 在大约 240 个选项以内都能可靠工作，而 75 个远在其范围内。
返回的答案带有 choice（胜出的组）、probabilities（75 个选项各自的权重）以及 confidence（该分布的集中程度）。本方法读取的是 confidence 而非胜出选项自身的概率。胜出选项为 0.45、亚军为 0.44，与胜出选项为 0.45、其余权重零散分布，是两种不同的局面；而 confidence 正是将它们区分开的指标。
QUESTION = (
"Which broad industry does this company operate in? Judge the company's own operations "
"as this filing describes them."
)
def questions() -> dict:
return {
"group": Choice(
instructions=QUESTION,
criteria={group: describe(group) for group in sorted(GROUPS)},
)
}
@json_cache
def ask(filing_id: str, text: str) -> dict:
response = client.system_one(
state=text, questions=questions(), model=TYPESAFE_MODEL
)
answer = response.answers["group"]
return {
"group": answer.choice,
"confidence": answer.confidence,
"probabilities": dict(answer.probabilities),
}确定时返回行业组，不确定时返回其门类
下面这四行就是全部要点。置信度达到 0.9 及以上时，答案以行业组（industry group）形式给出；低于 0.9 时，同一份答案会以该行业组所属的门类（division）形式给出。
每份申报文件依然会带有一个可用的标签回来。模型无法有把握地分类的那些，会沿层级上升一级返回，而不是被丢弃或转走。如果门类对你的应用来说粒度太粗，那就把这条分支上的任务交给人工处理。
def classify(filing: dict) -> dict:
answer = ask(filing["id"], filing["text"])
sure = answer["confidence"] >= CONFIDENT
return {
"level": "group" if sure else "division",
"label": answer["group"] if sure else division(answer["group"]),
"confidence": answer["confidence"],
"group": answer["group"],
}
def show(filing: dict) -> None:
result = classify(filing)
named = describe(result["group"]).split(" — ")[0][:46]
print(
f" {filing['id']:>13} conf {result['confidence']:.2f} -> {result['level']:<8} "
f"{result['label']:<14} (group {result['group']}: {named})"
)
print("three filings the model was sure about:")
for f in sorted(FILINGS, key=lambda f: -ask(f["id"], f["text"])["confidence"])[:3]:
show(f)
print("\nthree it was not:")
for f in sorted(FILINGS, key=lambda f: ask(f["id"], f["text"])["confidence"])[:3]:
show(f)three filings the model was sure about:
310158_1996 conf 1.00 -> group 28 (group 28: chemicals & allied products)
33416_1998 conf 1.00 -> group 63 (group 63: life insurance; accident & health insurance; h)
352541_1996 conf 1.00 -> group 49 (group 49: electric, gas & sanitary services)
three it was not:
1372167_2013 conf 0.22 -> division manufacturing (group 38: search, detection, navagation, guidance, aeron)
1398633_2009 conf 0.23 -> division wholesale trade (group 50: wholesale-durable goods)
46653_1999 conf 0.29 -> division services (group 87: services-engineering, accounting, research, ma)置信度与每份文件的分类难度相对应。置信度为 1.00 的三份分别来自一家制药商、一家人寿保险公司和一家公用事业公司；三家在文件上都只是控股公司，但每一家都有一项占主导的业务，文件中也直接点明。底部那三份更难，难度原因在文件正文中就能读到：两家是处于开发阶段的公司，描述的是它们打算开展的业务（Nevaeh「intends to operate as a software developer」，Barricode「organized to enter into the computer security software industry」），第三家有两个分部，并在提交申报前几周出售了其中一个。这三份返回的是门类，而不是行业组。
classify() 就是全部要点。把 ask() 指向你自己的文档，为你自己的分类体系重写 describe()，其余部分可以直接沿用。
更宽泛的答案能带来什么
全部 60 份文件，以每家申报人选定的 SIC 编码为基准，对两种策略分别打分：始终给出行业组，或者仅在置信度低于 0.9 时给出门类。
def correct(filing: dict, result: dict) -> bool:
gold_group = filing["sic"][:2]
if result["level"] == "group":
return result["label"] == gold_group
return result["label"] == division(gold_group)
results = [(f, classify(f)) for f in FILINGS]
sure = [(f, r) for f, r in results if r["level"] == "group"]
unsure = [(f, r) for f, r in results if r["level"] == "division"]
forced = sum(r["group"] == f["sic"][:2] for f, r in results)
broadened = sum(correct(f, r) for f, r in results)
print(f"forced to name a group every time {forced}/{len(results)} right")
print(
f" of those, the {len(sure)} it was sure about "
f"{sum(r['group'] == f['sic'][:2] for f, r in sure)}/{len(sure)} right"
)
print(
f" and the {len(unsure)} it was not "
f"{sum(r['group'] == f['sic'][:2] for f, r in unsure)}/{len(unsure)} right"
)
print(
f"\nletting it answer coarsely when unsure {broadened}/{len(results)} useful answers"
)forced to name a group every time 39/60 right
of those, the 30 it was sure about 27/30 right
and the 30 it was not 12/30 right
letting it answer coarsely when unsure 48/60 useful answers在模型有把握的地方，它给出的行业组九成是对的。在它没把握的地方，给出行业组的正确率只有 40%，错的比对的还多。而把这些答案换算成门类，正确率就能提到 70%。
图表把两种策略并排展示，并按模型是否有把握分组。
labels = ["sure\n(group reported)", "unsure\n(division reported)"]
forced_split = [
sum(r["group"] == f["sic"][:2] for f, r in sure) / len(sure),
sum(r["group"] == f["sic"][:2] for f, r in unsure) / len(unsure),
]
broad_split = [
sum(correct(f, r) for f, r in sure) / len(sure),
sum(correct(f, r) for f, r in unsure) / len(unsure),
]
fig, ax = plt.subplots(figsize=(7, 3.6))
x = range(len(labels))
ax.bar(
[i - 0.19 for i in x],
forced_split,
0.38,
label="always name a group",
color="#c8ccd4",
)
ax.bar(
[i + 0.19 for i in x],
broad_split,
0.38,
label="answer broadly when unsure",
color="#3b6ea5",
)
for i, (a, b) in enumerate(zip(forced_split, broad_split)):
ax.text(i - 0.19, a + 0.02, f"{a:.0%}", ha="center", fontsize=9)
ax.text(i + 0.19, b + 0.02, f"{b:.0%}", ha="center", fontsize=9)
ax.set_xticks(list(x))
ax.set_xticklabels(
[f"{lab}\nn={n}" for lab, n in zip(labels, [len(sure), len(unsure)])]
)
ax.set_ylabel("labels that are right")
ax.set_ylim(0, 1.12)
ax.set_title("Where the broader answer helps: the filings it was unsure about")
ax.legend(frameon=False, loc="upper right")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
display(fig)
在 playground 中打开
这个分享链接包含一份文件以及那道有 75 个选项的问题，你不用写任何代码就能看到分布情况和模型给出的置信度。
playground_link = make_playground_link(
example["text"], questions(), models=[TYPESAFE_MODEL]
)
display(
Markdown(
f"🔗 [Open the filing + question in the TypeSafe playground]({playground_link})"
)
)