博文
AGI-AIGC-GPT测评DIKWP（全球）实验室（测评系列之十五）:民事裁判文书解析指令集
|
Evaluation on AGI/GPT based on the DIKWP XV: An Instruction Set for Civial Judgment Document Analysis
April 2023
DOI: 10.13140/RG.2.2.36168.39686
Evaluation on AGI/GPT based on the DIKWP XV:
An Instruction Set for Civial Judgment Document Parsing
基于DIKWP的AGI/GPT测评系列十五：
民事裁判文书解析指令集
Chengxiang Ren1, Yingtian Mei2, Xiaoying Tan3, Yucong Duan4, Dawei Ding5*
d202220044@xs.ustb.edu.cn1, myingtian@126.com2, tanxiaoing@cjbdi.com3, duanyucong@hotmail.com4, dingdawei@ustb.edu.cn4
China Justice Big Data Institute CO. Ltd1,3
University of Science and Technology Beijing1,5
AGI-AIGC-GPT Test and Evaluation DIKWP (Global) Lab, Hainan University2,4
摘要：以Transformer架构为主流大型语言模型在意图理解和逻辑推理等方面展示出通用人工智能的迹象，相关产品与传统软件系统甚至人类创作的场景存在着功能的交叉重叠，正在取代司法裁判等领域大量原本需要处理专用任务的模型算法，探索和评估此类AGI/GPT模型的能力边界正在成为关注问题。目前现有的针对AGI/GPT类模型的测试与评价方法对同样的结果的解读过于主观，缺乏一个有效、客观、完整的评测体系，与现有关于系统和软件质量的评价方法也存在差异。面向为AGI/GPT类技术的能力提供一个相对独立、完整、一致、以人的智能为中心的测试框架的需求，本研究在DIKWP框架的评价测试原则的基础上，构建了一套案由无关的民事裁判文书解析任务指令文本集，指令包含详细的任务信息，表达民事裁判文书解析各任务要求，以人类相对习惯的方式与AGI/GPT类模型进行交互。进一步对AGI/GPT类模型在法律领域多层次、体系化场景下的功能、性能与质量测评进行了整体分析，旨在为AGI/GPT类模型在专业性、多层次、体系化场景下的评测提供设计实施思路。
关键词：AGI, GPT, DIKWP, 民事裁判文书, 评测, 指令文本
1. 研究背景
近些年学者和工程技术人员在以Transformer架构[1]为主流的大型语言模型（Large Language Models, LLMs）领域投入了大量资源[2-4]，发现以涌现性为代表的一系列现象[5]，包括通用性的泛化[6]、指令遵从、思维链（Chaing of Thought）少样本以及少样本/零样本上下文学习能力[7-8]等，显示AGI/GPT类模型可在意图理解和逻辑推理等方面展示出通用人工智能（Artificial General Intelligence, AGI）的迹象。以基于生成式预训练Transformer（Generative Pretrained Transformer，GPT）的ChatGPT[9]为代表，基于此类技术的GPT4、Claude、Open Assistant、Stanford Alpaca[10]、ChatGLM[4]、MOSS、SenseChat、百度千言，阿里通义千问等新一代对话产品和模型，正在从以文本为主的自然语言理解和生成任务多模态、实时性、现实性等维度快速走向更加通用和广泛的任务场景[11]，给图像、视频和3D模型生成等AIGC领域带来了新的突破[12-13]，并且以AGI/GPT类模型为核心，可以创建Toolformer[14]、HuggingGPT[15]等具备使用工具与API的应用，以及AutoGPT等自主人工智能程序。医疗、编程[16]、数学、法律、数字创意等具体人类工作领域的相关研究显示在众多任务上此类模型和产品可以达到类似人类专家的水平。
基于AGI/GPT类模型的产品和应用通常以多轮对话作为交互方式，可以理解并接受广泛的自然语言等形式的指令文本作为输入内容。相对于GUI和CLI界面，大大减少了对用户交互方式和交互内容的限制，使相关产品应用于前所未有的广泛场景，每个场景包含了多种多样的任务，对评估此类模型的能力边界构成了巨大挑战。现有的针对AGI/GPT类模型的测试与评价方法主要体现为零碎、发散和过于重视经验的问题交互，且对同样的结果的解读过于主观，缺乏一个有效、客观、完整的评测体系。另一方面，虽然明确的工程需求使得关键领域中既有系统和软件工程质量评价体系仍将发挥着不可替代的作用，但AGI/GPT类模型的应用与传统软件系统甚至人类创作的场景存在着功能的交叉重叠，替代了大量原本需要处理专用任务的模型算法。与处理专用任务的算法相比，通用任务处理（或通用人工智能）类的模型可以从系统和整体的角度，一般性地辅助人类提升感知、认知、决策的执行的能力，通过人机交互方式执行对内容的分析理解、转换生成和解释评估任务，可以与人类共享更多的情境上下文，覆盖数据、信息、知识、智慧在内的各层次，与现有系统和软件的质量评价存在差异。在数据、信息、知识、智慧和意图（DIKWP）框架[17-18]的基础上，Y.Li等人提出针对AGI/GPT类模型的测试同时包含模型对DIKWP自身知识概念的理解以及运用DIKWP解决各领域问题的能力。相关测试在确认模型对DIKWP具有一定程度的概念理解能力的基础上，将自然语言蕴含的信息映射到DIKWP模型与表达体系上，通过在DIKWP体系上的有序交互，评价AGI相关模型产品在数据、信息、知识、智慧和意图层的抽取、分析、理解、运用和解释能力。相关测试报告显示有关框架对各类AGI/GPT模型的产品具有广泛适用性[19-21]，并初步应用至教育[22]、医学[23]和金融[24]等领域，为AGI和GPT等技术的评价提供了一个相对独立、完整、一致、以人的智能为中心的测评方法。
我国在智慧法院建设的进程中走在世界前列，推进审判体系和审判能力现代化是当前深化智慧法院建设的重要内容[25]。司法裁判作为法院核心业务，实现全方位智能服务是深化智慧法院建设重要的技术要求之一。当前在司法裁判领域的人工智能研究已涉及案件认知[26]、事实认定[27]、裁判推理[29]、说理生成[30]、类案检索与推荐[32]、司法领域舆情监测[33]等领域，正向着”计算法学”[34-35]实践化的方向进步。受人们法制和权利意识增强、司法便民政策的完善的影响，以民事诉讼为主体的司法诉讼案件快速增长，在这种背景下，减轻法官审理办案压力，缓解案多人少问题成为司法领域人工智能技术的应用的重要目标之一。另一方面，司法人工智能技术广泛应用的同时也不应当忽视维护司法的公平正义，提高司法服务质量。全国性的历史裁判文书的公开为司法领域人工智能提供了充足的样本，促进了司法裁判领域人工智能技术的进步。民事裁判文书解析任务主要针对民事判决书等案件文书，定位于将裁判文书中的案件基本信息、审理情况、诉称辩称等文本通过一系列文本处理任务，在尽可能保持语义的情况下整理成更友好易读的的数据和信息形式，可以有效辅助对民事案件要素的进一步分析，对于梳理案情具有很强的实用价值，也是类案检索和裁判推理等研究的基础工作。当前民事裁判文书解析任务通常采用一系列基于规则或者专有模型的方法，为文书结构分解、案件基本信息抽取、诉辩事实理由观点分析等任务分别设计规则或者模型算法，理解能力弱，面对质量参差不齐的民事裁判文书，变通性差，系统集成复杂，影响结果可读性。AGI/GPT类模型带来的技术范式转移，为贯穿民事裁判文书解析整个场景提供了一站式解决的可能。但在当前AGI/GPT类模型的能力水平下，为发挥在民事裁判文书解析场景中的能力，仍依赖于为各种任务精心设计各类提示信息，独立于各类民事案件法律问题和案由的途径仍需继续探索，急需形成能够处理民事裁判文书数据、信息、知识等各层面形式和语义的完整的、体系化的方案。
本文针对AGI/GPT类模型在民事裁判文书解析场景的应用问题，从目的性、体系性、普适性、人本性、全面性和可操作性出发，首先在归纳总结任务设计和指令设计原则的基础上，将民事裁判文书解析任务分解为裁判文书重排、文本语义理解、案件数据抽取和案件信息关联四个阶段，然后设计构建覆盖民事裁判文书各类数据和信息解析任务的一套描述详细、案由无关的指令集，并在样例文书上给出参考结果，然后分析了基于民事裁判文书指令集对AGI/GPT类模型在民事裁判文书解析场景的进行体系化、完整化的测试与评价的可能途径，结合AGI/GPT类模型与人工审核开发了一个最小可行的预标注程序界面。所提指令集可以为AGI/GPT类模型在民事案件解析场景下的应用提供能够完成所有任务的实现方式，针对解析结果的表示方式和生成过程可为多任务场景下的AGI/GPT类模型的任务处理能力提供实现途径，对解析结果的测试与评价路线分析为AGI/GPT类模型在法律领域多层次、体系化场景下的功能、性能与质量测评进行了整体分析，可为AGI/GPT类模型在通用任务处理以及领域场景应用提供有益参考。
本文后续内容安排如下：第 2节介绍了完成民事裁判文书解析场景下各阶段任务的指令集，并给出参考数据，以Anthropic的Claude为参考模型，给出样例结果。第3节归纳探讨了基于民事裁判文书解析指令集构建评价与测试体系的要素构成与实施途径并给出指令集抽取预标注的参考实现。第 4节总结当前工作并提出后续改进思路。
2. 民事裁判文书解析指令集
从裁判文书网中随机选择一篇信用卡纠纷案例，在不影响主要内容的情况下，对相关文书中涉及到的人名及案做进一步处理。文书共计1318字，长度适中，文书各部分内容相对完整，情节较为简单清晰。本部分中原始文书以及中间阶段的人工参照数据以蓝色信息框的形式展示，给AGI/GPT模型的指令以绿色提示框的形式展示，参照Claude模型的结果以橙色警告框的形式展示（指示该部分文本可能存在错误）。同时，每个文本框标题以多级路径的形式显示了该部分文本或半结构化数据在民事裁判文书解析结果中所在的位置，所有结果共同构成对一篇民事裁判文书的完整解析结果，结果易于被重构成计算机可识别的JSON格式，用户可以通过查询方式获得对应各个结果项。
每个指令包含详细而尽可能准确的任务信息，详细表达任务需求，便于人类以清晰的方式与AGI/GPT类模型进行交互。对于每个任务，采用了相对容易获得的Anthropic的Claude产品给出参考结果，便于直接观察现有AGI/GPT产品的任务完成情况。出于排版原因，本文对Claude网页界面中含有的\xa0等不可见字符进行了去除，对表格、代码、公式等富文本结果进行了渲染处理，相关处理过程不改变模型的实际输出。
|
广东省佛山市禅城区人民法院 |
2.1裁判文书重排
裁判文书重排用于提取案件中案件基本信息、审理情况、原告诉称、被告辩称等成分，并在重排过程中完成对各自成分进行无损的概括，形成结构清楚的文书结构。裁判文书重排任务的描述长度通常较长（比如指令537字，完整的文书1318字，需要向模型一次性输入1855字），需要模型具有长文本输入和处理等能力。针对该任务，可以用输出成功率、完整性、吞吐量进行模型能力的评价，针对任务的结果，分别计算案件基本信息等部分各自的识别正确率、语义完整性、相对参考答案的Rouge 1/2/L、BERT Score等指标。
|
指令：你是一个司法裁判文书分段器，实现根据内容及结构识别每句话在文书中所属主题的任务，在返回结果过程不要丢失输入文书中的任何一句话。 |
|
### 案件基本信息 |
|
### 案件基本信息 |
基于生成的带有有标题的Markdown文本，可以通过程序方法或者直接使用AGI/GPT类模型，将文书中的审理情况、原告诉称、被告辩称、审理查明、本院认为、裁判结果、审判人员信息等字段转换为JSON格式，以进行更细致的评价。
|
- 广东省佛山市禅城区人民法院 |
2.2案件数据抽取
相关指令要求民事裁判文书中的案件基本信息、审理情况、法条引用和审判人员信息解析为半结构化数据，输出格式为计算机可处理的JSON格式，语义类型用键名表示，语义值用值表示，能够被相应的软件程序成功进行反序列化，生成半结构化的字典。在输入模型时，除一句话输出完整指令，还可以在多轮对话环境中引用上文简化交互方式。除忠实性、有用性、无害性，这部分测试结果还可以从调用成功性、数据字段提取的全面性、规范性、准确性、类型正确性等角度进行评价。比如对于法条信息，格式化为法律法规名称，条文编号（条、款、项）等字段，尽可能以整型的形式返回条、款、项数值，如果没有则字段设为null。
|
指令：从下面Markdown表格中按照主题抽取原文中的各类信息，以JSON字典的格式输出，其中一级字典的键名为主题名，主题名下面包含各类别信息。 |
|
{ |
|
指令：你是一个能够输出JSON的信息抽取器，从下面Markdown表格中抽取案件审理中的各种信息，以JSON字典的格式输出，JSON数据放在Markdown的CodeBlock中。 |
|
{ |
|
指令：提取下面文本中包含的法条信息，以JSON数组的格式输出。 要求： 1. 数组的每个元素是一个字典，字典包括法律法规名称、条文编号两个属性，其中条文编号为”第XX条第XX款”的形式。 2. 如果同一法律法规名称有多个条款，则在输出时分成多条 输入： {{input}} 结果： |
|
[ |
|
指令：提取下面文本中包含的审判人员等相关信息，以JSON数组的格式输出。 要求： 数组的每个元素是一个字典，字典包括属性名和属性值两个属性。 属性名为审判员、书记员、审判日期等字段，如果没有则对应属性不输出 输入： {{审判人员信息.文本}} 结果： |
|
[ |
2.3文本语义理解
文本语义理解用于将复杂的案件情况和事件描述拆分和转换成相对简单、结构清晰、要素完整的句子，在过程中不改变语句原本意义，也不损失原文对应部分的语义信息，适用于审理情况、原告诉称、被告辩称等一系列以情节连贯的自然语句为主体构成的段落。相关指令要求以有序列表的形式列出所整理出的语句，并且将同一句子中涉及多笔费用信息，按照事实认定的要求，拆分成各条。文本语义理解阶段使用统一指令，依次在前文提取出的案件基本信息、审理情况、原告诉称、被告辩称、审理查明、本院认为、裁判结果、法条引用和审判人员信息段落中重复调用，分别为各个部分生成语义信息的列表。从DIKWP框架的角度，可以理解为将每条内容转换成表达一个直接的数据、信息、知识或者意图的能力。
对此类任务的评测，除忠实性外，还包括分析粒度是否足够详细，简洁性、抵抗句法错误等主面，客观指标可以采用文本相似度，Rouge1/2/L，BERT Score，基于各项内容计算本部分各构成项整体的准确率、召回率。
|
指令：你是一个专业而忠实的句子拆条器，负责将复杂句子拆分成多个结构简单但内容完整的句子，不要丢掉原句子中的任何一条信息。 |
|
1. 广东省佛山市禅城区人民法院受理一起民事纠纷案。 |
|
原告广州银行股份有限公司佛山分行与被告黄某某存在信用卡纠纷, 并提起诉讼。 本院于2022年3月24日立案受理该案。 本院适用小额诉讼程序对该案进行公开开庭审理。 原告委托诉讼代理人杨某某和被告黄某某到庭参加诉讼。 本案目前已经审理终结。 原告因被告拖欠信用卡欠款,根据违约规定向法院提出诉讼请求。 请求法院判令被告向原告偿还信用卡卡号6250 6427的欠款本金34464.01元。 请求法院判令被告向原告支付卡号6250 6427的利息3390.99元。 请求法院判令被告向原告支付卡号6250 6427的违约金4299.13元。 请求法院判令被告向原告支付卡号6250 6427的手续费1547.21元。 上述费用暂计至2020年5月5日,之后至实际清偿日的利息、违约金等按领用协议的约定计至实际清偿日止。 请求法院判令被告承担本案全部诉讼费用。 |
|
1. 原告因被告拖欠信用卡欠款,构成违约。 2. 原告根据违约为由,向法院提出诉讼请求。 3. 请求一:判令被告向原告偿还信用卡卡号6250 6427的欠款本金34464.01元。 4. 请求一:判令被告向原告支付卡号6250 6427的利息3390.99元。 5. 请求一:判令被告向原告支付卡号6250 6427的违约金4299.13元。 6. 请求一:判令被告向原告支付卡号6250 6427的手续费1547.21元。 7. 上述各项费用暂计至2020年5月5日,此后至实际清偿日的利息、违约金等按领用协议的约定计至实际清偿日止。 8. 请求二:本案的全部诉讼费用由被告承担。 |
|
被告在答辩意见中表示对欠款金额不存在异议。 |
|
1. 原告根据诉讼请求,依法提交证据。 2. 被告对原告提交的证据无异议。 3. 法院确认原告提交的证据,并在法庭记录中备注证明。 4. 法院另查明:原告主张被告从2019年10月29日开始逾期还款。 5. 法院确认原告主张被告从2019年10月29日开始逾期还款。 |
|
1. 本院认为:被告使用信用卡消费后未按时还款,构成违约。 2. 原告要求被告立即偿还拖欠的信用卡本金的主张应予支持。 3. 原告要求被告支付诉请所述各项利息与费用,依法有据,应予支持。 4. 法院根据原告提供的证据以及公平合理原则,确认支持暂计至2020年5月5日止的利息与各项费用总计为4365元。 5. 2020年5月5日之后的利息以尚欠本金为基数,按日万分之五计算。 6. 因原告要求一次性收取全部欠款,法院不再支持2020年5月5日之后至实际清偿日的违约金。 7. 如果被告在2020年5月5日之后有还款,则按照相关规定进行冲还。 8. 1-90天(含)内还款的,按照先利息或各项费用,后本金的顺序进行冲还。 9. 91天以上还款的,按照先本金,后利息或各项费用的顺序进行冲还。 10. 最终清算本案债权债务时,按照上述规则直接扣减已还款项。 |
|
1. 判决被告黄某某在本判决发生法律效力之日起十日内向原告广州银行股份有限公司佛山分行支付信用卡欠款本金34464.01元。 2. 判决被告黄某某在本判决发生法律效力之日起十日内向原告广州银行股份有限公司佛山分行支付利息等共计4365元。 3. 上述利息暂计至2020年5月5日,之后至实际清偿日的利息以尚欠本金为基数,按日万分之五计算。 4. 驳回原告广州银行股份有限公司佛山分行的其他诉讼请求。 5. 如果被告未在指定期间内履行支付金钱义务,应当依法加倍支付迟延履行期间的债务利息。 6. 案件受理费446元,由被告黄某某负担。 7. 该费用原告已预交,法院不再退还,被告应在本判决生效后十日内直接支付原告。 8. 本判决为终审判决。 |
裁判结果部分会重点关注法院对当事人请求的判决支持情况，为此，设计指令在抽取裁判结果各细项的同时，对应给出法院判决支持情况，以三列表格的形式给出结果。模型输出的支持原告、支持被告、驳回请求、部分支持等结果，可以从分类准确性等角度进行判别。
|
指令：你是一个法律内容理解器，根据文本中的原文，抽取出裁判结果详情，以Markdown表格的形式输出。 |
|
| 序号裁判结果支持情况1被告黄某某在本判决发生法律效力之日起十日内向原告广州银行股份有限公司佛山分行支付信用卡欠款本金34464.01元以及利息等共计4365元(暂计至2020年5月5日,此后至实际清偿之日止的利息以尚欠本金为基数按日利率万分之五计算);支持原告2驳回原告广州银行股份有限公司佛山分行的其他诉讼请求。驳回原告请求3如被告未按本判决指定的期间履行给付金钱义务,应当依照《中华人民共和国民事诉讼法》第二百六十条之规定,加倍支付迟延履行期间的债务利息。部分支持4案件受理费446元,由被告黄某某负担。该费用原告已预交,本院不再退还,由被告于本判决生效后十日内迳付原告。支持原告5本判决为终审判决。
|
2.4案件信息关联
测试正确关联各方信息和数据的能力。指令集要求依次将原有列表形式展示的项在二维表格空间中进行扩展，以列表的形式给出各字段要求模型填充对应的字段。指令集可以按照指令提示信息的详细程度分为简略和单样本两个版本。 可从表格项的正确性、语义和情节分离的程度、句子流畅性、公式语义的正确性、计算结果的正确性等进行结果评价。针对公式本身的正确性，需要提取结果中的LaTeX公式，识别其中公式中各符号项的关系以检查与标准公式的语义的符合性。
2.4.1原告诉称
|
指令：你是一个能够句子分解和内容解析器，从下面包含原告诉讼请求的文本中提取出原告每一个具体的讼诉请求。 |
|
| 诉讼请求内容概括主题关键词判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元偿还信用卡欠款本金34464.01元偿还债务信用卡,本金,欠款判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元偿还信用卡利息3390.99元偿还债务信用卡,利息判令被告向原告偿还信用卡(卡号:6250 6427)违约金4299.13元偿还信用卡违约金4299.13元违约责任信用卡,违约金判令被告向原告偿还信用卡(卡号:6250 6427)手续费1547.21元偿还信用卡手续费1547.21元偿还债务信用卡,手续费本案的全部诉讼费用由被告承担被告承担全部诉讼费用诉讼费责任诉讼费用 计算公式: 利息=本金×利率×天数÷365 违约金=本金×违约比率×违约天数 手续费=操作次数×每次手续费标准 |
|
指令：你是一个能够句子分解和内容解析器，从下面包含原告诉讼请求的文本中提取出原告每一个具体的讼诉请求 |
|
| 诉讼请求内容概括主题关键词判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元偿还信用卡欠款本金34464.01元偿还债务信用卡,本金,欠款判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元偿还信用卡利息3390.99元偿还债务信用卡,利息判令被告向原告偿还信用卡(卡号:6250 6427)违约金4299.13元偿还信用卡违约金4299.13元违约责任信用卡,违约金判令被告向原告偿还信用卡(卡号:6250 6427)手续费1547.21元偿还信用卡手续费1547.21元偿还债务信用卡,手续费本案的全部诉讼费用由被告承担被告承担全部诉讼费用诉讼费责任诉讼费用 计算公式: 利息=本金×利率×天数÷365 违约金=本金×违约比率×违约天数 手续费=操作次数×每次手续费标准
|
2.4.2被告辩称
使用案由无关、内容无关的指令，以原告诉称表格和被告辩称的列表项作为基础输入，将被告辩称的内容填补到以原告诉称为基础的表格项中，然后输出被告辩称表格。任务可用于测试模型关联不同来源数据的能力。
|
提示：你是一个司法内容理解器并且能够输出Markdown表格，接下来根据原告的诉讼请求与被告的答复，完成一个填表任务并输出。 |
|
| 原告请求项概括被告意见相关证据判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元偿还信用卡欠款本金34464.01元无异议null判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元偿还信用卡利息3390.99元无异议null判令被告向原告偿还信用卡(卡号:6250 6427)违约金4299.13元偿还信用卡违约金4299.13元无异议null判令被告向原告偿还信用卡(卡号:6250 6427)手续费1547.21元偿还信用卡手续费1547.21元无异议null本案的全部诉讼费用由被告承担被告承担全部诉讼费用无异议null
|
2.4.3审理查明
指令以被告辩称表格和审理查明的文本列表作为输入，然后被充证据、认定等情况到以被告辩称为基础的表格项中。
指令：你是一个法律内容理解器，根据原告给的每条信息和被告给的每条信息，公正地对原告和被告所说的内容进行判断，输出证据和请求的确认情况，以Markdown表格的形式输出。 |
|
| 原告请求项概括被告意见相关证据证据认定情况支持情况判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元偿还信用卡欠款本金34464.01元无异议null有效支持判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元偿还信用卡利息3390.99元无异议null有效支持判令被告向原告偿还信用卡(卡号:6250 6427)违约金4299.13元偿还信用卡违约金4299.13元无异议null有效支持判令被告向原告偿还信用卡(卡号:6250 6427)手续费1547.21元偿还信用卡手续费1547.21元无异议null有效支持本案的全部诉讼费用由被告承担被告承担全部诉讼费用无异议null有效支持
|
2.4.4本院认为
|
指令：你是一个法律内容理解器，根据文本中的原文，抽取出法院的观点，以Markdown表格的形式输出。 要求： 输出是一个Markdown的表格，表格的每一列是法院针对每个问题的观点。如果一条原文中包含多个问题，则输出成多个列； 第一列是序号，从1开始递增； 第二列是来自原文中原告或者被告的主张和请求，列名是“相关主张”。 第三列是原告或者被告的主张和请求时，法院认定的的事实或者理由，列名是“事实认定”。 第四列是法院的支持情况（列名“支持情况”），如果支持则填入“支持”、如果不支持则填入“不支持”，不能确定填“其他” 法院审理查明： {{审理查明.表格}} 法院认为： {{本院认为.列表}} 结果： |
|
| 序号相关主张事实认定支持情况1判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元被告持卡消费后未按期还款,构成违约支持2判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元与各项费用总计4299.13元原告主张被告依据相关协议约定支付诉请所述的各项利费,依法有据支持3暂计至2020年5月5日止的利息与各项费用总计4365元本院根据原告提供的证据以及公平合理原则,经核算支持4此后至实际清偿之日止的利息以尚欠本金为基数按日万分之五计收本院根据原告提供的证据以及公平合理原则,经核算支持5对2020年5月5日之后至实际清偿之日止的违约金不再支持原告主张一次性计收全部欠款不支持6如果本案第一项诉请主张的暂计日2020年5月5日之后被告有还款的,则按照《商业银行信用卡业务监督管理办法》第五十七条规定,在逾期1-90天(含)内还款的,按照先应收利息或各项费用、后本金的顺序进行冲还;逾期91天以上还款的,按照先本金、后应收利息或各项费用的顺序进行冲还。法院依据规定,对还款的顺序提出要求支持7原、被告双方在最终清算本案债权债务时按照上述规则直接扣减已还款项。法院对清算本案债权债务的方式提出要求支持
|
2.4.5裁判结果
|
指令：你是一个法律内容理解器，根据法院认为相关联的表格，与对应的裁判结果相关联，以Markdown表格的形式输出。 |
|
| 序号相关主张裁判结果支持情况1判令被告向原告偿还信用卡(卡号:6250 6427)欠款本金34464.01元被告黄某某在本判决发生法律效力之日起十日内向原告广州银行股份有限公司佛山分行支付信用卡欠款本金34464.01元支持原告2判令被告向原告偿还信用卡(卡号:6250 6427)利息3390.99元与各项费用总计4299.13元以及利息等共计4365元(暂计至2020年5月5日)支持原告3暂计至2020年5月5日止的利息与各项费用总计4365元 支持原告4此后至实际清偿之日止的利息以尚欠本金为基数按日万分之五计收 支持原告5对2020年5月5日之后至实际清偿之日止的违约金不再支持驳回原告广州银行股份有限公司佛山分行的其他诉讼请求。驳回请求6如果本案第一项诉请主张的暂计日2020年5月5日之后被告有还款的,则按照《商业银行信用卡业务监督管理办法》第五十七条规定,在逾期1-90天(含)内还款的,按照先应收利息或各项费用、后本金的顺序进行冲还;逾期91天以上还款的,按照先本金、后应收利息或各项费用的顺序进行冲还。 支持原告7原、被告双方在最终清算本案债权债务时按照上述规则直接扣减已还款项。 支持原告
|
3. 测试与评价路线
基于AGI/GPT类模型的民事裁判文书解析场景可以理解为，通过一系列与AGI/GPT类模型交互的具体子任务，逐步推进，从民事判决书文本中忠实、完整地解析出案件基本信息、审理情况、原告诉称、被告辩称等成分，并对各个成分进行数据和信息层面结构和语义的分解，将结果分解至各个字段中。具体包括：
针对民事裁判文书解析场景评测的总体目标是基于上述指令集评价和测试模型完成各个任务的功能、性能的质量情况，并结合评价AGI/GPT类模型结果时常用的有效性、可靠性等方式。各个任务也允许使用非AGI/GPT类模型的方式完成。
针对民事裁判文书解析场景，可以根据符合人类分析过程的思想分解场景为各层次任务需求，形成裁判文书重排、案件数据抽取、文本语义理解、案件信息关联四个构成部分，并针对每个部分形成案件基本情况等具体的抽取任务，针对每个任务设计指令或复用指令，集合形成指令集。指令集主要用于描述解决领域问题的要求，避免经过事先的训练。
通过执行执行集形成任务结果，可以对照人类标准进行测试与评价，从指标类型、准确率、文本相似度、是否成功、执行速度、一致性、可复现性、BERT Score, Rouge 1/2/L、数值相等等维度形成指标项和指标体系。针对相关结果的指标评价也适用于非AGI/GPT类模型的方式。
考虑到人工交叉审核等因素，包括AGI/GPT类模型在内的软件和系统在完成民事裁判文书解析完整场景任务时，可以采用两类执行模式：
点对点模式(Point to Point，P2P)，每个任务单独进行评测，针对每个任务设计完整的指令模板，指令模板中的变量替换为人工标注的正确数据，将各个任务的结果集合形成完整解析结果。比如针对文本语义理解阶段的任务，案件基本信息段落为人工标注的正确结果而非模型在裁判文书重排阶段任务的输出。本模式适合于人在回路的设计，其中人类参与民事裁判文书解析全过程的核验工作。
端到端模式(End to End, E2E)，用户仅需要输入裁判文书，后续每个任务以流水线方式，根据任务依赖关系图，接受被测试AGI/GPT类模型或者传统软件系统中间阶段的输入，直到返回全部结果。本模式下中间任务受到模型在所依赖任务上的结果正确性，可以更完整地测试AGI/GPT类模型在民事裁判文书解析场景下的表现。在E2E模式下可以更好地测试多轮对话的性能：完成中间任务所用的指令可以每次均以完整的形式描述，也可以通过省略和引用的形式由模型理解用户对指令之间依赖信息的理解。
针对民事裁判文书解析场景的人工案件检验需求设计了面向交互式环境和预标注的最小可行系统（Minimum Viable System）。见图1。系统采用流水线结构，逐步解析裁判文书，可以一次由模型全部调用，也可以人工修改上游结果，通过接口自动更新下游结果。列表和字段信息可以拖动和编辑修改，字段信息显示信息所在路径及解析结果。
(a) 裁判文书及重排结果界面
|
|
(b) 案件数据抽取结果界面 | (c) 文本语义理解结果界面 |
图1: 民事裁判文书解析场景辅助程序
4. 总结与展望
面向AGI/GPT类模型在更加完整、体系和复杂的任务场景下的评价与测试问题，本文设计并提供了一套针对民事裁判文书解析的任务指令集，形成覆盖裁判文书重排、案件数据抽取、文本语义理解、案件信息关联等子场景的完整任务描述。后续可以丰富人工标注样例，在更多民事公开文书、更多AGT/GPT类模型上进行对比和交叉测试。通过测试与评价路线分析，指令集支持以P2P或者E2E等模式以单轮或者带有情境上下文的方式执行，通过捕获AGI/GPT类模型在民事裁判文书解析场景的详细结果，后续可针对AGI/GPT类模型在民事裁判文书解析场景的全面评测问题设计功能、性能、质量等方面的评测指标项，形成自动化评测套件。
民事裁判文书分解任务以在数据和信息层面形式和语义的处理为主，覆盖了代码/配置文件生成、格式转换、文本语义理解和转换、信息关联等维度。后续将在本文框架的基础上，挖掘民事裁判问题，探讨从事实认定和裁判说理背后的法律知识等因素的影响，全面覆盖DIKWP体系的各个层面，并分析民事裁判问题上数据、信息、知识、智慧和意图之间的转换特点，对AGI/GPT类模型通用人工智能的能力进行更全面的评估。面对领域无关的文本结构解析系统的构建问题，后续将研究更加通用的任务需求和处理流程描述方法，从语言/数学/常识知识领域出发，持续改进民事裁判文书解析等领域基础场景的指令泛化性能。
5. 参考文献
[1] A. Vaswani et al., “Attention is all you need,” in Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, CA, USA, 2017, pp. 5998–6008.
[2] R. Thoppilan et al., “LaMDA: Language models for dialog applications,” CoRR, vol. abs/2201.08239, 2022.
[3] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023.
[4] Z. Du et al., “GLM: General language model pretraining with autoregressive blank infilling,” in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), ACL 2022, dublin, ireland, may 22-27, 2022, 2022, pp. 320–335.
[5] J. Kaplan et al., “Scaling laws for neural language models,” CoRR, vol. abs/2001.08361, 2020.
[6] H. W. Chung et al., “Scaling instruction-finetuned language models,” CoRR, vol. abs/2210.11416, 2022.
[7] T. B. Brown et al., “Language models are few-shot learners,” CoRR, vol. abs/2005.14165, 2020.
[8] J. Wei et al., “Finetuned language models are zero-shot learners,” in The tenth international conference on learning representations, ICLR 2022, virtual event, april 25-29, 2022, 2022.
[9] L. Ouyang et al., “Training language models to follow instructions with human feedback,” CoRR, vol. abs/2203.02155, 2022.
[10] R. Taori et al., “Stanford alpaca: An instruction-following LLaMA model,” GitHub repository, 2023. [Online]. Available: https://crfm.stanford.edu/2023/03/13/alpaca.html. [Accessed: 24-Apr-2023].
[11] S. Bubeck et al., “Sparks of artificial general intelligence: Early experiments with GPT-4,” CoRR, vol. abs/2303.12712, 2023.
[12] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual ChatGPT: Talking, drawing and editing with visual foundation models,” CoRR, vol. abs/2303.04671, 2023.
[13] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models.” 2023.
[14] T. Schick et al., “Toolformer: Language models can teach themselves to use tools,” CoRR, vol. abs/2302.04761, 2023.
[15] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in HuggingFace,” CoRR, vol. abs/2303.17580, 2023.
[16] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig, “Language models of code are few-shot commonsense learners,” CoRR, vol. abs/2210.07128, 2022.
[17] Y. Li, Y. Duan, Z. Maamar, H. Che, A.-B. Spulber, and S. Fuentes, “Swarm differential privacy for purpose-driven data-information-knowledge-wisdom architecture,” Mob. Inf. Syst., vol. 2021, pp. 6671628:1–6671628:15, 2021.
[18] Y. Mei, Y. Duan, L. Chen, Z. Feng, L. Yu, and Z. Guo, “Purpose driven disputation modeling, analysis and resolution based on DIKWP graphs,” in 24th IEEE int conf on high performance computing & communications; 8th int conf on data science & systems; 20th int conf on smart city; 8th int conf on dependability in sensor, cloud & big data systems & application, HPCC/DSS/SmartCity/DependSys 2022, hainan, china, december 18-20, 2022, 2022, pp. 2118–2125.
[19] Y. Li and Y. Duan, “The performance of GPT-4 on education domain with DIKWP analysis,” Apr-2023.
[20] Y. L. Chengxiang Ren Lixu Shao and Y. Duan, “Evaluation on AGI/GPT based on the DIKWP for: Anthropic’s claude,” Apr-2023.
[21] Y. L. Chengxiang Ren and Y. Duan, “Evaluation on AGI/GPT based on the DIKWP for ERNIE bot,” Apr-2023.
[22] Y. Li and Y. Duan, “The performance of GPT-4 on education domain with DIKWP analysis,” Apr-2023.
[23] Y. Li, Y. Zhou, Y. Mei, Z. Hu, and Y. Duan, “The capability evaluation of GPT-4 on medical/medicine domain with DIKWP analysis,” Apr-2023.
[24] F. Tang, Y. Duan, E. Dou, and Y. Yang, “The capability evaluation of GPT-4 and claude on financial domain with DIKWP+TIF analysis.” Apr-2023.
[25] 许建峰., 黄国栋., 柳叶., “全面建设智慧法院 促进审判体系和审判能力现代化,” 行政管理改革, no. 5, pp. 86–91, 2019.
[26] 李锦烨., 黄瑞章., 秦永彬., 陈艳平., 田小瑜., “基于反绎学习的裁判文书量刑情节识别,” 计算机应用, vol. 42, no. 6, pp. 1802–1807, 2022.
[27] S. Surana, S. Dembla, and P. Bihani, “Identifying contradictions in the legal proceedings using natural language models,” SN Comput. Sci., vol. 3, no. 3, p. 187, 2022.
[28] L. Zhao et al., “Legal judgment prediction with multiple perspectives on civil cases,” in Artificial intelligence - first CAAI international conference, CICAI 2021, hangzhou, china, june 5-6, 2021, proceedings, part I, 2021, vol. 13069, pp. 712–723.
[29] Y. Feng, C. Li, and V. Ng, “Legal judgment prediction via event extraction with constraints,” in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), ACL 2022, dublin, ireland, may 22-27, 2022, 2022, pp. 648–664.
[30] W. Chen, J. Tian, L. Xiao, H. He, and Y. Jin, “A semantically consistent and syntactically variational encoder-decoder framework for paraphrase generation,” in Proceedings of the 28th international conference on computational linguistics, COLING 2020, barcelona, spain (online), december 8-13, 2020, 2020, pp. 1186–1198.
[31] B. Liu et al., “Investigating conversational agent action in legal case retrieval,” in Advances in information retrieval - 45th european conference on information retrieval, ECIR 2023, dublin, ireland, april 2-6, 2023, proceedings, part I, 2023, vol. 13980, pp. 622–635.
[32] B. Liu et al., “Query generation and buffer mechanism: Towards a better conversational agent for legal case retrieval,” Inf. Process. Manag., vol. 59, no. 5, p. 103051, 2022.
[33] 胡东瑶., “面向司法领域的舆情监测技术研究与系统实现,” 硕士, 哈尔滨工业大学, 2019.
[34] 申卫星. 刘云., “探索可计算的法律发展道路,” 浙江社会科学, no. 6, pp. 33–41+156, 2022.
[35] 左卫民., “中国计算法学的未来：审思与前瞻,” 清华法学, vol. 16, no. 3, pp. 196–208, 2022.
Evaluation on AGI/GPT based on the DIKWP XV:
An Instruction Set for Civial Judgment Document Parsing
Chengxiang Ren1, Yingtian Mei2, Xiaoying Tan3, Yucong Duan4, Dawei Ding5*
d202220044@xs.ustb.edu.cn1, myingtian@126.com2, tanxiaoing@cjbdi.com3, duanyucong@hotmail.com4, dingdawei@ustb.edu.cn4
China Justice Big Data Institute CO. Ltd1,3
University of Science and Technology Beijing1,5
AGI-AIGC-GPT Test and Evaluation DIKWP (Global) Lab, Hainan University2,4
Abstract: Large-scale language models with Transformer architecture as the mainstream show signs of general-purpose AI in areas such as intent understanding and logical reasoning, and related products with cross-functional overlap with traditional software systems and even human-created scenarios are replacing a large number of model algorithms that would otherwise need to handle specialized tasks in areas such as judicial adjudication, and exploring and evaluating the boundaries of the capabilities of such AGI/GPT type models are becoming issues of concern. The existing test and evaluation methods for AGI/GPT type models interpret the same results too subjectively and lack a valid, objective and complete evaluation system, which also differs from the existing evaluation methods about system and software quality. Facing the need to provide a relatively independent, complete, consistent, and human intelligence-centered testing framework for the capabilities of AGI/GPT type technologies, this study constructs a case-independent text set of civil adjudication document parsing task instructions based on the evaluation testing principles of the DIKWP framework. The instructions contain detailed task information expressing the requirements of each task of civil adjudication document parsing and interact with AGI/GPT type models in a manner relatively accustomed to humans. The overall analysis of the function, performance and quality measurement of AGI/GPT type models in multi-level and systematic scenarios in the legal field is further carried out, aiming to provide design and implementation ideas for the evaluation of AGI/GPT type models in professional, multi-level and systematic scenarios.
Keywords: AGI, GPT, DIKWP, Civil adjudication documents, Evaluation measurement, Command text
1. Research Background
In recent years, scholars and engineers have invested a lot of resources[1-3] in the field of Large Language Models (LLMs) dominated by Transformer architecture[4] and found a series of phenomena represented by emergent properties[5], including generalization of generality[6], instruction compliance, Chaing of Thought less samples, and less/zero sample context learning capabilities[7-8], showing that AGI/GPT type models can exhibit signs of Artificial General Intelligence (AGI) in areas such as intent understanding and logical reasoning.Represented by ChatGPT [9] based on Generative Pretrained Transformer (GPT), GPT4, Claude, Open Assistant, Stanford Alpaca[10], based on such technologies ChatGLM[4], MOSS, SenseChat, Baidu Thousand Words, Ali Tongyi Thousand Questions, and other next-generation conversational products and models are rapidly moving from text-based natural language understanding and generation tasks with dimensions of multimodality, real-time, and realism to more general and extensive task scenarios[11], bringing new AIGC domains such as image, video, and 3D model generation breakthroughs[12-13], and with AGI/GPT type models as the core, applications with tools and APIs for use such as Toolformer[14] and HuggingGPT[15], as well as autonomous AI programs such as AutoGPT, can be created. Relevant studies in specific human work areas such as medicine, programming[16], mathematics, law, and digital creativity have shown that such models and products can reach a level similar to that of human experts on numerous tasks.
Products and applications based on AGI/GPT type models typically use multiple rounds of dialogue as the interaction method, and can understand and accept a wide range of command text in the form of natural language and other forms as input content. Compared to GUI and CLI interfaces, the restrictions on how and what users can interact with are greatly reduced, allowing the relevant products to be applied to an unprecedentedly wide range of scenarios, each containing a wide variety of tasks that pose a significant challenge in assessing the boundaries of the capabilities of such models. The existing testing and evaluation methods for AGI/GPT type models mainly reflect fragmented, diffuse and overly empirical problem interactions, and the interpretation of the same results is too subjective, lacking a valid, objective and complete evaluation system. On the other hand, while clear engineering requirements make it clear that established systems and software engineering quality evaluation systems in critical areas will still play an irreplaceable role, the application of AGI/GPT type models has a functional crossover with traditional software systems and even human creation scenarios, replacing a large number of model algorithms that would otherwise need to handle specialized tasks. Compared to algorithms dealing with dedicated tasks, models in the generic task processing (or generic AI) category can generally assist humans in enhancing the ability to perform perception, cognition, and decision making from a systemic and holistic perspective, perform analytical understanding, transformation generation, and interpretation evaluation tasks of content through human-computer interaction, can share more contextual contexts with humans, cover various levels including data, information, knowledge, and intelligence, and differ from the quality evaluation of existing systems and software. Based on the Data, Information, Knowledge, Wisdom and Intention (DIKWP) framework[17-18], Y. Li et al. proposed that tests for AGI/GPT type models include both the model's understanding of the DIKWP's own knowledge concepts and the ability to apply the DIKWP to solve problems in various domains. Based on the confirmation that the model has a certain degree of conceptual understanding of DIKWP, the correlation test maps the information embedded in natural language onto the DIKWP model and expression system, and evaluates the ability of AGI related model products to extract, analyze, understand, apply and interpret data, information, knowledge, wisdom and intention layers through orderly interaction on the DIKWP system. Relevant test reports have shown the broad applicability of the framework in question to various types of AGI/GPT model products[19-21], with initial applications to education[22], medicine[23], and finance[24], providing a relatively independent, complete, consistent, and human intelligence-centered measure for the evaluation of technologies such as AGI and GPT.
China is at the forefront of the world in the process of wisdom court construction, and promoting the modernization of the trial system and trial capacity is an important element of the current deepening of wisdom court construction[25]. Judicial adjudication as the core business of the court, the realization of a full range of intelligent services is one of the important technical requirements for deepening the construction of smart courts. The current AI research in the field of judicial adjudication has been involved in case cognition[26], fact finding[27], decision reasoning[29], reasoning generation[30], case retrieval and recommendation[32], and monitoring of public opinion in the judicial field[33], and is progressing toward the practicalization of "computational jurisprudence" [34-35]. Influenced by people's increased awareness of the legal system and rights, and the improvement of judicial convenience policy, judicial litigation cases, mainly civil litigation, are growing rapidly. In this context, reducing the pressure on judges to hear and handle cases, and alleviating the problem of a large number of cases has become one of the important goals of the application of artificial intelligence technology in the judicial field. On the other hand, the wide application of judicial artificial intelligence technology should not neglect the maintenance of judicial justice and improve the quality of judicial services. The nationwide public availability of historical adjudication documents provides ample samples for AI in the judicial field and promotes the advancement of AI technology in judicial adjudication. The task of parsing civil adjudication documents mainly focuses on civil judgments and other case documents, and is located in a series of text processing tasks to organize the basic information of cases, trials, claims and arguments in adjudication documents into a more friendly and readable form of data and information while maintaining semantics as much as possible, which can effectively assist in further analysis of the elements of civil cases and has a strong practical value for sorting out the facts of cases, and is also the basis for research on case search and adjudication reasoning. The current civil adjudication documents parsing tasks usually adopt a series of rule-based or proprietary model approaches, designing rules or model algorithms separately for tasks such as decomposition of document structure, extraction of basic case information, analysis of litigation and defense facts and reasons and viewpoints, which are weak in comprehension, poor in variability and complex in system integration in the face of civil adjudication documents of varying quality, and affect the readability of results. The technology paradigm shift brought by AGI/GPT type models provides the possibility of one-stop solution across the entire scenario of civil adjudication document parsing. However, at the current level of capability of AGI/GPT class models, in order to bring into play the capability in the civil adjudication document parsing scenario, it still relies on elaborating various types of hints for various tasks, and the ways independent of various civil case legal issues and causes still need to continue to be explored, and there is an urgent need to form a complete and systematic solution capable of handling all levels of forms and semantics of civil adjudication document data, information and knowledge.
This paper addresses the application of AGI/GPT class model in the scenario of civil adjudication document parsing, and starts from purposeful, systematic, universal, human-oriented, comprehensive and operable, firstly, on the basis of summarizing the principles of task design and instruction design, decomposes the civil adjudication document parsing task into four stages of adjudication document rearrangement, text semantic understanding, case data extraction and case information association, and then Then, we analyze the possible ways to systematically and completely test and evaluate the AGI/GPT class model in the civil judgment document parsing scenario based on the instruction set of civil judgment document. A minimum feasible pre-labeling program interface is developed. The proposed instruction set can provide a way for AGI/GPT class models to perform all tasks in civil case parsing scenarios, and the representation and generation process of parsing results can provide a way for AGI/GPT class models to achieve task processing capabilities in multi-task scenarios. The analysis of testing and evaluation routes of parsing results provides an overall analysis of the functionality, performance and quality measurement of AGI/GPT class models in multi-level and systematic scenarios in the legal field, which can provide useful references for AGI/GPT class models in general task processing and domain scenarios.
This paper is organized as follows: Section 2 introduces the set of instructions for completing each stage of the tasks in the civil adjudication document parsing scenario, and gives reference data and sample results using Claude of Anthropic as the reference model. Section 3 summarizes and discusses the elements and implementation of the evaluation and testing system based on the instruction set for parsing civil adjudication documents and gives a reference implementation of the instruction set extraction pre-labeling. Section 4 summarizes the current work and proposes subsequent improvement ideas.
2. The set of instructions for the analysis of civil adjudication documents
A credit card dispute case was randomly selected from the referee's website, and the names of people and cases involved in the relevant instruments were further processed without affecting the main content. The total length of the instrument is 1318 words, which is moderate. The content of each part of the instrument is relatively complete, and the plot is relatively simple and clear. The original instrument and the manual reference data of the intermediate stages are presented in this part in the form of blue information boxes, the instructions given to the AGI/GPT model are presented in the form of green alert boxes, and the results of the reference Claude model are presented in the form of orange warning boxes (indicating that there may be errors in this part of the text). At the same time, the title of each text box shows the location of the part of text or semi-structured data in the parsing result of the civil judgment document in the form of a multi-level path, and all the results together constitute the complete parsing result of a civil judgment document, and the result is easily reconstructed into a computer-recognizable JSON format, and the user can obtain the corresponding individual result items by query.
Each instruction contains detailed and as accurate as possible task information, expressing the task requirements in detail and facilitating human interaction with AGI/GPT-like models in a clear manner. For each task, the Claude product of Anthropic, which is relatively easy to obtain, is used to give reference results and facilitate direct observation of task completion for existing AGI/GPT products. For typographic reasons, invisible characters such as \xa0 contained in the Claude web interface are removed in this paper, and rich text results such as tables, codes, and formulas are rendered, with the relevant processing not changing the actual output of the model.
|
Foshan Chancheng District People's Court, Guangdong Province May 17, 2020 |
2.1 Adjudicative reordering
Adjudicative document rearrangement is used to extract the basic case information, trial, plaintiff's claims, defendant's arguments and other components of the case, and to complete a non-destructive overview of the respective components in the rearrangement process to form a clearly structured document structure. The description length of the adjudication document rearrangement task is usually long (e.g., 537 words for an instruction and 1,318 words for a complete document, requiring a one-time input of 1,855 words to the model), requiring the model to have capabilities such as long text input and processing. For this task, the model capability can be evaluated in terms of output success rate, completeness, and throughput. For the results of the task, the respective recognition correctness, semantic completeness, Rouge 1/2/L relative to the reference answer, and BERT Score of the case basic information and other parts are calculated separately.
|
Instruction: You are a segmenter of judicial documents, implementing the task of identifying the subject to which each sentence belongs in the document based on its content and structure, without losing any sentence in the input document during the return of the result. Requirements: 5. the defendant's defense describes the overall situation and specifics of the defendant's appearance in court and submission of the defense; 6. the trial ascertained described the defendant's arguments followed by the court's organization of the parties to cross-examine, pre-identify and confirm the evidence and find the facts; 7. the court considered describes the specific content of "the court considered" after the hearing; 8. the citation of the law describes the court's opinion after the court, a combination of various accounts, relied on the law; 9. the result of the decision describes the various elements of the court's decision in accordance with the provisions of the law after the citation of the law. 10. adjudicator information describes the trial judge, the clerk, and the date of the decision that appeared during the trial of the case. {{referee}} Result: |
|
### Basic information of the case - The court according to the evidence provided by the plaintiff and the principle of fairness and reasonableness, after accounting, support the interest and fees until May 5, 2020, a total of 4365 yuan, and thereafter to the actual date of settlement of the interest on the basis of the principal amount still owed at five ten thousandths of the day.
|
|
###Basic information of the case (2022)Cantonese 0604 first-instance civil No.0000 |
Based on the generated Markdown text with headings, the fields in the instrument such as trial status, plaintiff's claims, defendant's arguments, trial findings, the Court's opinion, decision results, and trial staff information can be converted to JSON format for more detailed evaluation, either through procedural methods or directly using the AGI/GPT type model.
|
- Foshan Chancheng District People's Court, Guangdong Province |
2.2 Case data extraction
The relevant instructions require that the basic case information, trial status, legal references and trial staff information in the civil adjudication documents are parsed into semi-structured data, and the output format is a computer-processable JSON format with semantic types represented by key names and semantic values represented by values that can be successfully deserialized by the corresponding software program to generate a semi-structured dictionary. In addition to outputting complete instructions in a single sentence when inputting the model, the above can be referenced in a multi-round dialogue environment to simplify the interaction. In addition to fidelity, usefulness, and harmlessness, the results of this part of the test can be evaluated in terms of call success, comprehensiveness of data field extraction, regularity, accuracy, and type correctness. For example, for law information, format the fields as law and regulation name, article number (article, paragraph, item), etc., and return the article, paragraph, item value as an integer if possible, or set the field to null if there is none.
|
Command: Extract all kinds of information in the original text from the following Markdown table according to the topic, and output it in the format of JSON dictionary, where the key of the first-level dictionary is the topic name, and the information of each category is included under the topic name. Requirements: 1. the original text with complex content inside is split into individual entries according to each type of information; 2. if the plaintiff or defendant is a unit, extract the name of the unit, address, social credit code, principal person in charge, position of the principal person in charge, the name of the delegated litigation agent and its unit respectively, or set to empty if there is none; 3. if the plaintiff or the defendant is a natural person, extract the name, gender, date of birth, nationality, address and other information respectively; if not, set to empty Enter: {{basic case information. Text}} Result: |
|
{ "Court Information": { "Court Name": "Foshan Chancheng District People's Court, Guangdong Province", "Case Type": "Civil Judgment", "Case No.": (2022)Cantonese 0604 first-instance civil No.0000 }, "Plaintiff Information": { "Unit name": "Bank of Guangzhou Corporation Foshan Branch", "Address": "P33 first floor store, P32 first floor and second floor store, No. 63, Chaoan South Road, Chancheng District, Foshan City, Guangdong Province", "Social Credit Code": "914406005778646632", "Principal": "Hu", "Principal position": "President of the bank", "Attorney-in-fact": "Yang", "Attorney at Law": "Beijing Tianchi Juntai (Guangzhou) Law Firm" }, "Name": "Huang", "Gender": "Male", "Date of Birth": "April 1, 1987", "Ethnicity": "Han Chinese", "address": "Yunfu City, Guangdong Province, Yunan County" } } |
|
Instruction: You are an information extractor capable of outputting JSON, extracting various information from the following Markdown table for the case hearing and outputting it in the format of a JSON dictionary, with the JSON data placed in the CodeBlock of Markdown. Requirements: 1. complex content inside the original text is split into individual entries according to various types of information, and a dictionary can be set up again under each piece of information to form a nested structure; 2. output the name of the case, the name of the cause (that is, the type of case), the time of acceptance, the trial procedure (the trial procedure applied by law), whether the trial was held (the value is Boolean, true or false), the name of the person who appeared in court on the plaintiff's side and his role, the name of the person who appeared in court on the defendant's side and his role, whether the trial procedure was completed (the value is Boolean, true or false), and other information. If the corresponding information is missing, the value is null. 3. 3. Note that the name of the cause of action usually refers to the type of dispute or charge, whether it is a civil action, criminal action, or administrative action. Generally, it is XX case. 4. Use Chinese for the key names in the JSON dictionary. Enter: {{hearing. text}} Result: |
|
{ |
|
Command: Extract the law bar information contained in the following text and output it as a JSON array. Requirements: 1: 1. each element of the array is a dictionary, which includes two attributes: law and regulation name and article number, where the article number is in the form of "Article XX, Paragraph XX". 2. If there are more than one article in the same law or regulation name, the output is divided into multiple articles. Input: {{input}} Result: |
|
[ |
2.3 Text Semantic Understanding
Text semantic understanding is used to split and convert complex case situations and event descriptions into relatively simple, clearly structured and elementally complete sentences, without changing the original meaning of the statements or losing the semantic information of the corresponding parts of the original text in the process, and is applicable to a series of paragraphs composed of natural statements with a coherent plot, such as the trial situation, the plaintiff's claims and the defendant's arguments. The relevant instructions require listing the collated statements in the form of an ordered list and splitting the same sentence involving multiple cost information into individual articles in accordance with the requirements of fact finding. The text semantic understanding stage uses unified instructions to generate lists of semantic information for each section in turn by repeating calls in the previously extracted paragraphs of basic case information, trial situation, plaintiff's claim, defendant's defense, trial finding, the Court's opinion, decision result, law citation and trial staff information. From the perspective of the DIKWP framework, this can be understood as the ability to convert each entry into an expression of a direct data, information, knowledge or purpose.
The evaluation of such tasks, in addition to fidelity, includes the main aspects of whether the analysis granularity is sufficiently detailed, conciseness, resistance to syntactic errors, etc. Objective metrics can be used for text similarity, Rouge1/2/L, BERT Score, based on each content to calculate the overall accuracy and recall of each constituent item in this part.
|
Instructions: You are a professional and faithful sentence splitter, responsible for splitting complex sentences into multiple sentences with simple structure but complete content, without losing any piece of information in the original sentence. Requirements: 1. output in the form of a Markdown list with no more than 30 words per entry, completing a complete sentence with subject-verb-object and other components based on the core meaning. 2. **Special attention** should be paid to splitting the different cost cases into individual entries. Input: {{Basic case information. Text}} Result: |
|
1. A civil dispute case was accepted by the People's Court of Chancheng District, Foshan City, Guangdong Province. 2. the case number is (2022)Cantonese 0604 first-instance civil No.0000. The plaintiff is Bank of Guangzhou Co., Ltd. Foshan Branch, domiciled in P33 first floor store, P32 first floor and second floor store, No. 63 Chaoan South Road, Chancheng District, Foshan City, Guangdong Province, with a unified social credit code of 914406005778646632. 4. the plaintiff is mainly responsible for Hu, the position of the president. 5. the plaintiff entrusted litigation agent for Yang Moumou, the law firm for Beijing Tianchi Juntai (Guangzhou) law firm. 6. The defendant is Huang Moumou, male, born on April 1, 1987, Han nationality, living in Yunfu City, Guangdong Province, Yunan County. 7. the defendant was ordered to repay to the plaintiff credit card number 6250 6427 of the principal amount of 34,464.01 yuan. 8. order the defendant to pay the plaintiff card number 6250 6427 interest 3390.99 yuan. 9. to order the defendant to pay the plaintiff $4299.13 as default interest on card number 6250 6427. 10. order the defendant to pay the plaintiff the handling fee of card No. 6250 6427 in the amount of RMB 1547.21. 11. The above costs are provisionally calculated until May 5, 2020, after which the interest and liquidated damages until the actual date of settlement shall be calculated according to the agreement of the subscription agreement until the actual date of settlement. |
|
1. The plaintiff Bank of Guangzhou Co., Ltd. Foshan Branch and the defendant Huang Moumou credit card dispute, and filed a lawsuit. 2. The case was filed and accepted by the court on March 24, 2022. 3. The court applied the small claims procedure to the case for public hearing. 4. The plaintiff entrusted the litigation agent Yang and the defendant Huang to attend the proceedings. 5. The case has been heard and concluded. 6. The plaintiff, due to the defendant's default on credit card debts, filed a request to the court according to the default provisions. 7. Request the court to order the defendant to repay to the plaintiff the principal amount of 34,464.01 yuan of credit card number 6250 6427. 8. Request the court to order the defendant to pay the plaintiff 3390.99 yuan of interest on card number 6250 6427. 9. Request the court to order the defendant to pay the plaintiff $4299.13 for the breach of contract of card number 6250 6427. 10. Request the court to order the defendant to pay the plaintiff the handling fee of card No. 6250 6427 of RMB 1547.21. 11. The above fees are provisionally calculated until May 5, 2020, after which the interest and liquidated damages until the actual date of settlement shall be calculated according to the agreement of the subscription agreement until the actual date of settlement. 12. We request the court to order the defendant to bear all the litigation costs of this case. |
|
1. The plaintiff is in breach of contract because the defendant has defaulted on the credit card payment. 2. The plaintiff filed a lawsuit to the court based on the breach of contract. 3. Request 1: Order the defendant to repay to the plaintiff the principal amount of 34,464.01 yuan of credit card number 6250 6427. 4. Request 1: Order the defendant to pay 3390.99 yuan of interest to the plaintiff for card number 6250 6427. 5. Request 1:Order the defendant to pay the plaintiff 4299.13 yuan for the breach of contract of card number 6250 6427. 6. Request 1:Order the defendant to pay the plaintiff the handling fee of card number 6250 6427 1547.21 yuan. 7. The above costs are provisionally counted until May 5, 2020, after which the interest and liquidated damages to the actual date of settlement will be counted according to the agreement of the adoption agreement until the actual date of settlement. 8. Request 2: The defendant shall bear all the litigation costs of this case. |
|
In its reply, the defendant indicated that it did not dispute the amount of money owed. |
|
1. The plaintiff submitted evidence according to the lawsuit request. 2. The defendant did not object to the evidence submitted by the plaintiff. 3. The court confirmed the evidence submitted by the plaintiff, and made a note in the court record to prove. 4. The court also found that: the plaintiff claimed that the defendant started to make late payments from October 29, 2019. 5. The court confirmed the plaintiff's claim that the defendant began to make late payments on October 29, 2019. |
|
1. The Court is of the opinion that the defendant has failed to repay the credit card on time after using it, which constitutes a breach of contract. 2. The plaintiff requested the defendant to immediately repay the principal of the credit card should be supported. 3. The plaintiff requested the defendant to pay the interest and costs mentioned in the claim, according to the law, should be supported. 4. According to the evidence provided by the plaintiff and the principle of fairness and reasonableness, the court confirms that the total amount of interest and fees supported until May 5, 2020 is 4365 yuan. 5. May 5, 2020 after the interest on the basis of the outstanding principal, calculated at five ten thousandths of a percent per day. 6. Since the plaintiff requested to collect the entire amount in one lump sum, the court will not support the liquidated damages after May 5, 2020 to the actual date of settlement. 7. If the defendant has made repayment after May 5, 2020, the repayment will be made in accordance with the relevant regulations. 8. If the repayment is made within 1-90 days (inclusive), the repayment will be made in the order of interest or fees first, followed by the principal. 9.If the repayment is made in more than 91 days, the repayment shall be made in the order of the principal first, followed by the interest or fees. 10. When the debt is finally settled, the amount repaid will be directly deducted according to the above rules. |
|
1. Judgment that the defendant Huang Moumou pay the principal amount of credit card debt of 34,464.01 yuan to the plaintiff Bank of Guangzhou Co. 2. The defendant Huang Moumou was sentenced to pay interest to the plaintiff Bank of Guangzhou Co. 3. The above-mentioned interest is provisionally calculated until May 5, 2020, after which the interest until the actual date of settlement shall be calculated on the basis of the principal amount still owed, at the rate of five ten thousandths of a percent per day. 4. The plaintiff Bank of Guangzhou Co., Ltd., Foshan Branch's other claims are rejected. 5. If the defendant does not fulfill the obligation to pay money within the specified period, he shall double the interest on the debt during the period of delayed performance according to law. 6. The case fee of 446 yuan, by the defendant Huang Moumou burden. 7. The plaintiff has paid in advance, the court will not be returned, the defendant shall pay the plaintiff directly within ten days after the effective date of this judgment. 8. This verdict is a final judgment. |
The adjudication results section will focus on the court's decision support of the parties' requests. For this reason, the design instruction gives the corresponding court decision support while extracting each line item of the adjudication results, and the results are given in the form of a three-column table. The model outputs the results of supporting the plaintiff, supporting the defendant, rejecting the request, and partially supporting, which can be discriminated from the perspective of classification accuracy and so on.
|
Instruction: You are a legal content comprehendor that extracts the details of the adjudication results from the original text and outputs them as a Markdown table. Requirements: 1. The output is a Markdown table with three columns in total, the serial number, the adjudication result and the support. Each column of the table breaks down the content of the original text in the input to form one or more decision result items; 2. the first column is the serial number, in increasing order from 1 3. the second column is the corresponding decision result (column name "Decision Result") 3. the third column is the court's support. If the court supports the plaintiff's claim, write "Support the plaintiff"; if the court supports the defendant's claim, write "Support the defendant"; if the court supports only part of the plaintiff's claim, write "Partial support ", if the court does not support the plaintiff's claim, write "dismissed".
Enter: {{Results. List}} Result: |
|
| Serial numberJudgment ResultsSupport1The defendant Huang in the date of legal effect of this judgment within ten days to the plaintiff Bank of Guangzhou Co., Ltd. Foshan branch to pay the principal amount of 34,464.01 yuan credit card debts and interest totaling 4365 yuan (provisionally until May 5, 2020, after the date of actual settlement of interest on the basis of the principal amount still owed at a daily rate of five ten thousandths).Support Plaintiff2The other claims of the plaintiff, Bank of Guangzhou Corporation, Foshan Branch, are rejected.Plaintiff's request is denied3If the defendant fails to perform the obligation to pay money within the period specified in this judgment, he shall pay the interest for the period of delayed performance doubly in accordance with the provisions of Article 260 of the Civil Procedure Law of the People's Republic of China.Partial Support4The case fee of 446 yuan, by the defendant Huang Moumou burden. The plaintiff has paid in advance, the court will not be returned, the defendant in the judgment within ten days after the effective date of the plaintiff.Support Plaintiff5This judgment is final.
|
2.4 Case Information Link
Test the ability to correctly associate information and data from all parties. The instruction set requires that the items presented in the original list form are sequentially expanded in a two-dimensional table space, and each field is given in the form of a list requiring the model to populate the corresponding field. The instruction set can be divided into abbreviated and single-sample versions according to the level of detail of the instruction prompt information. The results can be evaluated in terms of the correctness of the table items, the degree of semantic and plot separation, sentence fluency, the semantic correctness of the formulas, and the correctness of the calculation results. For the correctness of the formulas themselves, it is necessary to extract the LaTeX formulas from the results and identify the relationship of the symbolic terms in the formulas therein to check the conformity with the semantics of the standard formulas.
2.4.1 The plaintiff alleged that
|
Instructions: You are a sentence decomposer and content parser capable of extracting each of the plaintiff's specific claims from the text below, which contains the plaintiff's claims. Requirements: 1. break down each of the plaintiff's claims to be specific to each money and each matter, one entry for each money and one entry for each matter, and return it as a Markdown table. 2. the first column of the table is the content of the specific lawsuit request, including the subject, predicate, and amount of information including the complete content. 3. The second column is a brief summary of the first column, the third column is the type or subject of the claim, and the fourth column lists keywords. 4. For requests for reimbursement of credit card and related expenses, after the output Markdown form, list the various types of money involved in the request, with the relevant money-related formulae listed in separate columns (the formulae show how to calculate one amount from one type of amount to another). Input: {{Original name. Text}} Result: |
|
| Content of litigation requestOverviewTopicsKeywordsOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuanRepayment of the principal amount owed on the credit card of ¥34,464.01Debt repaymentCredit card, principal, outstanding amountOrder the defendant to repay the plaintiff 3390.99 yuan of interest on the credit card (card number: 6250 6427)Repayment of credit card interest of ¥3,390.99Debt repaymentCredit Card, InterestOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) default amount of ¥4299.13Repayment of credit card defaults of ¥4,299.13Liability for breach of contractCredit Card, Default FeeOrder the defendant to reimburse the plaintiff the credit card (card number: 6250 6427) handling fee of 1547.21 yuanReimbursement of credit card fees of ¥1547.21Debt repaymentCredit Card, FeesAll costs of this case shall be borne by the defendantThe defendant bears the full cost of litigationLiability for legal feesLitigation costs Calculation formula.: Interest=Principal×Interest Rate×Number of days÷365 Liquidated damages=Principal×Default rate×Number of days of default Handling fee=Number of operations×Each handling fee rate |
|
Instructions: You are a sentence decomposer and content parser capable of extracting each of the plaintiff's specific claims from the text below, which contains the plaintiff's claims Requirements: 1. break down each of the plaintiff's claims to be specific to each sum of money and each matter, one entry for each sum of money and one entry for each matter, and return them in a Markdown table. For example, Order the defendant to repay the plaintiff the principal amount of credit card debt of ¥34,464.01, interest of ¥3,390.99, liquidated damages of ¥4,299.13, and handling fee of ¥1,547.21, split into Defendant to repay the plaintiff the principal amount of credit card debt of ¥34,464.01, Defendant to repay the plaintiff the interest of credit card debt of ¥3,390.99, Defendant to repay the plaintiff the credit card The first column of the table is specific to the plaintiff. 2. the first column of the table is the content of the specific claim, including the subject, predicate and amount of information including the complete content. 3. The second column is a brief summary of the first column, the third column is the type or subject of the claim, and the fourth column lists key words. 4. For the request for reimbursement of credit card and related expenses, after the output of the Markdown form, list the various types of money involved in the request, and list the relevant money-related calculation formula in separate columns (the formula shows how to calculate another amount from one type of amount), e.g., The court fee of ¥100 is reduced by half and the defendant is responsible for ¥50; the remaining ¥50 is to be refunded to the plaintiff., output
- $$ Costs borne by defendant = Costs * percentage charged = 100 * 1/2 = $50$$ - $$ Remaining costs = Costs - Costs borne by defendant = 100 - 50 = $50$$ - $$ Costs to plaintiff = Remaining litigation costs = $50$$
Input: {{plaintiff's claim. Text}} Result: |
|
| Content of litigation requestOverviewTopicsKeywordsOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuanRepayment of the principal amount owed on the credit card of ¥34,464.01Debt repaymentCredit card, principal, outstanding amountOrder the defendant to repay the plaintiff 3390.99 yuan of interest on the credit card (card number: 6250 6427)Repayment of credit card interest of ¥3,390.99Debt repaymentCredit Card, InterestOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) default amount of ¥4299.13Repayment of credit card defaults of ¥4,299.13Liability for breach of contractCredit Card, Default FeeOrder the defendant to reimburse the plaintiff the credit card (card number: 6250 6427) handling fee of 1547.21 yuanReimbursement of credit card fees of ¥1547.21Debt repaymentCredit Card, FeesAll costs of this case shall be borne by the defendantThe defendant bears the full cost of litigationLiability for legal feesLitigation costs Calculation formula.: Interest=Principal×Interest Rate×Number of days÷365 Liquidated damages=Principal×Default rate×Number of days of default Handling fee=Number of operations×Each handling fee rate Costs borne by the defendant=Litigation Fees×Collection rate=100×1/2=50 Remaining litigation costs=Litigation Fees-Costs borne by the defendant=100-50=50 Plaintiffs received fees=Remaining litigation costs=50 |
2.4.2 The defendant argues that
Using case-independent, content-independent instructions, the plaintiff's claim form and the defendant's contentions list items are used as base inputs, the defendant's contentions are filled into the plaintiff's claim-based form items, and the defendant's contentions form is then output. The task can be used to test the ability of the model to correlate data from different sources.
|
Hint: You are a judicial content comprehendor and can output a Markdown table. Next, complete and output a form based on the plaintiff's motion and the defendant's response. Requirements: 1. for each item of the plaintiff's claim, summarize it according to the defendant's response. 2. The table includes four columns, the first column is the plaintiff's request, the second column is a brief summary, the third column is the defendant's opinion (listed as "defendant's opinion"), the defendant's opinion can be summarized as no objections, objections, no description of one of the three. The fourth column is the relevant evidence provided by the defendant, if no relevant evidence is provided, the content inside the column is null. 3. 3. the format of the output is a Markdown table, not other content that is not in the markdown table. 4. the amount owed refers to all types of charges borne by the defendant when the charges are owed due to the information card.
Plaintiff alleges: {{Plaintiff alleges. Form}}
Defendant contends that: {{Defendant argues. List}} Result: |
|
| Plaintiff's request itemOverviewDefendant's opinionRelated EvidenceOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuanRepayment of the principal amount owed on the credit card of ¥34,464.01No objectionnullOrder the defendant to repay the plaintiff 3390.99 yuan of interest on the credit card (card number: 6250 6427)Repayment of credit card interest of ¥3,390.99No objectionnullOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) default amount of ¥4299.13Repayment of credit card defaults of ¥4,299.13No objectionnullOrder the defendant to reimburse the plaintiff the credit card (card number: 6250 6427) handling fee of 1547.21 yuanReimbursement of credit card fees of ¥1547.21No objectionnullAll costs of this case shall be borne by the defendantThe defendant bears the full cost of litigationNo objectionnull
|
2.4.3 The trial identified
The order takes as input a table of the defendant's arguments and a list of texts identified at the trial, and is then filled with evidence, findings, etc. into the table entries based on the defendant's arguments.
|
Instructions: You are a legal content comprehendor that impartially judges what the plaintiff and defendant say based on each piece of information given by the plaintiff and each piece of information given by the defendant, outputting the confirmation of evidence and requests in the form of a Markdown table. Requirements: 2. Based on each item of the plaintiff's request, the judge summarizes according to the defendant's response to the request, with the content of the summary corresponding to each item of the plaintiff's request. 3. the table consists of four columns, the first column containing the plaintiff's request items and the second column containing a brief summary; 4. the third column is the defendant's opinion (listed as "defendant's opinion"), and the defendant's opinion can be summarized as one of no objection, objection, or no explanation. If the defendant has no objection or approval, fill in "no objection"; if the defendant disagrees or has evidence to refute, fill in "objection"; in other cases, fill in "no explanation". 5. The fourth column is the information provided by the defendant. 5. The fourth column is the relevant evidence provided by the defendant, if no relevant evidence is provided, the content of the column is null. 6. 6. The fifth column is the court's determination of the relevant evidence, in the absence of objections, assuming that the plaintiff's statements are true and valid; if found valid, fill in "valid", if found invalid, fill in "invalid", not sure fill in "other". If it is found to be valid, enter "valid"; if it is found to be invalid, enter "invalid"; if it is not sure, fill in "other". 7. The sixth column is the court's support for the relevant evidence (listed as "support"). If supported, enter "supported", if not supported, enter "not supported", not sure fill in "other". 8.The format of the output is a Markdown table, not other content that is not in the markdown table. {{Defendant argues. Form}}
The Court finds: {{The court finds that. List}}
Output: |
|
| Plaintiff's request itemOverviewDefendant's opinionRelated evidenceEvidence finding situationSupportconditionOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuanRepayment of the principal amount owed on the credit card of ¥34,464.01No objectionnullEffectiveSupportOrder the defendant to repay the plaintiff 3390.99 yuan of interest on the credit card (card number: 6250 6427)Repayment of credit card interest of ¥3,390.99No objectionnullEffectiveSupportOrder the defendant to repay to the plaintiff the credit card (card number: 6250 6427) default amount of $4299.13Repayment of credit card defaults of ¥4,299.13No objectionnullEffectiveSupportOrder the defendant to reimburse the plaintiff the credit card (card number: 6250 6427) handling fee of 1547.21 yuanReimbursement of credit card fees of ¥1547.21No objectionnullEffectiveSupportAll costs of this case shall be borne by the defendantThe defendant bears the full cost of litigationNo objectionnullEffectiveSupport
|
2.4.4 The Court believes that
|
Instructions: You are a legal content comprehendor who extracts the court's opinion from the original text and outputs it as a Markdown table. Requirements: 1. the output is a Markdown table, with each column of the table being the court's opinion for each issue. If an original text contains more than one issue, the output is in more than one column; 2. the first column is the serial number, in increasing order from 1; 3. The second column is the claims and motions of the plaintiff or defendant from the original text, and the column name is "Related claims". 4. the third column is the plaintiff or the defendant's claims and requests, the court found the facts or reasons, the name is "factual findings". 5. The fourth column is the court's support (listed as "support"), if support, enter "support", if not, enter "not support", not sure fill in "Other" Court hearing found that: {{Discovery. Form}} The court finds that: {{The Court finds that. List}} Result: |
|
| Serial numberRelated claimsFindings of FactSupportcondition1Order the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuan.The defendant failed to repay the card on time, constituting a breach of contract.Support2Order the defendant to repay to the plaintiff credit card (card number: 6250 6427) interest 3390.99 yuan and various costs totaling 4299.13 yuan.The plaintiff claims that the defendant pays the benefits mentioned in the claim according to the relevant agreement, which is justified by law.Support3Provisional interest and fees totaling ¥4,365 through May 5, 2020According to the evidence provided by the plaintiff and the principle of fairness and reasonableness, the court has calculated.Support4Thereafter, interest shall be charged at the rate of five percent per day on the outstanding principal amount until the date of actual settlement.According to the evidence provided by the plaintiff and the principle of fairness and reasonableness, the court has calculated.Support5No further support for liquidated damages after May 5, 2020 until the date of actual settlement.The plaintiff claims a lump sum for the entire amount owed.Support6If the defendant has made repayment after May 5, 2020, the provisional accrual date claimed in the first claim of this case, the repayment shall be made in accordance with Article 57 of the Supervision and Administration of Commercial Bank Credit Card Business, in the order of interest or fees receivable first and the principal after, if the repayment is overdue for 1-90 days (inclusive); if the repayment is overdue for 91 days or more, the repayment shall be made in the order of principal first and interest or fees receivable after. The order of repayment.The court relied on the regulations and requested the order of repaymentSupport7The Plaintiff and the Defendant shall directly deduct the repaid amount in accordance with the above rules in the final settlement of the claims and debts in this case.The court made a request on the way to liquidate the debt in this case.Support
|
2.4.5 Judgment Results
|
Instruction: You are a legal content comprehendor that outputs as a Markdown table based on the tables that the court considers relevant and correlates with the corresponding decision results. Requirements: 1. the output is a Markdown table, with each column of the table being the court's opinion for each issue. If an original article contains multiple issues, the output is in multiple columns; 2. the first column is the serial number, in increasing order from 1; 3. the second column is the claims and motions of the plaintiff or defendant from the original text, and the column name is "Related Claims". 4. the third column is the result of the decision. 4. the third column is the result of the decision, indicating the outcome of the plaintiff's or defendant's claims and motions 4. the fourth column is the support of the court. If the court supports the plaintiff's claim, write "support the plaintiff"; if the court supports the defendant's claim, write "support the defendant"; if the court supports only part of the plaintiff's claim, write "partial support ", if the court does not support the plaintiff's claim, write "dismissed".
The court believes that the content of: {{The Court considers. Form}}
Result: |
|
| Serial numberRelated claimsFindings of FactSupportcondition1Order the defendant to repay to the plaintiff the credit card (card number: 6250 6427) the principal amount in arrears of 34,464.01 yuan.The defendant Huang paid the principal amount of credit card debt of 34,464.01 yuan to the plaintiff Bank of Guangzhou Limited Foshan Branch within ten days from the date of legal effect of this judgmentSupport Plaintiff2Order the defendant to repay to the plaintiff credit card (card number: 6250 6427) interest 3390.99 yuan and various costs totaling 4299.13 yuan.and interest, etc. totaling ¥4,365 (provisionally accrued through May 5, 2020)Support Plaintiff3Provisional interest and fees totaling ¥4,365 through May 5, 2020. Support Plaintiff4Thereafter, interest shall be charged at the rate of five percent per day on the outstanding principal amount until the date of actual settlement. Support Plaintiff5No further support for liquidated damages after May 5, 2020 until the date of actual settlement.The other claims of the plaintiff, Bank of Guangzhou Corporation, Foshan Branch, are rejected.Request for dismissal6If the defendant has made repayment after May 5, 2020, the provisional accrual date claimed in the first claim of this case, the repayment shall be made in accordance with Article 57 of the Supervision and Administration of Commercial Bank Credit Card Business, in the order of interest or fees receivable first and the principal after, if the repayment is overdue for 1-90 days (inclusive); if the repayment is overdue for 91 days or more, the repayment shall be made in the order of principal first and interest or fees receivable after. The order of repayment. Support Plaintiff7The Plaintiff and the Defendant shall directly deduct the repaid amount in accordance with the above rules in the final settlement of the claims and debts in this case. Support Plaintiff
|
3. Test and evaluation route
The scenario of parsing civil judgment documents based on the AGI/GPT type model can be understood as a series of specific subtasks that interact with the AGI/GPT type model and advance step by step to faithfully and completely parse out the components such as basic case information, trial status, plaintiff's claims, defendant's arguments, etc. from the civil judgment text, and decompose the data and information level structure and semantics of each component, and decompose the results decomposed into individual fields. Specifically, these include:
1. The overall goal of the evaluation for the civil adjudication document parsing scenario is to evaluate and test the quality of the model's functionality and performance in completing each task based on the set of instructions described above, combined with the validity and reliability commonly used when evaluating the results of AGI/GPT type models. Each task is also allowed to be completed using non-AGI/GPT type models.
2. For the civil adjudication document parsing scenario, the scenario can be decomposed into various levels of task requirements based on the idea of conforming to the human analysis process, forming four constituent parts of adjudication document rearrangement, case data extraction, text semantic understanding, and case information association, and forming specific extraction tasks such as basic case situation for each part, designing instructions or reusing instructions for each task, and assembling to form an instruction set. The instruction set is mainly used to describe the requirements for solving domain problems, avoiding prior training.
3. The task results are formed by executing the execution set, which can be tested and evaluated against human standards, forming metric items and metric systems from dimensions such as metric type, accuracy, text similarity, success or not, execution speed, consistency, reproducibility, BERT Score, Rouge 1/2/L, and numerical equality. The evaluation of metrics for the relevant results is also applied in the way of non-AGI/GPT type models.
Taking into account factors such as manual cross-vetting, software and systems including AGI/GPT type models can adopt two types of execution models when completing the task of parsing complete scenarios of civil adjudication documents:
In the Point to Point (P2P) mode, each task is evaluated separately, a complete instruction template is designed for each task, the variables in the instruction template are replaced with the correct data manually labeled, and the results of each task are collected to form the complete parsing result. For example, for the task in the text semantic understanding stage, the basic case information paragraphs are manually labeled correct results instead of the model's output in the task in the reordering stage of the adjudication documents. This model is suitable for human-in-the-loop design, in which humans are involved in
In the End to End (E2E) mode, the user only needs to input the adjudication documents, and each subsequent task accepts inputs from the AGI/GPT-type model under test or from the intermediate stages of the traditional software system in a pipelined manner according to the task dependency graph until all results are returned. The intermediate tasks in this mode are subject to the correctness of the model's results on the dependent tasks, allowing a more complete test of the performance of the AGI/GPT class model in a civil adjudication document resolution scenario. The performance of multi-round dialogues can be better tested in the E2E mode: the instructions used to complete intermediate tasks can be described in full form each time, or in the form of omissions and references by the model to understand the user's understanding of the dependency information between instructions. the verification of the whole process of parsing civil adjudication documents.
The Minimum Viable System (MVS) is designed for an interactive environment and pre-labeling for the manual case examination of the civil judgment document parsing scenario. The system adopts a pipeline structure to parse the documents step by step, which can be called by the model all at once, or manually modify the upstream results and automatically update the downstream results through the interface. The list and field information can be dragged and edited, and the field information shows the path where the information is located and the parsing result.
(a) Adjudication documents and rearrangement results interface
|
|
(b) Case data extraction results interface | (c) Text semantic understanding results interface |
Fig. 1: Auxiliary procedures for parsing scenarios of civil adjudication documents
4. Summary and Outlook
Facing the problem of evaluating and testing AGI/GPT type models under more complete, systematic and complex task scenarios, this paper designs and provides a set of task instruction sets for parsing civil adjudication documents, forming a complete task description covering sub-scenarios of adjudication document rearrangement, case data extraction, text semantic understanding, and case information association. The manual annotation samples can be enriched later to compare and cross-test on more civil public instruments and more AGT/GPT type models. Through testing and evaluation route analysis, the instruction set supports single-round or contextual execution in P2P or E2E mode, and by capturing detailed results of AGI/GPT type models in civil adjudicative document parsing scenarios, it can subsequently design functional, performance and quality evaluation index items for AGI/GPT type models in civil adjudicative document parsing scenarios with comprehensive evaluation questions The automated evaluation suite can be formed.
The task of decomposition of civil adjudication documents is based on the processing of form and semantics at the data and information levels, covering dimensions such as code/profile generation, format conversion, text semantic understanding and conversion, and information association. Subsequently, based on the framework of this paper, we will dig into civil adjudication problems, explore the influence of factors ranging from fact finding and legal knowledge behind adjudication reasoning, comprehensively cover all dimensions of the DIKWP system, and analyze the transformation characteristics between data, information, knowledge, wisdom and intention on civil adjudication problems, and conduct a more comprehensive assessment of the capability of AGI/GPT type models of general artificial intelligence. Facing the problem of building domain-independent text structure parsing systems, we will subsequently study more general task requirements and processing flow description methods, and continuously improve the instruction generalization performance of domain-based scenarios such as civil adjudication document parsing from the language/mathematics/common sense knowledge domain.
5. References
[1] Z. Du et al., “GLM: General language model pretraining with autoregressive blank infilling,” in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), ACL 2022, dublin, ireland, may 22-27, 2022, 2022, pp. 320–335.
[2] A. Vaswani et al., “Attention is all you need,” in Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, CA, USA, 2017, pp. 5998–6008.
[3] R. Thoppilan et al., “LaMDA: Language models for dialog applications,” CoRR, vol. abs/2201.08239, 2022.
[4] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023.
[5] J. Kaplan et al., “Scaling laws for neural language models,” CoRR, vol. abs/2001.08361, 2020.
[6] H. W. Chung et al., “Scaling instruction-finetuned language models,” CoRR, vol. abs/2210.11416, 2022.
[7] T. B. Brown et al., “Language models are few-shot learners,” CoRR, vol. abs/2005.14165, 2020.
[8] J. Wei et al., “Finetuned language models are zero-shot learners,” in The tenth international conference on learning representations, ICLR 2022, virtual event, april 25-29, 2022, 2022.
[9] L. Ouyang et al., “Training language models to follow instructions with human feedback,” CoRR, vol. abs/2203.02155, 2022.
[10] R. Taori et al., “Stanford alpaca: An instruction-following LLaMA model,” GitHub repository, 2023. [Online]. Available: https://crfm.stanford.edu/2023/03/13/alpaca.html. [Accessed: 24-Apr-2023].
[11] S. Bubeck et al., “Sparks of artificial general intelligence: Early experiments with GPT-4,” CoRR, vol. abs/2303.12712, 2023.
[12] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual ChatGPT: Talking, drawing and editing with visual foundation models,” CoRR, vol. abs/2303.04671, 2023.
[13] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models.” 2023.
[14] T. Schick et al., “Toolformer: Language models can teach themselves to use tools,” CoRR, vol. abs/2302.04761, 2023.
[15] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in HuggingFace,” CoRR, vol. abs/2303.17580, 2023.
[16] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig, “Language models of code are few-shot commonsense learners,” CoRR, vol. abs/2210.07128, 2022.
[17] Y. Li, Y. Duan, Z. Maamar, H. Che, A.-B. Spulber, and S. Fuentes, “Swarm differential privacy for purpose-driven data-information-knowledge-wisdom architecture,” Mob. Inf. Syst., vol. 2021, pp. 6671628:1–6671628:15, 2021.
[18] Y. Mei, Y. Duan, L. Chen, Z. Feng, L. Yu, and Z. Guo, “Purpose driven disputation modeling, analysis and resolution based on DIKWP graphs,” in 24th IEEE int conf on high performance computing & communications; 8th int conf on data science & systems; 20th int conf on smart city; 8th int conf on dependability in sensor, cloud & big data systems & application, HPCC/DSS/SmartCity/DependSys 2022, hainan, china, december 18-20, 2022, 2022, pp. 2118–2125.
[19] Y. Li and Y. Duan, “The performance of GPT-4 on education domain with DIKWP analysis,” Apr-2023.
[20] Y. L. Chengxiang Ren Lixu Shao and Y. Duan, “Evaluation on AGI/GPT based on the DIKWP for: Anthropic’s claude,” Apr-2023.
[21] Y. L. Chengxiang Ren and Y. Duan, “Evaluation on AGI/GPT based on the DIKWP for ERNIE bot,” Apr-2023.
[22] Y. Li and Y. Duan, “The performance of GPT-4 on education domain with DIKWP analysis,” Apr-2023.
[23] Y. Li, Y. Zhou, Y. Mei, Z. Hu, and Y. Duan, “The capability evaluation of GPT-4 on medical/medicine domain with DIKWP analysis,” Apr-2023.
[24] F. Tang, Y. Duan, E. Dou, and Y. Yang, “The capability evaluation of GPT-4 and claude on financial domain with DIKWP+TIF analysis.” Apr-2023.
[25] J Xu., G Huang., Y Liu., “Comprehensive construction of intelligent courts to promote modernization of trial system and trial capacity,” Administrative Reform, no. 5, pp. 86–91, 2019.
[26] J Li., R Huang., Y Qin., Y Chen., X Tian., “Anti-deduction learning-based sentencing circumstance identification for adjudication documents,” Computer Applications, vol. 42, no. 6, pp. 1802–1807, 2022.
[27] S. Surana, S. Dembla, and P. Bihani, “Identifying contradictions in the legal proceedings using natural language models,” SN Comput. Sci., vol. 3, no. 3, p. 187, 2022.
[28] L. Zhao et al., “Legal judgment prediction with multiple perspectives on civil cases,” in Artificial intelligence - first CAAI international conference, CICAI 2021, hangzhou, china, june 5-6, 2021, proceedings, part I, 2021, vol. 13069, pp. 712–723.
[29] Y. Feng, C. Li, and V. Ng, “Legal judgment prediction via event extraction with constraints,” in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), ACL 2022, dublin, ireland, may 22-27, 2022, 2022, pp. 648–664.
[30] W. Chen, J. Tian, L. Xiao, H. He, and Y. Jin, “A semantically consistent and syntactically variational encoder-decoder framework for paraphrase generation,” in Proceedings of the 28th international conference on computational linguistics, COLING 2020, barcelona, spain (online), december 8-13, 2020, 2020, pp. 1186–1198.
[31] B. Liu et al., “Investigating conversational agent action in legal case retrieval,” in Advances in information retrieval - 45th european conference on information retrieval, ECIR 2023, dublin, ireland, april 2-6, 2023, proceedings, part I, 2023, vol. 13980, pp. 622–635.
[32] B. Liu et al., “Query generation and buffer mechanism: Towards a better conversational agent for legal case retrieval,” Inf. Process. Manag., vol. 59, no. 5, p. 103051, 2022.
[33] D Hu., “Research and system implementation of public opinion monitoring technology for judicial field,” Master, Harbin Institute of Technology, 2019.
[34] W Shen. Y Liu., “Exploring the path of computable legal development,” Zhejiang Social Science, no. 6, pp. 33–41+156, 2022.
[35] W Zuo. , "The future of computational jurisprudence in China: reflections and foresight," Tsinghua Law, vol. 16, no. 3, pp. 196-208, 2022.
https://m.sciencenet.cn/blog-3429562-1385863.html
上一篇：AGI-AIGC-GPT测评DIKWP（全球）实验室（日常答疑系列）DIKWP处理的常见误区
下一篇：世界上首篇大模型游戏领域测评报告-AGI-AIGC-GPT测评DIKWP（全球）实验室（测评系列之十六）
