docs: add human-in-the-loop guides for OpenAI and Anthropic #3133
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
GregHolmes
force-pushed
the
AIT-160-Guide-documentation-human-in-the-loop
branch
3 times, most recently
from
72b74bf to
de294d2
Compare
GregHolmes
added
javascript
mschristensen
left a comment
mschristensen
left a comment
There was a problem hiding this comment.
Thanks Greg! It's a good start - overall I felt the code examples ended up being quite verbose, which makes it hard to follow in places. I thought I would put together a minimal example that includes code that I would be happy with. I've added it in a gist here: https://gist.github.com/mschristensen/c735e1f906fcad889e91190e52938273
It's not a million miles away from what you've done but I hope it's a bit more concise, and addresses some of the feedback in my comments. Please have a look at let me know if you think we can align these docs with this example.
Side point - I think for writing these guides it might make sense for us to write the code first, agree that, and then the actual docs content. Perhaps we need a repo for this? We could even make them public and link to them from guides for people to get started with that pattern. Curious to hear your thoughts on this.
|
|
||
| ## Step 1: Define a tool requiring approval <a id="step-1"/> | ||
|
|
||
| Define an OpenAI tool that represents a sensitive operation requiring human approval. This example uses a file deletion tool that should not execute without explicit user consent. |
There was a problem hiding this comment.
I think we should pick a different type of tool call. Generally, we are targeting developers building cloud-hosted agents, rather than agents which run on your local machine like claude code (the requirement for client/agent sync doesn't exist in these cases, since these are no separate components that need to communicate).
Therefore I would suggest using a tool call with a more natural correspondence with operations that might be performed by a cloud agent. Perhaps something a bit more general that has a clear mapping to RBAC, e.g. publish_blog_post or similar?
| <Code> | ||
| ```javascript | ||
| import OpenAI from 'openai'; | ||
| import Ably from 'ably'; |
There was a problem hiding this comment.
I think this should only be specified in the step 2 code block, where it is used (similar to other guides).
(I know there's a bit of a pain point in communicating changes to files progressively in documentation like this, I have a draft implementation of a Code component with line highlighting which I think will help, which I aim to finish up soon).
| const channel = realtime.channels.get('ai:{{RANDOM_CHANNEL_NAME}}'); | ||
|
|
||
| // Track pending approval requests | ||
| const pendingApprovals = new Map(); |
There was a problem hiding this comment.
I think this should also be in the next code block in step 3, where it is used
| const parameters = JSON.parse(toolCall.arguments); | ||
| await channel.publish('approval-request', { | ||
| requestId, | ||
| tool: toolCall.name, |
There was a problem hiding this comment.
I the feature docs for HITL, we call this action. I actually think tool is a better name, because I think action risks confusion with the message.action, so can we please align the feature docs to use this field name too?
| return; | ||
| } | ||
|
|
||
| // Verify the approver (in production, check clientId or user claims) |
There was a problem hiding this comment.
Can we update the guide to show a concrete example of this? For example, we could use role based access, and assign an admin role to the user, and specify that only admins can delete data. I think it's quite an important part of the flow so worth describing a concrete example. Then, when we do the verification, we can show a code path which also rejects the promise.
|
|
||
| // Process each file with progress updates | ||
| const results = []; | ||
| for (let i = 0; i < files.length; i++) { |
There was a problem hiding this comment.
When we update this example to use e.g. a more generic delete_data, we can simplify this code. Instead of a loop etc, it will be clearer to just call a function like deleteData(parameters); the important bit is the surrounding code for handling the comms
| results.push({ file, deleted: true }); | ||
|
|
||
| // Stream progress for each file | ||
| await channel.publish('tool-progress', { |
There was a problem hiding this comment.
We didn't cover this pattern in the HITL feature docs and I don't think we should introduce it here. However, I think it might be valid to cover this in the "Tool calls" feature docs, e.g. updates that don't come directly from the model stream, but rather from the execution of a tool by the agent. There's a variant of this that uses LiveObjects for progress % etc. So, we can remove this from here, but I have created a ticket: https://ably.atlassian.net/browse/AIT-312
|
|
||
| <Code> | ||
| ```javascript | ||
| async function processToolCall(toolCall) { |
There was a problem hiding this comment.
I don't really know how this function is adding value other than calling requestApproval and executing the tool call. I think the rest just adds cognitive overhead. Can we simplify the code as much as possible?
| const anthropic = new Anthropic(); | ||
|
|
||
| // Define a tool that requires human approval | ||
| const tools = [ |
There was a problem hiding this comment.
Rather than defining static data, I think we should show the code that calls the model at this step, i.e. invoking the model the first time with the tools. I think generally we should aim to show how the pieces are used in context with the code that actually does stuff, rather than leaving random bits out in isolation. Wdyt?
| <Code> | ||
| ```javascript | ||
| async function requestApproval(toolUse) { | ||
| const requestId = crypto.randomUUID(); |
There was a problem hiding this comment.
We dont need to generate an ID, since OpenAI gives us a tool call id
Add two new AI Transport guides demonstrating how to implement human-in-the-loop approval workflows for AI agent tool calls: - openai-human-in-the-loop.mdx: HITL with OpenAI function calling - anthropic-human-in-the-loop.mdx: HITL with Anthropic tool use Both guides cover: - Defining tools requiring human approval - Publishing approval requests with requestId correlation - Subscribing to and processing approval decisions - Progress streaming during long-running tool execution - Handling rejection gracefully
GregHolmes
force-pushed
the
AIT-160-Guide-documentation-human-in-the-loop
branch
from
de294d2 to
20c1aa3
Compare
Add two new AI Transport guides demonstrating how to implement human-in-the-loop approval workflows for AI agent tool calls:
Both guides cover: