feat(eval): add datasets CLI commands by nborges-aws · Pull Request #1911 · aws/agentcore-cli

nborges-aws · 2026-08-04T20:59:23Z

Description

Adds command-line CRUDL plus versioning for AgentCore evaluation datasets:

eval dataset create
eval dataset get
eval dataset list
eval dataset delete
eval dataset publish

eval dataset update will land in follow up PR

DataSourceType union on CreateDataset allowing users to pass in s3 uri or file path for data set creation.
source.tsx - Responsible for translating file path in dataSourceType union to the service API expected inlineExample format. inlineExamples is a dict, with list of JSONL items inside. For user convenience and ability to pass their JSONL draft file as a whole -- we accept filepath, and parse the file into inlineExamples list before call to service
--file-path flag conditional logic on delete: eval datasets get without file path flag will fetch metadata and print to command line (getDataset API). When file path flag is included, CLI will pull content from pre-signed downloadUrl provided in metadata, and writes content to provided file location (streamed into temp file and renamed into place)
New error types in src/errors: DatasetDownloadError (SERVICE, covers a missing or unfetchable presigned URL) and DatasetWriteError (USER, covers local write issues)
EvalClient now receives the injected CoreFetch from CoreClient, since dataset content is served from S3
create, publish, delete are all async operation which return CREATING / UPDATING / DELETING

Related Issue

Closes #

Documentation PR

Type of Change

Testing

Manually tested commands using bun run ... against personal account, happy path and negative validation.

Full suite against latest HEAD after rebasing onto refactor:

bun test (729 pass, 0 fail)
bun run typecheck clean
bun run lint:check clean
bun run format:check clean
bun run build clean

How have you tested the change?

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

codecov-commenter · 2026-08-04T21:01:19Z

Codecov Report

❌ Patch coverage is 99.46809% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.37%. Comparing base (ae6e810) to head (f3fe5d7).

Files with missing lines	Patch %	Lines
src/io/atomicWrite.ts	92.30%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1911      +/-   ##
============================================
+ Coverage     96.28%   96.37%   +0.09%     
============================================
  Files           245      253       +8     
  Lines         12105    12471     +366     
============================================
+ Hits          11655    12019     +364     
- Misses          450      452       +2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-04T21:21:12Z

+    handle: async (ctx, flags) => {
+      if (!flags["id"]) throw new InputValidationError("required option '--id <id>' not specified");
+
+      // Publishing an unmodified DRAFT creates a version identical to the last


current implementation will allow user to publish a version that is unmodified from the last. If silently publishing unmodified version feels wrong here, we can check draftStatus for UNMODIFIED and use that to prompt/warn/reject publishing.

AlexanderRichey · 2026-08-05T18:34:03Z

+// renamed into place once the transfer completes. Streams so a large dataset
+// is never held in memory. A failed or aborted transfer removes the temp file and
+// leaves any existing file at `filePath` untouched.
+async function streamToFile(


This seems like it should be in the io package.

AlexanderRichey · 2026-08-05T18:35:40Z

+ * A downloaded dataset could not be written to its destination.
+ * Local and user-fixable: a missing directory, a permission denial, etc.
+ */
+export class DatasetWriteError extends AgentCoreCLIError {


Can these exceptions be made more general? Maybe NetworkingError and FileWriteError?

jariy17

Pretty Good! Just look at the error handling for the handlers

jariy17 · 2026-08-05T18:34:23Z

+    // ingesting has no URL to offer yet. Report the status, which is what tells
+    // the caller whether to retry.
+    if (!dataset.downloadUrl) {
+      throw new DatasetDownloadError(


This isn't really a user fault, more of service issue. Should we add a retry mechanism here?

I've classified this as a SERVICE error. I considered adding a retry mechanism, but it was plausible that after a short-polling loop it still wouldn't be active. I lead towards classifying this as a service-side retryable exception, and informing the user to retry once ACTIVE as the right call. Updated code does exactly that

jariy17 · 2026-08-05T18:34:34Z

+    try {
+      const response = await this.fetch(dataset.downloadUrl, { signal });
+      if (!response.ok) {
+        throw new DatasetDownloadError(


Again this could be a service issue or an Internal issue.

I think for this case we're good to leave as service. If we hit !response.ok then we successfully made the http call and got a non-2xx

jariy17 · 2026-08-05T18:52:56Z

+        });
+      }
+      body = response.body;
+    } catch (error) {


Concern about the catch-all wrap here. Blanket-tagging every unknown fetch failure as SERVICE is a guess, and it can mislabel things:
1. A bug in our own request construction is really INTERNAL, not SERVICE.
2. A runtime or polyfill issue is also INTERNAL.

'd only wrap the cases we can positively attribute to the service, like the non-ok HTTP status above. Let anything unrecognized fall through to fromError so it categorizes based on what it actually observes.

jariy17 · 2026-08-05T18:55:10Z

+// renamed into place once the transfer completes. Streams so a large dataset
+// is never held in memory. A failed or aborted transfer removes the temp file and
+// leaves any existing file at `filePath` untouched.
+async function streamToFile(


There should be a library package for this

jariy17 · 2026-08-05T19:21:33Z

+      const interrupt = () => controller.abort();
+      process.once("SIGINT", interrupt);
+      try {
+        const response = await core.eval.downloadDataset(


We should add filePath to the response here. Right now the output is just metadata, so nothing in it confirms where the file was written. Returning { ...dataset, filePath } gives the customer clear confirmation of the download and its location.

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 4, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 4, 2026

nborges-aws commented Aug 4, 2026

View reviewed changes

AlexanderRichey requested changes Aug 5, 2026

View reviewed changes

jariy17 requested changes Aug 5, 2026

View reviewed changes

nborges-aws added 3 commits August 5, 2026 17:14

feat: add datasets scaffolding and create operation

66741dd

feat: add get, list, delete commands to eval datasets

95d36a9

feat: add support for datasets publish command

2779595

nborges-aws force-pushed the datasets-cli branch from 180627e to 1b22111 Compare August 5, 2026 22:15

fix: improve error handling and update user response on download dataset

f3fe5d7

nborges-aws force-pushed the datasets-cli branch from 1b22111 to f3fe5d7 Compare August 5, 2026 22:26

Navigation Menu

Uh oh!

feat(eval): add datasets CLI commands - #1911

feat(eval): add datasets CLI commands #1911
nborges-aws wants to merge 4 commits into
refactorfrom
datasets-cli

nborges-aws commented Aug 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 4, 2026 •

edited

Loading

Uh oh!

nborges-aws Aug 4, 2026

Uh oh!

AlexanderRichey Aug 5, 2026

Uh oh!

AlexanderRichey Aug 5, 2026

Uh oh!

jariy17 left a comment

Uh oh!

jariy17 Aug 5, 2026

Uh oh!

nborges-aws Aug 5, 2026

Uh oh!

jariy17 Aug 5, 2026

Uh oh!

nborges-aws Aug 5, 2026

Uh oh!

jariy17 Aug 5, 2026

Uh oh!

jariy17 Aug 5, 2026

Uh oh!

jariy17 Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

nborges-aws commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Documentation PR

Type of Change

Testing

Checklist

Uh oh!

codecov-commenter commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nborges-aws commented Aug 4, 2026 •

edited

Loading

codecov-commenter commented Aug 4, 2026 •

edited

Loading