Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions apps/docs/content/docs/en/tools/gmail.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ Send emails using Gmail
| `body` | string | Yes | Email body content |
| `cc` | string | No | CC recipients \(comma-separated\) |
| `bcc` | string | No | BCC recipients \(comma-separated\) |
| `attachments` | file[] | No | Files to attach to the email |

#### Output

Expand All @@ -91,6 +92,7 @@ Draft emails using Gmail
| `body` | string | Yes | Email body content |
| `cc` | string | No | CC recipients \(comma-separated\) |
| `bcc` | string | No | BCC recipients \(comma-separated\) |
| `attachments` | file[] | No | Files to attach to the email draft |

#### Output

Expand Down
96 changes: 94 additions & 2 deletions apps/docs/content/docs/en/tools/s3.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: S3
description: View S3 files
description: Upload, download, list, and manage S3 files
---

import { BlockInfoCard } from "@/components/ui/block-info-card"
Expand Down Expand Up @@ -62,12 +62,37 @@ In Sim, the S3 integration enables your agents to retrieve and access files stor

## Usage Instructions

Integrate S3 into the workflow. Can get presigned URLs for S3 objects. Requires access key and secret access key.
Integrate S3 into the workflow. Upload files, download objects, list bucket contents, delete objects, and copy objects between buckets. Requires AWS access key and secret access key.



## Tools

### `s3_put_object`

Upload a file to an AWS S3 bucket

#### Input

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessKeyId` | string | Yes | Your AWS Access Key ID |
| `secretAccessKey` | string | Yes | Your AWS Secret Access Key |
| `region` | string | Yes | AWS region \(e.g., us-east-1\) |
| `bucketName` | string | Yes | S3 bucket name |
| `objectKey` | string | Yes | Object key/path in S3 \(e.g., folder/filename.ext\) |
| `file` | file | No | File to upload |
| `content` | string | No | Text content to upload \(alternative to file\) |
| `contentType` | string | No | Content-Type header \(auto-detected from file if not provided\) |
| `acl` | string | No | Access control list \(e.g., private, public-read\) |

#### Output

| Parameter | Type | Description |
| --------- | ---- | ----------- |
| `url` | string | URL of the uploaded S3 object |
| `metadata` | object | Upload metadata including ETag and location |

### `s3_get_object`

Retrieve an object from an AWS S3 bucket
Expand All @@ -87,6 +112,73 @@ Retrieve an object from an AWS S3 bucket
| `url` | string | Pre-signed URL for downloading the S3 object |
| `metadata` | object | File metadata including type, size, name, and last modified date |

### `s3_list_objects`

List objects in an AWS S3 bucket

#### Input

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessKeyId` | string | Yes | Your AWS Access Key ID |
| `secretAccessKey` | string | Yes | Your AWS Secret Access Key |
| `region` | string | Yes | AWS region \(e.g., us-east-1\) |
| `bucketName` | string | Yes | S3 bucket name |
| `prefix` | string | No | Prefix to filter objects \(e.g., folder/\) |
| `maxKeys` | number | No | Maximum number of objects to return \(default: 1000\) |
| `continuationToken` | string | No | Token for pagination |

#### Output

| Parameter | Type | Description |
| --------- | ---- | ----------- |
| `objects` | array | List of S3 objects |

### `s3_delete_object`

Delete an object from an AWS S3 bucket

#### Input

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessKeyId` | string | Yes | Your AWS Access Key ID |
| `secretAccessKey` | string | Yes | Your AWS Secret Access Key |
| `region` | string | Yes | AWS region \(e.g., us-east-1\) |
| `bucketName` | string | Yes | S3 bucket name |
| `objectKey` | string | Yes | Object key/path to delete |

#### Output

| Parameter | Type | Description |
| --------- | ---- | ----------- |
| `deleted` | boolean | Whether the object was successfully deleted |
| `metadata` | object | Deletion metadata |

### `s3_copy_object`

Copy an object within or between AWS S3 buckets

#### Input

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessKeyId` | string | Yes | Your AWS Access Key ID |
| `secretAccessKey` | string | Yes | Your AWS Secret Access Key |
| `region` | string | Yes | AWS region \(e.g., us-east-1\) |
| `sourceBucket` | string | Yes | Source bucket name |
| `sourceKey` | string | Yes | Source object key/path |
| `destinationBucket` | string | Yes | Destination bucket name |
| `destinationKey` | string | Yes | Destination object key/path |
| `acl` | string | No | Access control list for the copied object \(e.g., private, public-read\) |

#### Output

| Parameter | Type | Description |
| --------- | ---- | ----------- |
| `url` | string | URL of the copied S3 object |
| `metadata` | object | Copy operation metadata |



## Notes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
Code2,
Database,
DollarSign,
HardDrive,
Users,
Workflow,
} from 'lucide-react'
Expand Down Expand Up @@ -42,6 +43,7 @@ interface PricingTier {
*/
const FREE_PLAN_FEATURES: PricingFeature[] = [
{ icon: DollarSign, text: '$10 usage limit' },
{ icon: HardDrive, text: '5GB file storage' },
{ icon: Workflow, text: 'Public template access' },
{ icon: Users, text: 'Community support' },
{ icon: Database, text: 'Limited log retention' },
Expand Down
84 changes: 64 additions & 20 deletions apps/sim/app/api/files/parse/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { isSupportedFileType, parseFile } from '@/lib/file-parsers'
import { createLogger } from '@/lib/logs/console/logger'
import { validateExternalUrl } from '@/lib/security/input-validation'
import { downloadFile, isUsingCloudStorage } from '@/lib/uploads'
import { extractStorageKey } from '@/lib/uploads/file-utils'
import { UPLOAD_DIR_SERVER } from '@/lib/uploads/setup.server'
import '@/lib/uploads/setup.server'

Expand Down Expand Up @@ -69,13 +70,13 @@ export async function POST(request: NextRequest) {

try {
const requestData = await request.json()
const { filePath, fileType } = requestData
const { filePath, fileType, workspaceId } = requestData

if (!filePath || (typeof filePath === 'string' && filePath.trim() === '')) {
return NextResponse.json({ success: false, error: 'No file path provided' }, { status: 400 })
}

logger.info('File parse request received:', { filePath, fileType })
logger.info('File parse request received:', { filePath, fileType, workspaceId })

if (Array.isArray(filePath)) {
const results = []
Expand All @@ -89,7 +90,7 @@ export async function POST(request: NextRequest) {
continue
}

const result = await parseFileSingle(path, fileType)
const result = await parseFileSingle(path, fileType, workspaceId)
if (result.metadata) {
result.metadata.processingTime = Date.now() - startTime
}
Expand Down Expand Up @@ -117,7 +118,7 @@ export async function POST(request: NextRequest) {
})
}

const result = await parseFileSingle(filePath, fileType)
const result = await parseFileSingle(filePath, fileType, workspaceId)

if (result.metadata) {
result.metadata.processingTime = Date.now() - startTime
Expand Down Expand Up @@ -153,7 +154,11 @@ export async function POST(request: NextRequest) {
/**
* Parse a single file and return its content
*/
async function parseFileSingle(filePath: string, fileType?: string): Promise<ParseResult> {
async function parseFileSingle(
filePath: string,
fileType?: string,
workspaceId?: string
): Promise<ParseResult> {
logger.info('Parsing file:', filePath)

if (!filePath || filePath.trim() === '') {
Expand All @@ -174,7 +179,7 @@ async function parseFileSingle(filePath: string, fileType?: string): Promise<Par
}

if (filePath.startsWith('http://') || filePath.startsWith('https://')) {
return handleExternalUrl(filePath, fileType)
return handleExternalUrl(filePath, fileType, workspaceId)
}

const isS3Path = filePath.includes('/api/files/serve/s3/')
Expand Down Expand Up @@ -216,10 +221,16 @@ function validateFilePath(filePath: string): { isValid: boolean; error?: string

/**
* Handle external URL
* If workspaceId is provided, checks if file already exists and saves to workspace if not
*/
async function handleExternalUrl(url: string, fileType?: string): Promise<ParseResult> {
async function handleExternalUrl(
url: string,
fileType?: string,
workspaceId?: string
): Promise<ParseResult> {
try {
logger.info('Fetching external URL:', url)
logger.info('WorkspaceId for URL save:', workspaceId)

const urlValidation = validateExternalUrl(url, 'fileUrl')
if (!urlValidation.isValid) {
Expand All @@ -231,6 +242,34 @@ async function handleExternalUrl(url: string, fileType?: string): Promise<ParseR
}
}

// Extract filename from URL
const urlPath = new URL(url).pathname
const filename = urlPath.split('/').pop() || 'download'
const extension = path.extname(filename).toLowerCase().substring(1)

logger.info(`Extracted filename: ${filename}, workspaceId: ${workspaceId}`)

// If workspaceId provided, check if file already exists in workspace
if (workspaceId) {
const { fileExistsInWorkspace, listWorkspaceFiles } = await import(
'@/lib/uploads/workspace-files'
)
const exists = await fileExistsInWorkspace(workspaceId, filename)

if (exists) {
logger.info(`File ${filename} already exists in workspace, using existing file`)
// Get existing file and parse from storage
const workspaceFiles = await listWorkspaceFiles(workspaceId)
const existingFile = workspaceFiles.find((f) => f.name === filename)

if (existingFile) {
// Parse from workspace storage instead of re-downloading
const storageFilePath = `/api/files/serve/${existingFile.key}`
return handleCloudFile(storageFilePath, fileType)
}
}
}

const response = await fetch(url, {
signal: AbortSignal.timeout(DOWNLOAD_TIMEOUT_MS),
})
Expand All @@ -251,9 +290,23 @@ async function handleExternalUrl(url: string, fileType?: string): Promise<ParseR

logger.info(`Downloaded file from URL: ${url}, size: ${buffer.length} bytes`)

const urlPath = new URL(url).pathname
const filename = urlPath.split('/').pop() || 'download'
const extension = path.extname(filename).toLowerCase().substring(1)
// If workspaceId provided, save to workspace storage
if (workspaceId) {
try {
const { getSession } = await import('@/lib/auth')
const { uploadWorkspaceFile } = await import('@/lib/uploads/workspace-files')

const session = await getSession()
if (session?.user?.id) {
const mimeType = response.headers.get('content-type') || getMimeType(extension)
await uploadWorkspaceFile(workspaceId, session.user.id, buffer, filename, mimeType)
logger.info(`Saved URL file to workspace storage: ${filename}`)
}
} catch (saveError) {
// Log but don't fail - continue with parsing even if save fails
logger.warn(`Failed to save URL file to workspace:`, saveError)
}
}

if (extension === 'pdf') {
return await handlePdfBuffer(buffer, filename, fileType, url)
Expand Down Expand Up @@ -281,16 +334,7 @@ async function handleExternalUrl(url: string, fileType?: string): Promise<ParseR
*/
async function handleCloudFile(filePath: string, fileType?: string): Promise<ParseResult> {
try {
let cloudKey: string
if (filePath.includes('/api/files/serve/s3/')) {
cloudKey = decodeURIComponent(filePath.split('/api/files/serve/s3/')[1])
} else if (filePath.includes('/api/files/serve/blob/')) {
cloudKey = decodeURIComponent(filePath.split('/api/files/serve/blob/')[1])
} else if (filePath.startsWith('/api/files/serve/')) {
cloudKey = decodeURIComponent(filePath.substring('/api/files/serve/'.length))
} else {
cloudKey = filePath
}
const cloudKey = extractStorageKey(filePath)

logger.info('Extracted cloud key:', cloudKey)

Expand Down
44 changes: 44 additions & 0 deletions apps/sim/app/api/files/upload/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ export async function POST(request: NextRequest) {
logger.info(
`Uploading files for execution-scoped storage: workflow=${workflowId}, execution=${executionId}`
)
} else if (workspaceId) {
logger.info(`Uploading files for workspace-scoped storage: workspace=${workspaceId}`)
}

const uploadResults = []
Expand All @@ -83,6 +85,7 @@ export async function POST(request: NextRequest) {
const bytes = await file.arrayBuffer()
const buffer = Buffer.from(bytes)

// Priority 1: Execution-scoped storage (temporary, 5 min expiry)
if (workflowId && executionId) {
const { uploadExecutionFile } = await import('@/lib/workflows/execution-file-storage')
const userFile = await uploadExecutionFile(
Expand All @@ -100,6 +103,47 @@ export async function POST(request: NextRequest) {
continue
}

// Priority 2: Workspace-scoped storage (persistent, no expiry)
if (workspaceId) {
try {
const { uploadWorkspaceFile } = await import('@/lib/uploads/workspace-files')
const userFile = await uploadWorkspaceFile(
workspaceId,
session.user.id,
buffer,
originalName,
file.type || 'application/octet-stream'
)

uploadResults.push(userFile)
continue
} catch (workspaceError) {
// Check error type
const errorMessage =
workspaceError instanceof Error ? workspaceError.message : 'Upload failed'
const isDuplicate = errorMessage.includes('already exists')
const isStorageLimitError =
errorMessage.includes('Storage limit exceeded') ||
errorMessage.includes('storage limit')

logger.warn(`Workspace file upload failed: ${errorMessage}`)

// Determine appropriate status code
let statusCode = 500
if (isDuplicate) statusCode = 409
else if (isStorageLimitError) statusCode = 413

return NextResponse.json(
{
success: false,
error: errorMessage,
isDuplicate,
},
{ status: statusCode }
)
}
}

try {
logger.info(`Uploading file: ${originalName}`)
const result = await uploadFile(buffer, originalName, file.type, file.size)
Expand Down
Loading