Skip to main content
Search...

chapter 1: your first upload

Set up the upload engine, add files, and track progress.

Goal

By the end of this chapter you will have a working upload pipeline. You will create an upload client, add a file, start the upload, and watch it progress through the state machine phases.

Loading diagram...

Step by Step

Install @gentleduck/upload


npm install @gentleduck/upload

npm install @gentleduck/upload

This is the only package you need. It includes the core engine, React bindings, and all built-in strategies. It works with npm, yarn, pnpm, and bun.

Define your types

Create src/upload.ts. First, define the intent map, cursor map, purpose union, and result type that describe your upload pipeline:

src/upload.ts
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// Intent map: which strategies exist and what data they need
type PhotoIntentMap = {
  post: PostIntent
}
 
// Cursor map: resume state per strategy
type PhotoCursorMap = {
  post: PostCursor
}
 
// Purpose: what kind of upload is this?
type PhotoPurpose = 'photo'
 
// Result: what the backend returns on completion
type PhotoResult = UploadResultBase & {
  url: string
}
src/upload.ts
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// Intent map: which strategies exist and what data they need
type PhotoIntentMap = {
  post: PostIntent
}
 
// Cursor map: resume state per strategy
type PhotoCursorMap = {
  post: PostCursor
}
 
// Purpose: what kind of upload is this?
type PhotoPurpose = 'photo'
 
// Result: what the backend returns on completion
type PhotoResult = UploadResultBase & {
  url: string
}

The intent map defines all upload strategies your app supports. Each key is a strategy name, and the value is the data shape your backend returns. For now we use the built-in PostIntent.

The purpose is a string union that categorizes uploads. PhotoDuck only has 'photo' for now, but a real app might have 'avatar' | 'cover' | 'attachment'.

Implement the UploadApi

The UploadApi is the contract between the upload engine and your backend. It has two required methods: createIntent and complete. For now, we mock them:

src/upload.ts
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    // In a real app, this calls your backend to get a presigned POST URL
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    // In a real app, this tells your backend the upload finished
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}
src/upload.ts
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    // In a real app, this calls your backend to get a presigned POST URL
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    // In a real app, this tells your backend the upload finished
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}

createIntent is called when a file enters the pipeline. Your backend decides the strategy, generates presigned URLs, and returns an intent object. The engine then hands this intent to the matching strategy.

complete is called after the bytes are transferred. Your backend can mark the file as uploaded in the database, generate thumbnails, or return metadata.

Create the upload client

Now wire everything together with createUploadClient:

src/upload.ts
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// ... types and api from above ...
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})
src/upload.ts
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// ... types and api from above ...
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})

createStrategyRegistry creates a registry where you register strategy implementations. PostStrategy() creates the built-in POST strategy for simple file uploads. createXHRTransport() creates a browser-native XHR transport with upload progress support.

Add files and start uploading

Create src/main.ts:

src/main.ts
import { uploadClient } from './upload'
 
// Add files to the pipeline
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Listen to progress events
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
// Listen to completion
uploadClient.on('upload.completed', (event) => {
  console.log(`Upload completed: ${event.localId}`, event.result)
})
 
// Listen to errors
uploadClient.on('upload.error', (event) => {
  console.log(`Upload failed: ${event.localId}`, event.error.message)
})
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })
src/main.ts
import { uploadClient } from './upload'
 
// Add files to the pipeline
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Listen to progress events
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
// Listen to completion
uploadClient.on('upload.completed', (event) => {
  console.log(`Upload completed: ${event.localId}`, event.result)
})
 
// Listen to errors
uploadClient.on('upload.error', (event) => {
  console.log(`Upload failed: ${event.localId}`, event.error.message)
})
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })

dispatch is the single entry point for all commands. addFiles puts files into the pipeline. startAll begins uploading every file that is in the ready phase.

How the State Machine Works

Every upload item flows through a sequence of phases. The phase determines what the engine is doing with the file at any given moment:

Loading diagram...

PhaseMeaning
validatingFile is being checked against validation rules (size, type)
creating_intentEngine is calling api.createIntent() to get upload instructions
readyIntent received, waiting for user or autoStart to trigger upload
queuedUpload requested, waiting for a concurrency slot
uploadingBytes are being transferred
completingBytes sent, calling api.complete() to finalize
completedUpload finished successfully
errorSomething went wrong (may be retryable)
pausedUpload paused by user (resumable strategies can pick up where they left off)
canceledUpload canceled by user

When you dispatch({ type: 'addFiles', ... }), the engine:

  1. Creates a local ID and fingerprint for each file
  2. Moves the item to validating phase
  3. Runs validation rules (Chapter 5)
  4. On success, moves to creating_intent and calls api.createIntent()
  5. On success, moves to ready with the intent attached

When you dispatch({ type: 'start', localId }) or dispatch({ type: 'startAll' }):

  1. Items in ready move to queued
  2. The scheduler picks items from the queue (respecting maxConcurrentUploads)
  3. The matching strategy's start() method runs, moving the item to uploading
  4. On success, the item moves to completing and the engine calls api.complete()
  5. On success, the item reaches completed

Commands Reference

All user actions go through dispatch(). Here are the available commands:

CommandEffect
{ type: 'addFiles', files: File[], purpose: P }Add files to the pipeline
{ type: 'start', localId: string }Start a single upload
{ type: 'startAll', purpose?: P }Start all ready uploads (optionally filtered by purpose)
{ type: 'pause', localId: string }Pause a single upload
{ type: 'pauseAll', purpose?: P }Pause all active uploads
{ type: 'resume', localId: string }Resume a paused upload
{ type: 'cancel', localId: string }Cancel a single upload
{ type: 'cancelAll', purpose?: P }Cancel all uploads
{ type: 'retry', localId: string }Retry a failed upload
{ type: 'remove', localId: string }Remove an item from state
{ type: 'rebind', localId: string, file: File }Re-attach a File to a persisted paused item

Events Reference

Subscribe to events with uploadClient.on(eventName, callback). The on method returns an unsubscribe function:

const unsub = uploadClient.on('upload.progress', (event) => {
  console.log(event.pct)
})
 
// Later, stop listening
unsub()
const unsub = uploadClient.on('upload.progress', (event) => {
  console.log(event.pct)
})
 
// Later, stop listening
unsub()
EventPayload
file.added{ localId, purpose, file, fingerprint }
file.rejected{ file, reason }
intent.creating{ localId }
intent.created{ localId, intent }
intent.failed{ localId, error, retryable }
upload.started{ localId }
upload.progress{ localId, pct, uploadedBytes, totalBytes }
upload.paused{ localId, cursor }
upload.canceled{ localId }
upload.completing{ localId }
upload.completed{ localId, result, completedBy }
upload.error{ localId, error, retryable }

Reading State with getSnapshot

You can read the current state at any time:

const snapshot = uploadClient.getSnapshot()
 
// snapshot.items is a Map<string, UploadItem>
for (const [localId, item] of snapshot.items) {
  console.log(`${localId}: phase=${item.phase}, purpose=${item.purpose}`)
 
  if (item.phase === 'uploading') {
    console.log(`  progress: ${item.progress.pct.toFixed(1)}%`)
  }
 
  if (item.phase === 'error') {
    console.log(`  error: ${item.error.message} (retryable: ${item.retryable})`)
  }
}
const snapshot = uploadClient.getSnapshot()
 
// snapshot.items is a Map<string, UploadItem>
for (const [localId, item] of snapshot.items) {
  console.log(`${localId}: phase=${item.phase}, purpose=${item.purpose}`)
 
  if (item.phase === 'uploading') {
    console.log(`  progress: ${item.progress.pct.toFixed(1)}%`)
  }
 
  if (item.phase === 'error') {
    console.log(`  error: ${item.error.message} (retryable: ${item.retryable})`)
  }
}

The UploadItem type is a discriminated union on phase. TypeScript narrows the type when you check the phase, giving you access to phase-specific fields like progress, intent, error, and cursor.

Waiting for Uploads

The waitFor method returns a promise that resolves when all given uploads reach a terminal state (completed, error, or canceled):

uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Get the localId from the snapshot
const snapshot = uploadClient.getSnapshot()
const localId = Array.from(snapshot.items.keys())[0]
 
uploadClient.dispatch({ type: 'start', localId })
 
const [outcome] = await uploadClient.waitFor([localId])
 
if (outcome.status === 'completed') {
  console.log('Upload done:', outcome.result)
} else if (outcome.status === 'error') {
  console.log('Upload failed:', outcome.error)
}
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Get the localId from the snapshot
const snapshot = uploadClient.getSnapshot()
const localId = Array.from(snapshot.items.keys())[0]
 
uploadClient.dispatch({ type: 'start', localId })
 
const [outcome] = await uploadClient.waitFor([localId])
 
if (outcome.status === 'completed') {
  console.log('Upload done:', outcome.result)
} else if (outcome.status === 'error') {
  console.log('Upload failed:', outcome.error)
}

Checkpoint

Your project should look like this:

photoduck/
  src/
    upload.ts    -- types + api + client
    main.ts      -- dispatch commands + event listeners
  package.json
  tsconfig.json
Full src/upload.ts
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// --- Types ---
 
type PhotoIntentMap = {
  post: PostIntent
}
 
type PhotoCursorMap = {
  post: PostCursor
}
 
type PhotoPurpose = 'photo'
 
type PhotoResult = UploadResultBase & {
  url: string
}
 
// --- Backend API ---
 
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}
 
// --- Upload Client ---
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// --- Types ---
 
type PhotoIntentMap = {
  post: PostIntent
}
 
type PhotoCursorMap = {
  post: PostCursor
}
 
type PhotoPurpose = 'photo'
 
type PhotoResult = UploadResultBase & {
  url: string
}
 
// --- Backend API ---
 
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}
 
// --- Upload Client ---
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})
Full src/main.ts
import { uploadClient } from './upload'
 
// Listen to events
uploadClient.on('file.added', (event) => {
  console.log(`File added: ${event.localId} (${event.purpose})`)
})
 
uploadClient.on('intent.created', (event) => {
  console.log(`Intent created: ${event.localId}`, event.intent)
})
 
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
uploadClient.on('upload.completed', (event) => {
  console.log(`Completed: ${event.localId}`, event.result)
})
 
uploadClient.on('upload.error', (event) => {
  console.log(`Error: ${event.localId}`, event.error.message)
})
 
// Add a file
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })
import { uploadClient } from './upload'
 
// Listen to events
uploadClient.on('file.added', (event) => {
  console.log(`File added: ${event.localId} (${event.purpose})`)
})
 
uploadClient.on('intent.created', (event) => {
  console.log(`Intent created: ${event.localId}`, event.intent)
})
 
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
uploadClient.on('upload.completed', (event) => {
  console.log(`Completed: ${event.localId}`, event.result)
})
 
uploadClient.on('upload.error', (event) => {
  console.log(`Error: ${event.localId}`, event.error.message)
})
 
// Add a file
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })

Chapter 1 FAQ

Why use dispatch instead of direct method calls like client.addFiles()?

The dispatch pattern (command pattern) centralizes all state mutations through a single entry point. This makes the engine predictable, testable, and debuggable. Every action goes through the same pipeline: command -> reducer -> state update -> effects -> events. Plugins and hooks can observe all commands in one place. It also allows the engine to batch and schedule work without race conditions.

What is a "purpose" and why do I need it?

A purpose is a string that categorizes the upload. It serves three roles: (1) your backend uses it to decide where to store the file (e.g., avatars go to one bucket, photos to another), (2) the engine uses it to apply purpose-specific validation rules (e.g., avatars max 5MB, photos max 50MB), and (3) commands like startAll and cancelAll can filter by purpose. Define your purposes as a string union type for type safety.

What is a file fingerprint?

When a file is added, the engine computes a fingerprint from the file name, size, type, and last modified timestamp. This fingerprint is used to deduplicate files (avoid adding the same file twice) and to match files on resume. You can provide a custom fingerprinting function via the fingerprint option if you need SHA-256 hashing or other schemes.

What is a localId?

The localId is a client-generated unique identifier for each upload item. It is created when you addFiles and persists throughout the upload lifecycle. It is different from the fileId which comes from your backend via the intent. Use localId for all client-side operations (start, pause, cancel, retry). Use fileId for server-side references.

Can uploads start automatically without calling startAll?

Yes. Pass autoStart in the config: createUploadClient({ ..., config: { autoStart: ['photo'] } }). This makes files with purpose 'photo' start uploading immediately after the intent is created. You can also pass a function: autoStart: (purpose) => purpose !== 'draft' for dynamic control.

How many files upload at the same time?

The default maxConcurrentUploads is determined by the engine defaults. You can configure it: createUploadClient({ ..., config: { maxConcurrentUploads: 3 } }). When more files are ready than slots available, they queue in the queued phase and the scheduler picks them up as slots become free.

What is the difference between subscribe() and on()?

subscribe(listener) fires on every state change, regardless of which item or phase changed. It is designed for UI frameworks that need to re-render on any update (React uses this internally via useSyncExternalStore). on(eventName, callback) fires only for specific events like upload.progress or upload.completed. Use subscribe for rendering and on for side effects (logging, analytics, notifications).

Why use XHR instead of fetch?

The createXHRTransport() uses XMLHttpRequest because fetch does not yet support upload progress events in a standard cross-browser way. XHR provides xhr.upload.onprogress which gives deterministic byte-level progress tracking. The transport is an abstraction -- you can swap it for a fetch-based or test transport by implementing the UploadTransport interface.


Next: Chapter 2: Strategies & Backends