Errors and retries

What the engine does when your code fails, and how you change it. All the code is in examples/ts/errors.ts and examples/go/errors/main.go.

A failed step is tried again

A step fails when its callback throws (Go: returns an error, or panics). The engine then retries that step: it waits, then calls your app again. Steps that are done keep their results and do not run again.

• Each step gets retries + 1 tries. retries is a function option: default 4, from 0 to 20. It counts per step: every step gets its own tries.
• The wait between tries doubles: about 1 s, 2 s, 4 s, 8 s and so on, up to 1 hour. Each wait is cut by a random part (to between half and all of it), so many failing runs do not all retry at the same moment. On a local engine, the waits between the four tries of a failing step were 0.8 s, 1.9 s and 3.4 s.
• An error outside a step (in the function body) fails that call, and the engine retries the call the same way.

The dashboard shows each failed try of a step, with its error, under Failed tries on the step.

When a step runs out of tries

The step's error comes back into your function: the step call throws a StepError (Go: returns a *step.StepError). It has the step's error name and message. If you do not catch it, the run ends as Failed.

Catch it to undo earlier work. Put the undo in a step of its own:

TypeScript

const chargeOrder = flow.createFunction(
  { id: "charge-order", retries: 3 },
  { event: "order/checkout" },
  async ({ event, step }) => {
    const hold = await step.run("hold-stock", () => holdStock(event.data.orderId));

    try {
      await step.run("charge-card", ({ idempotencyKey }) => chargeCard(event.data.card, idempotencyKey));
    } catch (err) {
      // A StepError means the step failed for good: no tries are left.
      if (!(err instanceof StepError)) throw err;
      // Undo the earlier step, in a step of its own.
      await step.run("release-stock", () => releaseStock(hold.holdId));
      return { status: "payment-failed", reason: err.message };
    }

    return { status: "paid" };
  },
);

Go

// Checkout is the data of an order/checkout event.
type Checkout struct {
	OrderID string `json:"orderId"`
	Card    string `json:"card"`
}

func addChargeOrder(client *flow.Client) error {
	_, err := flow.CreateFunction(client,
		flow.FunctionOpts{ID: "charge-order", Retries: flow.IntPtr(3)},
		flow.EventTrigger("order/checkout", nil),
		func(ctx context.Context, in flow.Input[Checkout]) (any, error) {
			holdID, err := step.Run(ctx, "hold-stock", func(ctx context.Context) (string, error) {
				return holdStock(ctx, in.Event.Data.OrderID)
			})
			if err != nil {
				return nil, err
			}

			_, err = step.Run(ctx, "charge-card", func(ctx context.Context) (string, error) {
				return chargeCard(ctx, in.Event.Data.Card, step.IdempotencyKey(ctx))
			})
			if err != nil {
				// A *step.StepError means the step failed for good: no tries are left.
				var failed *step.StepError
				if !errors.As(err, &failed) {
					return nil, err
				}
				// Undo the earlier step, in a step of its own.
				if _, err := step.Run(ctx, "release-stock", func(ctx context.Context) (bool, error) {
					return true, releaseStock(ctx, holdID)
				}); err != nil {
					return nil, err
				}
				return map[string]string{"status": "payment-failed", "reason": failed.Message}, nil
			}

			return map[string]string{"status": "paid"}, nil
		},
	)
	return err
}

On a local engine, a card set to fail every time used all 4 tries of charge-card; then release-stock ran, and the run Completed with {"status": "payment-failed", "reason": "the payment API timed out"}.

In TypeScript, catch only what you mean to: rethrow anything that is not a StepError. In Go, if you recover() around step calls, call step.Repanic(r) first.

Stop retrying

When trying again can not help (bad input, a record that is gone), throw NonRetriableError (Go: wrap the error with flow.NoRetryError(err)).

• In a step: the step fails at once, with no more tries. The StepError comes back into your function as above. On a local engine, a declined card failed charge-card after 1 try, and release-stock ran.
• Outside a step: the run fails at once.

import { NonRetriableError } from "@clevix/flow";

throw new NonRetriableError("no such user");

return nil, flow.NoRetryError(errors.New("no such user"))

Retry later

When an API tells you when to come back, throw RetryAfterError with a time (Go: flow.RetryAtError(err, at)). The engine tries the step again at that time.

TypeScript

const pushContacts = flow.createFunction(
  { id: "push-contacts" },
  { event: "crm/contacts-changed" },
  async ({ event, step }) => {
    return step.run("push-to-crm", async () => {
      const answer = await crmUpdate(event.data.contactIds);
      if (answer.status === 429) {
        // Try this step again when the API says, not on the normal schedule.
        throw new RetryAfterError("the CRM is rate limiting us", `${answer.retryAfterSeconds}s`);
      }
      return answer.updated;
    });
  },
);

Go

// Contacts is the data of a crm/contacts-changed event.
type Contacts struct {
	ContactIDs []string `json:"contactIds"`
}

func addPushContacts(client *flow.Client) error {
	_, err := flow.CreateFunction(client,
		flow.FunctionOpts{ID: "push-contacts"},
		flow.EventTrigger("crm/contacts-changed", nil),
		func(ctx context.Context, in flow.Input[Contacts]) (any, error) {
			return step.Run(ctx, "push-to-crm", func(ctx context.Context) (int, error) {
				status, retryAfter, updated := crmUpdate(ctx, in.Event.Data.ContactIDs)
				if status == http.StatusTooManyRequests {
					// Try this step again when the API says, not on the normal schedule.
					return 0, flow.RetryAtError(errors.New("the CRM is rate limiting us"), time.Now().Add(retryAfter))
				}
				return updated, nil
			})
		},
	)
	return err
}

In TypeScript the time is a Date, ms since 1970, or a duration from now like "30s". It still counts as a try.

When a run fails for good

TypeScript: the onFailure option runs after a run fails for good (all tries used, or a NonRetriableError).

TypeScript

const importUsers = flow.createFunction(
  {
    id: "import-users",
    // Runs once a run of import-users fails for good.
    onFailure: async ({ error, event, step }) => {
      // event is inngest/function.failed; event.data.event is the event that
      // started the failed run.
      await step.run("alert-the-team", () =>
        postToChat(`import of ${event.data.event.data.file} failed: ${error.message}`),
      );
    },
  },
  { event: "import/requested" },
  async ({ event, step }) => {
    const rows = await step.run("read-file", () => readFile(event.data.file));
    if (rows.length === 0) {
      // Trying again can not help: fail now, with no more retries.
      throw new NonRetriableError(`${event.data.file} is empty`);
    }
    return { rows: rows.length };
  },
);

Go

// Import is the data of an import/requested event.
type Import struct {
	File string `json:"file"`
}

func addImportUsers(client *flow.Client) error {
	importUsers, err := flow.CreateFunction(client,
		flow.FunctionOpts{ID: "import-users"},
		flow.EventTrigger("import/requested", nil),
		func(ctx context.Context, in flow.Input[Import]) (any, error) {
			rows, err := step.Run(ctx, "read-file", func(ctx context.Context) ([]string, error) {
				return readFile(ctx, in.Event.Data.File)
			})
			if err != nil {
				return nil, err
			}
			if len(rows) == 0 {
				// Trying again can not help: fail now, with no more retries.
				return nil, flow.NoRetryError(fmt.Errorf("%s is empty", in.Event.Data.File))
			}
			return map[string]int{"rows": len(rows)}, nil
		},
	)
	if err != nil {
		return err
	}

	// Go has no onFailure option: make a second function on the event the engine
	// sends when a run fails for good, for this function only.
	type Failed struct {
		FunctionID string `json:"function_id"` // "<appId>-<functionId>"
		RunID      string `json:"run_id"`
		Error      struct {
			Name    string `json:"name"`
			Message string `json:"message"`
		} `json:"error"`
		Event flow.GenericEvent[Import] `json:"event"` // the event that started the failed run
	}
	_, err = flow.CreateFunction(client,
		flow.FunctionOpts{ID: "import-users-failure"},
		flow.EventTrigger("inngest/function.failed",
			flow.StrPtr("event.data.function_id == '"+importUsers.FullyQualifiedID()+"'")),
		func(ctx context.Context, in flow.Input[Failed]) (any, error) {
			failed := in.Event.Data
			return step.Run(ctx, "alert-the-team", func(ctx context.Context) (bool, error) {
				return true, postToChat(ctx, fmt.Sprintf("import of %s failed: %s", failed.Event.Data.File, failed.Error.Message))
			})
		},
	)
	return err
}

How it works: when a run fails, the engine sends the event inngest/function.failed. onFailure is a second function, with the ID <id>-failure (here import-users-failure), whose trigger is that event, for this function only. Its event's data has:

Field	What
function_id	The failed function, as <appId>-<functionId>
run_id	The failed run
error	name, message and stack
event	The event that started the failed run
attempts	How many times the call that failed the run was tried. A NonRetriableError in a step gives 1. 0 when the run failed with no failed try, for example when its state grew too large. In TypeScript, onFailure also gets it as attempts.

Go has no onFailure option: make that second function yourself, as above. On a local engine, the empty file failed import-users at once, and the handler posted import of empty-users.csv failed: empty-users.csv is empty.

• It runs only when a run ends Failed, never when it is Cancelled. Runs started by step.invoke count too.
• To act on a cancelled run, trigger a function on inngest/function.cancelled, the same way. Its data has the same fields but attempts; error is there when the cancel had a reason. A run that passes timeouts.finish is cancelled, so it sends this event, not the failed one.
• A run started by one of these two events sends neither, so a handler that fails can not start itself again.

Timeouts

TypeScript

const exportData = flow.createFunction(
  {
    id: "export-data",
    timeouts: {
      step: "2m", // one call to your app may take up to 2 minutes
      finish: "1h", // the whole run may take up to 1 hour, then it is cancelled
    },
  },
  { event: "export/requested" },
  async ({ event, step }) => {
    return step.run("export", () => runExport(event.data.accountId));
  },
);

Go

// ExportRequest is the data of an export/requested event.
type ExportRequest struct {
	AccountID string `json:"accountId"`
}

func addExportData(client *flow.Client) error {
	_, err := flow.CreateFunction(client,
		flow.FunctionOpts{
			ID: "export-data",
			Timeouts: &flow.Timeouts{
				Step:   2 * time.Minute, // one call to your app may take up to 2 minutes
				Finish: time.Hour,       // the whole run may take up to 1 hour, then it is cancelled
			},
		},
		flow.EventTrigger("export/requested", nil),
		func(ctx context.Context, in flow.Input[ExportRequest]) (any, error) {
			return step.Run(ctx, "export", func(ctx context.Context) (string, error) {
				return runExport(ctx, in.Event.Data.AccountID)
			})
		},
	)
	return err
}

• timeouts.step limits how long one call to your app may take. Default 5 minutes; any value is kept between 1 second and 14 minutes. A call that takes longer fails with TimeoutError, and is retried like any other failure. For longer work, split it into more steps.
• timeouts.finish limits how long the whole run may take, counted from its first call to your app. Time spent queued before that does not count; every sleep and wait after it does. A run still going then is cancelled with FunctionTimeout, and the engine sends inngest/function.cancelled. There is no limit by default.

Error names

The error's name tells you where it came from:

Name	Where it comes from
Error (or your error's own name in TypeScript)	Your code threw or returned an error
NonRetriableError	NonRetriableError / flow.NoRetryError
RetryAfterError	RetryAfterError / flow.RetryAtError
Panic	Go: a panic in a step.Run callback (it counts as a failed try)
TimeoutError	A call took longer than timeouts.step
FunctionTimeout	The run took longer than timeouts.finish
InvokeTimeout	step.invoke waited longer than its timeout
FunctionNotFound	The engine called, or invoked, a function that is not there
OutputTooLarge	A step or the function returned more than 4 MB
StateOverflowed	The run holds more than 32 MB in all
AppUnavailable	The app could not be reached for 7 days

Going to production has more, with what to do.