Skip to content

v0.6.33: polling improvements, jsm forms tools, credentials reactquery invalidation, HITL docs#4080

Merged
waleedlatif1 merged 8 commits intomainfrom
staging
Apr 9, 2026
Merged

v0.6.33: polling improvements, jsm forms tools, credentials reactquery invalidation, HITL docs#4080
waleedlatif1 merged 8 commits intomainfrom
staging

Conversation

@waleedlatif1
Copy link
Copy Markdown
Collaborator

icecrasher321 and others added 8 commits April 8, 2026 23:22
* improvement(deploy): improve auto-generated version descriptions

* fix(deploy): address PR review - log dropdown errors, populate first-deploy details

* lint
…ng handlers (#4067)

* improvement(polling): fix correctness and efficiency across all polling handlers

- Gmail: paginate history API, add historyTypes filter, differentiate 403/429,
  fetch fresh historyId on fallback to break 404 retry loop
- Outlook: follow @odata.nextLink pagination, use fetchWithRetry for all Graph
  calls, fix $top alignment, skip folder filter on partial resolution failure,
  remove Content-Type from GET requests
- RSS: add conditional GET (ETag/If-None-Match), raise GUID cap to 500, fix 304
  ETag capture per RFC 9111, align GUID tracking with idempotency fallback key
- IMAP: single connection reuse, UIDVALIDITY tracking per mailbox, advance UID
  only on successful fetch, fix messageFlagsAdd range type, remove cross-mailbox
  legacy UID fallback
- Dispatch polling via trigger.dev task with per-provider concurrency key;
  fall back to synchronous Redis-locked polling for self-hosted

* fix(rss): align idempotency key GUID fallback with tracking/filter guard

* removed comments

* fix(imap): clear stale UID when UIDVALIDITY changes during state merge

* fix(rss): skip items with no identifiable GUID to avoid idempotency key collisions

* fix(schedules): convert dynamic import of getWorkflowById to static import

* fix(imap): preserve fresh UID after UIDVALIDITY reset in state merge

* improvement(polling): remove trigger.dev dispatch, use synchronous Redis-locked polling

* fix(polling): decouple outlook page size from total email cap so pagination works
* feat(jsm): add ProForma/JSM Forms discovery tools

Add three new tools for discovering and inspecting JSM Forms (ProForma) templates
and their structure, enabling dynamic form-based workflows:

- jsm_get_form_templates: List form templates in a project with request type bindings
- jsm_get_form_structure: Get full form design (questions, layout, conditions, sections)
- jsm_get_issue_forms: List forms attached to an issue with submission status

All endpoints validated against the official Atlassian Forms REST API OpenAPI spec.
Uses the Forms Cloud API base URL (jira/forms/cloud/{cloudId}) with X-ExperimentalApi header.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(jsm): add input validation and extract shared error parser

- Add validateJiraIssueKey for projectIdOrKey in templates and structure routes
- Add validateJiraCloudId for formId (UUID) in structure route
- Extract parseJsmErrorMessage to shared utils.ts (was duplicated across 3 routes)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(jsm): remove unused FORM_QUESTION_PROPERTIES constant

Dead code — the get_form_structure tool passes the raw design object
through as JSON, so this output constant had no consumers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(trigger): add ServiceNow webhook triggers

* fix(trigger): add webhook secret field and remove non-TSDoc comment

Add webhookSecret field to ServiceNow triggers (matching Salesforce pattern)
so users are prompted to protect the webhook endpoint. Update setup
instructions to include Authorization header in the Business Rule example.
Remove non-TSDoc inline comment in the block config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(trigger): add ServiceNow provider handler with event matching

Add dedicated ServiceNow webhook provider handler with:
- verifyAuth: validates webhookSecret via Bearer token or X-Sim-Webhook-Secret
- matchEvent: filters events by trigger type and table name using
  isServiceNowEventMatch utility (matching Salesforce/GitHub pattern)

The event matcher handles incident created/updated and change request
created/updated triggers with table name enforcement and event type
normalization. The generic webhook trigger passes through all events
but still respects the optional table name filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs(openapi): add Human in the Loop API endpoints

Add HITL pause/resume endpoints to the OpenAPI spec covering
the full workflow pause lifecycle: listing paused executions,
inspecting pause details, and resuming with input.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(openapi): add 403 and 500 responses to HITL endpoints

Address PR review feedback: add missing 403 Forbidden response
to all HITL endpoints (from validateWorkflowAccess), and 500
responses to resume endpoints that have explicit error paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel bot commented Apr 9, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Apr 9, 2026 9:01pm

Request Review

@cursor
Copy link
Copy Markdown

cursor bot commented Apr 9, 2026

PR Summary

Medium Risk
Changes touch webhook polling (Gmail/IMAP/Outlook/RSS) and add new ServiceNow/JSM endpoints, which can affect event ingestion correctness and external API behavior; however changes are scoped and largely additive with validation and tests added for workflow diff formatting.

Overview
Human-in-the-Loop API documentation expanded: OpenAPI now includes a new Human in the Loop tag and endpoints to list/get paused executions (/api/workflows/{id}/paused...) and fetch/resume pause contexts via /api/resume/..., plus docs updates describing async resume responses with jobId/statusUrl polling.

Jira Service Management Forms support added: Introduces three new JSM tools/endpoints (jsm_get_form_templates, jsm_get_form_structure, jsm_get_issue_forms) with corresponding Next.js API routes, tool configs, block UI params, and integration metadata updates.

Webhook/polling reliability improvements: Refines Gmail history polling (pagination, error handling, refresh historyId), overhauls IMAP polling to reuse a single connection and persist uidValidity per mailbox to avoid stale UIDs, adds Outlook pagination with retry and safer folder filtering, and improves RSS polling with ETag/Last-Modified caching, larger GUID history, and more robust item IDs.

Misc fixes and UX improvements: ServiceNow block now exposes triggers and registers a new servicenow webhook provider with secret verification and event matching; workflow deployment diff descriptions become more readable (field labels, capped output, edge/variable details) with extensive tests; credential mutations now invalidate OAuth credential queries; small cleanup in form submission stream draining, resume sync/async branching, schedule execute import, webhook polling lock handling, and a modal size tweak.

Reviewed by Cursor Bugbot for commit c393791. Configure here.

@waleedlatif1 waleedlatif1 merged commit 3c8bb40 into main Apr 9, 2026
11 of 12 checks passed
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c393791. Configure here.

} catch {
continue
}
const lastUidForMailbox = latestUidByMailbox[mailboxPath]
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMAP drops legacy UID fallback causing email reprocessing

High Severity

The fetchNewEmails function no longer falls back to config.lastProcessedUid when determining the last processed email UID. This affects older IMAP webhooks that rely solely on lastProcessedUid for tracking, causing them to lose their position and re-process previously handled emails.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c393791. Configure here.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 9, 2026

Greptile Summary

This release bundles several independent improvements: pagination fixes for Gmail (history API), IMAP (UIDVALIDITY tracking, shared client), and Outlook (hard-max pagination loop) polling handlers; cross-cache OAUTH_CREDENTIALS_KEY invalidation so credential mutations now bust the OAuth credential picker cache; new JSM ProForma/Forms discovery tools with proper input validation; and ServiceNow webhook triggers with a shared event-matching utility.

The polling refactors are generally well-structured — the releaseLock fix in the poll route and IMAP's shared-client pattern are clear improvements. The main concern is a cross-domain import of fetchWithRetry from @/lib/knowledge/documents/utils into Outlook polling code.

Confidence Score: 5/5

Safe to merge — all findings are P2 style/design suggestions with no blocking correctness issues.

The two issues raised are both P2: the cross-domain fetchWithRetry import is a code-organisation smell with no runtime impact, and the ServiceNow event-type set overlap is an edge-case documentation/UX concern rather than a data-correctness bug. All core logic changes (lock release, UIDVALIDITY tracking, Gmail pagination, credential cache invalidation) are sound.

apps/sim/lib/webhooks/polling/outlook.ts (cross-domain import), apps/sim/triggers/servicenow/utils.ts (ambiguous event sets)

Vulnerabilities

  • ServiceNow webhook auth accepts either Authorization: Bearer or X-Sim-Webhook-Secret headers (apps/sim/lib/webhooks/providers/servicenow.ts). Both validate the same secret, so no credential bypass is possible, but the dual-header surface is worth documenting.
  • JSM Forms API routes correctly use checkInternalAuth + validateJiraCloudId/validateJiraIssueKey guards before proxying. No injection surface identified.
  • No secrets are logged or exposed in new code paths.

Important Files Changed

Filename Overview
apps/sim/app/api/webhooks/poll/[provider]/route.ts Fixes a lock-release gap by moving the acquire/release into a nested try-finally, ensuring the distributed lock is always freed even when pollProvider throws.
apps/sim/lib/webhooks/polling/gmail.ts Adds paginated history-API fetching, restricts to messageAdded events, and refreshes historyId from the profile endpoint after an invalid-history error to prevent polling stalls.
apps/sim/lib/webhooks/polling/imap.ts Shares one ImapFlow client between fetch and mark-as-read phases, adds UIDVALIDITY tracking to discard stale UIDs after folder resets, and fixes the messageFlagsAdd call to pass the UID directly.
apps/sim/lib/webhooks/polling/outlook.ts Adds paginated Graph API fetching with a hard 200-email cap; replaces bare fetch with fetchWithRetry — but imports the utility from @/lib/knowledge/documents/utils, creating an unintended cross-domain dependency.
apps/sim/hooks/queries/credentials.ts Adds OAUTH_CREDENTIALS_KEY cross-cache invalidation to create, update, and delete credential mutations so the OAuth picker updates immediately after credential changes.
apps/sim/hooks/queries/credential-sets.ts Converts all mutation cache-invalidation callbacks from onSuccess to onSettled across 7 mutations, ensuring cache is reconciled even when a mutation fails.
apps/sim/triggers/servicenow/utils.ts Shared event-matching logic for ServiceNow triggers; INCIDENT_CREATED and CHANGE_REQUEST_CREATED share generic event tokens (insert, created) that can match both trigger types when no table name is present in the payload.
apps/sim/lib/webhooks/providers/servicenow.ts New ServiceNow webhook provider handler with token-based auth (accepts both Authorization Bearer and X-Sim-Webhook-Secret headers) and dynamic import of event-matching logic.
apps/sim/tools/jsm/get_issue_forms.ts New JSM tool for listing ProForma forms attached to an issue; delegates to /api/tools/jsm/forms/issue with proper OAuth params.
apps/sim/app/api/tools/jsm/forms/issue/route.ts API route for JSM issue forms with internal auth check, cloudId and issueKey validation, and well-structured error forwarding.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Cron: GET /api/webhooks/poll/provider] --> B{Auth OK?}
    B -- No --> Z[401 Unauthorized]
    B -- Yes --> C{Provider known?}
    C -- No --> Z2[404 Not Found]
    C -- Yes --> D[acquireLock provider-polling-lock]
    D -- Already locked --> E[202 Skipped]
    D -- Acquired --> F[pollProvider]
    F --> G{Provider}
    G -- gmail --> H[Gmail: paginated historyId loop - messageAdded filter only]
    G -- imap --> I[IMAP: shared ImapFlow client - UIDVALIDITY tracking]
    G -- outlook --> J[Outlook: paginated Graph API - hard cap 200 emails]
    G -- rss --> K[RSS: ETag / Last-Modified - MAX_GUIDS 500]
    H & I & J & K --> L[processEmails / trigger workflows]
    L --> M[releaseLock - always in finally]
    M --> N[200 Completed]

    subgraph ServiceNow Webhook
        SN1[POST /api/webhooks/provider/servicenow] --> SN2{verifyAuth Bearer or X-Sim-Webhook-Secret}
        SN2 -- fail --> SN3[401]
        SN2 -- pass --> SN4[matchEvent isServiceNowEventMatch]
        SN4 -- no match --> SN5[skip]
        SN4 -- match --> SN6[execute workflow]
    end
Loading

Reviews (1): Last reviewed commit: "docs(openapi): add Human in the Loop API..." | Re-trigger Greptile

@@ -1,5 +1,6 @@
import { htmlToText } from 'html-to-text'
import { pollingIdempotency } from '@/lib/core/idempotency/service'
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Cross-domain import of fetchWithRetry

fetchWithRetry is only exported from @/lib/knowledge/documents/utils, a module scoped to the knowledge-base feature. Importing it here couples the webhook polling domain to an unrelated feature — any refactor or removal of the knowledge module would silently break Outlook polling.

The function should be lifted to a shared location (e.g. @/lib/core/utils/fetch) and imported from there in both the knowledge code and here.

Suggested change
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
import { fetchWithRetry } from '@/lib/core/utils/fetch'

Comment on lines +167 to +191
const INCIDENT_CREATED = new Set([
'incident_created',
'insert',
'created',
'create',
'after_insert',
'afterinsert',
])

const INCIDENT_UPDATED = new Set([
'incident_updated',
'update',
'updated',
'after_update',
'afterupdate',
])

const CHANGE_REQUEST_CREATED = new Set([
'change_request_created',
'insert',
'created',
'create',
'after_insert',
'afterinsert',
])
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Ambiguous event-type sets across incident and change-request triggers

INCIDENT_CREATED and CHANGE_REQUEST_CREATED share the same generic tokens ('insert', 'created', 'create', 'after_insert', 'afterinsert'). When a payload arrives with no tableName field and no configuredTableName is set on the trigger, isServiceNowEventMatch will return true for both servicenow_incident_created and servicenow_change_request_created. Any workflow that users configure with the wrong default would fire on every insert from either table.

Consider requiring tableName in the payload or in trigger configuration before accepting generic event tokens, or document that table-name configuration is mandatory for users who have both trigger types active.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants