df — the data flow (.df)

Audience: people building a tool, and the AI working on their behalf. Both read this same page.

What it is

A .df is the data-fetching step of one widget or one page: call the network, pull fields out, compute derived values, and finally expose the keys the rendering layer needs. It only fetches and transforms — no UI, no navigation, no user involvement — because it runs on the rendering path, possibly at moments the user cannot see (a scheduled home-screen widget refresh, the dashboard's first frame).

Anything that needs the user's finger (a confirmation, opening a page, writing parameters back, a haptic tap) belongs to the other file type, .af. The two are two capability tiers of the same engine:

Getting the file type wrong has a very specific symptom: call an .af with runDataFlow from H5 and the app answers -6 flow not found. For how to write .af and the full action list, see numable docs af.

A minimal working example

A flow that really calls the network, really fails, and really exposes keys:

{
  "version": 1,
  "actions": [
    {
      "id": "resp",
      "action": "request",
      "params": {
        "url": "https://hn.algolia.com/api/v1/search?tags=front_page",
        "method": "GET",
        "formatType": "json",
        "timeout": "8000"
      }
    },
    {
      "op": "set",
      "props": { "key": "_b", "value": "1" },
      "_note": "Barrier: the step immediately after request/concurrent cannot see the result; put one empty set in between"
    },
    {
      "op": "if",
      "props": { "val": "$[if::(gt::(length::(${resp.hits}),0),0,1)]" },
      "items": [ { "action": "error", "params": { "errorMsg": "Could not load the leaderboard" } } ],
      "_note": "Essential field missing, so fail the flow: the platform falls back to the last good data. Never render an empty payload"
    },
    { "op": "set", "props": { "key": "t0", "value": "${resp.hits[0].title}" } },
    { "op": "set", "props": { "key": "p0", "value": "${resp.hits[0].points}" } },
    { "op": "set", "props": { "key": "id0", "value": "${resp.hits[0].objectID}" } },
    { "op": "set", "props": { "key": "at", "value": "$[formatDate::(${@time.nowMs},HH:mm)]" } },
    { "op": "set", "props": { "key": "hasT0", "value": "$[if::(gt::(length::(${t0}),0),1,0)]" } },
    { "action": "resultFilter", "params": { "keys": ["t0", "p0", "id0", "at", "hasT0"] } }
  ]
}

Run it:

numable run <package> --flow top1          # the name = the widget name (the .xwidget filename); for a page flow use the filename (e.g. rates) or page/rates

How to write it

File structure

Two top-level keys are required: { "version": 1, "actions": [ … ] }. Optionally i18n (the file's own content table, same shape as in an .af, referenced from the body as ${@i18n.key} — see "Fetching by language and producing text" below). A _note string can be added anywhere as a comment — keys starting with an underscore are always ignored by the runtime.

The three shapes of a step

Writing concurrent / sequential as {"op": "concurrent"} makes the whole block be skipped silently — the symptom is that the section "succeeds" in 3 milliseconds with every field empty.

How to write concurrency

Two independent requests run side by side, and the total time is the slower of the two:

{ "action": "concurrent", "items": [
    { "id": "a", "action": "request", "params": { "url": "https://api.example.com/a", "formatType": "json" } },
    { "id": "b", "action": "request", "params": { "url": "https://api.example.com/b", "formatType": "json" } } ] }

Put a barrier right after it, and only then start using ${a.…} / ${b.…}:

{ "op": "set", "props": { "key": "_b", "value": "1" } }

• The params of a concurrent branch are evaluated at the moment the branch is queued, against the data as it stands then. So branches cannot see each other, and they cannot see a key written by an op:set earlier in the same batch either — which is where the barrier rule comes from (a sequential run evaluates each step as it goes).
• If any branch fails, the remaining siblings are aborted. Keep branches that are allowed to fail out of the group: run them sequentially on their own and fall back after testing for emptiness.
• If the composite step carries an id, all branch results land inside that one key (${group.a.field}); without an id they are flattened into the shared data scope (${a.field}). The read path downstream differs between the two, so do not mix them.
• policy (wait / skip / detach) applies to action steps and composite steps only; written on an op:* step it does nothing.

id is the result key

When a step with an id finishes, its result is written into the data scope under that id, and later steps read it with ${id.field}. Two rules:

• No id = the result is discarded.
• A null result is discarded too — the key never appears in the data scope at all, so every downstream ${k} is permanently empty and nothing is reported. That is why data.get must always carry a default (even just {}): otherwise a cache miss empties everything downstream.

The actions you may use (allowlist; anything else fails)

Explicitly forbidden: toast / showLoading / hideLoading, plus every app-injected action (ui.* / nav.* / xpage.* / widget.* / installBundle). Those may appear only in an .af; numable check catches it (E).

• The pattern of clearCookie is a case-insensitive regex matched against the domain (not against cookie names): {"action":"clearCookie","params":{"pattern":"\\.example\\.com$"}}.
• Do not use op:include: in a data flow and in an action flow alike it always expands to zero steps, so that section does nothing while the flow reports success. There are only two ways to reuse common steps — copy them, or split them into their own .df and bind it from several depends.

Methods used inside expressions (things like $[calc::(…)]) are a separate table; see numable docs methods for the full list. Look a method up before you write its name: an unregistered method silently evaluates to empty, the flow still reports success, and the number after that symbol on the widget is simply gone.

Every parameter of request

GET: parameters go in queryParams

{ "id": "resp", "action": "request", "params": {
    "url": "https://api.example.com/v1/quote",
    "method": "GET",
    "queryParams": { "symbol": "${sym}", "range": "1d" },
    "formatType": "json",
    "timeout": "8000" } }

POST: body or formData, not both

{ "id": "resp", "action": "request", "params": {
    "url": "https://api.example.com/graphql",
    "method": "POST",
    "header": { "Content-Type": "application/json" },
    "body": "{\"query\":\"{ viewer { login } }\"}",
    "formatType": "json",
    "timeout": "8000" } }

• formData is assembled into the request only when method is POST. Carry it on a PUT / PATCH and it is silently dropped: the request goes out with body instead (an empty body if you wrote none), the server answers 400, and the flow itself looks perfectly healthy.
• If you write a body but no Content-Type in header, it is sent as application/json; charset=utf-8. Header keys are case-sensitive: content-type does not count and it still goes out as JSON.
• Redirects are followed automatically, but every hop's host must be in manifest.network — a hop that leaves the whitelist is refused (and credentials are not carried along). Same-host redirects, like GitHub's renamed-repository redirect, need nothing extra. numable run / render check every hop the same way: when a hop leaves the allowlist the request fails with network_blocked:redirect_escaped:<host> and the log says "X redirected to Y; Y is not in manifest.network". The simplest fix is to request the final address directly.
• To interpolate a variable into the body, land the value in a top-level key with op:set first and then build the string. This matters most for GraphQL: a $variable in the query string and a ${} expression on the same line are very easy to misread, so splicing the value straight into the query string is the least error-prone.

The six values of formatType

• formatType: "json" returns null when parsing fails — not the raw text, and no error either, so every ${resp.…} is empty. Since that looks the same as a wrong path, switch to string first and look at the raw text.
• When the response body is empty, string / json / xml / base64 all yield null; and a null result is not written into the data scope, so the key resp does not exist at all. tsv / csv are the exception: an empty body and a header-only table both give [].
• Any value outside these six falls through a "try JSON, then XML, then raw text" chain; the resulting shape is unstable, so do not rely on it.
• For deeply nested XML such as RSS, string + htmlParse or split:: is more reliable than formatType: "xml": strict XML parsing fails on the whole document at a single unclosed tag, while HTML parsing tolerates it.
• A gzip-compressed response (bytes starting with 1F 8B) is decompressed automatically first and then parsed by formatType — the same for all six values. If it exceeds 4MB once decompressed, or the compressed data is corrupt, the request fails with body_too_large / decode_failed in the error message — you never get an empty array posing as "0 today".
• tsv / csv details: decoded as UTF-8 with a leading BOM removed; blank lines are skipped; header cells are trimmed, data cells are kept as-is; a row shorter than the header is padded with empty strings, a longer one loses the extras. csv understands double-quoted fields (commas and line breaks allowed inside, "" for one quote); tsv does not treat quotes specially. To do arithmetic, hand the values to methods like groupSum / parseNumber; see numable docs methods.
• When a report has many columns and you only need a few, always set columns: keeping a handful out of dozens makes the data several times smaller, which matters most in desktop widgets.
• A package that uses tsv / csv (or mapField / groupSum / convertSum) must set manifest.minEngine to "3.0.0"; older app versions do not know them and do not report an error either (G51).

Referencing a credential

{ "id": "me", "action": "request", "params": {
    "url": "https://api.github.com/user", "formatType": "json", "credential": "gh" } }

The value of credential must be a literal id declared in manifest.credentials, never a ${} expression — a dynamic value is silently passed through at runtime and the request goes out anonymous, which reaches you as a 401 or as empty data. For the declaration format see numable docs layout; keys for local testing go in .numable/params/_credentials.json.

The secret itself never appears in the flow: injection happens at the app's network boundary. Also never put a credential or anything derived from it into resultFilter — once exposed it lands in the render cache and in shared screenshots. To let RCN know whether a credential is bound, expose a hasToken flag instead.

Scraping a page: htmlParse / xmlParse

When there is no API, scrape the page. Fetch the raw text with formatType: "string" and hand it to htmlParse:

{ "id": "page", "action": "htmlParse", "params": {
    "content": "${resp}",
    "rules": [
      { "key": "title", "selector": "h1.site-title", "defaultValue": "" },
      { "key": "posts", "selector": "ul.list > li", "isArray": "true",
        "items": [
          { "key": "name", "selector": "a.t" },
          { "key": "url", "selector": "a.t", "extractFrom": "href", "prefix": "https://example.com" },
          { "key": "hot", "selector": "span.n", "regex": "(\\d+)", "defaultValue": "0" }
        ] } ] } }

rules is an array, not an object — write it as an object like {"title": "h1"} and not a single key comes out, with no error. Each entry is one field:

Three traps:

• When a rule without isArray extracts nothing, that key is not written to the result at all (not even as an empty string). Downstream ${page.title} is permanently empty and nothing is reported — give every such rule a defaultValue, or raise a hasX flag as in rule 3 of the six below.
• Selectors run against the raw HTML. Content that the page renders with scripts cannot be scraped; the symptom is a selector that looks entirely correct returning nothing every time. When that happens, go find the endpoint the page itself calls.
• The rules of xmlParse have the same shape as htmlParse, plus an encoding (fill it in only when the parse comes out garbled).

Structured op:set

{ "op": "set", "props": { "key": "rows", "value": ["${v1}", "${v2}", "${v3}"] } }

value may be an array or object literal: the structure is preserved and the leaves are evaluated one by one. An array destined for sum:: / max:: has to be built this way — writing [...] inside a method argument string does not work, and that statistic silently disappears.

op:if and the two loops

{ "op": "if", "props": { "val": "$[eq::(${fresh},0)]" }, "items": [ … ] }

The condition key of op:if is props.val. Writing cond raises no error: the condition is permanently false, the branch body silently never runs, and the flow still reports success.

Use op:if only to dispatch actions, not to compute values: a key set by op:set inside a branch is unreachable outside it. Compute values with a nested if:: inside an expression.

For a fixed number of rounds, use op:for:

{ "op": "for", "props": { "count": "5", "index": "i" },
  "items": [ { "op": "set", "props": { "key": "y_${i}", "value": "$[calc::(48+${i}*48)]" } } ] }

To walk a collection, use op:forEach:

{ "op": "forEach", "props": { "items": "${resp.list}", "key": "it", "index": "i" },
  "items": [ { "op": "set", "props": { "key": "n_${i}", "value": "${it.name}" } } ] }

• Given an array, forEach walks it in order; given an object it walks the values in key order; given a single string or number it treats it as one element; given nothing (null, or a missing key) it runs not once, and reports no error.

• Loop variables are gone once the loop ends: ${i} / ${it} exist only inside items, and reading them afterwards gives empty, not the value from the last round.

• An op:set inside the body writes to the flow's shared data scope, not to a per-round copy: the same key is overwritten every round and only the last one survives. To keep every round, put the index in the key name (y_${i} above). A . in a key name is read as a path separator, so avoid stray dots in keys.

• ⚠️ But the first node of each round is one beat behind — it reads the snapshot taken as the loop was entered, not what the previous round wrote. From the second node on you get live values. This is the same thing as the barrier you need after request / concurrent / data.get; here the trigger is simply the loop itself.

  When it bites: whenever the body reads a key that accumulates across rounds. The classic case is "pick the first element that matches" — you raise a pickHas flag and guard the condition with eq::(${pickHas},0):

  { "op": "forEach", "props": { "items": "${resp.list}", "key": "it", "index": "i" }, "items": [
    { "op": "set", "props": { "key": "_lb", "value": "1" } },
    { "op": "set", "props": { "key": "sel", "value": "$[if::(and::(eq::(${pickHas},0),…),1,0)]" } },
    { "op": "set", "props": { "key": "pick",    "value": "$[if::(eq::(${sel},1),${it.date},${pick})]" } },
    { "op": "set", "props": { "key": "pickHas", "value": "$[if::(eq::(${sel},1),1,${pickHas})]" } } ] }
  

  That first _lb line is the loop barrier: its value is irrelevant, its only job is to burn the one beat that reads the snapshot so the nodes after it see a live pickHas. Without it, pickHas is forever the pre-loop 0 in the eyes of each round's first node, the guard never fires, and you pick the last matching element instead of the first.

  Why this one matters: it does not error. The flow reports success, the key set matches resultFilter exactly, check is 0 error, run is all green, and the widget shows a perfectly plausible wrong number. None of the three layers catches it — you only find it when you happen to know the right answer. (Reproduced 2026-09-10: on one real flow, removing the barrier turned "Mid-Autumn Festival · 15 days" into "National Day · 21 days", and both runs reported ✓ 1 passed / 0 failed.)

resultFilter decides what the rendering layer can see

{ "action": "resultFilter", "params": { "keys": ["t0", "p0", "id0", "at", "hasT0"] } }

Only the keys listed here reach RCN / XPage. Miss one and that slot on the widget is empty with no error (check G28 compares every ${} in the RCN against this key list). Two rules ride along:

• For widgets, always lift derived values into top-level scalars. A path like ${resp.hits[0].objectID} cannot be interpolated into an onClick navigation string; land it as id0 in the flow first. Flows used by pages are not bound by this: an H5 page receives the whole result object and an XPage iterates an array directly with forEach, so exposing an array of objects (such as items) in keys is fine.
• Avoid the names lang and theme for your own keys: when the flow does not expose them, the rendering layer fills in the current language and light/dark value, so that slot shows en-US / dark rather than nothing — which looks like data crossing wires.

data.*: this package's persistence

Reads and writes this package's private sandbox (the user's watchlist, the last selected item, the first-paint cache), preserved across launches. One line per action, eight of them:

{ "id": "saved",   "action": "data.get",    "params": { "key": "cities", "default": [ { "name": "Shanghai" } ] } },
{ "id": "hasCity", "action": "data.has",    "params": { "key": "cities" } },
{ "id": "allKeys", "action": "data.keys",   "params": {} },
{ "id": "all",     "action": "data.getAll", "params": {} },
{ "action": "data.set",    "params": { "key": "cityIdx", "value": "2" } },
{ "action": "data.merge",  "params": { "key": "prefs", "value": { "unit": "c", "sort": "hot" } } },
{ "action": "data.remove", "params": { "key": "c.wx.31_121" } },
{ "action": "data.clear",  "params": {} }

Key points:

• There is no scope parameter; the namespace is always this package. A scope you write is silently ignored.
• data.get must always be given a default (see "id is the result key").
• data.merge is a shallow merge: top-level keys of value replace keys of the same name, and anything deeper is swapped wholesale. When value is not an object (an array, a string, a number) it is treated as an empty object — nothing happens, and no error is raised. If what was stored under that key was not an object, the object simply replaces it.
• The three with no result (merge / remove / clear) give you no key even with an id, so do not use one as a "did it work" test; to confirm a write, read it back with data.get in the next step.
• key is coerced to a string; leaving key out gives you the empty-string key (legal, but nobody can read it).
• data.clear wipes all of this package's data and cannot be undone, so do not reach for it to clear a cache — clear a cache with data.remove on the named key.
• For local testing, seed the initial content in .numable/params/_datastore.json.

The first-paint cache (for pages) and the ${_cache} switch

A .df consumed by a page should implement three states, otherwise every visit waits a full network round trip for nothing:

1. cache present and fresh → use it, do not even open a connection;
2. cache present but stale → render it anyway, and expose a _stale flag so the page can revalidate in the background;
3. no cache / far too old → actually fetch.

{ "op": "set", "props": { "key": "cacheIn", "value": "${_cache}" } },
{ "op": "set", "props": { "key": "useCache", "value": "$[parseNumber::(${cacheIn},0)]" } },
{ "op": "set", "props": { "key": "ckey", "value": "$[connect::(c.wx.,${lat},_,${lon})]" } },
{ "id": "hit", "action": "data.get", "params": { "key": "${ckey}", "default": {} } }

Three hard requirements:

• The cache must be switched by the input parameter ${_cache}, and default to off. The page's depends passes "_cache": 1; a widget passes nothing → 0 → no caching. The whole point of a widget refresh is to skip the cache and fetch directly, so another cache layer underneath makes the entire refresh chain spin for nothing. The same .df is shared by the page and the widget — do not fork a second copy for this.
• Cache keys must not contain an index. Indexes drift as items are added and removed, and after a drift object A's cache is rendered for B, with no error. Use the object's stable identity (coordinates, a code, an id), and prefix the key with c. so it can be swept later.
• A cached page must be refreshable: the XPage root needs events.onRefresh, which first calls data.remove on the cache key and then xpage.reloadPage. Without that step, pull to refresh just hits the cache again — "refreshed" and unchanged, with clean logs.

The six rules you must keep

1. The barrier: the step immediately after request / concurrent / data.get cannot see the result (it lands one beat later), and so is the first node of a loop body (it reads the snapshot taken as the loop was entered). Put a {"op":"set","props":{"key":"_b","value":"1"}} in between. Likewise, do not read the id of a concurrent branch from inside an op:if scope — the evaluation scopes do not line up and you get empty.
2. A failed fetch must fail the flow. If an essential field is missing, go to error and the platform falls back to the last good data; reporting success and rendering an empty payload overwrites good data. Pin the test on the field "without which this widget is pointless"; an empty collection is a success, and a broken optional branch does not throw.
3. Always express emptiness as an explicit flag: hasX = $[if::(gt::(length::(${rawString}),0),1,0)], and the rendering layer looks only at hasX. Do not test emptiness with eq:: — if both sides convert to numbers it compares numerically, so eq::("00","") and eq::(empty,0) are both true. Pin length:: to a raw string field; on a number you computed yourself it is always 0.
4. A ${} inside a method argument cannot see the flow's input parameters, only keys already landed in the flow (results of actions with an id, keys from op:set). So the first thing an input parameter does in a flow is land: {"op":"set","props":{"key":"k","value":"$[findNotEmpty::(${pair},btcusd)]"}}.
5. A key referenced by op:set must appear before it. The engine runs in order and has no dependency graph; get the order wrong and the branch always falls to else while the code looks perfectly correct.
6. Guard the time anchor against empty. A live number has to tell people how old it is; feed formatDate:: an empty string and it renders 1970. Put a sentinel in front: $[if::(eq::(findNotEmpty::(${ts},__none__),__none__),,formatDate::(${ts},HH:mm))].

Fetching by language and producing text

A .df may read ${@app.language} directly (and @device.language / @time.locale); fetching by language is a legitimate, expected pattern (news headlines, city names, store descriptions). When the user switches language, the dashboard runs one widget.refresh per package (clear the fetch timer + a real fetch), so language-dependent data follows.

A .df has exactly the same shape as an .af: it may carry its own top-level i18n table (a flat { locale: { key: text } }), the body writes ${@i18n.key}, and the flow's input carries the host page's table ⊕ this file's table (plus @app). So finished sentences such as "About the same as yesterday" or "59% into today's range" can be produced right in the data layer:

{
  "version": 1,
  "actions": [
    { "id": "resp", "action": "request", "params": { "url": "https://api.example.com/today?lang=${@app.language}", "formatType": "json" } },
    { "op": "set", "props": { "key": "_b", "value": "1" } },
    { "op": "set", "props": { "key": "delta", "value": "${resp.delta}" } },
    { "op": "set", "props": { "key": "summary", "value": "$[if::(gt::(${delta},0),${@i18n.up},${@i18n.flat})]" } },
    { "action": "resultFilter", "params": { "keys": ["delta", "summary"] } }
  ],
  "i18n": {
    "zh-CN": { "up": "比昨天高", "flat": "和昨天差不多" },
    "en-US": { "up": "Higher than yesterday", "flat": "About the same as yesterday" }
  }
}

One sentence to remember: any flow file may have a top-level i18n table; write text as ${@i18n.key} and branch on language with ${@app.language}. Whether a plain word like "today" lives in the data layer or the rendering layer is your call — assembling it in the .rcn lets one set of data serve both languages, producing it in the .df saves the .rcn a join. numable check applies the same checks to a .df table as to an .af (G8: every referenced key must exist in the required locales, key sets match across locales, no unused keys).

The data-cache key includes the language: after a switch, no widget has stored data for the new language yet — a skeleton shows first, then the data once fetched; switching language offline leaves no data to show for that language (the empty or error state appears). That is a deliberate trade-off. The theme (light/dark) is different: it is a pure render parameter, switching theme only re-renders and never fetches, and the data cache is not split by theme either. The full rules for content tables are in numable docs i18n.

Rules (breaking one means rework)

When something goes wrong

See also

numable docs methods · numable docs builtins · numable docs af