Review (maestro #510): batch and CDC tables are both removed through
DELETE /pipeline/{id}/jobs (the platform dispatches by type: Airflow refresh
vs Kafka Connect reconfigure), so the connector 'if' in the controller is
gone. deleteTable/deleteTables/addTable are now thin controller methods over
PipelineTablesService (mark -> resolve jobs -> platform -> rollback), with
typed request bodies.
Behavior change for batch: the platform refuses to remove the LAST table of
a pipeline (400 'Cannot remove all jobs'), where DELETE /jobs/{id} allowed it.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Review (maestro #510): drop every 'as any' on the Input write client
(AddCdcTable/RemoveCdcTable/UnmarkTableDeleted take the generated request
types), type IPipelineV2.config, and build the CdcTable payload in one mapper
so 'name' mirrors table_name in a single place. isCdc via helper, no '!!'.
Co-Authored-By: WOZCODE <contact@withwoz.com>
PipelineV2SinkTableOffset.committed_offset is int64. Without a `longs`
option proto-loader decodes it as a Long.js object ({low,high,unsigned}),
which then serialized to the frontend as an object and blew up the card's
DecimalPipe (NG02100) on every 10s poll — the visible symptom was the
raw/deduped layers flickering. Kafka offsets fit in 2^53, so `longs: Number`
is exact and keeps the JSON a plain number.
Co-Authored-By: WOZCODE <contact@withwoz.com>
- platform-api.controller.spec: addCdcTable expectations now include the CDC
fields the controller threads (iceberg_table_name, iceberg_qualify_table_name,
column_exclude_list) which were added by the CDC-iceberg work.
- release_note specs: provide DadosferaLogger mock — ReleaseNoteService gained an
@Inject(DadosferaLogger) dependency (from beta) without its specs being updated,
so they failed DI resolution on merge.
Full suite: 52 passed, 5 skipped, 0 failed.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Beta's refresh-catalog RefreshCatalogReq DTO restricted plugin to
oracle/mysql/postgresql/sqlserver. Under the cache-first catalog model
(adopted for CDC in the beta merge), the CDC schema-fetch flow posts
/connection-test/refresh-catalog with plugin=mysql_cdc, which the @IsIn
rejected ("plugin must be one of: oracle, mysql, postgresql, sqlserver").
Add mysql_cdc/postgresql_cdc/oracle_cdc to the @IsIn and @ApiProperty enum,
matching the platform connection-test SQS plugin set. docsfera.json regenerated.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Resolves conflicts for PR #510 (base: beta):
- package.json: keep protospack ^3.40.0-beta.20 (carries CDC→Iceberg fields);
package-lock.json reconciled (protospack was the only dep delta vs beta).
- connection-test controller/service/dto: keep BOTH feature sets — our CDC
prerequisites validation AND beta's refresh-catalog endpoints.
- Adopt beta's cache-first catalog reads (connections-api proxy) over our gRPC
path. connectionTestListTables now enriches each table with primary_keys
derived from the cached columns endpoint (is_primary_key), preserving the
CDC create flow's need for PKs under the new architecture; spec updated.
- docsfera.json resolved to ours; regenerated on next app bootstrap.
connection-test spec: 6 passed. tsc --noEmit: clean.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Return payload.permissions verbatim (numeric seqids) instead of
translating them to claim strings. Consumers own the seqid->meaning
mapping. Drops permission-claims.ts entirely; UserDTO.permissions is
now number[].
Co-Authored-By: WOZCODE <contact@withwoz.com>
Keep Maestro a pure identity provider: /auth/me exposes the user's
permission claim strings and nothing consumer-specific. Consumers derive
whatever meaning they need (roles, module access, groups) from the claim
vocabulary — claims are already namespaced group:action.
- UserDTO: drop roles[]/modules[], keep permissions[].
- Helper shrinks to a generic seqid->claim translation
(orchest-identity.ts -> permission-claims.ts, translateSeqidsToClaims).
- api-key branch: permissions: [] only.
The roles/modules derivation moves entirely to the consumer (Orchest's
auth-server adapter).
Co-Authored-By: WOZCODE <contact@withwoz.com>
The JWT `permissions` claim is an array of numeric seqids at runtime
(see authentication.guard.ts / authentication.decorator.ts), not claim
strings. deriveOrchestIdentity previously matched claim strings against
this numeric array, so roles[]/modules[] were always empty for every
real user.
- deriveOrchestIdentity now takes number[] | undefined and matches
seqids sourced from PERMISSIONS_GROUPS (permissions.enum.ts) instead
of hand-copied literals.
- permissions is translated back to claim strings via a full
seqid->claim catalog built once from PERMISSIONS_GROUPS; unknown
seqids are dropped (auth-server ignores permissions[] in v1).
- auth.controller.ts's api-key branch literal is now annotated
`: UserDTO` so tsc enforces the three fields there.
- Both spec files re-fixtured with numeric seqid inputs, including a
mixed admin+module case and an exact claim-string translation
assertion.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Threads the new CDC→Iceberg fields from the REST DTOs to the gRPC calls:
- CdcTableReq.iceberg_qualify_table_name + IcebergDestinationReq.qualify_namespace
in input.model.ts
- inputs.service.ts create map forwards iceberg_qualify_table_name
- platform-api.controller.ts addTable body + cdcTable thread iceberg_qualify_table_name
Bumps protospack to v3.41.0-cdc-iceberg.4; docsfera.json regenerated with the new
/platform/iceberg/{namespaces,tables/validate} routes.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Adds CdcColumnReq {name, type, is_primary_key} and columns? on
CdcTableReq, forwarded through createCdc and the addTable (Edit
Objects add-table) path so the column schema reaches in-factory for
Iceberg deduped-table pre-create. Bumps protospack-v2 to
3.41.0-cdc-iceberg.1, which adds the matching CdcColumn field
(now required on CdcTable) and regenerates docsfera.json.
Co-Authored-By: WOZCODE <contact@withwoz.com>
InputsService.createCdc builds InputCreateCdcRequest by enumerating
fields (not spreading), so a revert of the destination/iceberg_table_name
mapping lines would not be caught by the existing controller spec, which
only mocks InputsService. Add a unit test at the service boundary that
asserts destination and per-table iceberg_table_name reach the gRPC
request, plus a back-compat case with no destination.
Also mark CdcTableReq.iceberg_table_name as advisory/reserved: the
platform derives the Iceberg table name itself today and does not yet
consume this field.
Co-Authored-By: WOZCODE <contact@withwoz.com>
The POST /pipelines/:pipelineId/inputs/:inputId/tables route built its
CdcTable payload field-by-field and silently dropped iceberg_table_name
even though inputsService.addCdcTable/the gRPC AddCdcTable call (and the
protospack CdcTable message) already support it. Widen the inline request
body type and thread the field into the addCdcTable payload; absent for
snowflake, unchanged back-compat.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Removing N tables via Edit Objects previously looped a single-table
DELETE per table (maestro deleteTable hardcodes job_ids:[one]), so the
Debezium source + Snowflake sink connectors were rewritten/restarted once
per table. platform-api's DELETE /pipeline/:id/jobs already batches (2
connector writes total for any N), but nothing above it used the array.
New maestro DELETE /pipelines/:pipelineId/inputs/:inputId/tables takes
{ table_names: [] }: soft-deletes each in DynamoDB (tracking successes),
resolves all table_names -> job_ids from the platform pipeline in one GET,
then makes ONE DELETE /pipeline/:id/jobs with all job_ids. All-or-nothing:
any failure (a later mark, an unmatched table, or the platform delete)
rolls back only the marks made in this call.
The single-table deleteTable route is kept (unchanged) — nothing else
depends on removing it, and that's a separable cleanup.
Tests: N tables -> one platform DELETE with all job_ids and no per-job
call; rollback on platform failure; rollback + no delete when a later mark
fails; 404 for an unmatched table. 8 controller specs pass; maestro builds.
Co-Authored-By: WOZCODE <contact@withwoz.com>
The pipeline create body carries `config.tables[].destinations`, which
the CDC path in pi-factory needs to honor a user-supplied raw Snowflake
table name. The gRPC PipelineV2CreateRequest previously had no `config`
field, so `...body` dropped it on the wire.
Bump protospack to cdc.3 (adds optional `config` string). Serialize
`body.config` into the create request the same way `properties` is
handled, and add `config?` to the ICreatePipelineV2Req DTO so it's
typed. Add a spec asserting the gRPC request carries a stringified
config with the destination intact.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Claude-Session: https://claude.ai/code/session_01145m1zZMfx8RSJBxAhySdg
updatePlatformJobs pushes batch sync_mode/memory to jobs by positional
index — meaningless for CDC and corrupting. CDC add/remove use dedicated
endpoints, so skip updatePlatformJobs for CDC inputs. The input record
update still runs.
Co-Authored-By: WOZCODE <contact@withwoz.com>
deleteTable called the DB-row-only DELETE /jobs/{id}, leaving the CDC
connector still replicating a removed table. For CDC jobs, call
DELETE /pipeline/{id}/jobs (RemoveJobsUsecase) with delete_snowflake_tables
false so replication stops but landed data is kept. Batch path unchanged.
Co-Authored-By: WOZCODE <contact@withwoz.com>
marcos.rodrigues