Compare commits

...
Author SHA1 Message Date
Claude 9c8c761529 fix(auth): environment-scoped session cookies and resilient /auth/me refresh
Apps behind an nginx auth_request (Orchest / Intelligence / Process) rely on
the ddf-* cookies and on GET /auth/me. Several problems made them lose the
session or show 500 pages:

- Cookies were written with a hard-coded Domain=.dadosfera.ai in every
  environment, so a staging login overwrote the production session (and
  vice versa) in the same browser. Names now take AUTH_COOKIE_SUFFIX and the
  domain comes from COOKIE_DOMAIN (staging: -stg / .stg.dadosfera.ai).
- maxAge was the token's absolute epoch (~56 years). It is now a real
  duration, SESSION_COOKIE_MAX_AGE_DAYS (default 30).
- /auth/me answered 500 when a refresh failed for anything but
  NotAuthorizedException, when the refreshed token failed validation, or
  when DUC hung (no deadline). It now answers 401 and expires the dead
  cookies for definitive failures, 503 for DUC transport errors/timeouts
  (AUTH_RPC_TIMEOUT_MS, default 10s), keeps 403 answers instead of masking
  them with a refresh, refreshes when only the refresh cookie is left, and
  stops logging request headers (they contain the cookies).
- /auth/refresh-access-token rewrites all three cookies from the session the
  frontend just proved valid, healing stale cookies from another login or
  environment. /auth/sign-out expires all three and always responds.

Helm values wire the new variables for prd and stg.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUEXmydTBX9G2CGJ97YQKL
2026-09-03 13:13:54 +00:00
marcos.rodrigues 740540d4dc Merge pull request #517 from dadosfera/bugfix/sync-proto
FIX: sync proto
2026-08-28 13:33:27 -03:00
marcos.rodrigues 94c9ab7939 FIX: add missing helm value 2026-08-28 12:25:44 -03:00
marcos.rodrigues a211b8593e FIX: sync proto 2026-08-28 12:23:25 -03:00
marcos.rodrigues 25661613d5 Merge pull request #516 from dadosfera/bugfix/add-jest-file-in-exclude-build
FIX: add-jest-file-in-exclude-build
2026-08-27 17:29:08 -03:00
marcos.rodrigues 3d8ea3576d FIX: add-jest-file-in-exclude-build 2026-08-27 17:10:37 -03:00
marcos.rodrigues e3a2eba5c3 Merge pull request #515 from dadosfera/release/2026-08-27-beta
Release/2026 08 27 beta
2026-08-27 16:09:16 -03:00
marcos.rodrigues 81c446354b FIX: mock process.env and dadosfera logger 2026-08-27 14:33:30 -03:00
RafaelandWOZCODE bf0314f5b1 FIX: trigger release for /auth/me permission seqids (PR #513)
PR #513 merged to beta but no semantic-release ran: its commits used
conventional-commits prefixes (feat(auth):, fix(auth):) which the
.releaserc.json eslint preset does not recognise, so commit-analyzer
found no release-worthy change. This empty FIX: commit matches the
eslint preset's releaseRules (tag FIX -> patch) to cut a beta release
that includes the /auth/me permission-seqids change, so stg can deploy it.

Co-Authored-By: WOZCODE <contact@withwoz.com>
2026-08-24 19:18:59 -03:00
Rafael Santana c97180ac95 Merge pull request #513 from dadosfera/feat/auth-me-orchest-identity
feat(auth): return the user's permission seqids from /auth/me
2026-08-24 17:00:15 -03:00
RafaelandWOZCODE cac36f2c60 refactor(auth): /auth/me returns raw permission seqids
Return payload.permissions verbatim (numeric seqids) instead of
translating them to claim strings. Consumers own the seqid->meaning
mapping. Drops permission-claims.ts entirely; UserDTO.permissions is
now number[].

Co-Authored-By: WOZCODE <contact@withwoz.com>
2026-08-24 16:56:23 -03:00
RafaelandWOZCODE cd21fd0b7b refactor(auth): /auth/me returns permissions only (drop roles/modules)
Keep Maestro a pure identity provider: /auth/me exposes the user's
permission claim strings and nothing consumer-specific. Consumers derive
whatever meaning they need (roles, module access, groups) from the claim
vocabulary — claims are already namespaced group:action.

- UserDTO: drop roles[]/modules[], keep permissions[].
- Helper shrinks to a generic seqid->claim translation
  (orchest-identity.ts -> permission-claims.ts, translateSeqidsToClaims).
- api-key branch: permissions: [] only.

The roles/modules derivation moves entirely to the consumer (Orchest's
auth-server adapter).

Co-Authored-By: WOZCODE <contact@withwoz.com>
2026-08-24 16:30:33 -03:00
RafaelandWOZCODE a5a685ee3f fix(auth): derive Orchest identity from numeric seqids (JWT carries seqids not claim strings)
The JWT `permissions` claim is an array of numeric seqids at runtime
(see authentication.guard.ts / authentication.decorator.ts), not claim
strings. deriveOrchestIdentity previously matched claim strings against
this numeric array, so roles[]/modules[] were always empty for every
real user.

- deriveOrchestIdentity now takes number[] | undefined and matches
  seqids sourced from PERMISSIONS_GROUPS (permissions.enum.ts) instead
  of hand-copied literals.
- permissions is translated back to claim strings via a full
  seqid->claim catalog built once from PERMISSIONS_GROUPS; unknown
  seqids are dropped (auth-server ignores permissions[] in v1).
- auth.controller.ts's api-key branch literal is now annotated
  `: UserDTO` so tsc enforces the three fields there.
- Both spec files re-fixtured with numeric seqid inputs, including a
  mixed admin+module case and an exact claim-string translation
  assertion.

Co-Authored-By: WOZCODE <contact@withwoz.com>
2026-08-24 16:03:20 -03:00
Rafael a16fefe691 feat(auth): return permissions/roles/modules from /auth/me (all branches) 2026-08-24 15:45:09 -03:00
Rafael 9c57485031 feat(auth): pure helper deriving Orchest identity from permissions 2026-08-24 15:39:40 -03:00
marcos.rodrigues 7051b21d86 Merge pull request #509 from dadosfera/release/2026-08-12
FIX: require collect module in endpoints
2026-08-12 15:06:04 -03:00
marcos.rodrigues 0c4888ccdb FIX: require collect module in endpoints 2026-08-12 15:02:37 -03:00
marcos.rodrigues e1b0e88bd8 Merge pull request #507 from dadosfera/bugfix/catalog-module-main
Bugfix/catalog module main
2026-08-12 09:59:08 -03:00
marcos.rodrigues cef1184908 Merge branch 'beta' into bugfix/catalog-module-main 2026-08-12 09:58:58 -03:00
marcos.rodrigues 31dda867d1 FIX: require collect module in endpoints 2026-08-12 09:57:23 -03:00
marcos.rodrigues 8a92da470c Merge pull request #506 from dadosfera/bugfix/catalog-module-main
FIX: skip nimbus update when customer haven't catalog module
2026-08-07 16:27:19 -03:00
marcos.rodrigues b89909ad66 FIX: skip nimbus update when customer haven't catalog module 2026-08-07 14:48:46 -03:00
marcos.rodrigues 2bb280e8de Merge pull request #505 from dadosfera/bugfix/catalog-module
Bugfix/catalog module
2026-08-07 12:26:21 -03:00
marcos.rodrigues 00cbadbb45 Merge branch 'beta' into bugfix/catalog-module 2026-08-07 12:26:11 -03:00
marcos.rodrigues 47ad527d38 FIX: skip nimbus update when customer haven't catalog module 2026-08-07 12:18:37 -03:00
iruy-fr 51044a23b3 FIX: trigger cache connections rollout 2026-08-03 09:19:27 -03:00
yuri.rodrigues bb29d126c1 Merge pull request #503 from dadosfera/feat/cache-connections-rollout
feat(connection-test): refresh connection catalog cache
2026-07-31 20:56:11 -03:00
iruy-fr 9d0f449eeb feat(connection-test): refresh connection catalog cache 2026-07-31 16:33:15 -03:00
iruy-fr 0eafa67e6f FIX: trigger cache connections deployment 2026-07-31 09:56:05 -03:00
yuri.rodrigues c3937472ec Merge pull request #502 from dadosfera/feat/cache-connections-rollout
feat: read connection metadata from catalog cache
2026-07-31 09:46:57 -03:00
iruy-fr 84d64424ca feat: read connection metadata from catalog cache 2026-07-30 10:17:28 -03:00
vinicius gadea f0bfc5c94b Merge pull request #500 from dadosfera/feat/custom-properties
FEAT: add color and emoji properties to CustomPropertyDto
2026-07-28 12:21:19 -03:00
viniciusgadea ad86a6a698 FEAT: simplify color and emoji property definitions in docsfera.json 2026-07-28 12:01:50 -03:00
viniciusgadea af3b11ad54 FEAT: add color and emoji properties to CustomPropertyDto 2026-07-28 07:42:18 -03:00
marcos.rodrigues d54f381998 Merge pull request #499 from dadosfera/release/2026-07-27
Release/2026 07 27
2026-07-27 17:59:56 -03:00
viniciusgadea 3fd586753e FEAT: add endpoint and logic to update data asset certification status 2026-07-27 17:07:19 -03:00
viniciusgadea 9b3894f9e4 FIX: pin npm version to 10.8.2 in Dockerfile for consistency 2026-07-27 17:07:10 -03:00
viniciusgadea 4a5f8f679a FEAT: rename documentation_status to certification_status in docs and update package.json for protospack versioning 2026-07-27 17:06:58 -03:00
viniciusgadea a444f5e5ec FEAT: add documentation status enum and property to data asset 2026-07-27 17:06:25 -03:00
marcos.rodrigues 0d30c1cf83 FEAT: remove deprecated protospack lib 2026-07-27 17:06:10 -03:00
vinicius gadea 15860519e1 Merge pull request #498 from dadosfera/feat/custom-properties
FEAT: add custom properties endpoints and DTOs for catalog management
2026-07-22 11:57:51 -03:00
viniciusgadea cffda86eab FEAT: add CustomPropertyDto and update IUpdateDataRequest to use custom properties array 2026-07-22 10:51:53 -03:00
viniciusgadea a93fbfbbd9 FEAT: remove custom properties endpoints and DTOs from catalog management 2026-07-21 16:32:38 -03:00
viniciusgadea e1cfc6e1a8 FEAT: add custom properties endpoints and DTOs for catalog management 2026-07-21 11:21:22 -03:00
vinicius gadea 1e63df6536 Merge pull request #497 from dadosfera/feat/documentation-status
FEAT: add endpoint and logic to update data asset certification status
2026-07-20 13:44:43 -03:00
viniciusgadea 8eb7fd0169 FEAT: add endpoint and logic to update data asset certification status 2026-07-17 09:55:17 -03:00
vinicius gadea f8c6a8b747 Merge pull request #496 from dadosfera/feat/documentation-status
FIX: pin npm version to 10.8.2 in Dockerfile for consistency
2026-07-16 17:06:04 -03:00
viniciusgadea 8a9d6c2f9c FIX: pin npm version to 10.8.2 in Dockerfile for consistency 2026-07-16 17:05:10 -03:00
vinicius gadea bcb2a0b7cb Merge pull request #495 from dadosfera/feat/documentation-status
Feat/documentation status
2026-07-16 16:47:52 -03:00
viniciusgadea 2e181e70af FEAT: rename documentation_status to certification_status in docs and update package.json for protospack versioning 2026-07-16 09:08:11 -03:00
vinicius gadea ef615adb8e Merge branch 'beta' into feat/documentation-status 2026-07-13 08:42:37 -03:00
viniciusgadea e03900811b FEAT: add documentation status enum and property to data asset 2026-07-13 08:33:35 -03:00
yuri.rodrigues 5bc5fb0977 Merge pull request #494 from dadosfera/feat/pipeline-run-jobs
feat: add endpoint to retrieve pipeline run jobs
2026-06-23 13:38:55 -03:00
iruy-fr 011032e3d4 FEAT: update API title in docsfera.json to reflect project name 2026-06-23 11:58:28 -03:00
iruy-fr 17363e74f4 FEAT: simplify pipeline run jobs handling and normalize run ID usage 2026-06-23 11:54:59 -03:00
iruy-fr 63efff6adf FEAT: enhance pipeline run jobs endpoint with error handling and response structure 2026-06-22 20:25:16 -03:00
iruy-fr b5f569e522 Merge branch 'beta' into feat/pipeline-run-jobs
# Conflicts:
#	docsfera.json
2026-06-19 17:08:34 -03:00
iruy-fr 5c77577992 feat: add endpoint to retrieve pipeline run jobs 2026-06-19 17:04:42 -03:00
marcos.rodrigues 4b9e113185 Merge pull request #493 from dadosfera/chore/remove-deprecated-lib
FEAT: remove deprecated protospack lib
2026-06-14 15:10:22 -03:00
marcos.rodrigues 06d505c50a FEAT: remove deprecated protospack lib 2026-06-13 21:07:14 -03:00
Marcos Rodrigues Silva c70abfc826 Merge pull request #492 from dadosfera/release/25-05
Release/25 05
2026-05-25 17:48:12 -03:00
marcos-silva-rodrigues db2c7d6c02 FIX: tests 2026-05-25 17:43:38 -03:00
marcos-silva-rodrigues 0a99ce1aa4 FEAT: update deployment to include firebase base url 2026-05-25 14:51:40 -03:00
marcos-silva-rodrigues 39c66f030a FIX: release note endpoint 2026-05-25 14:51:32 -03:00
marcos-silva-rodrigues 1223e21ac4 FEAT: pipeline upgrade route 2026-05-25 14:51:20 -03:00
Marcos Rodrigues Silva 2177f6725c Merge pull request #491 from dadosfera/hotfix/release-notes
Hotfix/release notes
2026-05-25 14:10:46 -03:00
yuri.rodrigues e8b982998f Merge pull request #490 from dadosfera/feat/platform-job-rout
Feat/platform job rout
2026-05-21 16:48:38 -03:00
iruy-fr ccd4159c59 fix: validate workflow 2026-05-21 16:43:37 -03:00
iruy-fr 9e49abb40d fix: validate workflow 2026-05-21 16:37:47 -03:00
iruy-fr 21a82b64f3 fix: validate workflow 2026-05-21 16:31:45 -03:00
iruy-fr 2414fcf21e fix: validate workflow 2026-05-21 16:23:45 -03:00
iruy-fr 94fbdb2226 fix: validate workflow 2026-05-21 16:21:30 -03:00
iruy-fr 985170d7ae chore: exposure from route pipeline run jobs to maestro 2026-05-21 16:12:35 -03:00
iruy-fr b38b8f51e2 FEAT: Add endpoint to fetch pipeline run jobs 2026-05-20 15:34:49 -03:00
Marcos Rodrigues Silva ea16e62d6b Merge pull request #489 from dadosfera/feature/pipeline-upgrade
FEAT: pipeline upgrade route
2026-05-11 12:58:06 -03:00
marcos-silva-rodrigues 6182705410 FEAT: pipeline upgrade route 2026-05-11 12:52:07 -03:00
51 changed files with 1926 additions and 583 deletions
+6
View File
@@ -71,6 +71,11 @@ jobs:
sudo mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff plugin
run: |
helm plugin install https://github.com/databus23/helm-diff --version v3.9.3
helm diff version
- name: Debug Helm env
run: |
helm env
@@ -102,4 +107,5 @@ jobs:
- name: Run Helmfile Diff
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
HELM_PLUGINS: /home/runner/.local/share/helm/plugins
run: helmfile -f deploy/helmfiles/${ENV}.yaml diff
+1 -1
View File
@@ -1,5 +1,5 @@
FROM node:20-alpine AS base_image
RUN npm install -g npm@latest
RUN npm install -g npm@10.8.2
FROM base_image AS build_base
WORKDIR /app
@@ -101,6 +101,17 @@ spec:
value: {{ .Values.maestro.dedicated_proxy }}
- name: COOKIE_SECRET
value: {{ .Values.maestro.cookie_secret }}
# Browser session cookies (ddf-auth*, ddf-refresh-auth*, ddf-user-id*).
# Scoped per environment so a staging login never overwrites a
# production session in the same browser (both live under dadosfera.ai).
- name: COOKIE_DOMAIN
value: {{ .Values.maestro.cookie_domain | default ".dadosfera.ai" | quote }}
- name: AUTH_COOKIE_SUFFIX
value: {{ .Values.maestro.auth_cookie_suffix | default "" | quote }}
- name: SESSION_COOKIE_MAX_AGE_DAYS
value: {{ .Values.maestro.session_cookie_max_age_days | default "30" | quote }}
- name: AUTH_RPC_TIMEOUT_MS
value: {{ .Values.maestro.auth_rpc_timeout_ms | default "10000" | quote }}
- name: REDIS_DATABASE
value: "{{ .Values.maestro.redis_database }}"
- name: REDIS_HOST
@@ -111,6 +122,8 @@ spec:
value: "{{ .Values.maestro.redis_tls }}"
- name: PLATFORM_API_URL
value: {{ .Values.maestro.platform_api_url }}
- name: CONNECTIONS_API_URL
value: {{ .Values.maestro.connections_api_url | default "" | quote }}
- name: STORAGE_EXPLORER_API_URL
value: {{ .Values.maestro.storage_explorer_api_url | quote }}
- name: FIREBASE_BASE_URL
+7
View File
@@ -7,8 +7,15 @@ maestro:
open_customer_id: b3e3dfe5-b992-4586-a73c-c0b0c00f615d
open_group_id: e3f98a2f-7748-4981-8505-7695c8ca8218
cookie_secret: "ff7bc13823edb2ae50d248e5780bddc9d4b31c36"
# Session cookies: staging gets its own names AND domain so that
# app.stg.dadosfera.ai and app.dadosfera.ai sessions never collide.
cookie_domain: ".stg.dadosfera.ai"
auth_cookie_suffix: "-stg"
session_cookie_max_age_days: "30"
auth_rpc_timeout_ms: "10000"
redis_database: "1"
platform_api_url: https://xs2hkhq07k.execute-api.us-east-1.amazonaws.com
connections_api_url: https://iy40eans64.execute-api.us-east-1.amazonaws.com
storage_explorer_api_url: "http://storage-explorer-{customer}.data-apps.svc.cluster.local:8000/api"
firebase_base_url: https://feature-flag-25bf6-default-rtdb.firebaseio.com/stg
+7
View File
@@ -55,7 +55,14 @@ maestro:
redis_database: "0"
redis_tls: "true"
cookie_secret: "13cc5e136d3074bcc05bec8697092ec1f5f376bf"
# Session cookies: production keeps the historical names on the apex domain.
cookie_domain: ".dadosfera.ai"
auth_cookie_suffix: ""
session_cookie_max_age_days: "30"
auth_rpc_timeout_ms: "10000"
firebase_base_url: https://feature-flag-25bf6-default-rtdb.firebaseio.com/prd
connections_api_url: https://wv6fw7ayj4.execute-api.us-east-1.amazonaws.com
autoscaling:
enabled: false
minReplicas: 1
+484 -87
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -11,6 +11,7 @@ const config: Config.InitialOptions = {
'<rootDir>/node_modules/',
'.*\\.module\\.[jt]s$',
],
setupFiles: ['<rootDir>/jest.setup.ts'],
// moduleDirectories: ['node_modules'], // default is already 'node_modules'
// rootDir: '.', //No need
// testEnvironment: 'node', //Defaults to 'node'
+3
View File
@@ -0,0 +1,3 @@
process.env.DUC_URL="duc:50051"
process.env.INFACTORY_URL="in-factory:50052"
process.env.PIFACTORY_URL="pi-factory:50053"
+4 -14
View File
@@ -16,8 +16,7 @@
"@aws-sdk/lib-dynamodb": "^3.414.0",
"@aws-sdk/signature-v4": "^3.370.0",
"@dadosfera/dadosfera-logs": "^1.0.0-beta.4",
"@dadosfera/protospack": "2.5.3",
"@dadosfera/protospack-v2": "3.40.0-beta.8",
"@dadosfera/protospack-v2": "3.40.0",
"@grpc/grpc-js": "^1.9.3",
"@grpc/proto-loader": "^0.7.9",
"@nestjs/cli": "^9.5.0",
@@ -1735,19 +1734,10 @@
"winston-log2gelf": "^2.4.0"
}
},
"node_modules/@dadosfera/protospack": {
"version": "2.5.3",
"resolved": "https://dadosfera-611330257153.d.codeartifact.us-east-1.amazonaws.com/npm/dadosfera-npm/@dadosfera/protospack/-/protospack-2.5.3.tgz",
"integrity": "sha512-yOLnd+s6n9VkPpZXO8HnUY27CQPHj/qs+ecddviA4Ldn0Gx4KGRgbVdsSSP45nPm0GHhCd2bHg4ap+la7xtRmA==",
"license": "ISC",
"dependencies": {
"rxjs": "^7.5.5"
}
},
"node_modules/@dadosfera/protospack-v2": {
"version": "3.40.0-beta.8",
"resolved": "https://dadosfera-611330257153.d.codeartifact.us-east-1.amazonaws.com/npm/dadosfera-npm/@dadosfera/protospack-v2/-/protospack-v2-3.40.0-beta.8.tgz",
"integrity": "sha512-JE5qMjqB3UOM+tCUxB1EwYLQW0PecsaQIa1KDpKEaG3lrzG/H13z8iJi3WH/DuVav2EI94i9VcJWJ1Y0F7ribw==",
"version": "3.40.0",
"resolved": "https://dadosfera-611330257153.d.codeartifact.us-east-1.amazonaws.com/npm/dadosfera-npm/@dadosfera/protospack-v2/-/protospack-v2-3.40.0.tgz",
"integrity": "sha512-tzYAGosFjGUgi3pAvXqv+Eqqjca9tH1kU9kJdRI5d1rLXNimo0fIuPWxWng0C4OBH/2sIgC1WLHseq+DZ6qgdw==",
"license": "ISC",
"dependencies": {
"@grpc/grpc-js": "^1.9.3",
+1 -2
View File
@@ -34,8 +34,7 @@
"@aws-sdk/lib-dynamodb": "^3.414.0",
"@aws-sdk/signature-v4": "^3.370.0",
"@dadosfera/dadosfera-logs": "^1.0.0-beta.4",
"@dadosfera/protospack": "2.5.3",
"@dadosfera/protospack-v2": "3.40.0-beta.8",
"@dadosfera/protospack-v2": "3.40.0",
"@grpc/grpc-js": "^1.9.3",
"@grpc/proto-loader": "^0.7.9",
"@nestjs/cli": "^9.5.0",
-2
View File
@@ -17,7 +17,6 @@ import { ConnectionTestModule } from './modules/connection-test/connection-test.
import { NetworkConfigModule } from './modules/network-config/network-config.module';
import { InputsModule } from './modules/inputs/inputs.module';
import { OauthModule } from './modules/oauth/oauth.module';
import { PipelinesModule } from './modules/pipelines/pipelines.module';
import { TransformationsModule } from './modules/transformations/transformations.module';
import { HealthModule } from './modules/health/health.module';
import { CatalogModule } from './modules/catalog/catalog.module';
@@ -60,7 +59,6 @@ import { ReleaseNoteModule } from './modules/release_note/release_note.module';
PermissionsModule,
TermsOfUseModule,
ConnectionTestModule,
PipelinesModule,
TransformationsModule,
UsersModule,
RolesModule,
+12
View File
@@ -357,6 +357,16 @@ export const PERMISSIONS_GROUPS = {
'es-es': 'Crear y editar atributos en el catálogo',
},
},
CERTIFY: {
seqid: 53,
claim: 'catalog:certify',
usage: PermissionUsages.PUBLIC,
name: {
'pt-br': 'Alterar o status de certificação dos Ativos',
'en-us': "Change Assets' certification status",
'es-es': 'Cambiar el estado de certificación de los Activos',
},
},
DELETE: {
seqid: 1,
claim: 'catalog:delete',
@@ -712,6 +722,8 @@ export const DADOSFERA_MODULES_KEYS = {
PII: 'pii',
EMBED: 'embedded-analytics',
EMBED_ASSIGNED: 'embed-assigned',
CATALOG: 'catalog',
COLLECT: 'collect',
}
export const DADOSFERA_MODULES: Array<DadosferaModule> = [
+42 -1
View File
@@ -35,4 +35,45 @@ So in order to have an SSO login, besides configuring the Strategy, we must have
// ... Rest of the logic
return { url: url.href };
}
```
```
## Browser session cookies
Besides the tokens returned in the sign-in body (kept by the frontend in
`localStorage` and sent as `Authorization`), Maestro also persists the session
as httpOnly cookies so that apps which cannot hold a token themselves —
Orchest / Intelligence / Process behind an nginx `auth_request` pointing at
`GET /auth/me` — can authenticate the browser.
| Cookie | Content |
| --- | --- |
| `ddf-auth<suffix>` | DUC access token |
| `ddf-refresh-auth<suffix>` | IdP refresh token |
| `ddf-user-id<suffix>` | DUC user id |
All three are `HttpOnly; Secure; SameSite=None; Path=/` and live for
`SESSION_COOKIE_MAX_AGE_DAYS` (default 30 — keep it aligned with the IdP
refresh-token validity). See `session-cookies.ts`.
They are **environment-scoped** so that a login on `app.stg.dadosfera.ai`
never overwrites a production session in the same browser:
| Env | `AUTH_COOKIE_SUFFIX` | `COOKIE_DOMAIN` |
| --- | --- | --- |
| prd | `""` | `.dadosfera.ai` |
| stg | `-stg` | `.stg.dadosfera.ai` |
`GET /auth/me` behaviour:
- valid access token → `200` + user (permissions are the raw seqids from the JWT);
- expired/invalid access token but refresh token present → refresh, rewrite the
cookies (`Set-Cookie`, access token first — nginx `auth_request` only relays
the first one) and `200`;
- refresh impossible (expired/revoked/foreign token, unknown user) → `401` and the
cookies are expired so the browser stops presenting a dead session;
- DUC unreachable / slower than `AUTH_RPC_TIMEOUT_MS` (default 10s) → `503`,
cookies untouched (transient, retry);
- resource belongs to another customer → `403`; new terms of use pending → `403`.
`POST /auth/refresh-access-token` re-writes all three cookies from the session
the frontend is actually using, and `POST /auth/sign-out` expires them.
+123 -35
View File
@@ -14,6 +14,8 @@ import {
Param,
Res,
UnauthorizedException,
HttpException,
ServiceUnavailableException,
} from '@nestjs/common';
import {
ApiHeaders,
@@ -37,6 +39,7 @@ import {
RequireAllPermissions,
} from 'src/decorators/authentication.decorator';
import { AuthClientService } from './auth.service';
import { UserDTO } from './dtos/login';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { GrpcToHttpExceptionFilter } from '../../error/grpc-to-http-exception.filter';
import { RequestUser, User } from 'src/decorators/user.decorator';
@@ -50,7 +53,10 @@ import {
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { AuthGuard } from '@nestjs/passport';
import { Request, Response } from 'express';
import { status as GrpcStatus } from '@grpc/grpc-js';
import ErrorCodes, { OauthErrors } from 'src/utils/errorCodes';
import ErrorBuilder from 'src/utils/ErrorBuilder';
import { clearSessionCookies, readAuthSession } from './session-cookies';
import jwt, { JwtPayload } from 'jsonwebtoken';
import { LanguageEnum } from 'src/utils/languages.enum';
import { Language } from 'src/decorators/language.decorator';
@@ -128,14 +134,16 @@ export class AuthController {
) {
try {
this.logger.info('/auth - SignOut');
this.authClient.cleanUpAuthSession(res);
return res.send();
this.authClient.cleanUpAuthSession(res);
} catch (error) {
this.logger.error('/auth - SignIn - ERROR', error);
this.logger.error('/auth - SignOut - ERROR', error);
}
// Always answer: a sign-out that hangs leaves the frontend logout
// waiting on a request that never completes.
return res.send();
}
@Post('refresh-access-token')
@@ -148,7 +156,7 @@ export class AuthController {
@Res() res: Response,
) {
this.logger.info('/auth - RefreshAccessToken');
const frontHost = origin.replace(/^https?:\/\//, '');
const frontHost = (origin || '').replace(/^https?:\/\//, '');
const { refreshToken, userId } = body;
const metadata = PackTheMetadata({
@@ -158,10 +166,14 @@ export class AuthController {
const data = await this.authClient.refreshAccessToken({ refreshToken, userId }, metadata);
// Re-sync ALL session cookies with the session the frontend is actually
// using. The IdP does not rotate refresh tokens (DUC returns an empty
// refreshToken), so fall back to the one that just proved valid — this
// heals cookies left behind by an older login or by another environment.
this.authClient.writeAuthSession(res, {
accessToken: data.accessToken,
refreshToken: data.refreshToken,
userId
refreshToken: data.refreshToken || refreshToken,
userId,
});
return res.send(data);
@@ -477,16 +489,28 @@ export class AuthController {
@Get('me')
async getMe(@Req() req: Request, @Res() res: Response) {
this.logger.info('GET /auth/me ')
this.logger.info(JSON.stringify(req.headers));
const resourceHost = (req.headers['x-original-url'] as string) || '';
// Never dump req.headers here: it contains the session cookies.
this.logger.info('GET /auth/me', { resourceHost });
// Check for API key header first
const apiKey = req.get('X-Api-key');
if (apiKey) {
this.logger.info('Authenticating via X-Api-key header');
const { api_key } = await this.apiKeyService.get(apiKey);
let api_key: Awaited<ReturnType<ApiKeyService['get']>>['api_key'];
try {
({ api_key } = await this.apiKeyService.get(apiKey));
} catch (error) {
if (this.isTransportError(error)) {
throw this.identityProviderUnavailable(error);
}
this.logger.error('/auth/me - invalid API key', {
error: error?.message,
});
throw new UnauthorizedException('Invalid API key');
}
const userDto = {
const userDto: UserDTO = {
id: api_key.user_id,
name: api_key.username,
email: api_key.username,
@@ -494,41 +518,105 @@ export class AuthController {
id: api_key.customer_id,
name: api_key.customer_name,
tier: api_key.customer_tier,
}
},
permissions: [],
};
return res.status(200).json(userDto);
}
// Get token and headers
const accessToken = req.cookies['ddf-auth'];
const refreshToken = req.cookies['ddf-refresh-auth'];
const userId = req.cookies['ddf-user-id'];
const resourceHost = req.headers["x-original-url"] as string || "" ;
const { accessToken, refreshToken, userId } = readAuthSession(req);
const canRefresh = Boolean(refreshToken) && Boolean(userId);
this.logger.info('Has User Session: ' + Boolean(accessToken || canRefresh));
const hasUserSession = Boolean(accessToken) && Boolean(userId);
this.logger.info('Has User Session: ' + hasUserSession);
if (!accessToken && !canRefresh) {
throw new UnauthorizedException();
}
if (!hasUserSession) {
throw new UnauthorizedException()
if (accessToken) {
try {
const userDto = await this.authClient.validateUserSession(
accessToken,
resourceHost,
);
return res.status(200).json(userDto);
} catch (error) {
// Definitive answers (e.g. 403: resource belongs to another
// customer) must not be masked by a refresh attempt.
if (error instanceof HttpException) throw error;
if (this.isTransportError(error)) {
throw this.identityProviderUnavailable(error);
}
if (!canRefresh) {
this.logger.error(
'/auth/me - access token rejected and no refresh token',
{ error: error?.message },
);
clearSessionCookies(res);
throw new ErrorBuilder(ErrorCodes.AUTH.INVALID_REFRESH_TOKEN);
}
this.logger.info('/auth/me - access token rejected, refreshing', {
error: error?.message,
});
}
}
try {
const userDto = await this.authClient.validateUserSession(accessToken, resourceHost);
return res.status(200).json(userDto);
} catch (error) {
if (!refreshToken) {
this.logger.error('Invalid refresh token or customer name');
throw new UnauthorizedException("Invalid refresh token or customer name");
};
const {
authSession,
user
} = await this.authClient.refreshUserSession(refreshToken, userId, resourceHost);
this.authClient.writeAuthSession(res, authSession);
const { authSession, user } = await this.authClient.refreshUserSession(
refreshToken,
userId,
resourceHost,
);
// Refresh tokens are not rotated by the IdP (DUC returns ''), so keep
// the one that just worked; this also slides the cookie lifetime.
this.authClient.writeAuthSession(res, {
...authSession,
refreshToken: authSession.refreshToken || refreshToken,
});
return res.status(200).json(user);
} catch (error) {
if (error instanceof HttpException) throw error;
if (this.isTransportError(error)) {
throw this.identityProviderUnavailable(error);
}
const code =
typeof error?.details === 'string' ? error.details : undefined;
if (code === ErrorCodes.TERMS_OF_USE.REQUIRED) {
// The session is fine; the user has to accept the new terms first.
throw new ErrorBuilder(code);
}
// Expired, revoked, or minted by another environment / user pool.
// Answer 401 (never 500) and drop the dead cookies so the browser
// does not keep presenting them on every request.
this.logger.error('/auth/me - session refresh failed', {
code,
error: error?.message,
});
clearSessionCookies(res);
throw new ErrorBuilder(ErrorCodes.AUTH.INVALID_REFRESH_TOKEN);
}
}
/** DUC unreachable or too slow — a transient failure, not an auth answer. */
private isTransportError(error: any): boolean {
return (
error?.name === 'TimeoutError' ||
error?.code === GrpcStatus.UNAVAILABLE ||
error?.code === GrpcStatus.DEADLINE_EXCEEDED
);
}
private identityProviderUnavailable(error: any): HttpException {
this.logger.error('/auth/me - identity provider unavailable', {
code: error?.code,
error: error?.message,
});
// 503 (not 500): callers such as nginx auth_request / Orchest's
// auth-server treat it as "retry later" instead of a broken session.
return new ServiceUnavailableException(
'Identity provider temporarily unavailable',
);
}
}
+43 -69
View File
@@ -8,7 +8,7 @@ import {
} from '@nestjs/common';
import { ClientGrpc } from '@nestjs/microservices';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { lastValueFrom } from 'rxjs';
import { lastValueFrom, timeout } from 'rxjs';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import {
@@ -35,12 +35,24 @@ import { BulkEditResponse, UserDTO } from './dtos/login';
import jwt, { JwtPayload } from 'jsonwebtoken';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { Request, Response } from 'express';
import {
AuthSession,
clearSessionCookies,
writeSessionCookies,
} from './session-cookies';
type AuthSession = {
accessToken?: string;
refreshToken?: string;
userId?: string;
};
export type { AuthSession };
/**
* Upper bound for the DUC round-trips on the session-validation path
* (`GET /auth/me`). Without it a hung DUC keeps every `auth_request`
* subrequest open until the ingress gives up (minutes), which surfaces to
* users as 500/504 pages instead of a fast, retryable 503.
*/
function authRpcTimeoutMs(): number {
const ms = Number(process.env.AUTH_RPC_TIMEOUT_MS);
return Number.isFinite(ms) && ms > 0 ? ms : 10_000;
}
@Injectable()
export class AuthClientService implements OnModuleInit {
@@ -70,7 +82,9 @@ export class AuthClientService implements OnModuleInit {
async getPublicKeys() {
this.logger.info('GetPublicKeys');
return lastValueFrom(this.authService.AuthGetPublicKeys({}));
return lastValueFrom(
this.authService.AuthGetPublicKeys({}).pipe(timeout(authRpcTimeoutMs())),
);
}
async snowflakeSignIn(input: AuthSnowflakeSignInRequest) {
@@ -132,10 +146,9 @@ export class AuthClientService implements OnModuleInit {
this.logger.info('RefreshAccessToken');
return lastValueFrom(
this.authService.AuthRefreshAccessToken(
{ refreshToken, userId },
metadata,
),
this.authService
.AuthRefreshAccessToken({ refreshToken, userId }, metadata)
.pipe(timeout(authRpcTimeoutMs())),
);
}
@@ -346,67 +359,18 @@ export class AuthClientService implements OnModuleInit {
};
}
/**
* Persists the session in the browser as httpOnly cookies. Names, domain
* and lifetime are environment-scoped — see `session-cookies.ts`.
*/
public writeAuthSession(res: Response, data: AuthSession) {
let exp = 1000 * 60 * 5; // 5 minutes
if (data.accessToken) {
const { exp: expiration } = jwt.decode(data.accessToken) as JwtPayload;
exp = (expiration - 30) * 1000; // exp em segundos, maxAge em ms
this.logger.info('Set Cookie ddf-auth');
res.cookie('ddf-auth', data.accessToken, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
if (data.refreshToken) {
this.logger.info('Set Cookie ddf-refresh-auth');
res.cookie('ddf-refresh-auth', data.refreshToken, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
if (data.userId) {
this.logger.info('Set Cookie ddf-refresh-auth');
res.cookie('ddf-user-id', data.userId, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
this.logger.info('Set session cookies');
writeSessionCookies(res, data);
}
/** Expires every session cookie this environment writes. */
public cleanUpAuthSession(res: Response) {
const exp = 1000 * 60 * 3;
res.cookie('ddf-auth', '', {
domain: 'dadosfera.ai',
maxAge: Date.now() - exp,
expires: new Date(),
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
res.cookie('ddf-refresh-auth', '', {
domain: 'dadosfera.ai',
maxAge: Date.now() - exp,
expires: new Date(),
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
clearSessionCookies(res);
this.logger.info('Clean cookie sessions');
}
@@ -432,9 +396,16 @@ export class AuthClientService implements OnModuleInit {
});
const { user } = await lastValueFrom(
this.userService.UserFindOneById({ id: payload.user_id }, metadata),
this.userService
.UserFindOneById({ id: payload.user_id }, metadata)
.pipe(timeout(authRpcTimeoutMs())),
);
if (!user) {
// Deleted/unknown user: a definitive "no session", not a crash.
throw new HttpException('User not found', HttpStatus.UNAUTHORIZED);
}
const userDto: UserDTO = {
id: user.id,
name: user.name,
@@ -447,6 +418,9 @@ export class AuthClientService implements OnModuleInit {
name: payload.customer_name,
tier: payload.customer_tier,
},
// Raw permission seqids from the JWT. Consumers own the seqid->meaning
// mapping (e.g. Orchest's auth-server); Maestro reports them as-is.
permissions: payload.permissions ?? [],
};
return userDto;
+2 -1
View File
@@ -152,5 +152,6 @@ export type UserDTO = {
id: string,
name: string,
tier: string,
}
},
permissions: number[],
}
+103
View File
@@ -0,0 +1,103 @@
import {
clearSessionCookies,
readAuthSession,
sessionCookieMaxAgeMs,
sessionCookieNames,
sessionCookieOptions,
writeSessionCookies,
} from './session-cookies';
const DAY_MS = 24 * 60 * 60 * 1000;
function fakeResponse() {
const cookie = jest.fn();
const clearCookie = jest.fn();
return { cookie, clearCookie };
}
describe('session cookies', () => {
it('uses the legacy names and domain when nothing is configured (production)', () => {
const env = {};
expect(sessionCookieNames(env)).toEqual({
accessToken: 'ddf-auth',
refreshToken: 'ddf-refresh-auth',
userId: 'ddf-user-id',
});
expect(sessionCookieOptions(env)).toMatchObject({
domain: '.dadosfera.ai',
path: '/',
httpOnly: true,
secure: true,
sameSite: 'none',
});
});
it('scopes names and domain per environment so staging never clobbers production', () => {
const env = { AUTH_COOKIE_SUFFIX: '-stg', COOKIE_DOMAIN: '.stg.dadosfera.ai' };
expect(sessionCookieNames(env)).toEqual({
accessToken: 'ddf-auth-stg',
refreshToken: 'ddf-refresh-auth-stg',
userId: 'ddf-user-id-stg',
});
expect(sessionCookieOptions(env).domain).toBe('.stg.dadosfera.ai');
});
it('treats maxAge as a duration (30 days by default), not as the token epoch', () => {
expect(sessionCookieMaxAgeMs({})).toBe(30 * DAY_MS);
expect(sessionCookieMaxAgeMs({ SESSION_COOKIE_MAX_AGE_DAYS: '7' })).toBe(7 * DAY_MS);
expect(sessionCookieMaxAgeMs({ SESSION_COOKIE_MAX_AGE_DAYS: 'nope' })).toBe(30 * DAY_MS);
expect(sessionCookieMaxAgeMs({ SESSION_COOKIE_MAX_AGE_DAYS: '-1' })).toBe(30 * DAY_MS);
});
it('reads the session using the environment-specific names', () => {
const env = { AUTH_COOKIE_SUFFIX: '-stg' };
const req = {
cookies: {
'ddf-auth': 'prod-token',
'ddf-auth-stg': 'stg-token',
'ddf-refresh-auth-stg': 'stg-refresh',
'ddf-user-id-stg': 'u-stg',
},
};
expect(readAuthSession(req as any, env)).toEqual({
accessToken: 'stg-token',
refreshToken: 'stg-refresh',
userId: 'u-stg',
});
expect(readAuthSession({ cookies: undefined } as any, env)).toEqual({
accessToken: undefined,
refreshToken: undefined,
userId: undefined,
});
});
it('writes the access-token cookie first and only the parts that are present', () => {
const res = fakeResponse();
writeSessionCookies(res as any, { accessToken: 'a', userId: 'u' }, {});
expect(res.cookie.mock.calls.map((c) => c[0])).toEqual(['ddf-auth', 'ddf-user-id']);
expect(res.cookie.mock.calls[0][2]).toMatchObject({
domain: '.dadosfera.ai',
path: '/',
maxAge: 30 * DAY_MS,
httpOnly: true,
secure: true,
sameSite: 'none',
});
});
it('clears all three cookies with the same domain and path they were set with', () => {
const res = fakeResponse();
const env = { AUTH_COOKIE_SUFFIX: '-stg', COOKIE_DOMAIN: '.stg.dadosfera.ai' };
clearSessionCookies(res as any, env);
expect(res.clearCookie.mock.calls.map((c) => c[0])).toEqual([
'ddf-auth-stg',
'ddf-refresh-auth-stg',
'ddf-user-id-stg',
]);
for (const call of res.clearCookie.mock.calls) {
expect(call[1]).toMatchObject({ domain: '.stg.dadosfera.ai', path: '/' });
}
});
});
+131
View File
@@ -0,0 +1,131 @@
import { CookieOptions, Request, Response } from 'express';
/**
* Browser session cookies written by Maestro and consumed by every app that
* authenticates through `GET /auth/me` (Orchest auth-server, nginx
* `auth_request` ingresses, ...).
*
* The names and the domain are environment-scoped so that a staging login
* never overwrites (or is confused with) a production session in the same
* browser: both platforms live under `dadosfera.ai`, so a cookie on the
* registrable domain is sent to *both* environments.
*
* ENV | AUTH_COOKIE_SUFFIX | COOKIE_DOMAIN
* prd | "" | .dadosfera.ai -> ddf-auth
* stg | "-stg" | .stg.dadosfera.ai -> ddf-auth-stg
*/
export type AuthSession = {
accessToken?: string;
refreshToken?: string;
userId?: string;
};
export type SessionCookieNames = {
accessToken: string;
refreshToken: string;
userId: string;
};
type Env = Record<string, string | undefined>;
const DEFAULT_COOKIE_DOMAIN = '.dadosfera.ai';
const DEFAULT_SESSION_COOKIE_MAX_AGE_DAYS = 30;
export function cookieDomain(env: Env = process.env): string {
return env.COOKIE_DOMAIN || DEFAULT_COOKIE_DOMAIN;
}
export function cookieSuffix(env: Env = process.env): string {
return env.AUTH_COOKIE_SUFFIX || '';
}
export function sessionCookieNames(env: Env = process.env): SessionCookieNames {
const suffix = cookieSuffix(env);
return {
accessToken: `ddf-auth${suffix}`,
refreshToken: `ddf-refresh-auth${suffix}`,
userId: `ddf-user-id${suffix}`,
};
}
/**
* How long the browser keeps the session cookies. This is a *duration*
* (express `maxAge` is in milliseconds from now), independent from the
* access token's own `exp`: an expired access token in the cookie is
* harmless — `/auth/me` transparently refreshes it — while a refresh token
* that outlives the cookie would make the browser forget a still-valid
* session. Keep it aligned with the IdP refresh-token validity.
*/
export function sessionCookieMaxAgeMs(env: Env = process.env): number {
const days = Number(env.SESSION_COOKIE_MAX_AGE_DAYS);
const safeDays =
Number.isFinite(days) && days > 0
? days
: DEFAULT_SESSION_COOKIE_MAX_AGE_DAYS;
return safeDays * 24 * 60 * 60 * 1000;
}
export function sessionCookieOptions(env: Env = process.env): CookieOptions {
return {
domain: cookieDomain(env),
path: '/',
httpOnly: true,
secure: true,
// Required: the cookies are sent cross-site (app.* -> maestro.*) and
// from embedded iframes (app.* embedding app-intelligence-*.*).
sameSite: 'none',
};
}
export function readAuthSession(
req: Pick<Request, 'cookies'>,
env: Env = process.env,
): AuthSession {
const names = sessionCookieNames(env);
const cookies = req.cookies ?? {};
return {
accessToken: cookies[names.accessToken] || undefined,
refreshToken: cookies[names.refreshToken] || undefined,
userId: cookies[names.userId] || undefined,
};
}
/**
* (Re)writes whichever parts of the session are present. The access-token
* cookie is written first on purpose: nginx `auth_request` relays only the
* first `Set-Cookie` of the auth subrequest (`$upstream_http_set_cookie`),
* and the access token is the one that changes on refresh.
*/
export function writeSessionCookies(
res: Pick<Response, 'cookie'>,
session: AuthSession,
env: Env = process.env,
): void {
const names = sessionCookieNames(env);
const options: CookieOptions = {
...sessionCookieOptions(env),
maxAge: sessionCookieMaxAgeMs(env),
};
if (session.accessToken) {
res.cookie(names.accessToken, session.accessToken, options);
}
if (session.refreshToken) {
res.cookie(names.refreshToken, session.refreshToken, options);
}
if (session.userId) {
res.cookie(names.userId, session.userId, options);
}
}
/** Expires all three session cookies (same name/domain/path they were set with). */
export function clearSessionCookies(
res: Pick<Response, 'clearCookie'>,
env: Env = process.env,
): void {
const names = sessionCookieNames(env);
const options = sessionCookieOptions(env);
res.clearCookie(names.accessToken, options);
res.clearCookie(names.refreshToken, options);
res.clearCookie(names.userId, options);
}
+103 -1
View File
@@ -17,6 +17,7 @@ import {
HttpStatus,
Res,
} from '@nestjs/common';
import { ValidationPipe } from '../../pipes/object-validation.pipe';
import {
ApiCreatedResponse,
ApiHeaders,
@@ -46,6 +47,7 @@ import {
IMakeAComment,
IOneDataAsset,
IPreviewResponse,
IUpdateCertificationStatusRequest,
IUpdateDataRequest,
TriggerCatalogReq,
TriggerCatalogRes,
@@ -83,6 +85,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async searchCatalog(
@User() user: RequestUser,
@Query() query: ICatalogAllRequest,
@@ -122,6 +127,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async dowloadAsserts(
@User() user: RequestUser,
@Query() query: ICatalogAllRequest,
@@ -165,6 +173,9 @@ export class CatalogController {
@ApiInternalOnlyEndpoint()
@Get('data-asset')
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async findByPipelineAndObject(@User() user: RequestUser, @Query() query) {
const { username, user_id, customer_id, customer_name, permissions } = user;
const { pipeline, object } = query;
@@ -223,6 +234,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async findAllTags(@Body() body) {
this.logger.info(`/catalog - ON FIND ALL TAGS ROUTE`, {
user: body.info.user_id,
@@ -269,11 +283,31 @@ export class CatalogController {
}
@Get('custom-properties')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
async getCustomPropertyDefinitions(@User() user: RequestUser) {
const { customer_id, customer_name, user_id, username } = user;
const metadata = PackTheMetadata({
customer_id,
customer_name,
user_id,
username,
});
return this.catalogService.getCustomPropertyDefinitions(metadata);
}
@Get('data-asset/:id')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async getDataAsset(
@User() user: RequestUser,
@Param('id') id: string,
@@ -385,6 +419,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async getDataAssetColumnsMetadata(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -416,6 +453,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async getDataAssetPreview(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -447,6 +487,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async getDataAssetDocs(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -478,6 +521,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async updateDataAsset(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -493,6 +539,8 @@ export class CatalogController {
language,
});
delete (body as any).certification_status;
const result = await this.catalogService.updateOneDataAsset({
body,
data_asset_id,
@@ -506,11 +554,44 @@ export class CatalogController {
return result;
}
@Put('data-asset/:id/certification-status')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.CERTIFY,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async updateDataAssetCertificationStatus(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@Param('id') data_asset_id: string,
@Body(new ValidationPipe()) body: IUpdateCertificationStatusRequest,
): Promise<IUpdateCertificationStatusRequest> {
const { customer_id, customer_name, user_id, username } = user;
const metadata = PackTheMetadata({
customer_id,
customer_name,
user_id,
username,
language,
});
return this.catalogService.updateCertificationStatus({
body,
data_asset_id,
metadata,
});
}
@Post('data-asset/:id/docs')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async manageDataAssetDocs(
@User() user: RequestUser,
@Headers() headers,
@@ -548,6 +629,9 @@ export class CatalogController {
@ApiInternalOnlyEndpoint()
@Put('data-asset/:id/manage-permissions')
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async manageDataAssetPermissions(
@Param('id') id: string,
@User() user: RequestUser,
@@ -570,6 +654,9 @@ export class CatalogController {
@ApiInternalOnlyEndpoint()
@Put('data-asset/:id/revoke-permissions')
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async revokeDataAssetPermissions(
@Param('id') id: string,
@User() user: RequestUser,
@@ -595,6 +682,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.CREATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async createDataAsset(
@User() user: RequestUser,
@Body() body: ICreateDataAsset,
@@ -619,6 +709,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async commentOnDataAsset(
@Param('id') id: string,
@User() user: RequestUser,
@@ -645,6 +738,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.DELETE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async deleteDataAsset(@Param('id') id: string, @User() user: RequestUser) {
const { customer_id, customer_name, user_id, username } = user;
const metadata = PackTheMetadata({
@@ -666,6 +762,9 @@ export class CatalogController {
PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async deleteComment(
@Param('id') id: string,
@User() user: RequestUser,
@@ -819,6 +918,9 @@ export class CatalogController {
@Get('nimbus-dashboards')
@RequireAllPermissions(PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER)
@RequireModule(
DADOSFERA_MODULES_KEYS.CATALOG
)
async getNimbusDashboards(
@User() user: RequestUser,
@Body() body: GetNimbusDashboardsRequest,
@@ -971,4 +1073,4 @@ export class CatalogController {
this.logger.error(error.message);
}
}
}
}
-3
View File
@@ -5,20 +5,17 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { CatalogController } from './catalog.controller';
import { CatalogClientConfiguration } from './catalog-client';
import { ClientsModule } from '@nestjs/microservices';
import { PipelinesModule as OldPipelineModule } from 'src/modules/pipelines/pipelines.module';
import { UsersModule } from '../users/users.module';
import { RolesModule } from '../roles/roles.module';
import { CustomersModule } from '../customers/customers.module';
import { ShareModule } from './share/share.module';
import { CatalogService } from './catalog.service';
import { MixpanelModule } from '../mixpanel/mixpanel.module';
const client = new CatalogClientConfiguration();
@Module({
imports: [
ClientsModule.register([client.providerOptions]),
OldPipelineModule,
UsersModule,
RolesModule,
CustomersModule,
+27
View File
@@ -29,6 +29,7 @@ import {
AssetReporter,
BatchRemoveRlsRulesRequest,
CreateDataDocsDTO,
IUpdateCertificationStatusRequest,
IUpdateDataRequest,
TriggerCatalogReq,
} from './dtos';
@@ -119,6 +120,10 @@ class CatalogService implements OnModuleInit {
}
}
async getCustomPropertyDefinitions(metadata: Metadata) {
return lastValueFrom(this.catalogReadService.GetCustomPropertyDefinitions({}, metadata));
}
async createDataAsset(data: Messages.CreateDataAssetRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
if (!data.embed) data.embed = undefined;
@@ -384,6 +389,28 @@ class CatalogService implements OnModuleInit {
return { data_asset: asset[0] };
}
async updateCertificationStatus(data: {
data_asset_id: string;
body: IUpdateCertificationStatusRequest;
metadata: Metadata;
}) {
const { body, data_asset_id, metadata } = data;
await lastValueFrom(
this.catalogWriteService.UpdateDataAsset(
{
id: data_asset_id,
changes: JSON.stringify({
certification_status: body.certification_status,
}),
},
metadata,
),
);
return { certification_status: body.certification_status };
}
async updateOneDataAsset(data: {
data_asset_id: string;
customer_id: string;
+43 -1
View File
@@ -1,4 +1,10 @@
import { ApiProperty, ApiPropertyOptional, PickType } from '@nestjs/swagger';
import {
IsEnum,
IsNotEmpty,
IsOptional,
IsString,
} from 'class-validator';
import { CreateDataAssetRequest } from '@dadosfera/protospack-v2/dist/lib/Catalog/interfaces/messages';
export enum DataAssetShareType {
@@ -6,6 +12,12 @@ export enum DataAssetShareType {
public = 'public',
private = 'private',
}
export enum CertificationStatus {
draft = 'draft',
in_review = 'in_review',
approved = 'approved',
deprecated = 'deprecated',
}
export enum OrderEnum {
asc = 'asc',
desc = 'desc',
@@ -191,6 +203,27 @@ export class IData {
day_opening: number;
}
export enum CustomPropertyType {
TEXT = 'text',
NUMBER = 'number',
DATE = 'date',
BOOLEAN = 'boolean',
}
export class CustomPropertyDto {
@ApiProperty()
key: string;
@ApiProperty()
value: string;
@ApiProperty({ enum: CustomPropertyType })
type: CustomPropertyType;
@ApiPropertyOptional()
color?: string;
@ApiPropertyOptional()
emoji?: string;
}
export class IUpdateDataRequest {
@ApiProperty()
name: string;
@@ -204,7 +237,16 @@ export class IUpdateDataRequest {
share_type?: DataAssetShareType;
@ApiPropertyOptional()
docs?: string;
@ApiPropertyOptional({ type: [CustomPropertyDto] })
custom_properties?: CustomPropertyDto[];
}
export class IUpdateCertificationStatusRequest {
@ApiProperty({ enum: CertificationStatus })
@IsEnum(CertificationStatus)
certification_status: CertificationStatus;
}
export class ICreateDataAsset implements CreateDataAssetRequest {
@ApiProperty()
display_name: string;
@@ -358,4 +400,4 @@ export type CreateDataDocsDTO = {
docs: string;
asset_type: string;
}
}
@@ -22,17 +22,24 @@ import {
ConnectionTestListTablesRes,
GetTableMetadataRes,
GetTableMetadataReq,
RefreshCatalogReq,
RefreshCatalogRes,
RefreshCatalogStatusReq,
} from './dto/connection-test';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { Authenticated } from 'src/decorators/authentication.decorator';
import { Authenticated, RequireModule } from 'src/decorators/authentication.decorator';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
import { DADOSFERA_MODULES_KEYS } from 'src/authentication/permissions.enum';
@ApiInternalOnlyController()
@ApiTags('Connection Test')
@Controller('connection-test')
@UseFilters(new GrpcToHttpExceptionFilter())
@Authenticated()
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
export class ConnectionTestController {
logger: any;
constructor(
@@ -84,7 +91,7 @@ export class ConnectionTestController {
});
return this.connectionTestService.connectionTestListSchemas(
body,
user.customer_name,
user,
);
}
@@ -101,7 +108,7 @@ export class ConnectionTestController {
});
return this.connectionTestService.connectionTestListTables(
body,
user.customer_name,
user,
);
}
@@ -118,7 +125,38 @@ export class ConnectionTestController {
});
return this.connectionTestService.getTableMetadata(
body,
user.customer_name,
user,
);
}
@Post('refresh-catalog')
@ApiOkResponse({ type: RefreshCatalogRes })
@HttpCode(HttpStatus.ACCEPTED)
async refreshCatalog(
@User() user: RequestUser,
@Body(new ValidationPipe()) body: RefreshCatalogReq,
) {
this.logger.info('/connection-test/refresh-catalog', {
user: user.user_id,
customer: user.customer_name,
connection: body.connection_id,
});
return this.connectionTestService.refreshCatalog(body, user);
}
@Post('refresh-catalog/status')
@ApiOkResponse({ type: RefreshCatalogRes })
@HttpCode(HttpStatus.OK)
async refreshCatalogStatus(
@User() user: RequestUser,
@Body(new ValidationPipe()) body: RefreshCatalogStatusReq,
) {
this.logger.info('/connection-test/refresh-catalog/status', {
user: user.user_id,
customer: user.customer_name,
connection: body.connection_id,
session: body.session_id,
});
return this.connectionTestService.refreshCatalogStatus(body, user);
}
}
@@ -5,10 +5,17 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { ClientsModule } from '@nestjs/microservices';
import { ConnectionTestClientConfiguration } from './connection-test-client.config';
import { ConnectionModule } from '../connection/connection.module';
import { ConnectionsApiModule } from '../connections-api/connections-api.module';
import { PlatformApiModule } from '../platform-api/platform-api.module';
const client = new ConnectionTestClientConfiguration();
@Module({
controllers: [ConnectionTestController],
providers: [ConnectionTestService, DadosferaLogger],
imports: [ClientsModule.register([client.providerOptions]), ConnectionModule],
imports: [
ClientsModule.register([client.providerOptions]),
ConnectionModule,
ConnectionsApiModule,
PlatformApiModule,
],
})
export class ConnectionTestModule {}
@@ -0,0 +1,205 @@
import { ConnectionTestService } from './connection-test.service';
import { RequestUser } from 'src/decorators/user.decorator';
describe('ConnectionTestService catalog cache', () => {
const user: RequestUser = {
user_id: 'user-id',
username: 'user@example.com',
permissions: [],
customer_id: 'customer-id',
customer_name: 'customer-name',
customer_tier: 'standard',
access_token: 'token',
customer_modules: [],
roles: [],
};
const grpcClient = { getService: jest.fn().mockReturnValue({}) };
const connectionsService = {};
const connectionsApiService = { proxy: jest.fn() };
const platformApiService = { proxy: jest.fn() };
let service: ConnectionTestService;
beforeEach(() => {
jest.clearAllMocks();
service = new ConnectionTestService(
grpcClient as any,
connectionsService as any,
connectionsApiService as any,
platformApiService as any,
);
});
it('keeps the existing schemas response contract', async () => {
connectionsApiService.proxy.mockResolvedValue({
schemas: [{ schema_name: 'analytics' }, { schema_name: 'public' }],
});
await expect(
service.connectionTestListSchemas(
{ connection_id: 'config-id', plugin: 'postgresql' },
user,
),
).resolves.toEqual({
operation_result: true,
schema_list: ['analytics', 'public'],
});
});
it('keeps the existing tables response contract', async () => {
connectionsApiService.proxy.mockResolvedValue({
tables: [{ table_name: 'customers' }, { table_name: 'orders' }],
});
await expect(
service.connectionTestListTables(
{
connection_id: 'config-id',
plugin: 'postgresql',
schema: 'public',
},
user,
),
).resolves.toEqual({
operation_result: true,
table_list: ['customers', 'orders'],
});
});
it('maps cached columns to the existing table metadata contract', async () => {
connectionsApiService.proxy.mockResolvedValue({
columns: [
{
column_name: 'id',
data_type: 'bigint',
is_primary_key: true,
},
],
});
await expect(
service.getTableMetadata(
{
connection_id: 'config-id',
plugin: 'postgresql',
schema: 'public',
table_list: ['customers'],
},
user,
),
).resolves.toEqual({
operation_result: true,
tables_metadata: [
{
table_name: 'customers',
columns: [
{
name: 'id',
type: 'bigint',
is_primary_key: true,
},
],
references: [],
},
],
});
expect(connectionsApiService.proxy).toHaveBeenCalledWith(
'GET',
'/connection_catalog/config-id/schemas/public/tables/customers/columns',
user,
);
});
it('submits a catalog refresh without holding the request open', async () => {
platformApiService.proxy.mockResolvedValue({
session_id: 'session-id',
date: '20260731',
});
await expect(
service.refreshCatalog(
{ connection_id: 'config-id', plugin: 'postgresql' },
user,
),
).resolves.toEqual({
operation_result: true,
status: 'PENDING',
session_id: 'session-id',
date: '20260731',
});
expect(platformApiService.proxy).toHaveBeenCalledWith(
'POST',
'/connection_test',
user,
{
customer_id: user.customer_name,
plugin: 'postgresql',
task: {
task_type: 'refresh_catalog',
connection: {
provider: 'connection_manager',
config_id: 'config-id',
},
},
},
);
});
it('keeps polling without changing the catalog pointer while pending', async () => {
platformApiService.proxy.mockResolvedValue({ status: 'PENDING' });
await expect(
service.refreshCatalogStatus(
{
connection_id: 'config-id',
plugin: 'postgresql',
session_id: 'session-id',
date: '20260731',
},
user,
),
).resolves.toEqual({
operation_result: false,
status: 'PENDING',
session_id: 'session-id',
date: '20260731',
});
expect(connectionsApiService.proxy).not.toHaveBeenCalled();
});
it('publishes the catalog pointer after the refresh finishes', async () => {
platformApiService.proxy.mockResolvedValue({ status: 'DONE' });
connectionsApiService.proxy.mockResolvedValue({
last_catalog_refresh_status: 'SUCCESS',
});
await expect(
service.refreshCatalogStatus(
{
connection_id: 'config/id',
plugin: 'postgresql',
session_id: 'session-id',
date: '20260731',
},
user,
),
).resolves.toEqual({
operation_result: true,
status: 'DONE',
session_id: 'session-id',
date: '20260731',
});
expect(connectionsApiService.proxy).toHaveBeenCalledWith(
'PUT',
'/connection_config/config%2Fid/catalog_metadata',
user,
{
last_catalog_refresh_status: 'SUCCESS',
last_catalog_connection_test_date: '20260731',
last_catalog_connection_test_session_id: 'session-id',
},
);
});
});
@@ -1,4 +1,4 @@
import { Inject, Injectable } from '@nestjs/common';
import { HttpException, HttpStatus, Inject, Injectable } from '@nestjs/common';
import { ClientGrpc } from '@nestjs/microservices';
import { ConnectionTest } from '@dadosfera/protospack-v2';
import { lastValueFrom } from 'rxjs';
@@ -13,6 +13,9 @@ import {
ConnectionTestPingRes,
GetTableMetadataReq,
GetTableMetadataRes,
RefreshCatalogReq,
RefreshCatalogRes,
RefreshCatalogStatusReq,
} from './dto/connection-test';
import { ConnectionClientService } from '../connection/client.service';
import {
@@ -21,6 +24,8 @@ import {
} from '../connection/dtos/connection';
import { RequestUser } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { ConnectionsApiService } from '../connections-api/connections-api.service';
import { PlatformApiService } from '../platform-api/platform-api.service';
@Injectable()
export class ConnectionTestService {
@@ -28,6 +33,8 @@ export class ConnectionTestService {
constructor(
@Inject('ConnectionTestGrpcClient') private readonly grpcClient: ClientGrpc,
private connectionsService: ConnectionClientService,
private connectionsApiService: ConnectionsApiService,
private platformApiService: PlatformApiService,
) {
this.connectionTestReadClient =
grpcClient.getService<ConnectionTest.ReadService.ConnectionTestReadServices>(
@@ -147,45 +154,137 @@ export class ConnectionTestService {
}
async connectionTestListSchemas(
body: ConnectionTestListSchemasReq,
customer_name: string,
user: RequestUser,
): Promise<ConnectionTestListSchemasRes> {
const { connection_id, plugin } = body;
return lastValueFrom(
this.connectionTestReadClient.ListSchemas({
connection_id,
customer_name,
plugin,
}),
const result = await this.connectionsApiService.proxy(
'GET',
`/connection_catalog/${encodeURIComponent(body.connection_id)}/schemas`,
user,
);
return {
operation_result: true,
schema_list: result.schemas.map((schema) => schema.schema_name),
};
}
async connectionTestListTables(
body: ConnectionTestListTablesReq,
customer_name: string,
user: RequestUser,
): Promise<ConnectionTestListTablesRes> {
const { connection_id, plugin, schema } = body;
return lastValueFrom(
this.connectionTestReadClient.ListTables({
connection_id,
customer_name,
plugin,
schema,
}),
const result = await this.connectionsApiService.proxy(
'GET',
`/connection_catalog/${encodeURIComponent(body.connection_id)}` +
`/schemas/${encodeURIComponent(body.schema)}/tables`,
user,
);
return {
operation_result: true,
table_list: result.tables.map((table) => table.table_name),
};
}
async getTableMetadata(
body: GetTableMetadataReq,
customer_name: string,
user: RequestUser,
): Promise<GetTableMetadataRes> {
const { schema, plugin, table_list, connection_id } = body;
return lastValueFrom(
this.connectionTestReadClient.GetTableMetadata({
connection_id,
customer_name,
plugin,
schema,
table_list,
const tables_metadata = await Promise.all(
body.table_list.map(async (table_name) => {
const result = await this.connectionsApiService.proxy(
'GET',
`/connection_catalog/${encodeURIComponent(body.connection_id)}` +
`/schemas/${encodeURIComponent(body.schema)}` +
`/tables/${encodeURIComponent(table_name)}/columns`,
user,
);
return {
table_name,
columns: result.columns.map((column) => ({
name: column.column_name,
type: column.data_type,
is_primary_key: column.is_primary_key,
})),
references: [],
};
}),
);
return { operation_result: true, tables_metadata };
}
async refreshCatalog(
body: RefreshCatalogReq,
user: RequestUser,
): Promise<RefreshCatalogRes> {
const task = await this.platformApiService.proxy(
'POST',
'/connection_test',
user,
{
customer_id: user.customer_name,
plugin: body.plugin,
task: {
task_type: 'refresh_catalog',
connection: {
provider: 'connection_manager',
config_id: body.connection_id,
},
},
},
);
if (!task.session_id || !task.date) {
throw new HttpException(
'Platform API did not return a catalog refresh task identifier',
HttpStatus.BAD_GATEWAY,
);
}
return {
operation_result: true,
status: 'PENDING',
session_id: task.session_id,
date: task.date,
};
}
async refreshCatalogStatus(
body: RefreshCatalogStatusReq,
user: RequestUser,
): Promise<RefreshCatalogRes> {
const result = await this.platformApiService.proxy(
'POST',
'/connection_test/status',
user,
{
session_id: body.session_id,
date: body.date,
},
);
if (result.status === 'DONE') {
await this.connectionsApiService.proxy(
'PUT',
`/connection_config/${encodeURIComponent(
body.connection_id,
)}/catalog_metadata`,
user,
{
last_catalog_refresh_status: 'SUCCESS',
last_catalog_connection_test_date: body.date,
last_catalog_connection_test_session_id: body.session_id,
},
);
} else if (result.status === 'ERROR' || result.status === 'EXPIRED') {
throw new HttpException(
`Catalog refresh finished with status ${result.status}`,
HttpStatus.BAD_GATEWAY,
);
}
return {
operation_result: result.status === 'DONE',
status: result.status,
session_id: body.session_id,
date: body.date,
};
}
}
@@ -1,5 +1,5 @@
import { ApiProperty, ApiPropertyOptional, OmitType } from '@nestjs/swagger';
import { IsString, IsOptional } from 'class-validator';
import { IsIn, IsString, IsOptional } from 'class-validator';
import { DatabaseConnectionPropertiesDto } from 'src/modules/connection/dtos/connection';
import { CreateConnectionDto } from 'src/modules/connection/dtos/connection';
export class ColumnDto {
@@ -7,6 +7,8 @@ export class ColumnDto {
name: string;
@ApiProperty()
type: string;
@ApiProperty()
is_primary_key: boolean;
}
export class TableMetadataDto {
@ApiProperty()
@@ -131,3 +133,37 @@ export class GetTableMetadataRes {
@ApiProperty({ type: [TableMetadataDto] })
tables_metadata: TableMetadataDto[];
}
export class RefreshCatalogReq {
@ApiProperty()
@IsString()
connection_id: string;
@ApiProperty({ enum: ['oracle', 'mysql', 'postgresql', 'sqlserver'] })
@IsIn(['oracle', 'mysql', 'postgresql', 'sqlserver'])
plugin: string;
}
export class RefreshCatalogStatusReq extends RefreshCatalogReq {
@ApiProperty()
@IsString()
session_id: string;
@ApiProperty()
@IsString()
date: string;
}
export class RefreshCatalogRes {
@ApiProperty()
operation_result: boolean;
@ApiProperty()
status: string;
@ApiProperty()
session_id: string;
@ApiProperty()
date: string;
}
@@ -16,8 +16,9 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import {
Authenticated,
RequireAllPermissions,
RequireModule,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { ValidationPipe } from '../../pipes/object-validation.pipe';
import {
@@ -39,6 +40,9 @@ const connectionPermissions = PERMISSIONS_GROUPS.CONNECTION.permissions;
@ApiTags('connections')
@Authenticated()
@Controller('connections')
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
export class ConnectionController {
logger: any;
constructor(
@@ -0,0 +1,11 @@
export const CONNECTIONS_API_CONFIG = {
getUrl: (): string => {
const url = process.env.CONNECTIONS_API_URL;
if (!url) {
throw new Error('CONNECTIONS_API_URL environment variable is not set');
}
return url;
},
region: process.env.AWS_REGION || 'us-east-1',
timeout: parseInt(process.env.CONNECTIONS_API_TIMEOUT || '30000', 10),
};
@@ -0,0 +1,10 @@
import { Module } from '@nestjs/common';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { ConnectionsApiService } from './connections-api.service';
@Module({
providers: [ConnectionsApiService, DadosferaLogger],
exports: [ConnectionsApiService],
})
export class ConnectionsApiModule {}
@@ -0,0 +1,99 @@
import { Injectable, Inject, HttpException } from '@nestjs/common';
import { SignatureV4 } from '@aws-sdk/signature-v4';
import { Sha256 } from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import axios, { AxiosResponse, Method } from 'axios';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { RequestUser } from '../../decorators/user.decorator';
import { CONNECTIONS_API_CONFIG } from './connections-api.config';
@Injectable()
export class ConnectionsApiService {
private signer: SignatureV4;
private logger: any;
constructor(@Inject(DadosferaLogger) dadosferaLogger: DadosferaLogger) {
this.logger = dadosferaLogger.logger;
this.signer = new SignatureV4({
service: 'execute-api',
region: CONNECTIONS_API_CONFIG.region,
credentials: defaultProvider(),
sha256: Sha256,
});
}
async proxy(
method: string,
path: string,
user: RequestUser,
body?: any,
query?: Record<string, string>,
): Promise<any> {
const baseUrl = CONNECTIONS_API_CONFIG.getUrl();
const url = new URL(`${baseUrl}${path}`);
if (query) {
Object.entries(query).forEach(([key, value]) => {
if (value !== undefined && value !== null) {
url.searchParams.set(key, String(value));
}
});
}
const headers: Record<string, string> = {
host: url.hostname,
'content-type': 'application/json',
customer_name: user.customer_name || '',
customer_id: user.customer_id || '',
'x-user-id': user.user_id || '',
'x-username': user.username || '',
'x-customer-tier': user.customer_tier || '',
'x-customer-id': user.customer_id || '',
};
const requestToSign = {
method: method.toUpperCase(),
protocol: url.protocol,
hostname: url.hostname,
port: url.port ? parseInt(url.port, 10) : undefined,
path: url.pathname + url.search,
headers,
body: body ? JSON.stringify(body) : undefined,
};
try {
const signedRequest = await this.signer.sign(requestToSign);
const response: AxiosResponse = await axios({
method: method as Method,
url: url.href,
headers: signedRequest.headers as Record<string, string>,
data: body,
timeout: CONNECTIONS_API_CONFIG.timeout,
validateStatus: () => true,
});
if (response.status >= 400) {
throw new HttpException(response.data, response.status);
}
return response.data;
} catch (error) {
this.logger.error('Connections API proxy error', {
error: error.message,
path,
method: method.toUpperCase(),
});
if (error instanceof HttpException) {
throw error;
}
if (error.response) {
throw new HttpException(error.response.data, error.response.status);
}
if (error.code === 'ECONNREFUSED') {
throw new HttpException('Connections API service unavailable', 503);
}
if (error.code === 'ETIMEDOUT' || error.code === 'ECONNABORTED') {
throw new HttpException('Connections API request timeout', 504);
}
throw new HttpException('Internal server error', 500);
}
}
}
+26 -1
View File
@@ -25,9 +25,10 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import {
Authenticated,
RequireAllPermissions,
RequireModule,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { Language } from 'src/decorators/language.decorator';
import { LanguageEnum } from 'src/utils/languages.enum';
import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
@@ -99,6 +100,9 @@ export class ConnectorController {
PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async getAllConnectors(
@Language() language: LanguageEnum,
@Query() queries: GetAllDto,
@@ -131,6 +135,9 @@ export class ConnectorController {
PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async getConnectorsTags() {
return await this.connectorClientService.getConnectorsTags();
}
@@ -143,6 +150,9 @@ export class ConnectorController {
PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async getConnector(
@Language() language: LanguageEnum,
@Param('plugin') plugin: string,
@@ -171,6 +181,9 @@ export class ConnectorController {
PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async getConnectorDetails(
@Language() language: LanguageEnum,
@Param('plugin') plugin: string,
@@ -193,6 +206,9 @@ export class ConnectorController {
@Put('/:plugin')
@RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
@ApiConsumes('multipart/form-data')
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async updateConnector(
@Param('plugin') plugin: string,
@Body() body: UpdateDto,
@@ -214,6 +230,9 @@ export class ConnectorController {
@Put('/:plugin/add-tag')
@RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async addTagOnConnector(
@Param('plugin') plugin: string,
@Body() body: AddTagDto,
@@ -241,6 +260,9 @@ export class ConnectorController {
@Put('/:plugin/remove-tag')
@RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async removeTagOnConnector(
@Param('plugin') plugin: string,
@Body() body: RemoveTagDto,
@@ -269,6 +291,9 @@ export class ConnectorController {
@Delete('/:plugin')
@RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.DELETE)
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
async deleteConnector(
@Param('plugin') plugin: string,
@Query('version') version: string,
+5 -1
View File
@@ -1,4 +1,8 @@
import { Info } from '@dadosfera/protospack/dist/lib/interfaces';
export interface Info {
user_id: string;
customer_id: string;
customer: string;
}
interface Values {
jdbc_user: string;
+5 -4
View File
@@ -23,6 +23,7 @@ import {
} from '@dadosfera/protospack-v2/dist/lib/Input/interfaces/messages';
import { Info } from '@dadosfera/protospack-v2/dist/lib/Input/interfaces/entities';
import { CreateInputReq } from './dtos/input.model';
import { Metadata } from '@grpc/grpc-js';
@Injectable()
@@ -73,10 +74,10 @@ export class InputsService {
objectCamelToSnake(createInputResponse);
return createInputResponse;
},
update: async (updateInputDTO: UpdateInputRequest): Promise<InputUpdateResponse> => {
update: async (updateInputDTO: UpdateInputRequest, metadata: Metadata): Promise<InputUpdateResponse> => {
this.logger.info('InputClientService - Update' + JSON.stringify(updateInputDTO));
const updateInputResponse = await lastValueFrom(
this.inputWriteService.InputUpdate(updateInputDTO),
this.inputWriteService.InputUpdate(updateInputDTO, metadata),
);
return updateInputResponse;
@@ -206,7 +207,7 @@ export class InputsService {
return findOneInputResponse;
}
async update(id: string, data, info: Info) {
async update(id: string, data, info: Info, metadata?: Metadata) {
// this.validateCron({ ...data, info });
try {
const {
@@ -217,7 +218,7 @@ export class InputsService {
id,
...data,
info,
});
}, metadata);
const updateInputResponse = this.adjustInputPayload(
input,
@@ -1,6 +1,6 @@
import { ClientsModule } from '@nestjs/microservices';
import { Test, TestingModule } from '@nestjs/testing';
// import { DucClient } from 'src/clients/duc/client.config';
// import { DucClient } from '../duc/client.config'
import { PermissionsController } from './permissions.controller';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PermissionsService } from './permissions.service';
@@ -22,7 +22,7 @@ describe.skip('PermissionsController', () => {
provide: DadosferaLogger,
useValue: { logger },
},
PermissionsService,
PermissionsService
],
}).compile();
-78
View File
@@ -1,78 +0,0 @@
import { ConflictException, Inject, OnModuleInit } from '@nestjs/common';
import { ClientGrpc } from '@nestjs/microservices';
import {
PipelineServicesNames,
PipelinesServiceInterface,
} from '@dadosfera/protospack';
import { lastValueFrom } from 'rxjs';
import { IIdRequest } from './interfaces';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PipelinesClientConfiguration } from './pipelines-client';
export class PipelinesClientService implements OnModuleInit {
private pipelineService: PipelinesServiceInterface;
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
@Inject(PipelinesClientConfiguration.name)
private readonly grpcClient: ClientGrpc,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.pipelineService =
this.grpcClient.getService<PipelinesServiceInterface>(
PipelineServicesNames.PipelineService,
);
}
async getPipelineStatus(data) {
this.logger.info('PipelinesClientService - GetPipelineStatus');
const statusPipelineResponse = await lastValueFrom(
this.pipelineService.getPipelineStatus(data),
)
.then((res) => {
const statusArray =
res.status?.sort((a, b) => {
if (a.id < b.id) {
return 1;
} else {
return -1;
}
}) || [];
return { status: statusArray };
})
.catch((err) => {
this.logger.error(err.message);
throw new Error(err);
});
this.logger.info('Done');
return statusPipelineResponse;
}
async runPipeline({ id, info }: IIdRequest) {
this.logger.info('PipelinesClientService - RunPipeline');
const statusPipelineResponse = await lastValueFrom(
this.pipelineService.triggerPipeline({ id, info }),
).catch((err) => {
this.logger.error(err.message);
throw new Error(err);
});
if (statusPipelineResponse.status == false) {
throw new ConflictException(
'This pipeline is not ready yet to execute, Try again later!',
);
}
this.logger.info('Done');
return statusPipelineResponse;
}
}
-36
View File
@@ -1,36 +0,0 @@
import { Info } from '@dadosfera/protospack/dist/lib/interfaces';
export interface ICreatePipelineDto {
input: IdRequest;
transformations: IdRequest[];
output: IdRequest;
tags: string[];
name: string;
description: string;
info: Info;
}
export interface IdRequest {
id: string;
}
export interface IIdRequest {
id: string;
info: Info;
}
export interface IUpdatePipelineRequest {
input: IdRequest;
transformations: IdRequest[];
output: IdRequest;
tags: string[];
name: string;
description: string;
id: string;
info: Info;
}
export interface IGetPipelineLogsRequest {
id: string;
details: string;
}
-33
View File
@@ -1,33 +0,0 @@
import {
ClientsProviderAsyncOptions,
GrpcOptions,
Transport,
} from '@nestjs/microservices';
import { PipelinePackages, PipelineProtoFilePath } from '@dadosfera/protospack';
import { credentials } from '@grpc/grpc-js';
const isLocalConnection =
process.env.PIFACTORY_URL.startsWith('pi-factory:') ||
process.env.PIFACTORY_URL.includes('0.0.0.0');
export class PipelinesClientConfiguration {
public name = 'PipelinesClientConfiguration';
private config: GrpcOptions = {
transport: Transport.GRPC,
options: {
url: process.env.PIFACTORY_URL,
package: PipelinePackages,
credentials: isLocalConnection ? undefined : credentials.createSsl(),
protoPath: PipelineProtoFilePath,
loader: {
keepCase: true,
enums: String,
defaults: false,
},
},
};
providerOptions: ClientsProviderAsyncOptions = {
name: this.name,
...this.config,
};
}
@@ -1,72 +0,0 @@
import { Body, Controller, Get, Inject, Param, Post } from '@nestjs/common';
import { ApiOperation, ApiTags } from '@nestjs/swagger';
import {
AuthenticateCondition,
Authenticated,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { PipelinesService } from './pipelines.service';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
@ApiInternalOnlyController()
@ApiTags('Pipelines')
@Controller('pipelines')
@Authenticated()
export class PipelinesController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private pipelineService: PipelinesService,
) {
this.logger = dadosferaLogger.logger;
}
@Post('start/:id')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@ApiOperation({
deprecated: true,
description:
'This method is deprecated. Please use route /pipelinesV2/start/:id instead',
})
async activate(@Param('id') id: string, @Body() body) {
const { info } = body;
this.logger.info(
process.env.DEV_URL + `/pipeline/start/${id} - ON START PIPELINE ROUTE`,
{
user: body.info.user_id,
customer: body.info.customer,
},
);
const response = await this.pipelineService.runPipeline({ id, info });
return response;
}
@Get(':id/status')
@ApiOperation({
deprecated: true,
description:
'This method is deprecated. Please use route /pipelinesV2/:id/status instead',
})
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineStatus(@Body() body, @Param('id') id: string) {
body.id = id;
this.logger.info(
process.env.DEV_URL + `/pipeline/${id} - ON GET PIPELINE STATUS ROUTE`,
{
user: body.info.user_id,
customer: body.info.customer,
},
);
const response = await this.pipelineService.getPipelineStatus(body);
return response;
}
}
-19
View File
@@ -1,19 +0,0 @@
import { Module } from '@nestjs/common';
import { ClientsModule } from '@nestjs/microservices';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PipelinesController } from './pipelines.controller';
import { PipelinesService } from './pipelines.service';
import { PipelinesClientConfiguration } from './pipelines-client';
import { PipelinesClientService } from './client.service';
const client = new PipelinesClientConfiguration();
@Module({
imports: [ClientsModule.register([client.providerOptions])],
controllers: [PipelinesController],
providers: [PipelinesService, PipelinesClientService, DadosferaLogger],
exports: [PipelinesService],
})
export class PipelinesModule {}
@@ -1,33 +0,0 @@
import { HttpException, HttpStatus, Injectable } from '@nestjs/common';
import { PipelinesClientService } from './client.service';
import { IIdRequest } from './interfaces';
import { objectCamelToSnake } from 'src/utils/CaseConverter';
@Injectable()
export class PipelinesService {
constructor(private pipelineClient: PipelinesClientService) {}
async getPipelineStatus(data: IIdRequest) {
try {
const pipelineStatusResponse =
await this.pipelineClient.getPipelineStatus(data);
return objectCamelToSnake(pipelineStatusResponse);
} catch (err) {
throw new HttpException(err.message, HttpStatus.NOT_FOUND);
}
}
async runPipeline({ id, info }: IIdRequest) {
try {
const triggerPipelineResponse = await this.pipelineClient.runPipeline({
id,
info,
});
return objectCamelToSnake(triggerPipelineResponse);
} catch (err) {
throw new HttpException(err.message, HttpStatus.NOT_FOUND);
}
}
}
+6 -1
View File
@@ -1,5 +1,4 @@
import { ApiProperty, ApiPropertyOptional, OmitType } from '@nestjs/swagger';
import { Info } from '@dadosfera/protospack/dist/lib/interfaces';
export class PipelineInputsDTO {
@ApiProperty()
@@ -62,6 +61,12 @@ export interface IIdRequest {
info: Info;
}
export interface Info {
user_id: string;
customer_id: string;
customer: string;
}
export interface IUpdatePipelineRequest {
input: IdRequest;
transformations: IdRequest[];
+23 -14
View File
@@ -25,16 +25,16 @@ import {
} from '@nestjs/swagger';
import {
RequireAllPermissions,
RequireModule,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { PipelinesService } from './pipelines.service';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { Messages } from '@dadosfera/protospack-v2/dist/lib/PipelineV2';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { PipelinesService as OldPipelineService } from 'src/modules/pipelines/pipelines.service';
import {
ICompleteUploadCSVFile,
ICreatePipelineCSVFile,
@@ -58,14 +58,15 @@ type PipelineTablesConfig = { input_id?: string; tables: PipelineTable[] };
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@UseFilters(new GrpcToHttpExceptionFilter())
@Controller('pipelinesV2')
@RequireModule(
DADOSFERA_MODULES_KEYS.COLLECT
)
export class PipelinesController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private pipelinesClientService: PipelinesService,
private oldPipelinesService: OldPipelineService,
) {
this.logger = dadosferaLogger.logger;
}
@@ -201,7 +202,7 @@ export class PipelinesController {
customer: body.info.customer,
});
const response = await this.oldPipelinesService.getPipelineStatus(body);
const response = await this.pipelinesClientService.getPipelineStatus(body);
return response;
}
@@ -320,7 +321,6 @@ export class PipelinesController {
) {
this.logger.info('PipelinesController - update', { user });
const { customer_id, customer_name, user_id, username } = user;
const info: Info = {
user_id: user.user_id,
customer: user.customer_name,
@@ -328,13 +328,7 @@ export class PipelinesController {
pipeline_id: pipelineId
};
const metadata = PackTheMetadata({
customer_id,
customer_name,
user_id,
username,
language,
});
const metadata = PackTheMetadata(user);
const response = await this.pipelinesClientService.updatePipelineInput(
pipelineId,
@@ -368,6 +362,21 @@ export class PipelinesController {
return response;
}
@Patch('/:id/upgrade')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
@HttpCode(HttpStatus.NO_CONTENT)
async upgradeConnector(
@Language() language: LanguageEnum,
@Param('id') id: string,
@User() user: RequestUser
) {
this.logger.info('PipelinesController - upgrade connector');
const metadata = PackTheMetadata(user);
await this.pipelinesClientService.upgrade(id, metadata);
}
@Delete(':id')
@ApiNoContentResponse()
@HttpCode(HttpStatus.NO_CONTENT)
@@ -497,7 +506,7 @@ export class PipelinesController {
},
);
const response = await this.oldPipelinesService.runPipeline({ id, info });
const response = await this.pipelinesClientService.runPipeline({ id, info });
return response;
}
@@ -7,7 +7,6 @@ import { PipelinesService } from './pipelines.service';
import { PipelinesClientConfiguration } from './pipelines-client';
import { PipelinesModule as OldPipelineModule } from 'src/modules/pipelines/pipelines.module';
import { ConnectorModule } from '../connector/connector.module';
import { InputsModule } from '../inputs/inputs.module';
import { TransformationsModule } from '../transformations/transformations.module';
@@ -21,7 +20,6 @@ const client = new PipelinesClientConfiguration();
@Module({
imports: [
ClientsModule.register([client.providerOptions]),
OldPipelineModule,
ConnectorModule,
InputsModule,
TransformationsModule,
+87 -29
View File
@@ -1,6 +1,7 @@
/* eslint-disable no-async-promise-executor */
import {
BadRequestException,
ConflictException,
HttpException,
HttpStatus,
Inject,
@@ -17,7 +18,7 @@ import { lastValueFrom } from 'rxjs';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PipelinesClientConfiguration } from './pipelines-client';
import { ICreatePipelineV2Req, UpdatePlatformInputRequest, UpdateTableDTO } from './interfaces';
import { ICreatePipelineV2Req, IIdRequest, UpdatePlatformInputRequest, UpdateTableDTO } from './interfaces';
import { PipelineV2CreateRequest } from '@dadosfera/protospack-v2/dist/lib/PipelineV2/interfaces/messages';
import { Metadata } from '@grpc/grpc-js';
import { ConnectorClientService } from '../connector/client.service';
@@ -176,6 +177,15 @@ export class PipelinesService implements OnModuleInit {
return updatePipelineResponse;
}
async upgrade(id: string, metadata: Metadata) {
await lastValueFrom(
this.pipelineWriteService.Upgrade(
{ id },
metadata,
),
);
}
async remove(data: { id: string; metadata: Metadata; user: RequestUser }) {
const { id, metadata, user } = data;
const info = {
@@ -373,7 +383,8 @@ export class PipelinesService implements OnModuleInit {
const updateInputResponse = await this.inputsService.update(
inputId,
updateInputDTO,
info
info,
metadata
);
const inputRollback = () => {
@@ -394,34 +405,36 @@ export class PipelinesService implements OnModuleInit {
const nimbusUpdates = updateInputResponse?.tablesUpdate || [];
nimbusUpdates.forEach(update => {
const nimbusRollback = () => {
return this.nimbusService.renameTable(
info.customer,
update.database,
{
table_name: update.table_name,
table_schema: update.table_schema
},
{
table_name: update.old_table_name,
table_schema: update.old_table_schema
}
);
if (user.customer_modules.includes('catalog')) {
nimbusUpdates.forEach(update => {
const nimbusRollback = () => {
return this.nimbusService.renameTable(
info.customer,
update.database,
{
table_name: update.table_name,
table_schema: update.table_schema
},
{
table_name: update.old_table_name,
table_schema: update.old_table_schema
}
);
}
rollback.push(nimbusRollback);
});
try {
await this.updateNimbus(info.customer, nimbusUpdates);
} catch (error) {
this.logger.error(error);
if (error instanceof AxiosError) {
this.logger.error(JSON.stringify(error.response.data));
}
await this.executeRenameRollback(rollback);
throw new Error("Error Nimbus updating tables");
}
rollback.push(nimbusRollback);
});
try {
await this.updateNimbus(info.customer, nimbusUpdates);
} catch (error) {
this.logger.error(error);
if (error instanceof AxiosError) {
this.logger.error(JSON.stringify(error.response.data));
}
await this.executeRenameRollback(rollback);
throw new Error("Error Nimbus updating tables");
}
try {
@@ -623,4 +636,49 @@ export class PipelinesService implements OnModuleInit {
return assets;
}
async getPipelineStatus(data) {
this.logger.info('PipelinesClientService - GetPipelineStatus');
const statusPipelineResponse = await lastValueFrom(
this.pipelineReadService.PipelineV2GetPipelineV2Status(data),
)
.then((res) => {
const statusArray =
res.status?.sort((a, b) => {
if (a.id < b.id) {
return 1;
} else {
return -1;
}
}) || [];
return { status: statusArray };
})
.catch((err) => {
this.logger.error(err.message);
throw new Error(err);
});
this.logger.info('Done');
return statusPipelineResponse;
}
async runPipeline({ id, info }: IIdRequest) {
this.logger.info('PipelinesClientService - RunPipeline');
const statusPipelineResponse = await lastValueFrom(
this.pipelineWriteService.PipelineV2TriggerPipelineV2({ id, info }),
).catch((err) => {
this.logger.error(err.message);
throw new Error(err);
});
if (statusPipelineResponse.status == false) {
throw new ConflictException(
'This pipeline is not ready yet to execute, Try again later!',
);
}
this.logger.info('Done');
return statusPipelineResponse;
}
}
@@ -14,16 +14,17 @@ import {
NotFoundException,
UseGuards,
} from '@nestjs/common';
import { ApiTags, ApiOperation } from '@nestjs/swagger';
import { ApiTags, ApiOperation, ApiOkResponse } from '@nestjs/swagger';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import {
Authenticated,
RequireAllPermissions,
RequireModule,
} from '../../decorators/authentication.decorator';
import { User, RequestUser } from '../../decorators/user.decorator';
import { PlatformApiService } from './platform-api.service';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { ElasticsearchService } from '../../services/elasticsearch';
import { DynamoDBService, ReferenceColumn } from '../../services/dynamodb';
import { CustomersService } from '../customers/customers.service';
@@ -49,6 +50,7 @@ type RenameTablesBody = {
@ApiTags('Platform API')
@Controller('platform')
@RequireModule(DADOSFERA_MODULES_KEYS.COLLECT)
export class PlatformApiController {
private logger: any;
@@ -72,6 +74,10 @@ export class PlatformApiController {
return id?.replace(/-/g, '_') || '';
}
private decodePathParam(value: string): string {
return value ? decodeURIComponent(value) : '';
}
/**
* Denormalize ID back to UUID format (replace _ with -).
* Used when we receive a normalized ID but need the original UUID.
@@ -832,6 +838,40 @@ export class PlatformApiController {
);
}
@Get('pipelines/:pipelineId/pipeline_run/:runId/jobs')
@ApiOperation({
summary: 'Get pipeline run jobs',
description: 'Proxies platform-api DB-backed job runs and returns `{ jobs: [...] }`.',
})
@ApiOkResponse({
description: 'DB-backed job runs for the selected pipeline run.',
schema: {
type: 'object',
properties: {
jobs: {
type: 'array',
items: { type: 'object' },
},
},
required: ['jobs'],
},
})
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineRunJobs(
@Param('pipelineId') pipelineId: string,
@Param('runId') runId: string,
@User() user: RequestUser,
) {
const normalizedPipelineId = this.normalizePipelineId(pipelineId);
const decodedRunId = this.decodePathParam(runId);
return this.platformApiService.proxy(
'GET',
`/pipeline/${normalizedPipelineId}/pipeline_run/${decodedRunId}/jobs`,
user,
);
}
// ==================== JOBS - COLUMN EDITING ROUTES ====================
@Put('jobs/:jobId/input')
@@ -1,14 +1,25 @@
import { Test, TestingModule } from '@nestjs/testing';
import { ReleaseNoteController } from './release_note.controller';
import { ReleaseNoteService } from './release_note.service';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
describe('ReleaseNoteController', () => {
let controller: ReleaseNoteController;
const logger = {
info: (...args) => args,
error: (...args) => args,
};
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
controllers: [ReleaseNoteController],
providers: [ReleaseNoteService],
providers: [
{
provide: DadosferaLogger,
useValue: { logger },
},
ReleaseNoteService
],
}).compile();
controller = module.get<ReleaseNoteController>(ReleaseNoteController);
@@ -1,12 +1,23 @@
import { Test, TestingModule } from '@nestjs/testing';
import { ReleaseNoteService } from './release_note.service';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
describe('ReleaseNoteService', () => {
let service: ReleaseNoteService;
const logger = {
info: (...args) => args,
error: (...args) => args,
};
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [ReleaseNoteService],
providers: [
{
provide: DadosferaLogger,
useValue: { logger },
},
ReleaseNoteService
],
}).compile();
service = module.get<ReleaseNoteService>(ReleaseNoteService);
+5 -2
View File
@@ -1,5 +1,8 @@
import { Info } from '@dadosfera/protospack/dist/lib/interfaces';
export interface Info {
user_id: string;
customer_id: string;
customer: string;
}
export interface ICreateTransformationsRequest {
transformations: Transformation[];
info: Info;
+1
View File
@@ -9,6 +9,7 @@ interface IMetadata {
details?: string;
sensitive?: string;
roles?: string[];
customer_modules?: string[];
is_data_manager?: boolean;
access_token?: string;
host?: string;
+1 -1
View File
@@ -1,4 +1,4 @@
{
"extends": "./tsconfig.json",
"exclude": ["node_modules", "test", "dist", "**/*spec.ts", "jest.config.ts"]
"exclude": ["node_modules", "test", "dist", "**/*spec.ts", "jest.*.ts"]
}