Compare commits

..
Author SHA1 Message Date
RafaelandClaude Opus 4.5 f61c241dde FIX: inject customer_id in pipeline execute/pause/unpause routes
Users no longer need to provide customer_id in the request body for
execute, pause, and unpause pipeline operations - it's now automatically
injected from the authenticated user's session.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-18 10:30:09 -03:00
RafaelandClaude Opus 4.5 9c55c22230 FIX: store reference_column as object with name and type
The protobuf definition expects reference_column to be an object with
name and type fields, but it was being stored as just a string (column
name). This caused pipeline fetching to fail with the error:
".NewTable.reference_column: object expected"

Changes:
- Update ReferenceColumn interface in DynamoDB service
- Update extractTablesFromJobs to create reference_column object
- Update syncJobInputToDynamoDB to handle reference_column object

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-17 13:47:57 -03:00
RafaelandClaude Opus 4.5 fb521f53cd FIX: pass type field to Elasticsearch createPipeline
The type field was missing from the createPipeline call,
causing ES documents to not have the type field set.

Maps: jdbc->database, singer->application, s3->file

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 21:09:07 -03:00
Rafael acb631e33d UPDATE: force deployment 2025-12-16 19:16:35 -03:00
RafaelandClaude Opus 4.5 e616061c21 fix: normalize IDs in pipeline run routes before calling Platform API
- Add normalization for pipelineId and runId in getPipelineRuns, getPipelineRun, and getPipelineRunLogs
- Remove unused createPipelineRun route

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 19:10:23 -03:00
Rafael e03b9e7a14 UPDATE: force deployment 2025-12-16 18:23:19 -03:00
RafaelandClaude Opus 4.5 2cc8f46418 feat: add Elasticsearch access to Maestro deployment
- Add ELASTICSEARCH_URL and ELASTICSEARCH_API_KEY env vars
- Pull credentials from {env}/microservices/elasticsearch secret

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 18:10:38 -03:00
Rafael 295f1f86ca Merge branch 'fix/redis-tls-configurable' of github.com:dadosfera/maestro into fix/redis-tls-configurable 2025-12-16 16:57:59 -03:00
Rafael 0a5e8001f9 UPDATE: adding platform_api_url 2025-12-16 16:57:25 -03:00
Rafael Santana b2700d4bb0 Merge branch 'beta' into fix/redis-tls-configurable 2025-12-16 16:39:11 -03:00
RafaelandClaude Opus 4.5 01c1087e07 FEAT: add schedule limit validation and improve ES update
- Add schedule limit validation against customer's scheduleLimit from DUC
- Improve ES update to only update provided fields
- Mark GET /platform/pipeline/:pipelineId/pipeline_run as READY

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 16:37:33 -03:00
vinicius gadea 7dedb3bd33 Merge pull request #409 from dadosfera/feature/catalog-update
FIX: Simplify searchDataAssets by owner logging and remove log code
2025-12-12 15:51:17 -03:00
viniciusgadea 1ff5589a2e FIX: Simplify searchDataAssets by owner logging and remove log code 2025-12-12 15:03:43 -03:00
Rafael Moraes d24e9a1d80 Merge pull request #408 from dadosfera/feature/catalog-update
FIX: Calendar Fix
2025-12-11 22:26:25 +00:00
Rafael Moraes 7d3ef1ef92 FIX: Conflit resolution 2025-12-11 22:25:42 +00:00
Rafael Moraes df3f2489f9 FIX: Calendar Fix 2025-12-11 22:10:32 +00:00
Rafael Moraes 53245b0067 Merge pull request #407 from dadosfera/feature/catalog-update
FIX: correção no validate k8s
2025-12-11 16:50:11 +00:00
Rafael Moraes dd699614ae FIX: Maestro helm fix 2025-12-11 16:47:28 +00:00
Rafael Moraes 6919a2a8d0 FIX: helmfile diff plugin 2025-12-11 14:14:02 +00:00
Rafael Moraes 5c29e07450 FIX: helmfile 2025-12-11 14:07:04 +00:00
Rafael Moraes 00304262ba FIX: wget change in helmfile 2025-12-11 13:52:58 +00:00
vinicius gadea bc931c6dd8 Merge pull request #406 from dadosfera/feature/catalog-update
catalogo: alteracao nas pastas e acrescimo de filtros
2025-12-11 09:18:57 -03:00
Rafael Moraes c444d6e956 Merge pull request #404 from dadosfera/feat/add-docs-assets
Feat/add docs assets
2025-12-10 19:42:42 +00:00
RafaelandClaude Opus 4.5 f7efb757bf FEAT: add schedule limit validation and improve ES update
- Add cron-validation utility with Airflow presets support (@once, @daily, etc.)
- Validate cron expressions against customer scheduleLimit from DUC
- Fetch scheduleLimit via gRPC to ensure up-to-date configuration
- Improve ES updatePipeline to fetch current doc and merge changes
- Only update fields explicitly provided in PATCH request body
- Preserve config.tables when updating config.cron

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 13:56:49 -03:00
Rafael Moraes 2f140d213a catalogo: alteracao nas pastas e acrescimo de filtros 2025-12-10 16:51:28 +00:00
RafaelandClaude Opus 4.5 df674dd441 FEAT: add Platform API proxy with Elasticsearch and DynamoDB sync
- Add Platform API module to proxy requests to Platform API service
- Add Elasticsearch service for pipeline catalog sync (CRUD operations)
- Add DynamoDB service for storing pipeline tables/inputs configuration
- Sync pipeline creation/update/delete to Elasticsearch index
- Extract tables from jobs with connector-specific mappings:
  - JDBC: table_name, load_type, column_include_list, incremental_column_name
  - Singer: replication_method -> full_load/incremental
  - S3: same as Singer
- Map connector types to DynamoDB types (jdbc->database, singer->application, s3->file)
- Validate connector type is provided in job input
- Normalize pipeline IDs for Platform API (replace - with _), keep UUIDs for ES

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 12:42:18 -03:00
viniciusgadea 09dced9fcd REFACTOR: add asset_type query parameter to getDataDocs and separate data docs methods 2025-12-10 11:21:46 -03:00
viniciusgadea 165b533172 FIX: removing unnecessary type parameter from GetDatasetDoc call 2025-12-10 11:17:43 -03:00
viniciusgadea b1dc567394 DOCS: update @dadosfera/protospack-v2 to version 3.38.0-beta.16 2025-12-10 11:12:13 -03:00
viniciusgadea a0a1303515 DOCS: add automatic asset_type query parameter to API schema and packages dependencies 2025-12-09 14:56:51 -03:00
viniciusgadea c21d977f7c FEAT: enhance catalog service to support any asset type in data docs creation 2025-12-09 14:56:51 -03:00
viniciusgadea 399d3492d3 DOCS: update @dadosfera/protospack-v2 to version 3.38.0-beta.15 2025-12-09 14:56:51 -03:00
Marcos Rodrigues Silva b0b557246e Merge pull request #403 from dadosfera/feat/user-properties
FIX:  change origin to host
2025-12-03 15:26:10 -03:00
Marcos Rodrigues bb671a90d6 FIX: change origin to host 2025-12-03 15:20:41 -03:00
Marcos Rodrigues Silva f6ababbe7a Merge pull request #402 from dadosfera/feat/user-properties
FIX: using header host
2025-12-03 14:46:14 -03:00
Marcos Rodrigues 82290285d0 FIX: using header host 2025-12-03 14:41:51 -03:00
Marcos Rodrigues Silva c490814a98 Merge pull request #401 from dadosfera/feat/user-properties
FEAT: add resource origin validate in me endpoint
2025-12-03 13:53:33 -03:00
Marcos Rodrigues 7616b1e32c FEAT: add resource origin validate in me endpoint 2025-12-03 13:51:32 -03:00
Rafael Santana 3ba2c91893 Merge pull request #400 from dadosfera/fix/redis-tls-configurable
UPDATE: force deployment
2025-12-02 15:30:03 -03:00
Rafael e789076ed4 UPDATE: force deployment 2025-12-02 15:29:36 -03:00
Rafael Santana 3b6ddaaee6 Merge pull request #399 from dadosfera/fix/redis-tls-configurable
fix: make Redis TLS configurable via REDIS_TLS env var
2025-12-02 15:18:05 -03:00
RafaelandClaude 8073194604 fix: make Redis TLS configurable via REDIS_TLS env var
- Change TLS check from ENV !== 'local' to REDIS_TLS === 'true'
- Add REDIS_TLS env var to Helm deployment template
- Default to true for OCI Redis, can override for self-hosted K8s

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 15:10:31 -03:00
Marcos Rodrigues Silva cd2c53b5c5 Merge pull request #398 from dadosfera/feat/user-properties
FIX: user response
2025-12-01 17:36:56 -03:00
Marcos Rodrigues adf1b3b97e FIX: user response 2025-12-01 17:30:37 -03:00
Marcos Rodrigues Silva d84b5e184b Merge pull request #397 from dadosfera/feat/user-properties
FEAT: include more user info in /me
2025-12-01 12:36:16 -03:00
Marcos Rodrigues 6d608a0457 FEAT: include more user info in /me 2025-12-01 12:26:33 -03:00
Marcos Rodrigues Silva b8be2c7803 Merge pull request #396 from dadosfera/fix/api-key-catalog-roles
FIX: api key catalog roles
2025-12-01 09:34:21 -03:00
Marcos Rodrigues 24fce721e3 FIX: api key catalog roles 2025-12-01 09:32:26 -03:00
Marcos Rodrigues Silva 23a9a27db1 Merge pull request #394 from dadosfera/fix/private-front-cors
FIX: 404 in  get public key
2025-11-28 16:44:54 -03:00
Marcos Rodrigues 0f5ed50af9 FIX: 404 in get public key 2025-11-28 16:37:38 -03:00
Marcos Rodrigues Silva 5f8f6a64ab Merge pull request #393 from dadosfera/fix/private-front-cors
FIX: include private frontend url in cors origins list
2025-11-28 10:53:45 -03:00
Marcos Rodrigues 8800ac2736 FIX: include private frontend url in cors origins list 2025-11-28 10:50:31 -03:00
Marcos Rodrigues Silva bdb82c2ce4 Merge pull request #392 from dadosfera/fix/authenticated-track-events
Fix/authenticated track events
2025-11-27 09:45:46 -03:00
Marcos Rodrigues 6d9ecc3568 FIX: mixpanel events 2025-11-27 09:43:43 -03:00
Marcos Rodrigues Silva 7764447adc Merge pull request #391 from dadosfera/fix/private-ingress-annotations
FIX: ci to oracle github worker
2025-11-25 11:37:29 -03:00
Marcos Rodrigues 576fdecf89 FIX: ci to oracle github worker 2025-11-25 11:27:04 -03:00
Marcos Rodrigues Silva 3de1e90fa8 Merge pull request #390 from dadosfera/fix/private-ingress-annotations
Fix/private ingress annotations
2025-11-25 11:12:58 -03:00
Marcos Rodrigues 5e90950660 FIX: restore whitelist annotation 2025-11-25 10:59:06 -03:00
Marcos Rodrigues 3ed26e648f FIX: ci 2025-11-25 10:48:06 -03:00
Marcos Rodrigues Silva fff3523152 Merge pull request #389 from dadosfera/fix/private-ingress-annotations
FIX: private ingress annotations
2025-11-25 10:38:52 -03:00
Marcos Rodrigues 19e4daeea4 FIX: private ingress annotations 2025-11-25 10:35:08 -03:00
Marcos Rodrigues Silva 8624d3f016 Merge pull request #388 from dadosfera/fix/reset-theme
Fix/reset theme
2025-11-21 11:01:48 -03:00
Marcos Rodrigues 4f4da5bebe FIX: reset theme 2025-11-21 10:51:31 -03:00
Marcos Rodrigues f00d2bf41d FIX: reset cstumer theme 2025-11-21 09:55:39 -03:00
Marcos Rodrigues Silva dc1d1400d8 Merge pull request #387 from dadosfera/feat/lineage
FIX: change wget to curl
2025-11-13 18:19:20 -03:00
Marcos Rodrigues d4451153a3 FIX: change wget to curl 2025-11-13 18:14:47 -03:00
Marcos Rodrigues Silva 06ba759ea3 Merge pull request #386 from dadosfera/feat/lineage
Feat/lineage
2025-11-13 17:58:33 -03:00
Marcos Rodrigues dd0d08ad6e FEAT: add permissions to lineage module 2025-11-13 17:48:55 -03:00
Marcos Rodrigues ec81082877 FIX: resolve cors from localhost 2025-11-13 17:48:38 -03:00
Marcos Rodrigues Silva 7f58dc7090 Merge pull request #385 from dadosfera/hotfix/import-files
FIX: permissions
2025-10-22 18:35:35 -03:00
Marcos Rodrigues 43aa379c06 FIX: permissions 2025-10-22 18:29:43 -03:00
Marcos Rodrigues Silva c284f8753c Merge pull request #384 from dadosfera/hotfix/import-files
FIX: pipelines v2 permissions
2025-10-22 17:48:26 -03:00
Marcos Rodrigues 3a8f2495c4 FIX: pipelines v2 permissions 2025-10-20 18:13:54 -03:00
Marcos Rodrigues Silva af248716ef Merge pull request #383 from dadosfera/beta
Beta
2025-10-16 18:15:08 -03:00
Marcos Rodrigues Silva a57ad41ad4 Merge pull request #382 from dadosfera/features/roles
Features/roles
2025-10-16 17:21:37 -03:00
Marcos Rodrigues ffddceec3b FEAT: roles 2025-10-16 17:17:01 -03:00
Marcos Rodrigues Silva b04d5bb402 Merge pull request #381 from dadosfera/feature/tableau-dash
UPDATE: default roles and user
2025-09-23 18:10:35 -03:00
Marcos Rodrigues 53df3caf3f UPDATE: default roles and user 2025-09-23 18:09:25 -03:00
Marcos Rodrigues Silva ab2e35b54f Merge pull request #380 from dadosfera/beta
Beta
2025-09-17 18:11:05 -03:00
Marcos Rodrigues Silva bfa77d8f7b Merge pull request #379 from dadosfera/feature/auth-cookie
Feature/auth cookie
2025-09-16 17:40:10 -03:00
marcos-silva-rodrigues 5002d147ad UPDATE: add all dadosfera domains 2025-09-16 17:33:19 -03:00
Marcos Rodrigues Silva cd55dc0dc4 Merge pull request #378 from dadosfera/hotfix/uptime-ip
UPDATE: uptime internal ip
2025-09-15 18:04:14 -03:00
marcos-silva-rodrigues f1d56e4c2c UPDATE: uptime internal ip 2025-09-15 10:34:36 -03:00
Marcos Rodrigues Silva bc49f79cb3 Merge pull request #373 from dadosfera/hotfix/oracle
Network policies
2025-09-09 18:08:28 -03:00
marcos-silva-rodrigues 00163ad894 UPDATE: network policies 2025-09-09 11:17:45 -03:00
Marcos Rodrigues Silva a2fbeb97cc Merge pull request #377 from dadosfera/feature/auth-cookie
UPDATE: test cors domain
2025-09-08 15:25:05 -03:00
marcos-silva-rodrigues 288a796f46 UPDATE: test cors domain 2025-09-08 15:22:26 -03:00
Marcos Rodrigues Silva b5a1e93770 Merge pull request #376 from dadosfera/feature/auth-cookie
UPDATE: cookie path
2025-09-08 10:32:32 -03:00
marcos-silva-rodrigues 1ee49ceab8 UPDATE: cookie path 2025-09-08 10:30:56 -03:00
Marcos Rodrigues Silva 6eaf9cf6d0 Merge pull request #375 from dadosfera/feature/auth-cookie
UDATE: test cookie domain dadosfera.ai
2025-09-08 09:32:21 -03:00
marcos-silva-rodrigues 075ca747df FIX: ghost commit 2025-09-08 09:26:47 -03:00
marcos-silva-rodrigues 9ead4588c1 UDATE: test cookie domain dadosfera.ai 2025-09-08 09:25:14 -03:00
Marcos Rodrigues Silva 92b64bb362 Merge pull request #374 from dadosfera/hotfix/oracle
Hotfix/oracle
2025-09-05 10:25:38 -03:00
marcos-silva-rodrigues cb8798d871 FIX: enable restricted ips 2025-09-05 10:21:11 -03:00
marcos-silva-rodrigues be70f4da09 FIX: disabled network policies 2025-09-04 10:02:20 -03:00
Marcos Rodrigues Silva b436d7de7a Merge pull request #372 from dadosfera/hotfix/oracle
Hotfix/oracle
2025-09-03 16:49:31 -03:00
Marcos Rodrigues Silva 6780167f2b Merge pull request #371 from dadosfera/hotfix/oracle
FIX: infinite loading when login is incorrect
2025-09-03 16:31:28 -03:00
marcos-silva-rodrigues 2a3ab4228f FIX: infinite loading when login is incorrect 2025-09-03 16:25:46 -03:00
Marcos Rodrigues Silva caeaf62a9d Merge pull request #370 from dadosfera/hotfix/oracle
Hotfix/oracle
2025-09-03 12:21:17 -03:00
marcos-silva-rodrigues 8e605aa361 FIX: affinity and tls config local 2025-09-03 12:19:30 -03:00
Marcos Rodrigues Silva fa4267f54a Merge pull request #369 from dadosfera/hotfix/oracle
UPDATE: oracle vpn
2025-08-29 10:16:45 -03:00
marcos-silva-rodrigues 47d8ca2760 UPDATE: oracle vpn 2025-08-29 10:12:58 -03:00
Marcos Rodrigues Silva f3c51e5328 Merge pull request #368 from dadosfera/beta
UPDATE: fixed cloud oracle
2025-08-28 18:54:02 -03:00
marcos-silva-rodrigues dcf70fdaae UPDATE: fixed cloud oracle 2025-08-28 18:40:43 -03:00
Marcos Rodrigues Silva 70f663374d Merge pull request #366 from dadosfera/beta
MIgração para oracle
2025-08-28 18:32:59 -03:00
Marcos Rodrigues Silva 38f2317bb6 Merge pull request #367 from dadosfera/chore/oracle
UPDATE: affinity and log
2025-08-28 17:05:32 -03:00
marcos-silva-rodrigues 31e0ca6c91 UPDATE: affinity and log 2025-08-28 16:52:52 -03:00
Marcos Rodrigues Silva dc063cbf3f Merge pull request #365 from dadosfera/chore/oracle
Chore/oracle
2025-08-27 17:45:34 -03:00
marcos-silva-rodrigues a4c82ae4a8 UPDATE:migration to oracle 2025-08-27 17:38:26 -03:00
Marcos Rodrigues Silva 11a65e11d1 Merge pull request #364 from dadosfera/chore/oracle
UPDATE: redis host
2025-08-27 14:15:27 -03:00
marcos-silva-rodrigues 73d47f0ff5 UPDATE: merge 2025-08-27 14:13:17 -03:00
marcos-silva-rodrigues d36e532c8e UPDATE: redis host 2025-08-27 13:58:31 -03:00
Marcos Rodrigues Silva b2a24e3a49 Merge pull request #363 from dadosfera/feature/auth-session
UPDATE: test subdomain
2025-08-14 17:51:36 -03:00
marcos-silva-rodrigues 301b6e98ec UPDATE: test subdomain 2025-08-14 17:42:42 -03:00
Marcos Rodrigues Silva 381a401ebb Merge pull request #362 from dadosfera/feature/auth-session
CI: add cookie secret
2025-08-14 17:27:09 -03:00
marcos-silva-rodrigues ba53934068 CI: add cookie secret 2025-08-14 17:23:40 -03:00
Marcos Rodrigues Silva 1b6846532f Merge pull request #361 from dadosfera/feature/auth-session
CHORE: update nginx ingress annotations
2025-08-14 16:07:31 -03:00
marcos-silva-rodrigues 216349f303 CHORE: update nginx ingress annotations 2025-08-14 16:03:28 -03:00
Marcos Rodrigues Silva 878c9cf450 Merge pull request #360 from dadosfera/feature/auth-session
FIX: set cookies if exists tokens
2025-08-14 15:39:57 -03:00
marcos-silva-rodrigues a654baef13 FIX: set cookies if exists tokens 2025-08-14 15:37:17 -03:00
Marcos Rodrigues Silva df852449b2 Merge pull request #359 from dadosfera/feature/auth-session
Feature/auth session
2025-08-14 15:28:23 -03:00
marcos-silva-rodrigues 04d761ca14 Merge branch 'beta' into feature/auth-session 2025-08-14 15:26:15 -03:00
Marcos Rodrigues Silva 9b168612a4 Merge pull request #358 from dadosfera/beta
Beta
2025-08-08 11:23:02 -03:00
Marcos Rodrigues Silva a772804dcd Merge pull request #357 from dadosfera/chore/oracle
CI: fix multi cloud deploy and maestro env
2025-08-07 18:04:10 -03:00
marcos-silva-rodrigues 8a399fbbcf CI: fix multi cloud deploy and maestro env 2025-08-07 18:03:22 -03:00
Marcos Rodrigues Silva 94049d6e8e Merge pull request #356 from dadosfera/chore/oracle
CI: oracle
2025-08-07 12:27:08 -03:00
marcos-silva-rodrigues c28cc58ac7 CI: oracle 2025-08-07 12:24:44 -03:00
Marcos Rodrigues Silva 56a655bb90 Merge pull request #355 from dadosfera/chore/oracle
CI: fix env home
2025-08-07 12:16:02 -03:00
marcos-silva-rodrigues b3d4a6b028 CI: fix env home 2025-08-07 12:13:19 -03:00
Marcos Rodrigues Silva c7bd1f761a Merge pull request #354 from dadosfera/chore/oracle
CI: remove k8s-setup need
2025-08-07 12:01:43 -03:00
marcos-silva-rodrigues 09c045455e CI: remove k8s-setup need 2025-08-07 11:59:37 -03:00
Marcos Rodrigues Silva 52d651126e Merge pull request #353 from dadosfera/chore/oracle
CI: revert k8s-setup workflow for the step in job helmfile-deploy
2025-08-07 11:57:38 -03:00
marcos-silva-rodrigues 2f11b1e78f CI: revert k8s-setup workflow for the step in job helmfile-deploy 2025-08-07 11:56:54 -03:00
Marcos Rodrigues Silva 885e9b71bb Merge pull request #352 from dadosfera/fix/tracker-sso
Fix/tracker sso
2025-08-07 11:45:44 -03:00
marcos-silva-rodrigues 193031dfdd UPDATE: increase redis ttl 2025-08-07 10:41:48 -03:00
marcos-silva-rodrigues f626a9bb5e UPDATE: add more logger 2025-08-07 10:32:22 -03:00
Marcos Rodrigues Silva ac26ad9f44 Merge pull request #351 from dadosfera/chore/oracle
FIX: correct name
2025-08-01 18:06:27 -03:00
marcos-silva-rodrigues fbe00af6dc FIX: correct name 2025-08-01 18:05:19 -03:00
Marcos Rodrigues Silva bfa9929b69 Merge pull request #350 from dadosfera/chore/oracle
Chore/oracle
2025-08-01 18:01:38 -03:00
marcos-silva-rodrigues bada7000f6 CHORE: change ci to oracle cloud 2025-08-01 18:01:10 -03:00
Marcos Rodrigues Silva 2c908d8d95 Merge pull request #349 from dadosfera/beta
Beta
2025-07-31 15:24:28 -03:00
Marcos Rodrigues Silva da35633d42 Merge pull request #348 from dadosfera/hotfix/module-assigned
Hotfix/module assigned
2025-07-31 12:04:02 -03:00
marcos-silva-rodrigues cc7e06013f FIX: change module permission 2025-07-31 11:59:19 -03:00
Marcos Rodrigues Silva 5ec36a05b5 Merge pull request #344 from dadosfera/beta
Beta
2025-07-30 11:56:22 -03:00
Marcos Rodrigues Silva f61e46c7c1 Merge pull request #347 from dadosfera/release/07-29
FEAT: add permission embed access
2025-07-30 11:15:28 -03:00
marcos-silva-rodrigues b9013b1493 FEAT: add permission embed access 2025-07-30 11:12:08 -03:00
Marcos Rodrigues Silva 5267fa6d2f Merge pull request #346 from dadosfera/release/07-29
FIX: comment test
2025-07-29 18:05:26 -03:00
marcos-silva-rodrigues 7f40cbdeed FIX: comment test 2025-07-29 18:03:20 -03:00
Marcos Rodrigues Silva b590c9402d Merge pull request #345 from dadosfera/release/07-29
Release/07 29
2025-07-29 15:19:52 -03:00
marcos-silva-rodrigues 9546feda11 FIX: merge 2025-07-29 15:12:25 -03:00
Marcos Rodrigues Silva 3587f82976 Merge pull request #343 from dadosfera/bugfix/public-link
Bugfix/public link
2025-07-29 12:31:34 -03:00
marcos-silva-rodrigues 94a3962590 FIX: add needs steps 2025-07-29 12:26:21 -03:00
marcos-silva-rodrigues 3b2d79a451 CHORE: remove needs step 2025-07-29 12:06:33 -03:00
marcos-silva-rodrigues da56a645af CHORE: update affinity to azure 2025-07-29 11:57:10 -03:00
Marcos Rodrigues Silva af4e55af1d Merge pull request #342 from dadosfera/bugfix/public-link
CHORE: remove sudo
2025-07-29 11:35:09 -03:00
marcos-silva-rodrigues d6a08a8f82 CHORE: remove sudo 2025-07-29 11:32:02 -03:00
Marcos Rodrigues Silva c73eb8869f Merge pull request #341 from dadosfera/bugfix/public-link
CHORE: install kubectl
2025-07-29 11:15:24 -03:00
marcos-silva-rodrigues 929fc97ac6 CHORE: install kubectl 2025-07-29 11:12:47 -03:00
Marcos Rodrigues Silva 7594a75bea Merge pull request #340 from dadosfera/bugfix/public-link
CHORE: update ci to wait extract_environment job
2025-07-29 11:02:14 -03:00
marcos-silva-rodrigues ec36a056b2 CHORE: update ci to wait extract_environment job 2025-07-29 10:59:35 -03:00
Marcos Rodrigues Silva 8d30f6ef15 Merge pull request #339 from dadosfera/bugfix/public-link
CHORE: multi cloud deployment
2025-07-29 10:57:26 -03:00
marcos-silva-rodrigues 08d1e3164f CHORE: multi cloud deployment 2025-07-29 10:53:41 -03:00
Marcos Rodrigues Silva a5f3ef20fe Merge pull request #338 from dadosfera/bugfix/public-link
FIX: add user by bearer token if it exists
2025-07-29 09:51:17 -03:00
marcos-silva-rodrigues 467445fe05 FIX: add user by bearer token if it exists 2025-07-29 09:47:56 -03:00
Marcos Rodrigues Silva d0448f844a Merge pull request #337 from dadosfera/bugfix/google-oauth
UPDATE: more logs
2025-07-21 16:06:22 -03:00
marcos-silva-rodrigues b2299a5f2f UPDATE: more logs 2025-07-21 15:51:56 -03:00
Marcos Rodrigues Silva 9b42754c69 Merge pull request #336 from dadosfera/bugfix/google-oauth
Bugfix/google oauth
2025-07-21 15:06:49 -03:00
marcos-silva-rodrigues aee33334ac UPDATE: add log to debug 2025-07-21 15:01:41 -03:00
Marcos Rodrigues Silva cc530fb12d Merge pull request #335 from dadosfera/release/sso
Release/sso
2025-07-18 17:12:51 -03:00
marcos-silva-rodrigues 19877f1b29 FIX: merge with main 2025-07-18 16:46:19 -03:00
marcos-silva-rodrigues e9880bcf68 FIX: add logger in header 2025-07-18 15:56:23 -03:00
marcos-silva-rodrigues 4c7111dcb6 FIX: using referer headaer 2025-07-18 15:56:10 -03:00
marcos-silva-rodrigues e81cfdcce1 FIX: using only header origin 2025-07-18 15:55:56 -03:00
marcos-silva-rodrigues 3eba3f414f FIX: send redirect uri 2025-07-18 15:55:45 -03:00
marcos-silva-rodrigues fbeb8dfd91 FIX: redis env 2025-07-18 15:55:29 -03:00
marcos-silva-rodrigues dda3e8baaf FIX: add logger 2025-07-18 15:54:27 -03:00
marcos-silva-rodrigues c1e4f0b18a FIX: remove express session 2025-07-18 15:54:24 -03:00
Marcos Rodrigues Silva b0d9f3a468 Merge pull request #334 from dadosfera/chore/stg-env
FIX: revert env
2025-07-18 10:27:01 -03:00
marcos-silva-rodrigues 5b117acaa9 FIX: revert env 2025-07-18 10:23:10 -03:00
Marcos Rodrigues Silva a5cb607650 Merge pull request #333 from dadosfera/feature/sso-openid
FIX: change env to stg
2025-07-17 16:12:51 -03:00
marcos-silva-rodrigues fe626155c5 FIX: change env to stg 2025-07-17 16:10:42 -03:00
Marcos Rodrigues Silva debbe1c1b1 Merge pull request #332 from dadosfera/feature/sso-openid
Feature/sso openid
2025-07-16 18:50:04 -03:00
marcos-silva-rodrigues f937a6efe2 FIX: add logger in header 2025-07-16 18:49:27 -03:00
marcos-silva-rodrigues 30cb3c87eb FIX: using referer headaer 2025-07-16 18:45:47 -03:00
Marcos Rodrigues Silva fa46294154 Merge pull request #331 from dadosfera/feature/sso-openid
FIX: using only header origin
2025-07-16 18:37:53 -03:00
marcos-silva-rodrigues 72554fb27f FIX: using only header origin 2025-07-16 18:36:36 -03:00
Marcos Rodrigues Silva 4e6985b572 Merge pull request #330 from dadosfera/feature/sso-openid
FIX: send redirect uri
2025-07-16 18:16:27 -03:00
marcos-silva-rodrigues 4477647613 FIX: send redirect uri 2025-07-16 18:15:47 -03:00
Marcos Rodrigues Silva 2b3066abc9 Merge pull request #329 from dadosfera/feature/sso-openid
FIX: redis env
2025-07-16 17:57:26 -03:00
marcos-silva-rodrigues da1bcb38da FIX: redis env 2025-07-16 17:56:43 -03:00
Marcos Rodrigues Silva 4a7524422d Merge pull request #328 from dadosfera/feature/sso-openid
FIX: add logger
2025-07-16 17:41:40 -03:00
marcos-silva-rodrigues 275af03195 FIX: add logger 2025-07-16 17:39:15 -03:00
Marcos Rodrigues Silva 0c75cf4619 Merge pull request #327 from dadosfera/feature/sso-openid
Feature/sso openid
2025-07-16 12:33:44 -03:00
marcos-silva-rodrigues 80de277529 FIX: merge problems 2025-07-16 12:27:33 -03:00
marcos-silva-rodrigues 98f669db66 FIX: remove express session 2025-07-16 11:48:15 -03:00
marcos-silva-rodrigues 2c8dab5927 FIX: merge 2025-07-16 11:46:05 -03:00
marcos-silva-rodrigues cb66d4f6c0 FEAT: dynamic callback and issuer urls 2025-07-16 11:35:47 -03:00
marcos-silva-rodrigues 42a39ee891 FEAT: send user id to refresh token endpoint 2025-07-15 16:21:44 -03:00
marcos-silva-rodrigues c9bfe68ea5 FEAT: list identity providers link for sso 2025-07-14 15:16:19 -03:00
marcos-silva-rodrigues 8cb7144ff4 FIX: send accessToken in refresh token endpoint 2025-07-14 15:14:25 -03:00
Marcos Rodrigues Silva 68cc58e2fe Merge pull request #326 from dadosfera/release/11-07
Release/11 07
2025-07-14 10:54:14 -03:00
marcos-silva-rodrigues cbe5556ac9 FIX: auth callback 2025-07-14 10:51:36 -03:00
marcos-silva-rodrigues eb80967608 FIX: tests 2025-07-11 11:36:23 -03:00
marcos-silva-rodrigues 66fea0722d FEAT: enabled network check in app.dadosfera 2025-07-11 11:12:35 -03:00
marcos-silva-rodrigues c917061b97 Merge branch 'feature/export-users-and-assets' into release/11-07 2025-07-11 11:04:31 -03:00
Rafael Santana da39a07f25 Merge pull request #325 from dadosfera/feat/oracle-migration-and-affinity-standardization
FEAT: migrate maestro to Oracle OKE deployment structure and standardize affinity/resources
2025-07-09 13:02:48 -03:00
Rafael f2f65285de resolve: merge conflicts with beta branch
- Resolve ingress.yaml timeout annotation conflicts by keeping both configurations
- Resolve stg.yaml conflicts by merging affinity/resources config with restricted_ip
- Fix chart references in unimed-maestro and private-maestro sections
- Update all chart paths from ../maestro to ../helm-chart for consistency

This merge brings in the latest beta changes while preserving our Oracle migration
and affinity standardization configuration.
2025-07-09 13:00:52 -03:00
Rafael cac8f93330 CI: update validate-k8s.yml to use beta branch for staging
- Change trigger branches from main/stg to main/beta
- Update environment extraction to map beta → stg environment
- Align with standard branch naming convention (beta for staging)
- Consistent with deploy-manually.yml workflow configuration
2025-07-09 12:58:11 -03:00
Rafael bb03b18f4a FEAT: migrate maestro to Oracle OKE deployment structure and standardize affinity/resources
Oracle Migration Changes:
- Create deploy/helm-chart/ directory structure (moved from maestro/)
- Create deploy/helmfiles/ directory structure (moved from helmfiles/)
- Update chart references in helmfiles from ../maestro to ../helm-chart
- Update deploy-manually.yml from Azure AKS to Oracle OKE
- Update validate-k8s.yml from Azure AKS to Oracle OKE
- Replace Azure CLI with OCI CLI installation and configuration
- Replace Azure authentication with OCI authentication using secrets
- Replace az aks get-credentials with oci ce cluster create-kubeconfig
- Update helmfile paths from helmfiles/ to deploy/helmfiles/
- Remove DockerHub integration (push_to_dockerhub input and related steps)
- Change runner from [self-hosted, prd-azure] to [self-hosted, prd-oracle]
- Add environment field for proper Oracle deployment environment handling
- Add HOME environment variable for OCI CLI

Affinity/Resources Standardization:
- Add affinity configuration with name=general node selector in values.yaml
- Update deployment template to use conditional affinity and resources blocks
- Configure staging to disable both affinity and resources (null values)
- Enable production to use affinity targeting name=general nodes
- Enable production to use standard resource limits (100m-2000m CPU, 1500Mi-2Gi memory)

This brings maestro into full compliance with Oracle migration requirements and
implements the standardized affinity/resources pattern consistent with in-factory and duc.
2025-07-09 12:55:49 -03:00
Marcos Rodrigues Silva ab280ee152 Merge pull request #324 from dadosfera/chore/update-unimed-ips
CHORE: Update unimed ips
2025-07-08 09:22:26 -03:00
marcos-silva-rodrigues 16c8137160 CHORE: Update unimed ips 2025-07-07 17:26:26 -03:00
Marcos Rodrigues Silva b8b903c8cd Merge pull request #323 from dadosfera/feature/export-users-and-assets
Feature/export users and assets
2025-07-05 11:27:10 -03:00
marcos-silva-rodrigues c98a5ff896 FEAT: endpoint to dowload assets list 2025-07-05 11:21:05 -03:00
marcos-silva-rodrigues d68e436a8b FEAT: export users in csv 2025-07-05 09:38:16 -03:00
marcos-silva-rodrigues 8efa6790d6 FEAT: sign out endpoint 2025-07-04 18:03:08 -03:00
marcos-silva-rodrigues a381f50e88 FEAT: set cookie after user login 2025-07-03 15:52:55 -03:00
marcos-silva-rodrigues d3040cd5a8 FEAT: update and delete identity provider 2025-07-03 10:59:43 -03:00
marcos-silva-rodrigues f726a9883f FEAT: create user from identity provider 2025-06-27 13:21:59 -03:00
Marcos Rodrigues Silva c18a7baa37 Merge pull request #322 from dadosfera/chore/add-dadosfera-vpn-unimed-chart
FIX: add dadosfera vpn ip in unimed chart
2025-06-25 11:12:33 -03:00
marcos-silva-rodrigues d638d518f8 FIX: change unimed chart name 2025-06-25 11:06:41 -03:00
marcos-silva-rodrigues bdd371ffe8 FIX: add dadosfera vpn ip in unimed chart 2025-06-25 10:58:18 -03:00
marcos-silva-rodrigues 57e6e49c7c FEAT: init url callback 2025-06-25 10:28:26 -03:00
marcos-silva-rodrigues a5c8de496d FEAT: map permission to identity provider 2025-06-24 16:20:03 -03:00
Marcos Rodrigues Silva d478f32760 Merge pull request #321 from dadosfera/hotfix/nginx-timeout
FIX: add nginx proxy timeout 5 min
2025-06-24 11:02:29 -03:00
marcos-silva-rodrigues 5d8e908339 FIX: add nginx proxy timeout 5 min 2025-06-24 10:26:50 -03:00
Marcos Rodrigues Silva 606bae016b Merge pull request #320 from dadosfera/hotfix/maesto-unimed-hostname
FIX: correct helm value
2025-06-23 18:43:58 -03:00
marcos-silva-rodrigues 6910a2f04c FIX: correct helm value 2025-06-23 18:38:20 -03:00
Marcos Rodrigues Silva 8bee1788c1 Merge pull request #319 from dadosfera/hotfix/maesto-unimed-hostname
FIX: maestro custom hostname to unimed
2025-06-23 18:04:43 -03:00
marcos-silva-rodrigues b10459cb15 FIX: maestro custom hostname to unimed 2025-06-23 17:51:14 -03:00
Marcos Rodrigues Silva 3b48dd1613 Merge pull request #317 from dadosfera/release/2025-06
Release/2025 06
2025-06-23 17:07:55 -03:00
Marcos Rodrigues Silva e5cef90a64 Merge pull request #318 from dadosfera/release/2025-06
Release/2025 06
2025-06-23 16:04:59 -03:00
marcos-silva-rodrigues bac74c9577 FIX: default value and throw correct Error Class 2025-06-23 15:36:26 -03:00
marcos-silva-rodrigues 19d748ae09 CHORE: add restricted ip and unimed chart 2025-06-23 14:47:39 -03:00
marcos-silva-rodrigues 4db865371f Merge branch 'bugfix/update-pupperter-timeout' into release/2025-06 2025-06-23 14:12:25 -03:00
marcos-silva-rodrigues 75e8b3bf4b FIX: correct dto name 2025-06-17 14:58:35 -03:00
Marcos Rodrigues Silva 194cdc66b7 Merge pull request #316 from dadosfera/feature/public-and-assign-embed
Feature/public and assign embed
2025-06-17 08:57:53 -03:00
marcos-silva-rodrigues 53059c0a0d FIX: import 2025-06-16 18:09:25 -03:00
marcos-silva-rodrigues ed4ffc8a86 FIX: merge with beta 2025-06-16 18:01:53 -03:00
Marcos Rodrigues Silva 9967c172da Merge pull request #315 from dadosfera/feature/dedicated-maestro
FEAT: block user from login when maestro is dedicated
2025-06-16 10:37:47 -03:00
marcos-silva-rodrigues 0aa9c065a5 FEAT: block user from login when maestro is dedicated 2025-06-16 10:34:11 -03:00
Marcos Rodrigues Silva d0c2abd38d Merge pull request #314 from dadosfera/bugfix/update-pupperter-timeout
FIX: update timeout
2025-06-13 15:10:08 -03:00
marcos-silva-rodrigues cb98099a91 FIX: update timeout 2025-06-13 15:07:50 -03:00
Marcos Rodrigues Silva f446e0d5b1 Merge pull request #313 from dadosfera/feature/dedicated-maestro
FIX: send env DEDICATED_PROXY
2025-06-12 14:47:54 -03:00
marcos-silva-rodrigues 17de31f1a6 FIX: send env DEDICATED_PROXY 2025-06-12 14:45:13 -03:00
Marcos Rodrigues Silva d81a466089 Merge pull request #312 from dadosfera/feature/dedicated-maestro
FIX: open data ingress route
2025-06-12 14:08:13 -03:00
marcos-silva-rodrigues 9e597fadba FIX: open data ingress route 2025-06-12 14:04:45 -03:00
Marcos Rodrigues Silva 1e6aa0f4c2 Merge pull request #311 from dadosfera/feature/dedicated-maestro
FIX: chart private maestro
2025-06-12 13:59:11 -03:00
marcos-silva-rodrigues 41842f7c6a FIX: chart private maestro 2025-06-12 13:56:57 -03:00
Marcos Rodrigues Silva 0d5564990d Merge pull request #309 from dadosfera/feature/dedicated-maestro
Feature/dedicated maestro
2025-06-12 12:31:26 -03:00
marcos-silva-rodrigues 7f9755493a FIX: experimental hostname 2025-06-12 12:29:51 -03:00
marcos-silva-rodrigues fea587ad36 FEAT: control login by dedicated customer id injected 2025-06-12 11:51:56 -03:00
Marcos Rodrigues Silva b2b7537fee Merge pull request #308 from dadosfera/fix/wordpress-block-list
FIX: comment block list
2025-06-10 15:25:27 -03:00
Marcos Rodrigues Silva 57f99fe7af Merge pull request #307 from dadosfera/fix/wordpress-block-list
FIX: comment block list
2025-06-10 14:25:23 -03:00
marcos-silva-rodrigues 01263017e7 FIX: comment block list 2025-06-10 14:24:26 -03:00
Marcos Rodrigues Silva e66f9a7244 Merge pull request #306 from dadosfera/feature/api-key
FEAT: add api key endpoint
2025-06-06 15:55:53 -03:00
marcos-silva-rodrigues 54cb3f8e7a FEAT: add api key endpoint 2025-06-06 15:53:12 -03:00
Marcos Rodrigues Silva 4c861cf5e4 Merge pull request #305 from dadosfera/fix/maestro-stg-url
Fix/maestro stg url
2025-05-30 10:15:10 -03:00
marcos-silva-rodrigues a89f57e690 FIX: return pi_factory_url and tr_factory_url to stg url 2025-05-30 10:13:37 -03:00
marcos-silva-rodrigues d5141e5b6a REFACTOR: rename openid to identity provider 2025-05-26 13:37:25 -03:00
Rafael Santana 8aded98a11 Merge pull request #304 from dadosfera/beta
Beta
2025-05-26 10:51:13 -03:00
Rafael Santana 0b5a0d0d7e Merge pull request #303 from dadosfera/adjusting-nestjs-payload
Adjusting nestjs payload
2025-05-26 10:40:43 -03:00
Rafael 5a36762544 UPDATE: route /catalog/register-dataset will have an limit of 10mb 2025-05-26 10:24:48 -03:00
marcos-silva-rodrigues a4b7573ce7 FIX: sync with main 2025-05-23 14:07:05 -03:00
marcos-silva-rodrigues 6fc8c0b4a8 FEAT: add mixpanel service to track share access 2025-05-20 16:50:04 -03:00
marcos-silva-rodrigues ced2a97547 FIX: add logger and return info 2025-05-19 16:36:54 -03:00
Marcos Rodrigues Silva f579171024 Merge pull request #302 from dadosfera/beta
Default value to header origin
2025-05-15 11:18:17 -03:00
Marcos Rodrigues Silva 885e5f4568 Merge pull request #301 from dadosfera/fix/remove-default-host
Fix/remove default host
2025-05-15 11:00:45 -03:00
marcos-silva-rodrigues 81d8cd8a07 FIX: default value to header origin 2025-05-15 11:00:07 -03:00
Marcos Rodrigues Silva f153f41357 Merge pull request #300 from dadosfera/beta
Beta
2025-05-14 18:30:54 -03:00
marcos-silva-rodrigues 2b9e3ff3bc Merge branch 'main' into beta 2025-05-13 14:32:34 -03:00
marcos-silva-rodrigues d15cc4642a FEAT: assign embed and public key changes 2025-05-13 09:21:44 -03:00
marcos-silva-rodrigues 3c4f45678a CHORE: update protospack 2025-05-12 12:25:45 -03:00
marcos-silva-rodrigues b312849fff FEAT: catalog share endpoint for embeds 2025-05-12 12:15:16 -03:00
marcos-silva-rodrigues 10580dfa03 FEAT: share endpoint 2025-05-12 12:13:39 -03:00
marcos-silva-rodrigues 62d0e8b041 FEAT: assign pubic key endpoint 2025-05-12 12:12:58 -03:00
Marcos Rodrigues Silva 17cdaac380 Merge pull request #299 from dadosfera/fix/stg-envs
FIX: stg to prd for sbm environment
2025-05-09 11:07:22 -03:00
marcos-silva-rodrigues e4eb388e18 FIX: stg to prd for sbm environment 2025-05-09 11:06:23 -03:00
marcos-silva-rodrigues d94c5dee4e FEAT: init public routes to get data asset by public embed 2025-05-06 14:44:09 -03:00
Marcos Rodrigues Silva 4bea0816b6 Merge pull request #298 from dadosfera/feature/pii-rules
FIX: install chromiun in base_image
2025-04-29 16:22:22 -03:00
marcos-silva-rodrigues fb491328d6 FIX: install chromiun in base_image 2025-04-29 16:20:37 -03:00
Marcos Rodrigues Silva 4e043d93ce Merge pull request #297 from dadosfera/feature/pii-rules
REFACTOR: move template to assets folder
2025-04-29 15:41:16 -03:00
marcos-silva-rodrigues 0b525b835f REFACTOR: move template to assets folder 2025-04-29 15:40:37 -03:00
Marcos Rodrigues Silva 3c8c39ec26 Merge pull request #296 from dadosfera/feature/pii-rules
REFACTOR: create builder for parser reporter and fix pdf
2025-04-29 15:20:46 -03:00
marcos-silva-rodrigues 36997403e1 REFACTOR: create builder for parser reporterand fix pdf 2025-04-29 15:18:46 -03:00
Marcos Rodrigues Silva 1175591521 Merge pull request #295 from dadosfera/feature/pii-rules
Feature/pii rules
2025-04-29 14:04:06 -03:00
marcos-silva-rodrigues 04accb6689 FIX: correct stg urls 2025-04-29 14:02:24 -03:00
marcos-silva-rodrigues 58e3c7be94 FEAT: pii 2025-04-29 14:00:23 -03:00
marcos-silva-rodrigues 7a6db36554 FEAT: generate pii reporter 2025-04-27 13:00:49 -03:00
Marcos Rodrigues Silva 80ebd98f6f Merge pull request #294 from dadosfera/feature/logs_dashboard
Feature/logs dashboard
2025-04-23 17:46:18 -03:00
Marcos Rodrigues Silva 9e0495fc26 Merge pull request #293 from dadosfera/feat/enforce-mfa
Feat/enforce mfa
2025-04-22 14:50:39 -03:00
marcos-silva-rodrigues f2437a8b3c FIX: add decorator to require danger zone module 2025-04-22 14:46:39 -03:00
marcos-silva-rodrigues 9e87562e23 FEAT: update protospack 2025-04-22 09:43:34 -03:00
marcos-silva-rodrigues 8f5cb13cc3 FEAT: enforce mfa 2025-04-21 18:53:34 -03:00
Marcos Rodrigues Silva e0a0ec9911 Merge pull request #292 from dadosfera/feature/logs_dashboard
Feature/logs dashboard
2025-04-17 18:12:28 -03:00
marcos-silva-rodrigues 419c063c6c FEAT: add access dashboard 2025-04-17 18:04:18 -03:00
marcos-silva-rodrigues 2aed66d371 FEAT: add decorator to block endpoint if not found module 2025-04-17 15:20:27 -03:00
Marcos Rodrigues Silva 1f45436c13 Merge pull request #291 from dadosfera/fix/network-policy
FIX: send origin header
2025-04-16 17:13:55 -03:00
marcos-silva-rodrigues df8520afd3 FIX: send origin header 2025-04-15 14:24:41 -03:00
Marcos Rodrigues Silva 95187e6b80 Merge pull request #290 from dadosfera/debug/unimed-sbm
UPDATE: change in-factory, pi-factory, tr-factory to prod
2025-04-14 14:15:45 -03:00
marcos-silva-rodrigues c8a3ace052 UPDATE: change in-factory, pi-factory, tr-factory to prod 2025-04-14 14:14:46 -03:00
Marcos Rodrigues Silva c059ac45bf Merge pull request #289 from dadosfera/fix/network-policy
FIX: send origin removing http prefix
2025-04-14 08:46:56 -03:00
marcos-silva-rodrigues f4b8addda4 FIX: send origin removing http prefix 2025-04-14 08:45:59 -03:00
Marcos Rodrigues Silva aa3a1cc300 Merge pull request #288 from dadosfera/fix/network-policy
FIX: remove host and send custom host
2025-04-14 08:32:19 -03:00
marcos-silva-rodrigues 0a5f138829 FIX: remove host and send custom host 2025-04-14 08:30:38 -03:00
Marcos Rodrigues Silva 9f427f5873 Merge pull request #287 from dadosfera/fix/network-policy
UPDATE: send host
2025-04-11 18:03:11 -03:00
marcos-silva-rodrigues bacbf42a04 UPDATE: send host 2025-04-11 18:00:51 -03:00
Marcos Rodrigues Silva 56a9d52466 Merge pull request #286 from dadosfera/fix/network-policy
UPDATE: send origin header
2025-04-11 17:37:56 -03:00
marcos-silva-rodrigues 2c964253ec UPDATE: send header in metadata 2025-04-11 17:33:34 -03:00
marcos-silva-rodrigues e886bea05c UPDATE: send origin header 2025-04-11 17:18:45 -03:00
rafael-moraes-ddf 66affed338 Merge pull request #285 from dadosfera/feature/logs_dashboard
Feature/logs dashboard
2025-04-11 16:22:05 -03:00
Rafael Moraes a8ae07e6c4 FEAT: url fix 2025-04-11 16:06:55 -03:00
Rafael Moraes 0df286a5f7 FEAT: att docsfera 2025-04-11 15:23:41 -03:00
Marcos Rodrigues Silva 519a8feb14 Merge pull request #284 from dadosfera/fix/network-policy
UPDATE: send host
2025-04-11 15:08:01 -03:00
marcos-silva-rodrigues 16cb00f46f UPDATE: send host 2025-04-11 15:06:02 -03:00
Marcos Rodrigues Silva 3c59602666 Merge pull request #283 from dadosfera/fix/network-policy
FIX: remove host metadata
2025-04-10 11:55:32 -03:00
marcos-silva-rodrigues 3e0dbc7401 FIX: remove host metadata 2025-04-10 11:53:53 -03:00
Marcos Rodrigues Silva a24d29b56b Merge pull request #282 from dadosfera/fix/network-policy
FIX: send host metadata
2025-04-10 11:47:09 -03:00
marcos-silva-rodrigues deb32b5c0f FIX: send host metadata 2025-04-10 11:44:52 -03:00
Marcos Rodrigues Silva d4a6d2fb4a Merge pull request #281 from dadosfera/fix/network-policy
Fix/network policy
2025-04-10 11:34:00 -03:00
Marcos Rodrigues Silva 01ea0e9561 Merge branch 'beta' into fix/network-policy 2025-04-10 11:33:43 -03:00
marcos-silva-rodrigues 9676b51f01 FIX: comment host metadata 2025-04-10 11:27:07 -03:00
Marcos Rodrigues Silva b0d2f86eaf Merge pull request #280 from dadosfera/feat/network-policy
FEAT: networks get endpoint
2025-04-09 15:34:18 -03:00
marcos-silva-rodrigues f356efb42e Merge branch 'beta' into feat/network-policy 2025-04-09 15:32:10 -03:00
marcos-silva-rodrigues 559d98e884 FIX: merge with beta 2025-04-09 15:28:23 -03:00
Marcos Rodrigues Silva d8623b88df Merge pull request #277 from dadosfera/feat/network-policy
Feat/network policy
2025-04-09 11:36:32 -03:00
Marcos Rodrigues Silva 7044d6606e Merge pull request #278 from dadosfera/beta
Beta
2025-04-08 18:04:41 -03:00
Marcos Rodrigues Silva 92aab57d37 Merge pull request #279 from dadosfera/fix/reset-block-users
FIX: send all metadata
2025-04-08 14:35:10 -03:00
marcos-silva-rodrigues f661055ae8 FIX: send all metadata 2025-04-08 13:49:10 -03:00
Rafael Moraes d310ec2b93 FEAT: logs_dasboard 2025-04-07 11:55:06 -03:00
marcos-silva-rodrigues 301ad369c0 FEAT: networks get endpoint 2025-04-04 18:25:48 -03:00
marcos-silva-rodrigues 4c94a51a9d REFACTOR: openid module 2025-04-03 07:46:16 -03:00
Marcos Rodrigues Silva a2c1d79247 Merge pull request #276 from dadosfera/release/march
FIX: correct env
2025-04-02 17:06:25 -03:00
marcos-silva-rodrigues e5d9ac2fbf FEAT: network policy endpoint 2025-04-02 16:48:10 -03:00
Marcos Rodrigues Silva 67a15b36b3 Merge pull request #275 from dadosfera/release/march
FIX: correct env
2025-04-02 14:07:52 -03:00
marcos-silva-rodrigues f396cdb2d3 FIX: correct env 2025-04-02 14:01:54 -03:00
marcos-silva-rodrigues 9cc538cdcd FEAT: add host in metadata 2025-04-01 17:57:05 -03:00
Guilherme Maioli 01da6becff Merge pull request #274 from dadosfera/UPDATE/helmfile
UPDATE: return variable to stg.
2025-03-31 14:08:00 -03:00
Guilherme Maioli 003c9b176c UPDATE: return variable to stg. 2025-03-31 14:04:34 -03:00
Marcos Rodrigues Silva 0a5315fb95 Merge pull request #273 from dadosfera/feature/block_and_reset_users
FEAT: block, unblock and reset users
2025-03-28 17:00:49 -03:00
marcos-silva-rodrigues 5a4d578d13 FEAT: block, unblock and reset users 2025-03-28 12:30:55 -03:00
marcos-silva-rodrigues f1345472b5 UPDATE: add routes to login with idp 2025-03-25 09:18:27 -03:00
Marcos Rodrigues Silva 60214f0e57 Merge pull request #272 from dadosfera/feature/block_and_reset_users
Feature/block and reset users
2025-03-24 16:27:35 -03:00
marcos-silva-rodrigues 3a2ce72910 UPDATE: protospack 2025-03-24 14:46:24 -03:00
marcos-silva-rodrigues ea9c0502b7 UPDATE: new route to add identity provider 2025-03-24 07:58:26 -03:00
marcos-silva-rodrigues 074d3ce201 BUILD: remove unused packages 2025-03-21 18:09:09 -03:00
marcos-silva-rodrigues 05c80e2497 FEAT: reset, block and unblock all users 2025-03-21 17:04:58 -03:00
marcos-silva-rodrigues 8a363dd815 FIX: Merge main 2025-03-18 17:59:04 -03:00
Marcos Rodrigues Silva 1e5fe59d9c Merge pull request #271 from dadosfera/feature/new-mixpanel-properties
Add new mixpanel properties
2025-03-14 15:20:42 -03:00
Jonathan Witkosky 1a2b9617fe Add new mixpanel properties 2025-03-14 13:57:31 -03:00
Guilherme Maioli 1f92510a43 Merge pull request #270 from dadosfera/bugfixes/13-03
Bugfixes/13 03
2025-03-13 17:59:32 -03:00
marcos-silva-rodrigues f7fceb1ea2 Merge branch 'fix/white-label' into bugfixes/13-03 2025-03-13 17:41:12 -03:00
Rafael Moraes 2aec60a354 FEATURE: Reset and block users 2025-03-13 14:44:54 -03:00
Marcos Rodrigues Silva 9bc42a4af0 Merge pull request #269 from dadosfera/fix/private-route
FIX: cors list
2025-02-26 17:15:03 -03:00
marcos-silva-rodrigues 5c98cc1748 FIX: cors list 2025-02-26 17:13:57 -03:00
Marcos Rodrigues Silva 9ae14ab04a Merge pull request #268 from dadosfera/fix/private-route
FIX: correct ip
2025-02-26 16:00:20 -03:00
marcos-silva-rodrigues 5d014f4669 FIX: correct ip 2025-02-26 15:57:07 -03:00
Marcos Rodrigues Silva 5e8dea7255 Merge pull request #267 from dadosfera/fix/private-route
FIX: add ip range and domain list to route /open-data
2025-02-26 08:49:40 -03:00
marcos-silva-rodrigues 38fc9f521b FIX: add ip range and domain list to route /open-data 2025-02-25 17:10:39 -03:00
Marcos Rodrigues Silva 80d7ac6ff9 Merge pull request #264 from dadosfera/fix/white-label
Fix/white label
2025-02-24 09:35:37 -03:00
marcos-silva-rodrigues 9bacd7a0f5 UPDATE: merge with beta branch 2025-02-24 09:28:03 -03:00
Aldemir Humberto Soares Neto c89ac40039 Merge pull request #266 from dadosfera/fixing-build-ecr
FEAT: fixing package-lock
2025-02-22 07:42:46 -03:00
aldemirneto 6d6cf590ee FEAT: fixing package-lock 2025-02-22 07:39:41 -03:00
Aldemir Humberto Soares Neto f063a8f30e Merge pull request #265 from dadosfera/UNIVR-PII-FACILITIES
Univr pii facilities
2025-02-22 07:29:14 -03:00
aldemirneto 03536a2180 FEAT: ensuring compatibility with ^ in the version 2025-02-22 07:28:30 -03:00
aldemirneto 0efd5cc257 FEAT: Changing protospack version
PII enabling
2025-02-22 07:23:34 -03:00
marcos-silva-rodrigues 9439f13b7b UPdATE: new displayName prop 2025-02-21 17:31:49 -03:00
Rafael Santana d936806406 Merge pull request #262 from dadosfera/beta
[PRD] Enable Dataset Registration via API for External Sources
2025-02-20 14:03:56 -03:00
Rafael Santana 9d7ccd4201 Merge pull request #263 from dadosfera/feat/catalog-external-datasets-using-api-call
UPDATE: migrating maestro to backend nodepool
2025-02-20 13:47:44 -03:00
Rafael c8a4c1aae6 UPDATE: migrating maestro to backend nodepool 2025-02-20 13:46:30 -03:00
Rafael Santana 406db7ccff Merge pull request #261 from dadosfera/feat/catalog-external-datasets-using-api-call
UPDATE: improve logging for table metadata
2025-02-20 08:52:15 -03:00
Rafael f48a35c7ce UPDATE: improve logging for table metadata 2025-02-20 08:40:48 -03:00
Rafael Santana 8d25df74b7 Merge pull request #260 from dadosfera/feat/catalog-external-datasets-using-api-call
UPDATE: updating protospack for add table_schema for /catalog/registe…
2025-02-19 12:25:10 -03:00
Rafael 831a03402e UPDATE: updating protospack for add table_schema for /catalog/register-dataset rout 2025-02-19 12:19:59 -03:00
Marcos Rodrigues Silva a631338599 Merge pull request #259 from dadosfera/feat/update-customer-dash
UPDATE: customer dash
2025-02-17 19:02:52 -03:00
Marcos Rodrigues Silva d943721b4a Merge pull request #258 from dadosfera/feat/update-customer-dash
Feat/update customer dash
2025-02-17 18:47:11 -03:00
Rafael Santana 1bbf978cc4 Merge pull request #257 from dadosfera/feat/catalog-external-datasets-using-api-call
FEAT: Enable dataset registration via API for external sources
2025-02-14 16:57:58 -03:00
Rafael 856f913ce6 Merge branch 'feat/catalog-external-datasets-using-api-call' of github.com:dadosfera/maestro into feat/catalog-external-datasets-using-api-call 2025-02-14 15:07:13 -03:00
Rafael dc617b6925 REFACTOR: Improve logging with additional metadata
- Enhanced logs to include more detailed request body information.
- Standardized log calls to include extra metadata, such as service name, environment, customer name, user ID, HTTP method, and request path.
- Improved error logging to provide better debugging insights.

Example log format:
EXTRA[{"service":{"name":"maestro","environment":"stg"},"customer_name":"dadosfera","user_id":"047824d8-0021-709a-bf15-39b8dd069aaf","method":"POST","path":"/catalog/register-dataset"}]

Related tickets or issues: [TECN-9441]
2025-02-14 15:04:22 -03:00
Rafael Santana 3f80062672 Merge branch 'beta' into feat/catalog-external-datasets-using-api-call 2025-02-14 10:14:01 -03:00
Rafael 07eb423814 UPDATE: update version of protospack 2025-02-14 10:12:20 -03:00
Rafael 721232448e FEAT: Enable dataset registration via API for external sources
As part of our hybrid approach, we need to allow dataset cataloging from outside our cloud environment.
Instead of relying on our GRPC APIs, we leveraged an existing REST API to handle this process.

- Updated Maestro to call Nimbus directly for dataset registration.
- Modified protospack to generate the necessary TypeScript interfaces.
- Added the `/catalog/register-dataset` endpoint to the API.

This change enables seamless dataset registration from external sources while maintaining compatibility with our existing infrastructure.

Related tickets or issues: [TECN-9441]
2025-02-14 10:08:45 -03:00
Marcos Rodrigues Silva 0b0d261aae Merge pull request #256 from dadosfera/fix/white-label
UPDATE: remove user agent
2025-02-13 13:56:37 -03:00
marcos-silva-rodrigues 87a8d741d7 UPDATE: remove user agent 2025-02-13 13:55:37 -03:00
marcos-silva-rodrigues d03dd6129a UPDATE: customer dash 2025-02-13 12:20:04 -03:00
Marcos Rodrigues Silva 96c20995a3 Merge pull request #255 from dadosfera/beta
Beta
2025-02-03 18:22:05 -03:00
Marcos Rodrigues Silva 82470553db Merge pull request #254 from dadosfera/fix/white-label
FIX: remove support for svg
2025-02-03 12:29:19 -03:00
marcos-silva-rodrigues 2408505889 FIX: remove support for svg 2025-02-03 12:25:56 -03:00
Marcos Rodrigues Silva 7d85693a1f Merge pull request #253 from dadosfera/fix/white-label
Fix/white label
2025-02-03 09:53:38 -03:00
marcos-silva-rodrigues d86a011bfd Reapply "Merge pull request #250 from dadosfera/beta"
This reverts commit d3f7664116.
2025-02-01 16:49:18 -03:00
Marcos Rodrigues Silva 4d54ac744b Merge pull request #252 from dadosfera/feature/new-monitoring-dash
FEAT: update monitoring dash
2025-01-31 18:32:41 -03:00
marcos-silva-rodrigues 132614b551 FEAT: update monitoring dash 2025-01-31 18:24:08 -03:00
Rafael Santana 17c541548a Merge pull request #251 from dadosfera/rollback-main
Revert "Merge pull request #250 from dadosfera/beta"
2025-01-31 18:20:20 -03:00
Rafael d3f7664116 Revert "Merge pull request #250 from dadosfera/beta"
This reverts commit 2ea40004a6, reversing
changes made to 873f7ea314.
2025-01-31 18:13:33 -03:00
Marcos Rodrigues Silva 2ea40004a6 Merge pull request #250 from dadosfera/beta
Beta
2025-01-31 17:24:43 -03:00
Marcos Rodrigues Silva 82ec2d9493 Merge pull request #248 from dadosfera/feature/new-monitoring-dash
FEAT: change dashboard id
2025-01-31 14:40:02 -03:00
Marcos Rodrigues Silva 53bef174b3 Merge pull request #249 from dadosfera/feature/white-label-app
FIX: return null when not found theme
2025-01-31 13:40:07 -03:00
marcos-silva-rodrigues bcfef8359e FIX: return null when not found theme 2025-01-31 13:39:15 -03:00
Marcos Rodrigues Silva 5f18411aed Merge pull request #247 from dadosfera/feature/white-label-app
Feature/white label app
2025-01-30 17:57:09 -03:00
marcos-silva-rodrigues 1102a6e8e5 FEAT: remove personal data in logger and only endpoint for wordpress 2025-01-30 17:06:09 -03:00
marcos-silva-rodrigues 867aec092b FEAT: add svg 2025-01-30 16:06:13 -03:00
marcos-silva-rodrigues ea9e727fcf FEAT: change dashboard id 2025-01-30 15:09:27 -03:00
Marcos Rodrigues Silva 099476f618 Merge pull request #246 from dadosfera/feature/white-label-app
Feature/white label app
2025-01-27 15:07:22 -03:00
marcos-silva-rodrigues db78494b1a FIX: test log headers 2025-01-27 14:39:12 -03:00
marcos-silva-rodrigues e7ad6e5a7b CHORE: update env 2025-01-27 14:12:38 -03:00
Marcos Rodrigues Silva 3d7d051df2 Merge pull request #245 from dadosfera/feature/white-label-app
chore: update stg values
2025-01-27 11:50:43 -03:00
marcos-silva-rodrigues 8e827dc934 chore: update stg values 2025-01-27 11:47:23 -03:00
Marcos Rodrigues Silva 12b764710a Merge pull request #244 from dadosfera/feature/white-label-app
Feature/white label app
2025-01-27 09:49:29 -03:00
marcos-silva-rodrigues b5c95874bb chore: updated package 2025-01-27 09:14:42 -03:00
marcos-silva-rodrigues b896b1a9b8 REFACTOR: rename props 2025-01-26 16:44:31 -03:00
marcos-silva-rodrigues cf99acc682 FEAT: send images in stream 2025-01-26 16:37:37 -03:00
Marcos Rodrigues Silva 7fb20964dc Merge pull request #243 from dadosfera/feature/white-label-app
Feature/white label app
2025-01-23 08:39:26 -03:00
Rafael Santana 8751117946 Merge pull request #242 from dadosfera/decrease-replica-count-maestro
UPDATE: add toleration for spot instances
2025-01-22 11:24:35 -03:00
Rafael 10c4ed179b UPDATE: add toleration for spot instances 2025-01-22 11:24:01 -03:00
marcos-silva-rodrigues 8e13541b24 FEAT: update protospack 2025-01-21 17:28:13 -03:00
marcos-silva-rodrigues 4f6a9d4b66 FEAT: new property in customer obj 2025-01-21 17:00:34 -03:00
Rafael Santana 56411496c2 Merge pull request #241 from dadosfera/decrease-replica-count-maestro
Decrease replica count maestro
2025-01-20 18:50:14 -03:00
Rafael e55a1cb657 CI: removed deploy-k8s workflow 2025-01-20 18:47:44 -03:00
Rafael 3850d4b8ae UPDATE: blank commit to force deployment 2025-01-20 18:47:13 -03:00
Rafael Santana 1626d85d31 Merge pull request #240 from dadosfera/decrease-replica-count-maestro
UPDATE: decrease the number of replicas for stg environment
2025-01-20 18:41:54 -03:00
Rafael 2a6677e4d9 UPDATE: decrease the number of replicas for stg environment 2025-01-20 18:41:01 -03:00
Marcos Rodrigues Silva 74e62c001d Merge pull request #239 from dadosfera/feature/white-label-app
FIX: env fixed and add logger for the process.env
2025-01-17 08:40:08 -03:00
marcos-silva-rodrigues 8237160c6d FIX: env fixed and add logger for the process.env 2025-01-17 08:38:03 -03:00
Marcos Rodrigues Silva 41bf9d4856 Merge pull request #238 from dadosfera/feature/white-label-app
FIX: change env for stg
2025-01-16 18:47:21 -03:00
marcos-silva-rodrigues 44dcf1b281 FIX: change env for stg 2025-01-16 18:43:15 -03:00
Marcos Rodrigues Silva 104787bce8 Merge pull request #237 from dadosfera/feature/white-label-app
Feature/white label app
2025-01-15 11:04:20 -03:00
marcos-silva-rodrigues 38a330e578 FIX: change stg env for prd env 2025-01-15 10:55:08 -03:00
marcos-silva-rodrigues 7d0b8755c9 FEAT: image upload in theme 2025-01-13 17:35:56 -03:00
Marcos Rodrigues Silva 7aadf4d9d1 Merge pull request #236 from dadosfera/feature/white-label-app
FIX: add header content type and change status code to 200
2025-01-09 08:40:59 -03:00
marcos-silva-rodrigues af8fa72377 FIX: add header content type and change status code to 200 2025-01-09 08:40:02 -03:00
Marcos Rodrigues Silva c51f262474 Merge pull request #235 from dadosfera/feature/white-label-app
FIX:  param to language and message in json
2025-01-09 08:15:46 -03:00
marcos-silva-rodrigues 54d19a78ab FIX: param to language and message in json 2025-01-09 08:13:06 -03:00
Marcos Rodrigues Silva 66f33a2246 Merge pull request #234 from dadosfera/feature/white-label-app
FIX: remove language param
2025-01-08 18:15:41 -03:00
marcos-silva-rodrigues 55d9441d10 FIX: remove language param 2025-01-08 18:12:04 -03:00
126 changed files with 14626 additions and 2821 deletions
-73
View File
@@ -1,73 +0,0 @@
name: Deploy K8S Modifications
on:
push:
branches:
- main
- beta
jobs:
extract_environment:
runs-on: ubuntu-22.04
outputs:
environment: ${{ steps.extract_environment.outputs.environment }}
steps:
- name: Extract Environment
run: |
if [ ${GITHUB_REF} == "refs/heads/main" ]; then
echo "environment=prd" >> $GITHUB_OUTPUT
elif [ ${GITHUB_REF} == "refs/heads/beta" ]; then
echo "environment=stg" >> $GITHUB_OUTPUT
fi
id: extract_environment
helmfile-deploy:
needs: [extract_environment]
runs-on: [self-hosted, "prd-azure"]
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: 'v3.9.0'
- name: Install Azure ClI
run: |
curl -sL https://aka.ms/InstallAzureCLIDeb | bash
- uses: azure/login@v2
with:
creds: '{"clientId":"${{ secrets.ARM_CLIENT_ID }}","clientSecret":"${{ secrets.ARM_CLIENT_SECRET }}","subscriptionId":"${{ secrets.ARM_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.ARM_TENANT_ID }}"}'
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install Helmfile
run: |
wget https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
tar -xzf helmfile_0.148.0_linux_amd64.tar.gz
mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff Plugin
run: helm plugin install https://github.com/databus23/helm-diff || true
- name: Setup kubectl
uses: azure/setup-kubectl@v1
with:
version: 'v1.30.1'
- name: Authenticate with cluster
env:
CLUSTER_NAME: platform-${{ needs.extract_environment.outputs.environment }}
run: az aks get-credentials --resource-group dadosfera-prd --name ${CLUSTER_NAME} --overwrite-existing
- name: Run Helmfile Apply
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
run: helmfile -f helmfiles/${ENV}.yaml sync
+7 -72
View File
@@ -13,11 +13,6 @@ on:
options:
- stg
- prd
push_to_dockerhub:
description: "Push image to Dockerhub?"
required: true
type: boolean
default: false
jobs:
extract_environment:
@@ -118,23 +113,6 @@ jobs:
docker compose -f build.docker-compose.yml build
docker compose -f build.docker-compose.yml push
- name: Login to Docker Hub
if: ${{inputs.push_to_dockerhub}}
uses: docker/login-action@v2
with:
username: dadosfera
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Build, Tag, and Push Image to Dockerhub
if: ${{inputs.push_to_dockerhub}}
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
IMAGE_TAG: ${{ needs.semantic_release.outputs.new_release_version }}
ACCOUNT_ID: ${{ steps.aws.outputs.aws-account-id }}
run: |
docker compose -f build.docker-compose.dockerhub.yml build
docker compose -f build.docker-compose.dockerhub.yml push
# - name: Create ZIP file to Deploy AWS Beanstalk
# env:
# ENV: ${{ needs.extract_environment.outputs.environment }}
@@ -164,54 +142,11 @@ jobs:
docker system prune --volumes -a -f
docker system df
helmfile-deploy:
k8s-deploy:
needs: [extract_environment, semantic_release, build_ecr_image]
runs-on: [self-hosted, "prd-azure"]
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: 'v3.9.0'
- name: Install Azure ClI
run: |
curl -sL https://aka.ms/InstallAzureCLIDeb | bash
- uses: azure/login@v2
with:
creds: '{"clientId":"${{ secrets.ARM_CLIENT_ID }}","clientSecret":"${{ secrets.ARM_CLIENT_SECRET }}","subscriptionId":"${{ secrets.ARM_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.ARM_TENANT_ID }}"}'
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install Helmfile
run: |
wget https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
tar -xzf helmfile_0.148.0_linux_amd64.tar.gz
mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff Plugin
run: helm plugin install https://github.com/databus23/helm-diff || true
- name: Setup kubectl
uses: azure/setup-kubectl@v1
with:
version: 'v1.30.1'
- name: Authenticate with cluster
env:
CLUSTER_NAME: platform-${{ needs.extract_environment.outputs.environment }}
run: az aks get-credentials --resource-group dadosfera-prd --name ${CLUSTER_NAME} --overwrite-existing
- name: Run Helmfile Apply
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
IMAGE_TAG: ${{ needs.semantic_release.outputs.new_release_version }}
run: helmfile -f helmfiles/${ENV}.yaml sync --set image.tag=$IMAGE_TAG
uses: ./.github/workflows/k8s-deploy.yml
with:
cloud: 'oracle'
environment: ${{ needs.extract_environment.outputs.environment }}
image: ${{ needs.semantic_release.outputs.new_release_version }}
secrets: inherit
+137
View File
@@ -0,0 +1,137 @@
name : K8s deploy
on:
workflow_call:
inputs:
cloud:
description: "Cloud provider for the deployment"
required: true
default: "azure"
type: string
environment:
description: "Deployment environment"
required: true
default: "prd"
type: string
image:
description: "Image Tag"
required: true
type: string
jobs:
azure:
if: inputs.cloud == 'azure'
runs-on: [self-hosted, "prd-azure"]
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: 'v3.9.0'
- name: Install Azure ClI
run: |
curl -sL https://aka.ms/InstallAzureCLIDeb | bash
- uses: azure/login@v2
with:
creds: '{"clientId":"${{ secrets.ARM_CLIENT_ID }}","clientSecret":"${{ secrets.ARM_CLIENT_SECRET }}","subscriptionId":"${{ secrets.ARM_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.ARM_TENANT_ID }}"}'
- name: Authenticate with cluster
env:
CLUSTER_NAME: platform-${{ inputs.environment }}
run: az aks get-credentials --resource-group dadosfera-prd --name ${CLUSTER_NAME} --overwrite-existing
- name: Setup kubectl
uses: azure/setup-kubectl@v1
with:
version: 'v1.30.1'
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install Helmfile
run: |
curl -fsSLO https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
tar -xzf helmfile_0.148.0_linux_amd64.tar.gz
sudo mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff Plugin
run: helm plugin install https://github.com/databus23/helm-diff || true
- name: Run Helmfile Apply
env:
ENV: ${{ inputs.environment }}
IMAGE_TAG: ${{ inputs.image }}
run: helmfile -f deploy/helmfiles/${ENV}.yaml sync --set image.tag=$IMAGE_TAG
oracle:
if: inputs.cloud == 'oracle'
runs-on: [self-hosted, "prd-oracle"]
env:
HOME: /home/runner
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: 'v3.9.0'
- name: Install OCI CLI
env:
HOME: /home/runner
run: |
bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" -- --accept-all-defaults
echo "$HOME/bin" >> $GITHUB_PATH
- name: Configure OCI CLI
run: |
mkdir -p ~/.oci || true
echo "${{ secrets.OCI_CONFIG }}" > ~/.oci/config
echo "${{ secrets.OCI_PRIVATE_KEY }}" > ~/.oci/oci_api_key.pem
chmod 600 ~/.oci/oci_api_key.pem
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install Helmfile
run: |
curl -fsSLO https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
tar -xzf helmfile_0.148.0_linux_amd64.tar.gz
sudo mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff Plugin
run: helm plugin install https://github.com/databus23/helm-diff || true
- name: Authenticate with OKE cluster
env:
ENV: ${{ inputs.environment }}
STG_CLUSTER_ID: "ocid1.cluster.oc1.sa-saopaulo-1.aaaaaaaagh3jvln52a3ebm3dodx6emmhv5bmfs7i7sv2k4zkbcbrzcl6v37q"
PRD_CLUSTER_ID: "ocid1.cluster.oc1.sa-saopaulo-1.aaaaaaaanf3vptl6hc2tzd4enfd2hfpsht3wikxww5xejc3l7cwfm6l3sndq"
run: |
if [ "$ENV" = "stg" ]; then
CLUSTER_ID=$STG_CLUSTER_ID
elif [ "$ENV" = "prd" ]; then
CLUSTER_ID=$PRD_CLUSTER_ID
else
echo "Unknown environment: $ENV"
exit 1
fi
oci ce cluster create-kubeconfig --cluster-id ${CLUSTER_ID} --file $HOME/.kube/config --region sa-saopaulo-1 --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT
- name: Run Helmfile Apply
env:
ENV: ${{ inputs.environment }}
IMAGE_TAG: ${{ inputs.image }}
run: helmfile -f deploy/helmfiles/${ENV}.yaml sync --set image.tag=$IMAGE_TAG
+51 -25
View File
@@ -4,7 +4,7 @@ on:
pull_request:
branches:
- main
- stg
- beta
jobs:
extract_environment:
@@ -21,17 +21,13 @@ jobs:
fi
id: extract_environment
helmfile-deploy:
helmfile-check:
env:
HOME: /home/runner
needs: [extract_environment]
runs-on: [self-hosted, "prd-azure"]
environment: ${{ needs.extract_environment.outputs.environment }}
runs-on: [self-hosted, "prd-oracle"]
steps:
- name: Summary
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
run: |
echo "### :rocket: Deploy da branch \`$GITHUB_REF_NAME\` para o environment ($ENV)" >> $GITHUB_STEP_SUMMARY
- name: Checkout code
uses: actions/checkout@v3
@@ -40,13 +36,28 @@ jobs:
with:
version: 'v3.9.0'
- name: Install Azure ClI
- name: Determine DNS_HOST based on environment
id: set_dns
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
run: |
curl -sL https://aka.ms/InstallAzureCLIDeb | bash
if [ "$ENV" = "prd" ]; then
echo "dns_host=dadosfera.ai" >> $GITHUB_OUTPUT
elif [ "$ENV" = "stg" ]; then
echo "dns_host=stg.dadosfera.ai" >> $GITHUB_OUTPUT
fi
- name: Install OCI CLI
run: |
bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" -- --accept-all-defaults
echo "$HOME/bin" >> $GITHUB_PATH
- uses: azure/login@v2
with:
creds: '{"clientId":"${{ secrets.ARM_CLIENT_ID }}","clientSecret":"${{ secrets.ARM_CLIENT_SECRET }}","subscriptionId":"${{ secrets.ARM_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.ARM_TENANT_ID }}"}'
- name: Configure OCI CLI
run: |
mkdir -p ~/.oci || true
echo "${{ secrets.OCI_CONFIG }}" > ~/.oci/config
echo "${{ secrets.OCI_PRIVATE_KEY }}" > ~/.oci/oci_api_key.pem
chmod 600 ~/.oci/oci_api_key.pem
- name: Set up Python
uses: actions/setup-python@v4
@@ -55,25 +66,40 @@ jobs:
- name: Install Helmfile
run: |
wget https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
curl -fsSLO https://github.com/helmfile/helmfile/releases/download/v0.148.0/helmfile_0.148.0_linux_amd64.tar.gz
tar -xzf helmfile_0.148.0_linux_amd64.tar.gz
mv helmfile /usr/local/bin/
sudo mv helmfile /usr/local/bin/
helmfile --version
- name: Install Helm Diff Plugin
run: helm plugin install https://github.com/databus23/helm-diff || true
- name: Debug Helm env
run: |
helm env
echo "HOME=$HOME"
ls -R $HOME/.local/share/helm || true
- name: Authenticate with OKE cluster
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
STG_CLUSTER_ID: "ocid1.cluster.oc1.sa-saopaulo-1.aaaaaaaagh3jvln52a3ebm3dodx6emmhv5bmfs7i7sv2k4zkbcbrzcl6v37q"
PRD_CLUSTER_ID: "ocid1.cluster.oc1.sa-saopaulo-1.aaaaaaaanf3vptl6hc2tzd4enfd2hfpsht3wikxww5xejc3l7cwfm6l3sndq"
run: |
if [ "$ENV" = "stg" ]; then
CLUSTER_ID=$STG_CLUSTER_ID
elif [ "$ENV" = "prd" ]; then
CLUSTER_ID=$PRD_CLUSTER_ID
else
echo "Unknown environment: $ENV"
exit 1
fi
oci ce cluster create-kubeconfig --cluster-id ${CLUSTER_ID} --file $HOME/.kube/config --region sa-saopaulo-1 --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT
- name: Setup kubectl
uses: azure/setup-kubectl@v1
with:
version: 'v1.30.1'
- name: Authenticate with cluster
env:
CLUSTER_NAME: platform-${{ needs.extract_environment.outputs.environment }}
run: az aks get-credentials --resource-group dadosfera-prd --name ${CLUSTER_NAME} --overwrite-existing
- name: Run Helmfile Diff
env:
ENV: ${{ needs.extract_environment.outputs.environment }}
run: helmfile -f helmfiles/${ENV}.yaml diff
run: helmfile -f deploy/helmfiles/${ENV}.yaml diff
+24 -1
View File
@@ -4,9 +4,19 @@ FROM base_image AS build_base
WORKDIR /app
RUN apk update
# needed packages to build dependencies from source
RUN apk add --no-cache aws-cli
RUN apk add --no-cache \
aws-cli \
chromium \
nss \
freetype \
harfbuzz \
ca-certificates \
ttf-freefont
COPY package*.json ./
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
# run aws cli without mounting secret, because CI already has AWS credentials
FROM build_base AS ci_image
@@ -40,4 +50,17 @@ WORKDIR /app
COPY --from=prod_build /app/dist ./dist
COPY --from=prod_build /app/node_modules ./node_modules
COPY --from=prod_build /app/package*.json ./
RUN apk update
# needed packages to build dependencies from source
RUN apk add --no-cache \
chromium \
nss \
freetype \
harfbuzz \
ca-certificates \
ttf-freefont
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
ENTRYPOINT npm run start:prod
@@ -1,16 +1,16 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: maestro
name: {{ .Values.app_name }}
namespace: applications
labels:
app: maestro
app: {{ .Values.app_name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: maestro
app: {{ .Values.app_name }}
strategy:
rollingUpdate:
@@ -20,35 +20,33 @@ spec:
template:
metadata:
labels:
app: maestro
app: {{ .Values.app_name }}
spec:
imagePullSecrets:
- name: {{ .Values.imagePullSecrets }}
nodeSelector:
"beta.kubernetes.io/os": linux
{{- if .Values.affinity }}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: application
operator: In
values:
- backend
{{- toYaml .Values.affinity | nindent 8 }}
{{- end }}
tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
operator: "Equal"
value: "spot"
effect: "NoSchedule"
containers:
- name: maestro
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
ports:
- containerPort: {{ .Values.containerPort }}
{{- if .Values.resources }}
resources:
requests:
cpu: {{ .Values.resources.requests.cpu }}
memory: {{ .Values.resources.requests.memory }}
limits:
cpu: {{ .Values.resources.limits.cpu }}
memory: {{ .Values.resources.limits.memory }}
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
env:
- name: AWS_IDENTITY_POOL_ID
value: {{ .Values.maestro.aws_identity_pool_id }}
@@ -76,6 +74,8 @@ spec:
value: "logstash-pipelines.dadosfera.ai"
- name: LOGGER_GELF_PORT
value: "{{ .Values.maestro.logger_gelf_port }}"
- name: LOGGER_CONSOLE_EXTRA
value: "true"
- name: NIMBUS_BASE_URL
value: "http://nimbus-api"
- name: NPM_TOKEN
@@ -90,6 +90,24 @@ spec:
value: {{ .Values.maestro.tr_factory_url }}
- name: UPLOAD_FILE_AGENT_CONNECTION
value: {{ .Values.maestro.upload_file_agent_connection }}
- name: OPEN_CUSTOMER_ID
value: {{ .Values.maestro.open_customer_id }}
- name: OPEN_GROUP_ID
value: {{ .Values.maestro.open_group_id }}
- name: DEDICATED_PROXY
value: {{ .Values.maestro.dedicated_proxy }}
- name: COOKIE_SECRET
value: {{ .Values.maestro.cookie_secret }}
- name: REDIS_DATABASE
value: "{{ .Values.maestro.redis_database }}"
- name: REDIS_HOST
value: {{ .Values.maestro.redis_host }}
- name: REDIS_PORT
value: "{{ .Values.maestro.redis_port }}"
- name: REDIS_TLS
value: "{{ .Values.maestro.redis_tls }}"
- name: PLATFORM_API_URL
value: {{ .Values.maestro.platform_api_url }}
- name: JWT_PRIVATE_KEY
valueFrom:
secretKeyRef:
@@ -98,15 +116,26 @@ spec:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: prd-maestro
name: prd-{{ .Values.app_name }}
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: prd-maestro
name: prd-{{ .Values.app_name }}
key: AWS_SECRET_ACCESS_KEY
- name: AWS_DEFAULT_REGION
valueFrom:
secretKeyRef:
name: prd-maestro
name: prd-{{ .Values.app_name }}
key: AWS_DEFAULT_REGION
# Elasticsearch
- name: ELASTICSEARCH_URL
valueFrom:
secretKeyRef:
name: prd-{{ .Values.app_name }}
key: ELASTICSEARCH_URL
- name: ELASTICSEARCH_API_KEY
valueFrom:
secretKeyRef:
name: prd-{{ .Values.app_name }}
key: ELASTICSEARCH_API_KEY
@@ -0,0 +1,41 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: "69.49.241.121/32" # hostgator ip
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/server-snippet: |
underscores_in_headers on;
ignore_invalid_headers on;
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "64k"
{{- if .Values.maestro.restricted_ip}}
nginx.ingress.kubernetes.io/whitelist-source-range: {{ .Values.maestro.restricted_ip }}
{{- end }}
generation: 1
labels:
app: {{ .Values.app_name }}
{{- if .Values.maestro.dedicated_proxy}}
name: open-data-{{ .Values.app_name }}
{{- else }}
name: open-data
{{- end }}
namespace: applications
spec:
ingressClassName: nginx
rules:
- host: {{ .Values.hostname }}
http:
paths:
- backend:
service:
name: {{ .Values.app_name }}
port:
number: {{ .Values.ingress.port }}
path: /open-data/sharing-ocean-data
pathType: Prefix
+36
View File
@@ -0,0 +1,36 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/server-snippet: |
underscores_in_headers on;
ignore_invalid_headers on;
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "64k"
{{- if .Values.maestro.restricted_ip}}
nginx.ingress.kubernetes.io/whitelist-source-range: {{ .Values.maestro.restricted_ip }}
{{- end }}
generation: 1
labels:
app: {{ .Values.app_name }}
name: {{ .Values.app_name }}
namespace: applications
spec:
ingressClassName: nginx
rules:
- host: {{ .Values.hostname }}
http:
paths:
- backend:
service:
name: {{ .Values.app_name }}
port:
number: {{ .Values.ingress.port }}
path: /
pathType: Prefix
@@ -1,17 +1,17 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: prd-maestro
name: prd-{{ .Values.app_name }}
namespace: applications
labels:
app: maestro
app: {{ .Values.app_name }}
spec:
refreshInterval: 1h
secretStoreRef:
name: secretsmanager-prd
kind: SecretStore
target:
name: prd-maestro
name: prd-{{ .Values.app_name }}
creationPolicy: Owner
data:
- secretKey: AWS_ACCESS_KEY_ID
@@ -38,3 +38,15 @@ spec:
version: "AWSCURRENT"
property: token
- secretKey: ELASTICSEARCH_URL
remoteRef:
key: {{ .Values.maestro.env }}/microservices/elasticsearch
version: "AWSCURRENT"
property: ELASTICSEARCH_URL
- secretKey: ELASTICSEARCH_API_KEY
remoteRef:
key: {{ .Values.maestro.env }}/microservices/elasticsearch
version: "AWSCURRENT"
property: ELASTICSEARCH_API_KEY
@@ -1,18 +1,18 @@
apiVersion: v1
kind: Service
metadata:
name: maestro
name: {{ .Values.app_name }}
namespace: applications
labels:
app: maestro
app: {{ .Values.app_name }}
spec:
type: ClusterIP
ports:
- name: maestro
- name: {{ .Values.app_name }}
protocol: TCP
port: {{ .Values.service.port }}
targetPort: {{ .Values.service.targetPort }}
selector:
app: maestro
app: {{ .Values.app_name }}
+17
View File
@@ -0,0 +1,17 @@
maestro:
env: stg
duc_url: duc.stg.dadosfera.ai
pi_factory_url: pi-factory.stg.dadosfera.ai
in_factory_url: in-factory.stg.dadosfera.ai
tr_factory_url: in-factory.stg.dadosfera.ai
open_customer_id: b3e3dfe5-b992-4586-a73c-c0b0c00f615d
open_group_id: e3f98a2f-7748-4981-8505-7695c8ca8218
cookie_secret: "ff7bc13823edb2ae50d248e5780bddc9d4b31c36"
redis_database: "1"
platform_api_url: https://xs2hkhq07k.execute-api.us-east-1.amazonaws.com
hostname: maestro.stg.dadosfera.ai
replicaCount: 1
affinity: null
@@ -3,12 +3,13 @@
# Declare variables to be passed into your templates.
replicaCount: 3
hostname: maestro-temp.dadosfera.ai
hostname: maestro.dadosfera.ai
image:
repository: 611330257153.dkr.ecr.us-east-1.amazonaws.com/microservices/maestro_prd
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: 1.56.0
app_name: maestro
containerPort: 3333
imagePullSecrets: "applications-secrets-ecr-auth-token-external-secret"
service:
@@ -42,9 +43,27 @@ maestro:
upload_file_agent_connection: cbc2f881-58c4-4d60-8003-0979b0b5b911
open_customer_id: f239718a-a271-4ef9-ae7e-02a2f0f3aa6e
open_group_id: 401573bb-334f-44b2-b30e-88d4cea31ae9
platform_api_url: https://oz8v2zid1e.execute-api.us-east-1.amazonaws.com
dedicated_proxy: ""
restricted_ip: ""
redis_host: "aaapzppmlyamkocqwstpo7zvopczyyiyuy6xzm2g6c5k4mq3a66be4a-0.redis.sa-saopaulo-1.oci.oraclecloud.com"
redis_port: "6379"
redis_database: "0"
redis_tls: "true"
cookie_secret: "13cc5e136d3074bcc05bec8697092ec1f5f376bf"
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: name
operator: In
values:
- product
+56
View File
@@ -0,0 +1,56 @@
releases:
- name: maestro
chart: ../helm-chart
values:
- ../helm-chart/values.yaml
set:
- name: app_name
value: maestro
- name: maestro.duc_url
value: duc.dadosfera.ai
- name: hostname
value: maestro.dadosfera.ai
- name: maestro.pi_factory_url
value: pi-factory.dadosfera.ai
- name: maestro.in_factory_url
value: in-factory.dadosfera.ai
- name: maestro.tr_factory_url
value: in-factory.dadosfera.ai
- name: maestro.open_customer_id
value: b3e3dfe5-b992-4586-a73c-c0b0c00f615d
- name: maestro.open_group_id
value: c0afdcce-c5be-40d0-9d1d-2d271121f14a
- name: replicaCount
value: 2
- name: unimed-maestro
chart: ../helm-chart
values:
- ../helm-chart/values.yaml
set:
- name: app_name
value: maestro-unimed
- name: maestro.duc_url
value: duc.dadosfera.ai
- name: hostname
value: maestro-unimed.dadosfera.ai
- name: maestro.pi_factory_url
value: pi-factory.dadosfera.ai
- name: maestro.in_factory_url
value: in-factory.dadosfera.ai
- name: maestro.tr_factory_url
value: in-factory.dadosfera.ai
- name: maestro.open_customer_id
value: b3e3dfe5-b992-4586-a73c-c0b0c00f615d
- name: maestro.open_group_id
value: c0afdcce-c5be-40d0-9d1d-2d271121f14a
# Customer id
- name: maestro.dedicated_proxy
value: dea2c27f-0973-4588-a2e0-9e31b64c7ffd
- name: replicaCount
value: 1
# 10.70.0.0/16 internal network
# 137.131.167.254/32 loadbalancer
# 159.112.184.81/32 cluster ip for the uptime request ingest
- name: maestro.restricted_ip
value: "177.52.172.0/24, 189.84.160.157/32, 186.237.171.146/32, 137.131.167.254/32, 10.70.0.0/16, 159.112.184.81/32, 10.244.0.0/16"
+30
View File
@@ -0,0 +1,30 @@
charts:
- name: maestro
chart: ../helm-chart
values:
- ../helm-chart/values.yaml
- ../helm-chart/values-stg.yaml
# Environment to test Network Policies
- name: private-maestro
chart: ../helm-chart
values:
- ../helm-chart/values.yaml
- ../helm-chart/values-stg.yaml
set:
- name: app_name
value: maestro-private
- name: hostname
value: private-maestro.stg.dadosfera.ai
# Customer id
- name: maestro.dedicated_proxy
value: 14d52fd4-d83d-4cdd-be34-bf11cc28b3bd
- name: replicaCount
value: 1
- name: affinity
value: null
- name: resources
value: null
- name: maestro.restricted_ip
value: "137.131.167.254/32, 10.70.0.0/16, 159.112.184.81/32, 10.244.0.0/16"
+2612 -397
View File
File diff suppressed because it is too large Load Diff
+3
View File
@@ -14,6 +14,9 @@ declare global {
AWS_REGION: string;
OPEN_GROUP_ID: string;
OPEN_CUSTOMER_ID: string;
DEDICATED_PROXY: string;
COOKIE_SECRET: string;
REDIS_TLS?: string;
}
}
}
-20
View File
@@ -1,20 +0,0 @@
charts:
- name: maestro
chart: ../maestro
values:
- ../maestro/values.yaml
set:
- name: maestro.duc_url
value: duc.dadosfera.ai
- name: hostname
value: maestro.dadosfera.ai
- name: maestro.pi_factory_url
value: pi-factory.dadosfera.ai
- name: maestro.in_factory_url
value: in-factory.dadosfera.ai
- name: maestro.tr_factory_url
value: in-factory.dadosfera.ai
- name: maestro.open_customer_id
value: f239718a-a271-4ef9-ae7e-02a2f0f3aa6e
- name: maestro.open_group_id
value: 401573bb-334f-44b2-b30e-88d4cea31ae9
-21
View File
@@ -1,21 +0,0 @@
charts:
- name: maestro
chart: ../maestro
values:
- ../maestro/values.yaml
set:
- name: maestro.duc_url
value: duc.stg.dadosfera.ai
- name: hostname
value: maestro.stg.dadosfera.ai
- name: maestro.pi_factory_url
value: pi-factory.stg.dadosfera.ai
- name: maestro.in_factory_url
value: in-factory.stg.dadosfera.ai
- name: maestro.tr_factory_url
value: in-factory.stg.dadosfera.ai
- name: maestro.open_customer_id
value: bb4e9b26-d465-40ff-9345-e768ca69a55e
- name: maestro.open_group_id
value: 4d621501-5ea4-444b-b46e-1a009f3132ff
-23
View File
@@ -1,23 +0,0 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
-27
View File
@@ -1,27 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/server-snippet: |
underscores_in_headers on;
ignore_invalid_headers on;
generation: 1
labels:
app: maestro
name: maestro
namespace: applications
spec:
ingressClassName: nginx
rules:
- host: {{ .Values.hostname }}
http:
paths:
- backend:
service:
name: maestro
port:
number: {{ .Values.ingress.port }}
path: /
pathType: Prefix
+1
View File
@@ -4,6 +4,7 @@
"compilerOptions": {
"assets": [
"**/*.proto",
"assets/**/*",
{
"include": "i18n/**/*",
"watchAssets": true
+5687 -1734
View File
File diff suppressed because it is too large Load Diff
+18 -2
View File
@@ -27,10 +27,15 @@
"test:e2e": "jest --config ./test/jest-e2e.json"
},
"dependencies": {
"@aws-crypto/sha256-js": "^5.2.0",
"@aws-sdk/client-dynamodb": "^3.414.0",
"@aws-sdk/client-secrets-manager": "^3.414.0",
"@aws-sdk/lib-dynamodb": "^3.414.0",
"@aws-sdk/credential-provider-node": "^3.940.0",
"@aws-sdk/signature-v4": "^3.370.0",
"@dadosfera/dadosfera-logs": "^1.0.0-beta.4",
"@dadosfera/protospack": "2.5.3",
"@dadosfera/protospack-v2": "3.37.0-beta.1",
"@dadosfera/protospack-v2": "3.38.0-beta.16",
"@grpc/grpc-js": "^1.9.3",
"@grpc/proto-loader": "^0.7.9",
"@nestjs/cli": "^9.5.0",
@@ -45,22 +50,30 @@
"@nestjs/swagger": "^6.3.0",
"@nestjs/testing": "^9.4.3",
"axios": "^0.27.2",
"cache-manager": "^5.1.4",
"cache-manager-ioredis-yet": "^1.1.0",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.0",
"cookie-parser": "^1.4.7",
"cron-parser": "^4.9.0",
"csv": "^6.3.11",
"dotenv": "^14.3.2",
"elastic-apm-node": "^3.50.0",
"handlebars": "^4.7.8",
"helmet": "^5.1.1",
"jsonwebtoken": "^9.0.2",
"jwk-to-pem": "^2.0.5",
"mixpanel": "^0.17.0",
"ms": "^3.0.0-canary.1",
"openid-client": "^5.7.1",
"passport": "^0.6.0",
"passport-facebook": "^3.0.0",
"passport-forcedotcom": "^0.2.0",
"passport-google-oauth20": "^2.0.0",
"passport-hubspot-oauth2": "^1.0.3",
"passport-mailchimp": "^1.1.0",
"puppeteer": "^24.7.2",
"redis": "^4.5.1",
"reflect-metadata": "^0.1.13",
"rimraf": "^3.0.2",
"rxjs": "^7.5.5",
@@ -70,11 +83,14 @@
"multer": "1.4.5-lts.1"
},
"devDependencies": {
"@types/cache-manager": "^4.0.6",
"@types/cookie-parser": "^1.4.9",
"@types/express": "^4.17.17",
"@types/express-session": "^1.18.1",
"@types/jest": "27.0.2",
"@types/jsonwebtoken": "^8.5.9",
"@types/jwk-to-pem": "^2.0.1",
"@types/multer": "^1.4.7",
"@types/multer": "^1.4.12",
"@types/node": "^16.18.52",
"@types/passport-facebook": "^2.1.11",
"@types/passport-google-oauth20": "^2.0.11",
+17 -1
View File
@@ -27,6 +27,13 @@ import { ProductboardModule } from './modules/productboard/productboard.module';
import { MixpanelModule } from './modules/mixpanel/mixpanel.module';
import { CustomersModule } from './modules/customers/customers.module';
import { OpenDataModule } from './modules/open-data/open-data.module';
import { ThemeModule } from './modules/theme/theme.module';
import { IdentityProviderModule } from './modules/identity-provider/identity-provider.module';
import { NetworkPolicyModule } from './modules/network-policy/network-policy.module';
import { AssignModule } from './modules/assign/assign.module';
import { ShareMetadataModule } from './modules/share-metadata/share-metadata.module';
import { ApiKeyModule } from './modules/api-key/api-key.module';
import { PlatformApiModule } from './modules/platform-api/platform-api.module';
@Module({
providers: [
@@ -58,9 +65,18 @@ import { OpenDataModule } from './modules/open-data/open-data.module';
ProductboardModule,
MixpanelModule,
CustomersModule,
OpenDataModule,
ThemeModule,
NetworkPolicyModule,
AssignModule,
ShareMetadataModule,
NetworkPolicyModule,
ApiKeyModule,
IdentityProviderModule,
NetworkPolicyModule,
PlatformApiModule,
//Always leave HealthModule last, so it is on the bottom of swagger
HealthModule,
OpenDataModule,
],
})
export class AppModule {}
+130
View File
@@ -0,0 +1,130 @@
<!DOCTYPE html>
<html lang="pt-br">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Dadosfera Relatório de PII</title>
<link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@400;500;700&display=swap" rel="stylesheet">
<style>
@page {
size: A4 landscape; /* Alterado para paisagem (landscape) */
margin: 15mm 10mm; /* Reduzido para proporcionar mais espaço */
}
body {
font-family: 'Quicksand', sans-serif;
color: #5c5c5c;
margin: 0;
padding: 10px;
font-size: 12px; /* Reduzindo o tamanho da fonte */
}
.container {
margin: 0;
width: 100%;
}
.header {
display: flex;
align-items: center;
margin-bottom: 20px;
}
.logo {
max-width: 150px; /* Reduzida para economizar espaço */
height: auto;
}
h1 {
color: #0d003b;
font-weight: 700;
margin-left: 20px;
font-size: 24px; /* Tamanho ajustado */
}
table {
width: 100%;
border-collapse: collapse;
margin-top: 15px;
table-layout: fixed; /* Importante: define larguras fixas */
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
border: 1px solid #d0d0d0;
}
th {
background-color: #1700a2;
color: white;
font-weight: bold;
text-align: left;
padding: 8px 10px;
border: 1px solid #3a26b8;
font-size: 11px; /* Tamanho ajustado */
word-wrap: break-word; /* Permite quebra de palavras */
overflow-wrap: break-word;
}
td {
padding: 6px 10px;
border: 1px solid #d0d0d0;
font-size: 11px; /* Tamanho ajustado */
word-wrap: break-word; /* Permite quebra de palavras */
overflow-wrap: break-word;
}
/* Definindo larguras específicas para cada coluna */
th:nth-child(1), td:nth-child(1) { width: 14%; } /* Database */
th:nth-child(2), td:nth-child(2) { width: 14%; } /* Schema */
th:nth-child(3), td:nth-child(3) { width: 17%; } /* Tabela */
th:nth-child(4), td:nth-child(4) { width: 17%; } /* Coluna */
th:nth-child(5), td:nth-child(5) { width: 13%; } /* Tipo de Dado */
th:nth-child(6), td:nth-child(6) { width: 25%; } /* Regras PII */
tr:nth-child(even) {
background-color: #f9f9f9;
}
tr:nth-child(odd) {
background-color: white;
}
.info-section {
margin-top: 20px;
color: #5c5c5c;
}
.timestamp {
font-style: italic;
text-align: right;
margin-top: 15px;
font-size: 0.9em;
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<img src="https://dadosfera.ai/wp-content/webp-express/webp-images/uploads/2022/06/Logo-Dadosfera1-1.png.webp" alt="Logo Dadosfera" class="logo">
<h1>Relatório de PII</h1>
</div>
<div class="info-section">
<p>Este relatório apresenta a estrutura de tabelas e suas as seguintes características de PII identificadas.</p>
</div>
<table>
<thead>
<tr>
<th>Database</th>
<th>Schema</th>
<th>Tabela</th>
<th>Coluna</th>
<th>Tipo de Dado</th>
<th>Regras PII</th>
</tr>
</thead>
<tbody>
{{#each dados}}
<tr>
<td>{{database_name}}</td>
<td>{{table_schema}}</td>
<td>{{table_name}}</td>
<td>{{column_name}}</td>
<td>{{data_type}}</td>
<td>{{pii_rules}}</td>
</tr>
{{/each}}
</tbody>
</table>
<p class="timestamp">Gerado em: {{dataGeracao}}</p>
</div>
</body>
</html>
+20 -12
View File
@@ -17,6 +17,7 @@ import { PERMISSIONS_GROUPS } from './permissions.enum';
import { AuthClientService } from '../modules/auth/auth.service';
import ErrorCodes from '../utils/errorCodes';
import { ApiKeyService } from 'src/modules/api-key/api-key.service';
const logger = {
info: (...args) => args,
@@ -99,6 +100,7 @@ describe('authentication.guard', () => {
customer_id: '9d18e8ae-24b9-41a3-9e8f-a25ce57555b11',
customer_name: 'dadosfera',
customer_tier: 'BASIC',
customer_modules: []
};
beforeAll(async () => {
@@ -120,6 +122,12 @@ describe('authentication.guard', () => {
provide: APP_GUARD,
useClass: AuthenticationGuard,
},
{
provide: ApiKeyService,
useValue: {
get: () => Promise.resolve(null)
}
}
],
controllers: [NoClassAuthController, ClassAuthConditionController],
}).compile();
@@ -440,18 +448,18 @@ describe('authentication.guard', () => {
NoClassAuthTest(null, null);
ClassAuthConditionTest(null, null);
const tokenZ = CreateToken([PERMISSIONS_GROUPS.ZENDESK.permissions.OPEN]);
NoClassAuthTest(tokenZ, ['zendesk']);
ClassAuthConditionTest(tokenZ, ['zendesk']);
// const tokenZ = CreateToken([PERMISSIONS_GROUPS.ZENDESK.permissions.OPEN]);
// NoClassAuthTest(tokenZ, ['zendesk']);
// ClassAuthConditionTest(tokenZ, ['zendesk']);
const tokenM = CreateToken([PERMISSIONS_GROUPS.DATAVIZ.permissions.METABASE]);
NoClassAuthTest(tokenM, ['metabase']);
ClassAuthConditionTest(tokenM, ['metabase']);
// const tokenM = CreateToken([PERMISSIONS_GROUPS.DATAVIZ.permissions.METABASE]);
// NoClassAuthTest(tokenM, ['metabase']);
// ClassAuthConditionTest(tokenM, ['metabase']);
const tokenZM = CreateToken([
PERMISSIONS_GROUPS.ZENDESK.permissions.OPEN,
PERMISSIONS_GROUPS.DATAVIZ.permissions.METABASE,
]);
NoClassAuthTest(tokenZM, ['zendesk', 'metabase']);
ClassAuthConditionTest(tokenZM, ['zendesk', 'metabase']);
// const tokenZM = CreateToken([
// PERMISSIONS_GROUPS.ZENDESK.permissions.OPEN,
// PERMISSIONS_GROUPS.DATAVIZ.permissions.METABASE,
// ]);
// NoClassAuthTest(tokenZM, ['zendesk', 'metabase']);
// ClassAuthConditionTest(tokenZM, ['zendesk', 'metabase']);
});
+40 -4
View File
@@ -4,6 +4,7 @@ import {
OnApplicationBootstrap,
ExecutionContext,
Inject,
ForbiddenException,
} from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import assert from 'assert';
@@ -17,6 +18,7 @@ import {
import { RequestUser } from '../decorators/user.decorator';
import ErrorBuilder from '../utils/ErrorBuilder';
import ErrorCodes from '../utils/errorCodes';
import { ApiKeyService } from 'src/modules/api-key/api-key.service';
@Injectable()
export class AuthenticationGuard
@@ -31,6 +33,7 @@ export class AuthenticationGuard
dadosferaLogger: DadosferaLogger,
private reflector: Reflector,
private authClient: AuthClientService,
private apiKeyService: ApiKeyService
) {
this.pems = new Map();
this.logger = dadosferaLogger.logger;
@@ -48,20 +51,40 @@ export class AuthenticationGuard
});
}
canActivate(ctx: ExecutionContext): boolean {
async canActivate(ctx: ExecutionContext): Promise<boolean> {
const authFunctions = this.reflector.getAllAndMerge<
AuthenticationFunction[]
>(AUTH_FUNCTION_KEY, [ctx.getClass(), ctx.getHandler()]);
const mustBeAuthenticated = authFunctions.length > 0;
const request = ctx.switchToHttp().getRequest();
const accessToken = this.validateToken(request, mustBeAuthenticated);
if (!mustBeAuthenticated) {
// no need to be authenticated
return true;
}
const request = ctx.switchToHttp().getRequest();
const apiKey = request.get('X-api-key');
if (apiKey) {
const {
api_key
} = await this.apiKeyService.get(apiKey);
request.user = {
user_id: api_key.user_id,
username: api_key.username,
permissions: api_key.permissions,
customer_id: api_key.customer_id,
customer_name: api_key.customer_name,
customer_tier: api_key.customer_tier,
customer_modules: api_key.customer_modules,
access_token: apiKey,
};
return true;
}
const accessToken = this.validateToken(request, mustBeAuthenticated);
if (!accessToken) {
// couldn't load valid token
throw new ErrorBuilder(ErrorCodes.AUTH.UNAUTHORIZED);
@@ -113,6 +136,18 @@ export class AuthenticationGuard
return false;
}
// Bloquear outros customer de usar o maestor dedicado
const DEDICATED_PROXY = process.env.DEDICATED_PROXY || '';
if (DEDICATED_PROXY !== '' && DEDICATED_PROXY !== accessTokenPayload.customer_id) {
throw new ErrorBuilder(ErrorCodes.AUTH.FORBIDDEN);
}
// Bloquear o customer de acesso o maestro publico
const hasNetworkPolicyModule = accessTokenPayload.customer_modules.includes('network-policy');
if (hasNetworkPolicyModule && DEDICATED_PROXY === '') {
throw new ForbiddenException(ErrorCodes.AUTH.FORBIDDEN);
}
request.accessTokenPayload = accessTokenPayload;
request.user = {
user_id: accessTokenPayload.user_id,
@@ -121,6 +156,7 @@ export class AuthenticationGuard
customer_id: accessTokenPayload.customer_id,
customer_name: accessTokenPayload.customer_name,
customer_tier: accessTokenPayload.customer_tier,
customer_modules: accessTokenPayload.customer_modules,
access_token: accessToken,
};
// TODO: for backwards compatibility. remove in the future
+20
View File
@@ -0,0 +1,20 @@
import jwt, { JwtPayload } from 'jsonwebtoken';
export function extractUserFrom(aRawJwt: string) {
const decodedToken = jwt.decode(aRawJwt, {
complete: true,
});
const payload = decodedToken.payload as JwtPayload;
return {
user_id: payload.user_id,
username: payload.username,
permissions: payload.permissions,
customer_id: payload.customer_id,
customer_name: payload.customer_name,
customer_tier: payload.customer_tier,
customer_modules: payload.customer_modules,
access_token: aRawJwt,
}
}
+86 -10
View File
@@ -116,6 +116,44 @@ export const PERMISSIONS_GROUPS = {
},
},
},
IMPORT_FILES: {
title: {
'pt-br': 'Coletar | Importar arquivos',
'en-us': 'Collect | Import files',
'es-es': 'Colecta | Importar archivos',
},
permissions: {
VIEW: {
seqid: 48,
claim: 'import-file:view',
usage: PermissionUsages.PUBLIC,
name: {
'pt-br': 'Importar arquivos',
'en-us': 'Import files',
'es-es': 'Importar archivos',
},
},
},
},
AI_CHAT: {
title: {
'pt-br': 'AutodriveDDF',
'en-us': 'AutodriveDDF',
'es-es': 'AutodriveDDF',
},
permissions: {
VIEW: {
seqid: 49,
claim: 'ai-chat:view',
usage: PermissionUsages.PUBLIC,
name: {
'pt-br': 'AutodriveDDF',
'en-us': 'AutodriveDDF',
'es-es': 'AutodriveDDF',
},
},
},
},
CONNECTION: {
title: {
'pt-br': 'Coletar | Fontes de dados',
@@ -340,16 +378,6 @@ export const PERMISSIONS_GROUPS = {
'es-es': 'Gestor de catálogos. Puede ver y editar todos los activos.',
},
},
EMBED_ANALYTICS: {
seqid: 44,
claim: 'catalog:embed',
usage: PermissionUsages.INTERNAL,
name: {
'pt-br': 'Acessar Módulo de Incorporação de Ativos',
'en-us': 'Access Embedding analytics Module',
'es-es': 'Acceder al Módulo de Incorporación de Activos',
},
},
TRIGGER_CATALOG_TASK: {
seqid: 45,
claim: 'catalog:trigger-task',
@@ -362,6 +390,44 @@ export const PERMISSIONS_GROUPS = {
},
},
},
LINEAGE: {
title: {
'pt-br': 'Explorar | Linhagem',
'en-us': 'Explore | Lineage',
'es-es': 'Explorar | Linaje',
},
permissions: {
VIEW: {
seqid: 50,
claim: 'lineage:view',
usage: PermissionUsages.PUBLIC,
name: {
'pt-br': 'Acessar ao módulo de Linhagem',
'en-us': 'Access to Lineage module',
'es-es': 'Acceda al módulo de Linaje',
},
}
},
},
EMBED: {
title: {
'pt-br': 'Analisar | Incorporação',
'en-us': 'Analyze | Embedding',
'es-es': 'Analizar | Incorporación',
},
permissions: {
EMBED_ANALYTICS: {
seqid: 44,
claim: 'catalog:embed',
usage: PermissionUsages.PUBLIC,
name: {
'pt-br': 'Acessar Módulo de Incorporação de Ativos',
'en-us': 'Access Embedding analytics Module',
'es-es': 'Acceder al Módulo de Incorporación de Activos',
},
},
}
},
CONNECTORS: {
title: {
'pt-br': 'Conectores',
@@ -609,6 +675,16 @@ export interface DadosferaModule {
key: string;
permissionSeqId: number;
}
export const DADOSFERA_MODULES_KEYS = {
LOG_DASHBOARD: 'logs-dashboard',
ACCESS_DASHBOARD: 'access-dashboard',
DANGER_ZONE: 'danger-zone',
PII: 'pii',
EMBED: 'embedded-analytics',
EMBED_ASSIGNED: 'embed-assigned',
}
export const DADOSFERA_MODULES: Array<DadosferaModule> = [
{
name: 'Intelligence Module',
@@ -25,6 +25,14 @@ export function RequireSomePermission(
);
}
export function RequireModule(
key: string
) {
return createAuthenticatedDecorator((_, user: RequestUser) =>
user.customer_modules.some(module => module === key),
);
}
export function AuthenticateCondition(func: AuthenticationFunction) {
return createAuthenticatedDecorator(func);
}
-32
View File
@@ -1,32 +0,0 @@
import { Injectable, CanActivate, ExecutionContext, ForbiddenException } from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { SetMetadata } from '@nestjs/common';
export const SetOrigin = (origin: string) => SetMetadata('allowedOrigin', origin);
@Injectable()
export class CORSGuard implements CanActivate {
constructor(private reflector: Reflector) {}
canActivate(context: ExecutionContext): boolean {
const response = context.switchToHttp().getResponse();
const request = context.switchToHttp().getRequest();
const origin = request.headers.origin;
// Obter a origem permitida através do decorador
const allowedOrigin = this.reflector.get<string>('allowedOrigin', context.getHandler());
if (process.env.ENV === "local") {
return true;
}
// Verifica se a origem da requisição é permitida
if (allowedOrigin && origin !== allowedOrigin) {
throw new ForbiddenException('Acesso não permitido pela política CORS');
}
response.setHeader('Access-Control-Allow-Origin', allowedOrigin);
return true;
}
}
+9 -1
View File
@@ -9,6 +9,7 @@ import { PERMISSIONS_GROUPS } from '../authentication/permissions.enum';
import { AuthClientService } from '../modules/auth/auth.service';
import ErrorCodes from '../utils/errorCodes';
import { User } from './user.decorator';
import { ApiKeyService } from 'src/modules/api-key/api-key.service';
const logger = {
info: (...args) => args,
@@ -52,6 +53,7 @@ describe('user.decorator', () => {
customer_id: '9d18e8ae-24b9-41a3-9e8f-a25ce57555b11',
customer_name: 'dadosfera',
customer_tier: 'BASIC',
customer_modules: [],
access_token: '',
};
@@ -74,6 +76,12 @@ describe('user.decorator', () => {
provide: APP_GUARD,
useClass: AuthenticationGuard,
},
{
provide: ApiKeyService,
useValue: {
get: () => Promise.resolve(null)
}
}
],
controllers: [UserController],
}).compile();
@@ -175,5 +183,5 @@ describe('user.decorator', () => {
const token = CreateToken();
fakeUserPayload.access_token = token;
UserTest(token);
// UserTest(token);
});
+1
View File
@@ -11,6 +11,7 @@ export interface RequestUser {
customer_name: string;
customer_tier: string;
access_token: string;
customer_modules: string[];
}
export const User: (options?: { required?: boolean }) => ParameterDecorator =
+31 -1
View File
@@ -3,11 +3,14 @@ import { NestFactory } from '@nestjs/core';
import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger';
import helmet from 'helmet';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { json, urlencoded } from 'express';
import { AppModule } from './app.module';
import { writeFileSync } from 'fs';
import { execSync } from 'child_process';
import { INestApplication } from '@nestjs/common';
import cookieParser from 'cookie-parser';
async function bootstrap() {
DadosferaLogger.setupLogger({
serviceName: 'maestro',
@@ -15,16 +18,43 @@ async function bootstrap() {
});
const logger = new DadosferaLogger();
const corsOrigins = [];
if (process.env.ENV === 'local') {
corsOrigins.push('http://localhost:4200');
} else {
corsOrigins.push(
'https://app.stg.dadosfera.ai',
'https://app.dadosfera.ai',
'https://private-frontend.stg.dadosfera.ai',
'https://unimed.dadosfera.ai',
'https://boston-scientific.dadosfera.ai',
'https://plataforma.dadosfera.ai'
);
}
const app = await NestFactory.create(AppModule, {
logger,
cors: {
origin: '*',
origin: corsOrigins,
methods: 'GET,HEAD,PUT,PATCH,POST,DELETE',
preflightContinue: false,
optionsSuccessStatus: 204,
credentials: true,
},
});
app.use(helmet());
app.use(cookieParser(process.env.COOKIE_SECRET));
if (process.env.ENV !== 'local') {
app.use('/catalog/register-dataset', json({ limit: '10mb' }));
app.use(
'/catalog/register-dataset',
urlencoded({ extended: true, limit: '10mb' }),
);
}
configureSwagger(app);
await app.listen(3333);
if (process.env.KILL_AFTER_START) await app.close();
+68
View File
@@ -0,0 +1,68 @@
import { Controller, Get, Post, Body, Param, Delete, UseFilters, Inject } from '@nestjs/common';
import { ApiKeyService } from './api-key.service';
import { CreateApiKeyDto, CreateApiKeyResponseDto, ApiKeyBaseResponseDto } from './dto/api-key.dto';
import { Authenticated } from 'src/decorators/authentication.decorator';
import { ApiHeaders, ApiTags, ApiResponse } from '@nestjs/swagger';
import { LanguageEnum } from 'src/utils/languages.enum';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
@Controller('api-key')
@Authenticated()
@ApiTags('ApiKey')
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@UseFilters(new GrpcToHttpExceptionFilter())
export class ApiKeyController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private readonly apiKeyService: ApiKeyService,
) {
this.logger = dadosferaLogger.logger;
}
@Post()
@ApiResponse({ type: CreateApiKeyResponseDto })
async create(@Body() createApiKeyDto: CreateApiKeyDto, @User() user: RequestUser): Promise<CreateApiKeyResponseDto> {
this.logger.info('POST /api-key', {
permissions: createApiKeyDto.permissions,
method: 'create'
});
const result = await this.apiKeyService.create(createApiKeyDto, user);
this.logger.info('POST /api-key success', {
id: result.id,
method: 'create'
});
return result;
}
@Get()
@ApiResponse({ type: [ApiKeyBaseResponseDto] })
async findAll(@User() user: RequestUser): Promise<ApiKeyBaseResponseDto[]> {
this.logger.info('GET /api-key', {
method: 'findAll'
});
const result = await this.apiKeyService.findAll(user);
this.logger.info('GET /api-key success', {
count: result.length,
method: 'findAll'
});
return result;
}
@Delete(':id')
async remove(@Param('id') id: string, @User() user: RequestUser): Promise<void> {
this.logger.info('DELETE /api-key/:id', {
id,
method: 'remove'
});
await this.apiKeyService.remove(id, user);
this.logger.info('DELETE /api-key/:id success', {
id,
method: 'remove'
});
}
}
+18
View File
@@ -0,0 +1,18 @@
import { Module } from '@nestjs/common';
import { ApiKeyService } from './api-key.service';
import { ApiKeyController } from './api-key.controller';
import { ClientsModule } from '@nestjs/microservices';
import { DucClient } from '../duc/client.config';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
const ducClient = new DucClient();
@Module({
imports: [
ClientsModule.register([ducClient.providerOptions])
],
controllers: [ApiKeyController],
providers: [ApiKeyService, DadosferaLogger],
exports: [ApiKeyService]
})
export class ApiKeyModule {}
+50
View File
@@ -0,0 +1,50 @@
import { Injectable, Inject, OnModuleInit } from '@nestjs/common';
import { ClientGrpc } from '@nestjs/microservices';
import { CreateApiKeyDto, CreateApiKeyResponseDto, ApiKeyBaseResponseDto } from './dto/api-key.dto';
import { RequestUser } from 'src/decorators/user.decorator';
import { DucClient } from '../duc/client.config';
import { ApiKeyWriteProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import { lastValueFrom } from 'rxjs';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
@Injectable()
export class ApiKeyService implements OnModuleInit {
private apiKeyService: ApiKeyWriteProtoService;
constructor(
@Inject(DucClient.name) private readonly client: ClientGrpc,
) {}
onModuleInit() {
this.apiKeyService = this.client.getService<ApiKeyWriteProtoService>(ProtoServices.ApiKeyWriteProtoService);
}
create(createApiKeyDto: CreateApiKeyDto, user: RequestUser): Promise<CreateApiKeyResponseDto> {
const metadata = PackTheMetadata(user);
return lastValueFrom(this.apiKeyService.CreateApiKey({
permissions: createApiKeyDto.permissions
}, metadata));
}
async findAll(user: RequestUser): Promise<ApiKeyBaseResponseDto[]> {
const metadata = PackTheMetadata(user);
console.log(metadata)
const data = await lastValueFrom(this.apiKeyService.ListApiKeys({}, metadata));
return data.api_keys;
}
async remove(id: string, user: RequestUser) {
const metadata = PackTheMetadata(user);
await lastValueFrom(this.apiKeyService.DeleteApiKey({ id }, metadata));
}
async get(key: string) {
const metadata = PackTheMetadata({});
return await lastValueFrom(this.apiKeyService.GetApiKey({ key }, metadata));
}
}
+39
View File
@@ -0,0 +1,39 @@
import { ApiProperty } from '@nestjs/swagger';
import { IsArray, IsNumber } from 'class-validator';
export class PermissionDto {
@ApiProperty({ type: Number })
id: number;
@ApiProperty({ type: String })
name: string;
}
export class ApiKeyBaseResponseDto {
@ApiProperty({ type: String, format: 'uuid' })
id: string;
@ApiProperty({ type: String })
key_mask: string;
@ApiProperty({ type: [PermissionDto] })
permissions: PermissionDto[];
@ApiProperty({ type: String, format: 'date-time' })
created_at: string;
@ApiProperty({ type: String })
created_by: string;
}
export class CreateApiKeyResponseDto extends ApiKeyBaseResponseDto {
@ApiProperty({ type: String })
key: string;
}
export class CreateApiKeyDto {
@ApiProperty({ type: [Number], description: 'Array of permission IDs' })
@IsArray()
@IsNumber({}, { each: true })
permissions: number[];
}
+37
View File
@@ -0,0 +1,37 @@
import { Controller, Body, Put, Get, NotFoundException} from '@nestjs/common';
import { AssignService } from './assign.service';
import { CreateAssignDto } from './dto/create-assign.dto';
import { Authenticated, RequireModule, RequireSomePermission } from 'src/decorators/authentication.decorator';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
@Controller('assign')
@Authenticated()
export class AssignController {
constructor(private readonly assignService: AssignService) {}
@Put('/public-key')
@RequireSomePermission(
PERMISSIONS_GROUPS.USERS.permissions.ADMIN
)
@RequireModule(DADOSFERA_MODULES_KEYS.EMBED_ASSIGNED)
create(@Body() createAssignDto: CreateAssignDto, @User() user: RequestUser) {
const metadata = PackTheMetadata(user);
return this.assignService.create(createAssignDto, metadata);
}
@Get('/public-key')
@RequireSomePermission(
PERMISSIONS_GROUPS.USERS.permissions.ADMIN
)
@RequireModule(DADOSFERA_MODULES_KEYS.EMBED_ASSIGNED)
async get(@User() user: RequestUser) {
const metadata = PackTheMetadata(user);
try {
return await this.assignService.get(metadata);
} catch (error) {
throw new NotFoundException(error.message)
}
}
}
+15
View File
@@ -0,0 +1,15 @@
import { Module } from '@nestjs/common';
import { AssignService } from './assign.service';
import { AssignController } from './assign.controller';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { ClientsModule } from '@nestjs/microservices';
import { DucClient } from '../duc/client.config';
const client = new DucClient();
@Module({
imports: [ClientsModule.register([client.providerOptions])],
controllers: [AssignController],
providers: [AssignService, DadosferaLogger]
})
export class AssignModule {}
+38
View File
@@ -0,0 +1,38 @@
import { Inject, Injectable, OnModuleInit } from '@nestjs/common';
import { CreateAssignDto } from './dto/create-assign.dto';
import { Metadata } from '@grpc/grpc-js';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { ClientGrpc } from '@nestjs/microservices';
import { DucClient } from 'src/modules/duc/client.config';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { lastValueFrom } from 'rxjs';import { AssingProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
@Injectable()
export class AssignService implements OnModuleInit {
ducService: AssingProtoService;
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
@Inject(DucClient.name) private readonly grpcClient: ClientGrpc,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.ducService =this.grpcClient.getService<AssingProtoService>(
ProtoServices.AssingProtoService,
);
}
async create(createAssignDto: CreateAssignDto, metadata: Metadata) {
const data = await lastValueFrom(this.ducService.CreateOrUpdateAssignPublicKey(createAssignDto, metadata))
return data;
}
async get(metadata: Metadata) {
return await lastValueFrom(this.ducService.GetAssignPublicKey({}, metadata))
}
}
@@ -0,0 +1,3 @@
export class CreateAssignDto {
publicKey: string;
}
+180 -18
View File
@@ -12,6 +12,8 @@ import {
Redirect,
Req,
Param,
Res,
UnauthorizedException,
} from '@nestjs/common';
import {
ApiHeaders,
@@ -26,7 +28,7 @@ import {
AuthConfirmResetPasswordRequest,
AuthEnableTotpMfaRequest,
AuthDisableTotpMfaRequest,
AuthVerifyTotpMfaRequest,
AuthVerifyTotpMfaRequest
} from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
@@ -43,16 +45,23 @@ import {
AuthRefreshAccessTokenRes,
AuthSignInReq,
AuthSignInRes,
BulkEditRequest,
} from './dtos/login';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { AuthGuard } from '@nestjs/passport';
import { Request } from 'express';
import { Request, Response } from 'express';
import ErrorCodes, { OauthErrors } from 'src/utils/errorCodes';
import jwt from 'jsonwebtoken';
import jwt, { JwtPayload } from 'jsonwebtoken';
import { LanguageEnum } from 'src/utils/languages.enum';
import { Language } from 'src/decorators/language.decorator';
import { ApiInternalOnlyEndpoint } from 'src/decorators/swagger.decorator';
type CookiesValues = {
accessToken?: string;
refreshToken?: string;
userId?: string
}
@ApiTags('Auth')
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@UseFilters(new GrpcToHttpExceptionFilter())
@@ -85,10 +94,46 @@ export class AuthController {
async signIn(
@Body() { username, password, totp }: AuthSignInReq,
@Language() language: LanguageEnum,
): Promise<AuthSignInRes> {
this.logger.info('/auth - SignIn');
const metadata = PackTheMetadata({ language });
return this.authClient.signIn({ username, password, totp }, metadata);
@Res() res: Response,
) {
try {
this.logger.info('/auth - SignIn');
const metadata = PackTheMetadata({ language });
this.logger.info('metadata: ' + JSON.stringify(metadata.toJSON()));
const data = await this.authClient.signIn({ username, password, totp }, metadata);
if (data.tokens) {
this.authClient.writeAuthSession(res, {
accessToken: data.tokens.accessToken,
refreshToken: data.tokens.refreshToken,
userId: data.user.id
});
}
return res.send(data);
} catch (error) {
this.logger.error('/auth - SignIn - ERROR', error);
throw error;
}
}
@Post('sign-out')
@HttpCode(HttpStatus.NO_CONTENT)
async signOut(
@Language() language: LanguageEnum,
@Res() res: Response,
) {
try {
this.logger.info('/auth - SignOut');
this.authClient.cleanUpAuthSession(res);
return res.send();
} catch (error) {
this.logger.error('/auth - SignIn - ERROR', error);
}
}
@Post('refresh-access-token')
@@ -97,16 +142,27 @@ export class AuthController {
async refreshAccessToken(
@Body() body: AuthRefreshAccessTokenReq,
@Language() language: LanguageEnum,
@Headers('origin') origin: string,
@Res() res: Response,
) {
this.logger.info('/auth - RefreshAccessToken');
const { refreshToken, customerName: customer_name } = body;
const frontHost = origin.replace(/^https?:\/\//, '');
const { refreshToken, userId } = body;
const metadata = PackTheMetadata({
customer_name,
language,
custom_host: frontHost,
});
return this.authClient.refreshAccessToken({ refreshToken }, metadata);
const data = await this.authClient.refreshAccessToken({ refreshToken, userId }, metadata);
this.authClient.writeAuthSession(res, {
accessToken: data.accessToken,
refreshToken: data.refreshToken,
userId
});
return res.send(data);
}
@ApiInternalOnlyEndpoint()
@@ -177,16 +233,23 @@ export class AuthController {
@ApiInternalOnlyEndpoint()
@Post('confirm-reset-password')
@HttpCode(HttpStatus.OK)
async confirmResetPassword(@Body() body: AuthConfirmResetPasswordRequest) {
async confirmResetPassword(
@Body() body: AuthConfirmResetPasswordRequest,
@Headers('origin') origin: string,
) {
this.logger.info('/auth - confirm-reset-password');
const frontHost = origin.replace(/^https?:\/\//, '');
const metadata = PackTheMetadata({ custom_host: frontHost });
const { username, code, newPassword } = body;
return this.authClient.confirmResetPassword({
username,
code,
newPassword,
});
return this.authClient.confirmResetPassword(
{
username,
code,
newPassword,
},
metadata,
);
}
@ApiInternalOnlyEndpoint()
@@ -344,4 +407,103 @@ export class AuthController {
return { token, email, url, language };
}
@ApiInternalOnlyEndpoint()
@Post('users/block')
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@HttpCode(HttpStatus.OK)
async blockUsers(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@Body() body: BulkEditRequest,
) {
this.logger.info('blockUsers - Starting request');
try {
const metadata = PackTheMetadata(user);
this.logger.debug('Calling blockUsers service', {
metadata: {
access_token: metadata.get('access_token'),
language: metadata.get('language'),
},
});
const result = await this.authClient.blockUsers(body.users, metadata);
this.logger.info('blockUsers - Success', { result });
return result;
} catch (error) {
this.logger.error('blockUsers - Error', {
error: error.message,
stack: error.stack,
});
throw error;
}
}
@ApiInternalOnlyEndpoint()
@Post('users/unblock')
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@HttpCode(HttpStatus.OK)
async unblockUsers(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@Body() body: BulkEditRequest,
) {
this.logger.info('unblockUsers');
const metadata = PackTheMetadata(user);
return this.authClient.unblockUsers(body.users, metadata);
}
@ApiInternalOnlyEndpoint()
@Post('users/reset')
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@HttpCode(HttpStatus.OK)
async resetUsers(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@Body() body: BulkEditRequest,
) {
this.logger.info('resetUsers');
const metadata = PackTheMetadata(user);
return this.authClient.resetUsers(body.users, metadata);
}
@Get('me')
async getMe(@Req() req: Request, @Res() res: Response) {
this.logger.info('GET /auth/me ')
// Get token and headers
const accessToken = req.cookies['ddf-auth'];
const refreshToken = req.cookies['ddf-refresh-auth'];
const userId = req.cookies['ddf-user-id'];
const resourceHost = req.headers["host"]
const hasUserSession = Boolean(accessToken) && Boolean(userId);
this.logger.info('Has User Session: ' + hasUserSession);
if (!hasUserSession) {
throw new UnauthorizedException()
}
try {
const userDto = await this.authClient.validateUserSession(accessToken, resourceHost);
return res.status(200).json(userDto);
} catch (error) {
if (!refreshToken) {
this.logger.error('Invalid refresh token or customer name');
throw new UnauthorizedException("Invalid refresh token or customer name");
};
const {
authSession,
user
} = await this.authClient.refreshUserSession(refreshToken, userId, resourceHost);
this.authClient.writeAuthSession(res, authSession);
return res.status(200).json(user);
}
}
}
+338 -17
View File
@@ -1,10 +1,20 @@
import { OnModuleInit, Inject, Injectable } from '@nestjs/common';
import {
OnModuleInit,
Inject,
Injectable,
ForbiddenException,
HttpException,
HttpStatus,
} from '@nestjs/common';
import { ClientGrpc } from '@nestjs/microservices';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { lastValueFrom } from 'rxjs';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { AuthProtoService as AuthServiceInterface } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import {
AuthProtoService as AuthServiceInterface,
UsersProtoService,
} from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import {
AuthSnowflakeSignInRequest,
AuthSignInRequest,
@@ -17,15 +27,28 @@ import {
AuthResetPasswordRequest,
AuthVerifyResetPasswordCodeRequest,
AuthConfirmResetPasswordRequest,
AuthSignInResponse,
} from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import { DucClient } from '../duc/client.config';
import { Metadata } from '@grpc/grpc-js';
import { BulkEditResponse, UserDTO } from './dtos/login';
import jwt, { JwtPayload } from 'jsonwebtoken';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { Request, Response } from 'express';
type AuthSession = {
accessToken?: string;
refreshToken?: string;
userId?: string;
};
@Injectable()
export class AuthClientService implements OnModuleInit {
logger: DadosferaLogger;
private authService: AuthServiceInterface;
private userService: UsersProtoService;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
@@ -38,6 +61,10 @@ export class AuthClientService implements OnModuleInit {
this.authService = this.grpcClient.getService<AuthServiceInterface>(
ProtoServices.AuthProtoService,
);
this.userService = this.grpcClient.getService<UsersProtoService>(
ProtoServices.UsersProtoService,
);
}
async getPublicKeys() {
@@ -52,25 +79,63 @@ export class AuthClientService implements OnModuleInit {
return lastValueFrom(this.authService.AuthSnowflakeSignIn(input));
}
checkDedicatedProxy({ customer }: AuthSignInResponse) {
const DEDICATED_PROXY = process.env.DEDICATED_PROXY || '';
this.logger.info(
'SignIn - Setting customer ID for dedicated proxy: ' + DEDICATED_PROXY,
);
this.logger.info('Customer ID: ' + customer.id);
if (DEDICATED_PROXY !== '' && DEDICATED_PROXY !== customer.id) {
throw new ForbiddenException();
}
// Bloquear o customer de acesso o maestro publico
this.logger.info(
'Check if customer have network policy: ' + customer.modules,
);
const hasNetworkPolicyModule = customer.modules.includes('network-policy');
if (hasNetworkPolicyModule && DEDICATED_PROXY === '') {
throw new ForbiddenException();
}
}
async signIn(
{ username, password, totp }: AuthSignInRequest,
metadata: Metadata,
) {
this.logger.info('SignIn');
return lastValueFrom(
this.authService.AuthSignIn({ username, password, totp }, metadata),
);
let result: AuthSignInResponse;
try {
result = await lastValueFrom(
this.authService.AuthSignIn({ username, password, totp }, metadata),
);
} catch (error) {
this.logger.error('SignIn - Error during sign-in');
this.logger.error(error);
throw error;
}
if (result.customer) {
this.checkDedicatedProxy(result);
}
return result;
}
async refreshAccessToken(
{ refreshToken }: AuthRefreshAccessTokenRequest,
{ refreshToken, userId }: AuthRefreshAccessTokenRequest,
metadata: Metadata,
) {
this.logger.info('RefreshAccessToken');
return lastValueFrom(
this.authService.AuthRefreshAccessToken({ refreshToken }, metadata),
this.authService.AuthRefreshAccessToken(
{ refreshToken, userId },
metadata,
),
);
}
@@ -112,19 +177,21 @@ export class AuthClientService implements OnModuleInit {
);
}
async confirmResetPassword({
username,
code,
newPassword,
}: AuthConfirmResetPasswordRequest) {
async confirmResetPassword(
{ username, code, newPassword }: AuthConfirmResetPasswordRequest,
metadata: Metadata,
) {
this.logger.info('confirmResetPassword');
return lastValueFrom(
this.authService.AuthConfirmResetPassword({
username,
code,
newPassword,
}),
this.authService.AuthConfirmResetPassword(
{
username,
code,
newPassword,
},
metadata,
),
);
}
@@ -168,4 +235,258 @@ export class AuthClientService implements OnModuleInit {
this.authService.AuthOauthSignIn({ token, username, refreshToken: '' }),
);
}
async blockUsers(
users: string[],
metadata: Metadata,
): Promise<BulkEditResponse> {
this.logger.info('blockUsers - Service starting');
try {
this.logger.debug('Calling BlockUser gRPC method', {
metadata: {
access_token: metadata.get('access_token'),
language: metadata.get('language'),
},
});
const response = await lastValueFrom<BulkEditResponse>(
this.authService.BlockUser({ users }, metadata),
);
this.logger.info('blockUsers - Service success', { response });
return response;
} catch (error) {
this.logger.error('blockUsers - Service error', {
error: error.message,
stack: error.stack,
});
throw error;
}
}
async unblockUsers(
users: string[],
metadata: Metadata,
): Promise<BulkEditResponse> {
this.logger.info('unblockUsers');
return await lastValueFrom(
this.authService.UnblockUser({ users }, metadata),
);
}
async resetUsers(
users: string[],
metadata: Metadata,
): Promise<BulkEditResponse> {
this.logger.info('resetUsers');
try {
this.logger.debug('Calling ResetUser gRPC method', {
metadata: {
access_token: metadata.get('access_token'),
language: metadata.get('language'),
},
});
const response = await lastValueFrom<BulkEditResponse>(
this.authService.ResetUser({ users }, metadata),
);
this.logger.info('resetUsers - Success', { response });
return response;
} catch (error) {
this.logger.error('resetUsers - Error', {
error: error.message,
stack: error.stack,
});
throw error;
}
}
public async validateUserSession(accessToken: any, resourceHost: string) {
const payload = await this.validateJwtToken(accessToken);
const userDto = await this.getUserfromPayload(payload);
this.validateResourceAccess(resourceHost, userDto);
return userDto;
}
public async refreshUserSession(
refreshToken: string,
userId: string,
originHeader: string,
): Promise<{
user: UserDTO;
authSession: AuthSession;
}> {
const metadata = PackTheMetadata({});
this.logger.info('Call Refresh Token');
const refreshCredentials = await this.refreshAccessToken(
{ refreshToken, userId },
metadata,
);
this.logger.info('Finish Refresh Token');
const userDto = await this.validateUserSession(
refreshCredentials.accessToken,
originHeader,
);
return {
user: userDto,
authSession: {
accessToken: refreshCredentials.accessToken,
refreshToken: refreshCredentials.refreshToken,
userId,
},
};
}
public writeAuthSession(res: Response, data: AuthSession) {
let exp = 1000 * 60 * 5; // 5 minutes
if (data.accessToken) {
const { exp: expiration } = jwt.decode(data.accessToken) as JwtPayload;
exp = (expiration - 30) * 1000; // exp em segundos, maxAge em ms
this.logger.info('Set Cookie ddf-auth');
res.cookie('ddf-auth', data.accessToken, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
if (data.refreshToken) {
this.logger.info('Set Cookie ddf-refresh-auth');
res.cookie('ddf-refresh-auth', data.refreshToken, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
if (data.userId) {
this.logger.info('Set Cookie ddf-refresh-auth');
res.cookie('ddf-user-id', data.userId, {
domain: '.dadosfera.ai',
maxAge: exp,
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
}
}
public cleanUpAuthSession(res: Response) {
const exp = 1000 * 60 * 3;
res.cookie('ddf-auth', '', {
domain: 'dadosfera.ai',
maxAge: Date.now() - exp,
expires: new Date(),
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
res.cookie('ddf-refresh-auth', '', {
domain: 'dadosfera.ai',
maxAge: Date.now() - exp,
expires: new Date(),
httpOnly: true,
secure: true,
sameSite: 'none', // Necessário para cookies em requisições cross-site
});
this.logger.info('Clean cookie sessions');
}
private async validateJwtToken(token: string) {
const decoded: any = token && jwt.decode(token, { complete: true });
if (!decoded) throw new Error('Invalid token');
const { kid } = decoded.header;
// Busca a chave pública
const { keys } = await this.getPublicKeys();
const pemValue = keys.find((k) => k.kid === kid)?.pem;
if (!pemValue) throw new Error('Public key not found');
jwt.verify(token, pemValue);
return decoded.payload;
}
private async getUserfromPayload(payload: JwtPayload): Promise<UserDTO> {
this.logger.info('getUser');
const metadata = PackTheMetadata({
customer_id: payload.customer_id,
});
const { user } = await lastValueFrom(
this.userService.UserFindOneById({ id: payload.user_id }, metadata),
);
const userDto: UserDTO = {
id: user.id,
name: user.username,
jobTitle: user?.jobTitle || null,
department: user?.department || null,
hierarchy: user?.hierarchy || null,
customer: {
id: payload.customer_id,
name: payload.customer_name,
tier: payload.customer_tier,
},
};
return userDto;
}
private validateResourceAccess(host: string, user: UserDTO) {
this.logger.info(
"Validate whether the source URL is a resource belonging to the user's client",
);
this.logger.info('Host: ' + host);
this.logger.info('Customer: ' + user.customer.name);
const hostParts = host.split('.');
const domain = hostParts[0];
const isResouceStg = hostParts[1] === 'stg';
const notFoundCustomerInDomain = !domain.includes('-')
if (notFoundCustomerInDomain) {
this.logger.info(`Not found Customer Name in domain`);
return;
}
const domainParts = domain.split('-');
const customerInDomain = domainParts[domainParts.length - 1];
if (isResouceStg && process.env.ENV !== 'stg') {
this.logger.error(`Customer ${user.customer.name} cannot access ${host}`);
throw new HttpException(
`Customer ${user.customer.name} cannot access ${host}`,
HttpStatus.FORBIDDEN
);
}
if (customerInDomain != user.customer.name) {
this.logger.error(`Customer ${user.customer.name} cannot access ${host}`);
throw new HttpException(
`Customer ${user.customer.name} cannot access ${host}`,
HttpStatus.FORBIDDEN
);
}
return;
}
}
+36 -1
View File
@@ -67,16 +67,28 @@ export class AuthUser {
export class AuthCustomer {
@ApiProperty()
modules: string[];
@ApiProperty()
id: string;
@ApiProperty()
name: string;
@ApiProperty()
displayName: string;
@ApiProperty()
tier: string;
@ApiProperty()
scheduleLimit: string;
@ApiProperty()
links: Link[];
@ApiProperty()
themeEnabled: boolean;
@ApiProperty()
enforceMfa: boolean;
}
export class AuthSignInReq implements AuthSignInRequest {
@@ -110,7 +122,7 @@ export class AuthRefreshAccessTokenReq {
@ApiProperty()
refreshToken: string;
@ApiProperty()
customerName: string;
userId: string;
}
export class AuthRefreshAccessTokenRes {
@ApiProperty()
@@ -118,3 +130,26 @@ export class AuthRefreshAccessTokenRes {
@ApiProperty()
accessToken: string;
}
export interface BulkEditRequest {
users: string[];
}
export interface BulkEditResponse {
message: string;
successfulUsers: string[];
failedUsers: string[];
}
export type UserDTO = {
id: string,
name: string,
jobTitle?: string,
department?: string,
hierarchy?: string,
customer: {
id: string,
name: string,
tier: string,
}
}
@@ -26,6 +26,7 @@ export class GoogleLoginStrategy extends PassportStrategy(
callbackURL: oauthSecrets['google-login'].redirect_uri,
scope: ['email', 'profile', 'openid'],
};
console.log("GoogleLoginStrategy", options.clientID, options.callbackURL);
const verify = (
accessToken: string,
refreshToken: string,
+3
View File
@@ -5,6 +5,7 @@ import {
} from '@nestjs/microservices';
import { credentials } from '@grpc/grpc-js';
import { Catalog } from '@dadosfera/protospack-v2';
import { PlatformInterfaces } from '@dadosfera/protospack-v2';
const isLocalConnection =
process.env.PIFACTORY_URL.startsWith('pi-factory:') ||
@@ -19,11 +20,13 @@ export class CatalogClientConfiguration {
package: [
Catalog.ProtoPackages.ReadPackage,
Catalog.ProtoPackages.WritePackage,
PlatformInterfaces.ProtoPackages.WritePackage
],
credentials: isLocalConnection ? undefined : credentials.createSsl(),
protoPath: [
Catalog.ProtoPaths.ReadFilePath,
Catalog.ProtoPaths.WriteFilePath,
PlatformInterfaces.ProtoPaths.WriteFilePath
],
loader: {
keepCase: true,
+209 -9
View File
@@ -13,6 +13,9 @@ import {
Put,
Query,
UseFilters,
HttpException,
HttpStatus,
Res,
} from '@nestjs/common';
import {
ApiCreatedResponse,
@@ -23,12 +26,13 @@ import {
import {
Authenticated,
RequireAllPermissions,
RequireModule,
RequireSomePermission,
} from '../../decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { CatalogService } from './catalog.service';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { RequestUser, User } from 'src/decorators/user.decorator';
import {
BatchRemoveRlsRulesRequest,
@@ -54,7 +58,10 @@ import {
AddRlsRuleRequest,
GetNimbusDashboardsRequest,
GetRlsRulesRequest,
RegisterDatasetWithMetatadaRequest,
} from '@dadosfera/protospack-v2/dist/lib/Catalog/interfaces/messages';
import { Response } from 'express';
import { TypeParser } from 'src/utils/FileParser/parser-types';
@ApiTags('Catalog')
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@@ -110,6 +117,52 @@ export class CatalogController {
return res;
}
@Get('/download')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.GET,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
async dowloadAsserts(
@User() user: RequestUser,
@Query() query: ICatalogAllRequest,
@Res() res: Response
) {
const { user_id, customer_name, customer_id, username, permissions } = user;
this.logger.info(`/catalog/download - searchCatalog`, {
user_id,
customer_name,
});
const is_data_manager = permissions.includes(
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER.seqid,
);
const roles = await this.catalogService.getUserRolesIds(user_id);
const metadata = PackTheMetadata({
user_id,
customer_id,
customer_name,
username,
roles,
is_data_manager,
});
const {
file,
filename
} = await this.catalogService.downloadAssets(
query,
metadata,
customer_id,
);
res.setHeader('Content-Disposition', `attachment; filename="${filename}"`);
res.setHeader('Content-Type', 'text/csv');
res.end(file);
}
@ApiInternalOnlyEndpoint()
@Get('data-asset')
async findByPipelineAndObject(@User() user: RequestUser, @Query() query) {
@@ -340,7 +393,7 @@ export class CatalogController {
@Language() language: LanguageEnum,
@Param('id') id: string,
): Promise<IPreviewResponse> {
const { customer_name, customer_id, user_id, username } = user;
const { customer_name, customer_id, user_id, username, customer_modules } = user;
this.logger.info(`/catalog - ON GET DATA DOCS ROUTE`, {
user_id,
@@ -353,6 +406,7 @@ export class CatalogController {
user_id,
username,
language,
is_mask: customer_modules.some(mod => mod === 'pii')
});
const preview = await this.catalogService.getDatasetPreview(id, metadata);
@@ -369,6 +423,7 @@ export class CatalogController {
@User() user: RequestUser,
@Language() language: LanguageEnum,
@Param('id') id: string,
@Query('asset_type') asset_type: string,
): Promise<IDocsResponse> {
const { customer_name, customer_id, user_id, username } = user;
@@ -385,7 +440,7 @@ export class CatalogController {
language,
});
const docs = await this.catalogService.getDataDocs(id, metadata);
const docs = await this.catalogService.getDataDocs(id, asset_type, metadata);
return { docs };
}
@@ -433,21 +488,32 @@ export class CatalogController {
@Headers() headers,
@Param('id') table_id: string,
@Body('docs') docs: string,
@Query('asset_type') asset_type: string,
) {
const { user_id, customer_name } = user;
const { user_id, customer_name, customer_id, username } = user;
const metadata = PackTheMetadata({
customer_id,
customer_name,
user_id,
username,
});
this.logger.info(`/catalog - ON GET DATA DOCS ROUTE`, {
this.logger.info(`/catalog - ON POST DATA DOCS ROUTE`, {
user_id,
customer_name,
});
const res = await this.catalogService.createDataDocs({
const body = {
table_id,
docs,
asset_type,
info: {
customer: customer_name,
},
});
}
const res = await this.catalogService.createDataDocs(body, metadata);
return res;
}
@@ -743,4 +809,138 @@ export class CatalogController {
return JSON.parse(dashboards);
}
}
@Post('register-dataset')
@RequireSomePermission(
PERMISSIONS_GROUPS.CATALOG.permissions.CREATE,
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
)
async registerDatasetWithMetadataRequest(
@Body() body: RegisterDatasetWithMetatadaRequest,
@User() user: RequestUser,
) {
const { customer_id, customer_name, user_id, username } = user;
const logMetadata = {
customer_name: customer_name,
user_id: user_id,
method: 'POST',
path: '/catalog/register-dataset',
};
try {
const metadata = PackTheMetadata({
customer_id,
customer_name,
user_id,
username,
});
this.logger.log(
`Request from user ${user_id} for customer ${customer_name}`,
logMetadata,
);
// Create table metadata
const tableMetadataBody = {
table_metadata: body.table_metadata,
info: {
customer: customer_name,
},
logMetadata: logMetadata,
};
const table_metadata_id = await this.catalogService.createTableMetadata(
tableMetadataBody,
);
this.logger.info(`table_metadata_id: ${table_metadata_id}`, logMetadata);
// Create column metadata
const columnMetadataBody = {
column_metadata: body.column_metadata,
info: {
customer: customer_name,
},
};
this.logger.info(
`Creating column metadata for table ${table_metadata_id}`,
logMetadata,
);
const column_metadata_ids =
await this.catalogService.createColumnMetadata(columnMetadataBody);
// Create data preview
const dataPreviewBody = {
data_preview: body.data_preview,
info: {
customer: customer_name,
},
};
this.logger.debug(
`Creating data preview for table ${table_metadata_id}`,
logMetadata,
);
const data_preview_id = await this.catalogService.createDataPreview(
dataPreviewBody,
);
// Catalog dataset item
this.logger.info(
`Cataloging dataset item for table ${table_metadata_id}`,
logMetadata,
);
await this.catalogService.catalogDatasetItem(table_metadata_id, metadata);
this.logger.info(
`Dataset registration completed successfully for table ${table_metadata_id}`,
logMetadata,
);
return {
message: 'Dataset registered successfully',
table_metadata_id: table_metadata_id,
column_metadata_ids: column_metadata_ids,
data_preview_id: data_preview_id,
};
} catch (error) {
this.logger.error(
`Failed to register dataset. The following error occurred: ${error.response.data}`,
logMetadata,
);
throw new HttpException(
{
message: 'Ocorreu um erro ao registrar o dataset',
error: error.message,
code: 'REGISTRATION_FAILED',
details: error.message,
},
HttpStatus.INTERNAL_SERVER_ERROR,
);
}
}
@Get('pii-reporter')
@RequireSomePermission(
PERMISSIONS_GROUPS.USERS.permissions.ADMIN
)
@RequireModule(
DADOSFERA_MODULES_KEYS.PII
)
async getPiiReporter(@User() user: RequestUser, @Res() res: Response, @Query('type') contentType: TypeParser = "pdf") {
this.logger.info('GET pii-reporter');
const metadata = PackTheMetadata(user);
try {
const {
file,
filename,
type
} = await this.catalogService.getPiiReporter(metadata, contentType);
res.setHeader('Content-Disposition', `attachment; filename="${filename}"`);
res.setHeader('Content-Type', type);
// use res.end to send buffer
return res.end(file);
} catch (error) {
console.error(error)
this.logger.error(error.message);
}
}
}
+6 -1
View File
@@ -3,12 +3,15 @@ import { Module } from '@nestjs/common';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { CatalogController } from './catalog.controller';
import { CatalogService } from './catalog.service';
import { CatalogClientConfiguration } from './catalog-client';
import { ClientsModule } from '@nestjs/microservices';
import { PipelinesModule as OldPipelineModule } from 'src/modules/pipelines/pipelines.module';
import { UsersModule } from '../users/users.module';
import { RolesModule } from '../roles/roles.module';
import { CustomersModule } from '../customers/customers.module';
import { ShareModule } from './share/share.module';
import { CatalogService } from './catalog.service';
import { MixpanelModule } from '../mixpanel/mixpanel.module';
const client = new CatalogClientConfiguration();
@@ -18,6 +21,8 @@ const client = new CatalogClientConfiguration();
OldPipelineModule,
UsersModule,
RolesModule,
CustomersModule,
ShareModule,
],
controllers: [CatalogController],
providers: [CatalogService, DadosferaLogger],
+310 -10
View File
@@ -5,6 +5,11 @@ import {
ProtoServices,
Messages,
} from '@dadosfera/protospack-v2/dist/lib/Catalog';
import {
Messages as PlatformInterfaceMessages,
WriteService as PlatformInterfaceWriteService,
ProtoServices as PlatformInterfacesProtoServices,
} from '@dadosfera/protospack-v2/dist/lib/PlatformInterfaces';
import {
BadRequestException,
HttpException,
@@ -19,8 +24,11 @@ import { CatalogClientConfiguration } from './catalog-client';
import { UsersService } from '../users/users.service';
import { RolesService } from '../roles/roles.service';
import { Metadata } from '@grpc/grpc-js';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import {
AssetReporter,
BatchRemoveRlsRulesRequest,
CreateDataDocsDTO,
IUpdateDataRequest,
TriggerCatalogReq,
} from './dtos';
@@ -28,11 +36,16 @@ import {
AddRlsRuleRequest,
GetNimbusDashboardsRequest,
GetRlsRulesRequest,
PiiMetadata,
} from '@dadosfera/protospack-v2/dist/lib/Catalog/interfaces/messages';
import { TypeParser } from 'src/utils/FileParser/parser-types';
import { ParserBuilder } from 'src/utils/FileParser/parser.builder';
class CatalogService implements OnModuleInit {
catalogReadService: ReadService.CatalogReadServices;
catalogWriteService: WriteService.CatalogWriteServices;
platformWriteService: PlatformInterfaceWriteService.PlatformInterfacesWriteServices;
logger: any;
constructor(
@Inject(DadosferaLogger)
@@ -45,6 +58,7 @@ class CatalogService implements OnModuleInit {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.catalogReadService =
this.grpcClient.getService<ReadService.CatalogReadServices>(
@@ -54,33 +68,87 @@ class CatalogService implements OnModuleInit {
this.grpcClient.getService<WriteService.CatalogWriteServices>(
ProtoServices.CatalogWriteServices,
);
this.platformWriteService =
this.grpcClient.getService<PlatformInterfaceWriteService.PlatformInterfacesWriteServices>(
PlatformInterfacesProtoServices.PlatformInterfacesWriteServices,
);
}
_getNimbusUrl(body) {
this.logger.debug(`Body: ${JSON.stringify(body)}`);
const customer = body.info.customer.toLowerCase();
if (process.env.ENV === 'prd') {
return `https://nimbus-${customer}.dadosfera.ai`;
}
return `https://nimbus-${customer}.${process.env.ENV.replace(
'local',
'stg',
)}.dadosfera.ai`;
}
async getPiiReporter(metadata: Metadata, type: TypeParser) {
this.logger.info('getPiiReporter: ' + type)
try {
const {
data
} = await lastValueFrom(
this.catalogWriteService.GetPiiReporter({}, metadata)
)
this.logger.info("Finish grpc call")
const parser = ParserBuilder.build<PiiMetadata>(type);
this.logger.info('parser file to: ' + type)
const file = await parser.parse(data)
this.logger.info('finish parser')
const mimeTypes: Record<TypeParser, string> = {
'csv': 'text/csv',
'html': 'text/html',
'pdf': 'application/pdf'
}
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `relatorio-pii-${timestamp}.${type}`;
return {
file,
filename: filename,
type: mimeTypes[type]
}
} catch (error) {
this.logger.error(error.message);
throw error;
}
}
async createDataAsset(data: Messages.CreateDataAssetRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
if (!data.embed) data.embed = undefined;
return lastValueFrom(
this.catalogWriteService.CreateDataAsset(data, metadata),
);
}
async managePermissions(data: Messages.ManagePermissionRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
return lastValueFrom(
this.catalogWriteService.ManagePermission(data, metadata),
).catch((err) => {
@@ -91,9 +159,11 @@ class CatalogService implements OnModuleInit {
});
}
async revokePermissions(data: Messages.RevokePermissionRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
return lastValueFrom(
this.catalogWriteService.RevokePermission(data, metadata),
).catch((err) => {
@@ -104,9 +174,11 @@ class CatalogService implements OnModuleInit {
});
}
async commentOnDataAsset(data: Messages.MakeACommentRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
return lastValueFrom(
this.catalogWriteService.MakeAComment(data, metadata),
).catch((err) => {
@@ -117,9 +189,11 @@ class CatalogService implements OnModuleInit {
});
}
async deleteComment(data: Messages.UpdateACommentRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
return lastValueFrom(
this.catalogWriteService.UpdateAComment(data, metadata),
).catch((err) => {
@@ -130,9 +204,11 @@ class CatalogService implements OnModuleInit {
});
}
async deleteDataAsset(data: Messages.DeleteDataAssetRequest, metadata) {
this.logger.info('CatalogService - Manage Data assets permissions');
return lastValueFrom(
this.catalogWriteService.DeleteDataAsset(data, metadata),
).catch((err) => {
@@ -143,14 +219,20 @@ class CatalogService implements OnModuleInit {
});
}
async getUserRolesIds(userId: string) {
const result = await this.userService.findOneById(userId).catch(() => null);
const roles_ids = result.user.roles.map((role) => role.id);
return roles_ids;
if (result) {
return result.user.roles.map((role) => role.id);
}
return [];
}
async searchDataAssets(
query: Record<string, any>,
metadata: Metadata,
@@ -158,6 +240,7 @@ class CatalogService implements OnModuleInit {
) {
this.logger.info('CatalogService - searchDataAssets');
const { search, page, size, sort_by, order, ...filters } = query;
const { data_assets, total } = await lastValueFrom(
@@ -174,16 +257,54 @@ class CatalogService implements OnModuleInit {
),
);
const result = JSON.parse(data_assets);
const response = await this.getAssetsUsersAndRoles(
result.data_assets,
customer_id,
);
return { data_assets: response, total };
}
async downloadAssets(
query: Record<string, any>,
metadata: Metadata,
customer_id: string,
) {
const data = await this.searchDataAssets(query, metadata, customer_id);
const formatData = data.data_assets.map(asset => ({
id: asset.id,
display_name: asset.display_name,
data_asset_type: asset.data_asset_type,
created_at: asset.created_at,
tags: '[' + asset.tags.join(', ') + ']'
}))
const parser = ParserBuilder.build<AssetReporter>('csv');
const file = await parser.parse(formatData);
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `dadosfera_assets_${timestamp}.csv`;
return {
file,
filename
}
}
async getOneDataAsset(data: {
id: string;
customer_id: string;
@@ -204,9 +325,11 @@ class CatalogService implements OnModuleInit {
};
asset = await this.getAssetsUsersAndRoles([asset], customer_id);
return { data_asset: asset[0] };
}
async getOneDataAssetByPipelineAndObject(data: {
customer_id: string;
pipeline: string;
@@ -228,9 +351,11 @@ class CatalogService implements OnModuleInit {
};
asset = await this.getAssetsUsersAndRoles([asset], customer_id);
return { data_asset: asset[0] };
}
async updateOneDataAsset(data: {
data_asset_id: string;
customer_id: string;
@@ -239,6 +364,7 @@ class CatalogService implements OnModuleInit {
}) {
const { body, customer_id, data_asset_id, metadata } = data;
const { data_asset } = await lastValueFrom(
this.catalogWriteService.UpdateDataAsset(
{ id: data_asset_id, changes: JSON.stringify(body) },
@@ -253,18 +379,20 @@ class CatalogService implements OnModuleInit {
};
asset = await this.getAssetsUsersAndRoles([asset], customer_id);
return { data_asset: asset[0] };
}
async getDataDocs(id: string, metadata: Metadata) {
async getDataDocs(id: string, assetType: string, metadata: Metadata) {
const { documentation } = await lastValueFrom(
this.catalogReadService.GetDatasetDoc({ id, type: undefined }, metadata),
this.catalogReadService.GetDatasetDoc({ id }, metadata),
);
console.log(documentation);
const docs = JSON.parse(documentation);
return docs;
}
async getDatasetPreview(id: string, metadata: Metadata) {
const { preview } = await lastValueFrom(
this.catalogReadService.GetDatasetPreview(
@@ -276,6 +404,7 @@ class CatalogService implements OnModuleInit {
return result;
}
async getDatasetColumnsMetadata(id: string, metadata: Metadata) {
const { columns_metadata } = await lastValueFrom(
this.catalogReadService.GetDatasetColumnsMetadata(
@@ -287,7 +416,16 @@ class CatalogService implements OnModuleInit {
return result;
}
async createDataDocs(body) {
async createDataDocs(body: CreateDataDocsDTO, metadata: Metadata) {
if (body.asset_type === 'table' || body.asset_type === 'view') {
return this.createDataDocsViaNimbus(body);
}
return this.createDataDocsViaGrpc(body, metadata);
}
private async createDataDocsViaNimbus(body: CreateDataDocsDTO) {
this.logger.info('Creating data docs via Nimbus for table/view');
const nimbusUrl = this._getNimbusUrl(body);
const { data } = await axios.post(
`${nimbusUrl}/api/catalog/data-docs/`,
@@ -296,9 +434,35 @@ class CatalogService implements OnModuleInit {
return data;
}
private async createDataDocsViaGrpc(body: CreateDataDocsDTO, metadata: Metadata) {
this.logger.info('Creating data docs via gRPC for other asset types');
try {
const response: any = await lastValueFrom(
this.catalogWriteService.UpdateDataAssetDoc(
{
id: body.table_id,
docs: body.docs,
},
metadata,
),
);
return response;
} catch (error) {
this.logger.error('Error creating data asset docs:', error);
throw new HttpException(
'Failed to create data asset documentation',
HttpStatus.INTERNAL_SERVER_ERROR,
);
}
}
async findAllTags(data, metadata) {
this.logger.info('CatalogService - findAllCustomerTags');
const response = await lastValueFrom(
this.catalogReadService.GetCustomerTags(data, metadata),
)
@@ -311,6 +475,7 @@ class CatalogService implements OnModuleInit {
throw new Error(err);
});
return response;
}
async getAssetsUsersAndRoles(data_assets: Array<any>, customer_id: string) {
@@ -323,17 +488,22 @@ class CatalogService implements OnModuleInit {
return data_assets.map((data_asset) => {
const owner = customer_users.find(
(u) => u.id === data_asset.owner,
)?.username;
)?.email;
const roles = [];
const users = [];
for (const role_id of data_asset.roles) {
const data_asset_roles = data_asset?.roles || []
for (const role_id of data_asset_roles) {
const role = customer_roles.find((r) => r.id === role_id);
if (role) roles.push({ id: role.id, name: role.name });
}
for (const user_id of data_asset.users) {
const data_asset_users = data_asset?.users || []
for (const user_id of data_asset_users) {
const user = customer_users.find((r) => r.id === user_id);
if (user) users.push({ id: user.id, username: user.username });
if (user) users.push({ id: user.id, email: user.email });
}
return {
...data_asset,
@@ -344,6 +514,7 @@ class CatalogService implements OnModuleInit {
});
}
async triggerCatalog(data: TriggerCatalogReq, metadata: Metadata) {
const { session } = await lastValueFrom(
this.catalogWriteService.TriggerDatasetCataloging(data, metadata),
@@ -357,6 +528,7 @@ class CatalogService implements OnModuleInit {
return res;
}
async addRlsRule(data: AddRlsRuleRequest, metadata: Metadata) {
const res = await lastValueFrom(
this.catalogWriteService.AddRlsRule(data, metadata),
@@ -364,6 +536,7 @@ class CatalogService implements OnModuleInit {
return res;
}
async removeRlsRule(id: number, metadata: Metadata) {
const res = await lastValueFrom(
this.catalogWriteService.RemoveRlsRule({ id }, metadata),
@@ -371,12 +544,14 @@ class CatalogService implements OnModuleInit {
return res;
}
async batchRemoveRlsRule(
query: BatchRemoveRlsRulesRequest,
metadata: Metadata,
) {
const { id_rls, nimbus_dashboard_id } = query;
if (id_rls && nimbus_dashboard_id) {
throw new BadRequestException(
"You can't delete using both parameters. Choose either 'id_rls' or 'nimbus_dashboard_id'",
@@ -397,6 +572,7 @@ class CatalogService implements OnModuleInit {
return 'OK';
}
async getRlsRules(data: GetRlsRulesRequest, metadata: Metadata) {
const res = await lastValueFrom(
this.catalogReadService.GetRlsRules(data, metadata),
@@ -404,6 +580,7 @@ class CatalogService implements OnModuleInit {
return res.rls_rules;
}
async getOneRlsRule(id: number, metadata: Metadata) {
const res = await lastValueFrom(
this.catalogReadService.GetOneRlsRule({ id }, metadata),
@@ -411,6 +588,7 @@ class CatalogService implements OnModuleInit {
return res.rls_rule;
}
async getNimbusDashboards(
data: GetNimbusDashboardsRequest,
metadata: Metadata,
@@ -420,6 +598,128 @@ class CatalogService implements OnModuleInit {
);
return res.dashboards;
}
async createTableMetadata(body: any): Promise<number> {
const nimbusUrl = this._getNimbusUrl(body);
this.logger.info(`Nimbus URL: ${nimbusUrl}`, {...body.logMetadata});
const endpoint = `${nimbusUrl}/api/catalog/table-metadata/`;
this.logger.info(`Creating table metadata for table ${body.table_metadata.table_name}`, {...body.logMetadata});
this.logger.info(`Using endpoint: ${endpoint}`, {...body.logMetadata});
this.logger.debug(`Payload: ${JSON.stringify(body.table_metadata)}`, {...body.logMetadata});
try {
const { data, status } = await axios.post(endpoint, {...body.table_metadata});
this.logger.info(
`Table metadata created successfully with status ${status} for table ${body.table_metadata.table_name}`,
{...body.logMetadata},
);
return data.id;
} catch (error) {
this.logger.error(
`Failed to create table metadata for table ${body.table_metadata.table_name} failed with status ${
error.response?.status
} because of ${JSON.stringify(error.response?.data) || error.message}`, {...body.logMetadata});
throw new Error(error.response?.data?.message || error.message);
}
}
async createColumnMetadata(body: any): Promise<number[]> {
const nimbusUrl = this._getNimbusUrl(body);
this.logger.info(`Nimbus URL: ${nimbusUrl}`, body.logMetadata);
const endpoint = `${nimbusUrl}/api/catalog/column-metadata/`;
try {
this.logger.info(`Creating column metadata for table ${body.column_metadata.table_name}`, {...body.logMetadata});
this.logger.info(`Using endpoint: ${endpoint}`, {...body.logMetadata});
this.logger.debug(`Payload: ${JSON.stringify(body.column_metadata)}`, {...body.logMetadata});
const { data, status } = await axios.post(endpoint, body.column_metadata);
this.logger.info(
`Column metadata created successfully with status ${status} for table ${body.column_metadata.table_name}`,
{...body.logMetadata},
);
return data.map((column) => column.id);
} catch (error) {
this.logger.error(
`Failed to create column metadata failed with status for table ${body.column_metadata.table_name} ${
error.response?.status
} because of ${error.response?.data || error.message}`, {...body.logMetadata});
throw new Error(error.response?.data?.message || error.message);
}
}
async createDataPreview(body: any): Promise<number> {
const nimbusUrl = this._getNimbusUrl(body);
this.logger.info(`Nimbus URL: ${nimbusUrl}`, {...body.logMetadata});
const endpoint = `${nimbusUrl}/api/catalog/data-preview/`;
this.logger.info(`Creating data preview for table ${body.data_preview.table_name}`, {...body.logMetadata});
this.logger.info(`Using endpoint: ${endpoint}`, {...body.logMetadata});
this.logger.debug(`Payload: ${JSON.stringify(body.data_preview)}`, {...body.logMetadata});
try {
const { data, status } = await axios.post(endpoint, body.data_preview);
this.logger.info(
`Data preview created successfully with status ${status} for table ${body.data_preview.table_name}`,
{...body.logMetadata},
);
return data.id;
} catch (error) {
this.logger.error(
`Failed to create data preview for table ${body.data_preview.table_name} failed with status ${
error.response?.status
} because of ${error.response?.data || error.message}`,
{...body.logMetadata},
);
throw new Error(error.response?.data?.message || error.message);
}
}
async catalogDatasetItem(table_metadata_id: number, metadata: Metadata) {
const customer_name_raw = metadata.get('customer_name');
const customer_name = customer_name_raw?.[0]?.toString();
if (!customer_name) {
throw new BadRequestException('Customer name not found in metadata');
}
const res = await lastValueFrom(
this.platformWriteService.CatalogDataAssets(
{
data_assets: [
{
data_asset_id: table_metadata_id.toString(),
customer_name: customer_name,
data_asset_type: 'dataset',
},
],
},
metadata,
),
);
return res;
}
}
export { CatalogService };
+33
View File
@@ -147,6 +147,24 @@ export class ICatalogAllRequest {
description: 'Tipo de ordenação - `asc`: crescente; `desc`: decrescente ',
})
order?: OrderEnum;
@ApiPropertyOptional({
description: 'ID do usuário owner para filtrar data assets',
example: 'user-id-1,user-id-2',
})
owner?: string;
@ApiPropertyOptional({
description: 'Data inicial para filtro de catálogo (formato: YYYY-MM-DD)',
example: '2025-01-01',
})
catalog_date_from?: string;
@ApiPropertyOptional({
description: 'Data final para filtro de catálogo (formato: YYYY-MM-DD)',
example: '2025-12-31',
})
catalog_date_to?: string;
}
export class ICatalogAllResponse {
@@ -320,3 +338,18 @@ export class BatchRemoveRlsRulesRequest {
@ApiPropertyOptional()
id_rls?: string;
}
export type AssetReporter = {
id: string;
display_name: string;
data_asset_type: string;
created_at: string;
tags: string;
}
export type CreateDataDocsDTO = {
table_id: string;
docs: string;
asset_type: string;
}
+8
View File
@@ -0,0 +1,8 @@
export class PiiDto {
database_name: string;
table_schema: string;
table_name: string;
column_name: string;
data_type: string;
pii_rules: string;
}
@@ -0,0 +1,89 @@
import {
Controller,
Get,
Inject,
Param,
Req,
UseFilters,
} from '@nestjs/common';
import {
ApiHeaders,
ApiTags,
} from '@nestjs/swagger';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { RequestUser, User } from 'src/decorators/user.decorator';
import {
IColumnsMetadataResponse,
IDocsResponse,
IPreviewResponse,
} from '../dtos';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { Language } from 'src/decorators/language.decorator';
import { LanguageEnum } from 'src/utils/languages.enum';
import { ShareService } from './share.service';
import { Request } from 'express';
@ApiTags('Catalog')
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@Controller('catalog/data-asset/share')
@UseFilters(new GrpcToHttpExceptionFilter())
export class ShareController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private catalogShareService: ShareService,
) {
this.logger = dadosferaLogger.logger;
}
@Get('/:id')
async getShareDataAsset(
@Param('id') id: string,
@Req() request: Request
) {
this.logger.info(`GET //:id`);
return await this.catalogShareService.getOneDataAssetPublic(id, request);
}
@Get('/:id/columns-metadata')
async getShareDataAssetColumnsMetadata(
@Language() language: LanguageEnum,
@Param('id') id: string,
@Req() request: Request
): Promise<IColumnsMetadataResponse> {
this.logger.info(`GET /:id/columns-metadata`);
const columns_metadata =
await this.catalogShareService.getDatasetColumnsMetadata(id, request);
return { columns_metadata };
}
@Get('/:id/preview')
async getShareDataAssetPreview(
@Language() language: LanguageEnum,
@Param('id') id: string,
@Req() request: Request
): Promise<IPreviewResponse> {
this.logger.info(`GET /:id/preview`);
const preview = await this.catalogShareService.getDatasetPreview(id, request);
return { preview };
}
@Get('/:id/docs')
async getShareDataAssetDocs(
@Language() language: LanguageEnum,
@Param('id') id: string,
@Req() request: Request
): Promise<IDocsResponse> {
this.logger.info(`GET /:id/docs`);
const docs = await this.catalogShareService.getDataDocs(id, request);
return { docs };
}
}
+30
View File
@@ -0,0 +1,30 @@
import { Module } from "@nestjs/common";
import { CatalogClientConfiguration } from "../catalog-client";
import { ClientsModule } from "@nestjs/microservices";
import { RolesModule } from "src/modules/roles/roles.module";
import { UsersModule } from "src/modules/users/users.module";
import { CustomersModule } from "src/modules/customers/customers.module";
import { ShareMetadataModule } from "src/modules/share-metadata/share-metadata.module";
import { ShareController } from "./share.controller";
import DadosferaLogger from "@dadosfera/dadosfera-logs";
import { ShareService } from "./share.service";
import { MixpanelModule } from "src/modules/mixpanel/mixpanel.module";
import { AuthModule } from "src/modules/auth/auth.module";
const client = new CatalogClientConfiguration();
@Module({
imports: [
ClientsModule.register([client.providerOptions]),
UsersModule,
RolesModule,
CustomersModule,
ShareMetadataModule,
MixpanelModule,
AuthModule
],
controllers: [ShareController],
providers: [ShareService, DadosferaLogger],
exports: [ShareModule],
})
export class ShareModule {}
+275
View File
@@ -0,0 +1,275 @@
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import {
ProtoServices,
ReadService,
} from '@dadosfera/protospack-v2/dist/lib/Catalog';
import {
ForbiddenException,
Inject,
NotFoundException,
OnModuleInit,
} from '@nestjs/common';
import { CatalogClientConfiguration } from '../catalog-client';
import { ClientGrpc } from '@nestjs/microservices';
import { UsersService } from 'src/modules/users/users.service';
import { RolesService } from 'src/modules/roles/roles.service';
import { RequestUser } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { Metadata } from '@grpc/grpc-js';
import { lastValueFrom } from 'rxjs';
import { ShareMetadataService } from 'src/modules/share-metadata/share-metadata.service';
import { isJWT } from 'class-validator';
import { MixpanelService } from 'src/modules/mixpanel/mixpanel.service';
import { Request } from 'express';
import jwt from 'jsonwebtoken';
import { AuthClientService } from 'src/modules/auth/auth.service';
export class ShareService implements OnModuleInit {
catalogReadService: ReadService.CatalogReadServices;
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
@Inject(CatalogClientConfiguration.name)
private readonly grpcClient: ClientGrpc,
private readonly userService: UsersService,
private readonly roleService: RolesService,
private readonly shareMetadataService: ShareMetadataService,
private readonly mixpanelService: MixpanelService,
private authClient: AuthClientService,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.catalogReadService =
this.grpcClient.getService<ReadService.CatalogReadServices>(
ProtoServices.CatalogReadServices,
);
}
async getDatasetColumnsMetadata(id: string, request: Request) {
const shareMetadata = await this.getShareMetadata(id, request);
const metadata = PackTheMetadata({
customer_id: shareMetadata.customerId,
customer_name: shareMetadata.customerName,
});
const { columns_metadata } = await lastValueFrom(
this.catalogReadService.GetDatasetColumnsMetadata(
{ id: shareMetadata.assetId, type: undefined },
metadata,
),
);
const result = JSON.parse(columns_metadata);
return result;
}
async getDatasetPreview(id: string, request: Request) {
const shareMetadata = await this.getShareMetadata(id, request);
const metadata = PackTheMetadata({
customer_id: shareMetadata.customerId,
customer_name: shareMetadata.customerName,
});
const { preview } = await lastValueFrom(
this.catalogReadService.GetDatasetPreview(
{ id: shareMetadata.assetId, type: undefined },
metadata,
),
);
const result = JSON.parse(preview);
return result;
}
async getOneDataAssetPublic(id: string, request: Request) {
this.logger.info("getOneDataAssetPublic: " + JSON.stringify({
id
}))
try {
const user = await this.getUserFromRequest(request);
const shareMetadata = await this.getShareMetadata(id, request);
const mixpanelTracker = {
asset: shareMetadata.assetId,
type: isJWT(id) ? 'assigned' : shareMetadata.type,
customer: shareMetadata.customerName
}
if (user) {
await this.mixpanelService.track("share_page", user, request, mixpanelTracker);
} else {
await this.mixpanelService.trackShare(request, mixpanelTracker);
}
this.logger.info("shareMetadata: " + JSON.stringify(shareMetadata))
const metadata = PackTheMetadata({
customer_id: shareMetadata.customerId,
customer_name: shareMetadata.customerName,
});
const { data_asset } = await this.getOneDataAsset({
customer_id: shareMetadata.customerId,
id: shareMetadata.assetId,
metadata,
});
this.logger.info('found asset: ' + JSON.stringify(data_asset));
delete data_asset.p_roles;
delete data_asset.p_users;
data_asset.share_type = 'public';
if (data_asset.share_type !== 'public') throw new NotFoundException();
return { data_asset };
} catch (error) {
this.logger.error(error);
throw error;
}
}
private async getOneDataAsset(data: {
id: string;
customer_id: string;
metadata: Metadata;
}) {
const { customer_id, id, metadata } = data;
const { data_asset } = await lastValueFrom(
this.catalogReadService.GetOneDataAsset(
{ id, type: undefined },
metadata,
),
);
let asset = JSON.parse(data_asset);
asset = {
...asset,
p_roles: asset.roles,
p_users: asset.users,
};
asset = await this.getAssetsUsersAndRoles([asset], customer_id);
return { data_asset: asset[0] };
}
async getDataDocs(id: string, request: Request) {
const shareMetadata = await this.getShareMetadata(id, request);
const metadata = PackTheMetadata({
customer_id: shareMetadata.customerId,
customer_name: shareMetadata.customerName,
});
const { documentation } = await lastValueFrom(
this.catalogReadService.GetDatasetDoc({ id }, metadata),
);
console.log(documentation);
const docs = JSON.parse(documentation);
return docs;
}
private async getAssetsUsersAndRoles(
data_assets: Array<any>,
customer_id: string,
) {
const { users: customer_users } =
await this.userService.findAllUsersByCustomerId(customer_id);
const { roles: customer_roles } = await this.roleService.roleSearch(
{},
{ customer_id },
);
return data_assets.map((data_asset) => {
const owner = customer_users.find(
(u) => u.id === data_asset.owner,
)?.email;
const roles = [];
const users = [];
for (const role_id of data_asset.roles) {
const role = customer_roles.find((r) => r.id === role_id);
if (role) roles.push({ id: role.id, name: role.name });
}
for (const user_id of data_asset.users) {
const user = customer_users.find((r) => r.id === user_id);
if (user) users.push({ id: user.id, email: user.email });
}
return {
...data_asset,
roles,
users,
owner,
} as typeof data_asset;
});
}
private async getShareMetadata(id: string, request: Request) {
const metadata = PackTheMetadata({});
this.logger.info('GET share metadata')
const info = await this.shareMetadataService.get(id, metadata);
if (isJWT(id) && info ){
return info;
}
const user = await this.getUserFromRequest(request);
if (info.type === 'private') {
if (!user) {
throw new ForbiddenException(
'You do not have permission to access this data asset.',
);
}
const is_data_manager = user.permissions.includes(
PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER.seqid,
);
const is_get = user.permissions.includes(
PERMISSIONS_GROUPS.CATALOG.permissions.GET.seqid,
);
if (is_data_manager || is_get) {
return info;
}
throw new ForbiddenException(
'You do not have permission to access this data asset.',
);
}
return info;
}
private async getUserFromRequest(request: Request): Promise<RequestUser | null> {
const accessToken = request.get('Authorization');
if (accessToken) {
const accessTokenDecoded: any = jwt.decode(accessToken, {
complete: true,
});
const { kid } = accessTokenDecoded.header;
const { keys } = await this.authClient.getPublicKeys();
const pemValue = keys.find((key) => key.kid === kid);
if (!pemValue) {
return null;
}
jwt.verify(accessToken, pemValue.pem);
const accessTokenPayload = accessTokenDecoded.payload;
return {
user_id: accessTokenPayload.user_id,
username: accessTokenPayload.username,
permissions: accessTokenPayload.permissions,
customer_id: accessTokenPayload.customer_id,
customer_name: accessTokenPayload.customer_name,
customer_tier: accessTokenPayload.customer_tier,
customer_modules: accessTokenPayload.customer_modules,
access_token: accessToken,
};
}
return null;
}
}
@@ -20,7 +20,7 @@ import {
DatabaseConnectionPropertiesDto,
} from '../connection/dtos/connection';
import { RequestUser } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
@Injectable()
export class ConnectionTestService {
@@ -28,7 +28,7 @@ import {
UpdateConnectionDto,
} from './dtos/connection';
import { CreateConnectionDto } from './dtos/connection';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { Language } from 'src/decorators/language.decorator';
import { LanguageEnum } from 'src/utils/languages.enum';
+47 -51
View File
@@ -1,11 +1,9 @@
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { IdResponse } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import {
Body,
Controller,
Get,
HttpCode,
HttpException,
HttpStatus,
Inject,
Param,
@@ -15,18 +13,20 @@ import {
UseFilters,
} from '@nestjs/common';
import { ApiOkResponse, ApiProduces, ApiTags } from '@nestjs/swagger';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import {
Authenticated,
RequireAllPermissions,
RequireModule,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { CustomersService } from './customers.service';
import { CustomerLinkRequest, CustomerLinksResponse, CustomerThemeRequest, CustomerThemeResponse } from './dtos/customers';
import { CustomerLinkRequest, CustomerLinksResponse } from './dtos/customers';
import { RequestUser, User } from 'src/decorators/user.decorator';
import type { StringValue } from 'ms';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import ErrorCodes from 'src/utils/errorCodes';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { EnforceMfa } from './dtos/enforce-mfa';
@ApiTags('Customers')
@Controller('customers')
@@ -42,6 +42,15 @@ export class CustomersController {
this.logger = dadosferaLogger.logger;
}
@Post(':id/mfa')
@Authenticated()
@RequireSomePermission(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@RequireModule(DADOSFERA_MODULES_KEYS.DANGER_ZONE)
async enableMfaEnforce(@Param('id') id: string, @Body() data: EnforceMfa) {
this.logger.info('enableMfaEnforce', { id });
return await this.customersService.enableEnforceMfa(id, data.enabled);
}
@Get(':id/links')
@Authenticated()
@ApiOkResponse({ type: CustomerLinksResponse })
@@ -51,51 +60,6 @@ export class CustomersController {
return { links };
}
@Post(':id/theme')
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@ApiOkResponse({ type: CustomerThemeResponse })
async saveCustomertheme(@Param('id') id: string, @Body() data: CustomerThemeRequest) {
this.logger.info('saveCustomertheme', JSON.stringify({
id, theme: data
}));
try {
// const theme = await this.customersService.createThemeByCustomer(id, data);
this.logger.info('saveCustomertheme', JSON.stringify(data));
return { theme: null };
} catch (err) {
if (err.details === ErrorCodes.CUSTOMER.NOT_FOUND) {
this.logger.error('Error - saveCustomertheme - Expect CUSTOMER.NOT_FOUND');
throw new HttpException(err.details, HttpStatus.NOT_FOUND);
} else {
this.logger.error('Error - saveCustomertheme Unknown Error:' + err?.message);
return { theme: null };
};
}
}
@Get(':id/theme')
@ApiOkResponse({ type: CustomerThemeResponse })
async getCustomerTheme(@Param('id') id: string) {
this.logger.info('getCustomerTheme with id' + id);
try {
const data = await this.customersService.getThemeByCustomer(id);
this.logger.info('Success - getCustomerTheme with id'+ id);
return data;
}catch (err) {
if (err.details === ErrorCodes.CUSTOMER.NOT_FOUND) {
this.logger.error('Error - getCustomerTheme - Expect CUSTOMER.NOT_FOUND');
throw new HttpException(err.details, HttpStatus.NOT_FOUND);
} else {
this.logger.error('Error - getCustomerTheme Unknown Error:' + err?.message);
return { theme: null };
};
}
}
@Put(':id/links')
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@@ -140,4 +104,36 @@ export class CustomersController {
const metadata = PackTheMetadata(user);
return this.customersService.getMonitoringDashboardUrl(metadata);
}
@Get('logs-dashboard')
@Authenticated()
@RequireAllPermissions(
PERMISSIONS_GROUPS.USERS.permissions.ADMIN,
)
@RequireModule(DADOSFERA_MODULES_KEYS.LOG_DASHBOARD)
async getCustomerMixPanelLogsDashboard(
@User() user: RequestUser,
): Promise<{ url: string }> {
this.logger.info('getLogsDashboardUrl');
const metadata = PackTheMetadata(user);
const result = await this.customersService.getLogsDashboardUrl(metadata);
return result;
}
@Get('access-dashboard')
@Authenticated()
@RequireAllPermissions(
PERMISSIONS_GROUPS.USERS.permissions.ADMIN,
)
@RequireModule(DADOSFERA_MODULES_KEYS.ACCESS_DASHBOARD)
async getAccessDashboard(
@User() user: RequestUser,
): Promise<{ url: string }> {
this.logger.info('getAccessDashboard');
const metadata = PackTheMetadata(user);
const result = await this.customersService.getAccessDashboardUrl(user.customer_name, metadata);
return result;
}
}
+68 -35
View File
@@ -5,6 +5,7 @@ import {
HttpException,
HttpStatus,
InternalServerErrorException,
ForbiddenException,
} from '@nestjs/common';
import { firstValueFrom, lastValueFrom } from 'rxjs';
@@ -28,12 +29,12 @@ import {
} from '@dadosfera/protospack-v2/dist/lib/PipelineV2';
import { Metadata } from '@grpc/grpc-js';
import { PipelinesClientConfiguration } from '../pipelinesV2/pipelines-client';
import { CustomerThemeRequest, CustomerThemeResponse } from './dtos/customers';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
// This function will accept any string, which may result in a bug.
@Injectable()
export class CustomersService implements OnModuleInit {
private customerService: CustomersProtoService;
private logger: DadosferaLogger;
private pipelineReadService: ReadService.PipelineV2ReadService;
@@ -58,6 +59,14 @@ export class CustomersService implements OnModuleInit {
);
}
async getCustomer(customerId: string) {
return await lastValueFrom(
this.customerService.CustomerFindOneById({
id: customerId
})
)
}
async getLinks(customerId: string) {
try {
const result = await lastValueFrom(
@@ -90,39 +99,6 @@ export class CustomersService implements OnModuleInit {
}
}
async createThemeByCustomer(id: string, theme: CustomerThemeRequest): Promise<CustomerThemeResponse> {
if (!id) {
this.logger.error('Error - saveCustomertheme - not found id:' + id);
throw new HttpException(null, HttpStatus.BAD_REQUEST);
}
const result = await firstValueFrom(
this.customerService.CustomerCreateTheme({
customerId: id,
theme
}),
);
return { theme: result };
}
async getThemeByCustomer(id: string): Promise<CustomerThemeResponse> {
if (!id) {
this.logger.error('Error - getCustomerTheme - not found id:' + id);
throw new HttpException(null, HttpStatus.BAD_REQUEST);
}
const result = await firstValueFrom(
this.customerService.CustomerGetTheme({
id
}),
);
return {
theme: result
}
}
async generateToken(
expiresIn = '30m',
data: { customerId: string; userId: string; customerName: string },
@@ -173,13 +149,14 @@ export class CustomersService implements OnModuleInit {
// const decoded = jwt.decode(jwt_token, { complete: true });
return jwt_token;
}
async getMonitoringDashboardUrl(metadata: Metadata) {
logger.info('CustomersService - getMonitoringDashboardUrl');
const res = await lastValueFrom(
this.pipelineReadService.PipelineV2GetDashboardUrl(
{
dashboard_id: '45',
dashboard_id: '98',
exp: '15m',
metabase_customer_name: 'dadosferatech',
},
@@ -190,4 +167,60 @@ export class CustomersService implements OnModuleInit {
return res;
}
async getLogsDashboardUrl(metadata: Metadata) {
logger.info('CustomersService - getMixPanelLogsDashboardUrl');
const res = await lastValueFrom(
this.pipelineReadService.PipelineV2GetDashboardUrl(
{
dashboard_id: '103',
exp: '15m',
metabase_customer_name: 'dadosferatech',
},
metadata,
),
);
logger.info('Done');
return res;
}
async getAccessDashboardUrl(customerName: string, metadata: Metadata) {
/*
* TODO(Refactor): dar um jeito de exibir o dash da sbm diferente dos outros customer
* pois o signicado de department para sbm significa as instituições do usuários
*/
if (customerName !== 'sbmoffshorecom') {
throw new ForbiddenException();
}
logger.info('CustomersService - getAccessDashboardUrl');
const res = await this.getDashboardUrl('105', metadata);
logger.info('Done');
return res;
}
private async getDashboardUrl(dashboardId: string, metadata: Metadata) {
return await lastValueFrom(
this.pipelineReadService.PipelineV2GetDashboardUrl(
{
dashboard_id: dashboardId,
exp: '15m',
metabase_customer_name: 'dadosferatech',
},
metadata
)
);
}
async enableEnforceMfa(id: string, enabled: boolean) {
return await lastValueFrom(
this.customerService.CustomerUpdateEnforceMfa({
customerId: id,
enforceMfa: enabled
})
)
}
}
+1 -27
View File
@@ -1,4 +1,4 @@
import { Link, Theme } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/entities';
import { Link } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/entities';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
export class CustomerLink implements Link {
@@ -21,29 +21,3 @@ export class CustomerLinksResponse {
links: CustomerLink[];
}
export class CustomerTheme implements Theme {
@ApiProperty()
backgroundColor: string;
@ApiProperty()
textColor: string;
@ApiProperty()
logoWhite: string;
@ApiProperty()
logoBlack: string;
}
export class CustomerThemeResponse {
@ApiProperty()
theme: CustomerTheme;
}
export class CustomerThemeRequest {
@ApiProperty()
backgroundColor: string;
@ApiProperty()
textColor: string;
@ApiProperty()
logoWhite: string;
@ApiProperty()
logoBlack: string;
}
@@ -0,0 +1,6 @@
import { ApiProperty } from "@nestjs/swagger";
export class EnforceMfa {
@ApiProperty()
enabled: boolean
}
+2
View File
@@ -29,6 +29,8 @@ export class DucClient {
objects: true,
arrays: true,
},
maxSendMessageLength: 15 * 1024 * 1024, // 15 MB por mensagem
maxReceiveMessageLength: 15 * 1024 * 1024,
},
};
@@ -0,0 +1,47 @@
import { ApiProperty } from "@nestjs/swagger";
export class CreateIdentityProvider {
@ApiProperty()
name: string;
@ApiProperty()
clientId: string;
@ApiProperty()
clientSecret: string;
@ApiProperty()
issuerUrl: string;
@ApiProperty()
permissions: number[];
}
export class IdentityProviderResponse {
@ApiProperty()
id: string;
@ApiProperty()
name: string;
@ApiProperty()
clientId: string;
@ApiProperty()
issueUrl: string;
@ApiProperty()
permissions: {
id: number;
name: string;
}[];
}
export class IdentityProviderListResponse {
@ApiProperty()
providers: IdentityProviderResponse[]
}
@@ -0,0 +1,10 @@
export class SsoSignInDto {
readonly nonce: string;
readonly codeVerifier: string;
readonly state: string;
readonly id: string;
readonly clientId: string;
readonly clientSecret: string;
readonly issuerUrl: string;
readonly redirectUrls: string[];
}
@@ -0,0 +1,246 @@
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import {
Body,
Controller,
Delete,
Get,
HttpCode,
HttpStatus,
Inject,
Param,
Post,
Put,
Redirect,
Req,
} from '@nestjs/common';
import { IdentityProviderService } from './identity-provider.service';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { Language } from 'src/decorators/language.decorator';
import { LanguageEnum } from 'src/utils/languages.enum';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { ApiOkResponse } from '@nestjs/swagger';
import {
CreateIdentityProvider,
IdentityProviderListResponse,
IdentityProviderResponse,
} from './dto/identity-provider.dto';
import { Request } from 'express';
import ErrorCodes from 'src/utils/errorCodes';
import {
Authenticated,
RequireModule,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
@Controller('identity-providers')
export class IdentityProviderController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private identityProviderService: IdentityProviderService,
) {
this.logger = dadosferaLogger.logger;
}
@Post()
@HttpCode(HttpStatus.OK)
@ApiOkResponse({ type: IdentityProviderResponse })
@Authenticated()
@RequireModule('sso')
@RequireSomePermission(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
async addIdentityProvider(
@User() user: RequestUser,
@Body() body: CreateIdentityProvider,
@Language() language: LanguageEnum,
) {
this.logger.info('POST /identity-providers');
const metadata = PackTheMetadata({
...user,
language,
});
return await this.identityProviderService.create(body, metadata);
}
@Get()
@HttpCode(HttpStatus.OK)
@ApiOkResponse({ type: IdentityProviderListResponse })
@Authenticated()
@RequireModule('sso')
@RequireSomePermission(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
async getProviders(
@User() user: RequestUser,
@Language() language: LanguageEnum,
) {
this.logger.info('GET identity-providers');
const metadata = PackTheMetadata({
...user,
language,
});
const result = await this.identityProviderService.getList(metadata);
return result;
}
@Delete(':id')
@HttpCode(HttpStatus.NO_CONTENT)
@RequireModule('sso')
@RequireSomePermission(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
async deleteIdentityProvider(
@Param('id') id: string,
@User() user: RequestUser,
) {
this.logger.info('DELETE /identity-providers');
const metadata = PackTheMetadata({
...user,
});
return await this.identityProviderService.deleteIdentityProvider(
id,
metadata,
);
}
@Put(':id')
@HttpCode(HttpStatus.OK)
@Authenticated()
@RequireModule('sso')
@RequireSomePermission(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
async updateIdentityProviders(
@Param('id') id: string,
@Body() body: CreateIdentityProvider,
@User() user: RequestUser,
) {
this.logger.info('PUT /identity-providers');
const metadata = PackTheMetadata({
...user,
});
return await this.identityProviderService.updateIdentityProviders(
id,
body,
metadata,
);
}
@Post('/callback')
@HttpCode(HttpStatus.OK)
async callbackIdp(
@Req() req: Request,
@Language() language: LanguageEnum,
@Body()
body: {
state: string;
code: string;
},
) {
this.logger.info('GET /identity-providers/callback');
const { code, state } = body;
if (!code) {
this.logger.error('No code received from IDP');
throw new Error(ErrorCodes.IDENTITY_PROVIDER.INVALID_RESPONSE);
}
if (!state) {
this.logger.error('No state received from IDP');
throw new Error(ErrorCodes.IDENTITY_PROVIDER.INVALID_RESPONSE);
}
try {
const origin = req.headers['origin'] as string;
this.logger.info('Header Origin: ' + origin);
const lang =
language.substring(0, 2) + language.substring(2).toUpperCase();
const callbackUrl =
process.env.ENV !== 'prd'
? `${origin}/auth/callback`
: `${origin}/${lang}/auth/callback`;
this.logger.info('Callback URL: ' + callbackUrl);
return await this.identityProviderService.getTokenByIdp(
code,
state,
callbackUrl,
);
} catch (error) {
this.logger.error(error);
throw error;
}
}
@Get('/links')
@HttpCode(HttpStatus.OK)
async providerLinks(@Req() req: Request) {
this.logger.info('GET /identity-providers/links');
try {
const frontDomain = req.headers['origin'] as string;
this.logger.info('Header Origin: ' + frontDomain);
if (!frontDomain) {
this.logger.info('Not found front domain');
throw new Error(ErrorCodes.IDENTITY_PROVIDER.INVALID_HEADER);
}
const result =
await this.identityProviderService.identityProvidersLinksPerDomain(
frontDomain,
);
return result;
} catch (error) {
this.logger.error(error);
throw error;
}
}
@Get(':id')
@HttpCode(HttpStatus.OK)
@Redirect()
async loginIdp(
@Param('id') id: string,
@Req() req: Request,
@Language() language: LanguageEnum,
) {
this.logger.info('GET /identity-providers/:id');
try {
const frontDomain =
(req.headers['origin'] as string) || (req.headers['referer'] as string);
this.logger.info(`Front domain: ${frontDomain}`);
const host =
frontDomain.lastIndexOf('/') !== -1
? frontDomain.substring(0, frontDomain.lastIndexOf('/'))
: frontDomain;
const lang =
language.substring(0, 2) + language.substring(2).toUpperCase();
const callbackUrl =
process.env.ENV !== 'prd'
? `${host}/auth/callback`
: `${host}/${lang}/auth/callback`;
this.logger.info('Callback URL: ' + callbackUrl);
const redirectUrl =
await this.identityProviderService.loginIdentityProvider(
id,
callbackUrl,
);
this.logger.info(`Redirecting to: ${redirectUrl}`);
return {
url: redirectUrl,
};
} catch (error) {
this.logger.error(error);
throw error;
}
}
}
@@ -0,0 +1,16 @@
import { Module } from '@nestjs/common';
import { IdentityProviderController } from './identity-provider.controller';
import { IdentityProviderService } from './identity-provider.service';
import { ClientsModule } from '@nestjs/microservices';
import { DucClient } from '../duc/client.config';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { ServicesModule } from 'src/services/service.module';
const client = new DucClient();
@Module({
imports: [ClientsModule.register([client.providerOptions]), ServicesModule],
controllers: [IdentityProviderController],
providers: [IdentityProviderService, DadosferaLogger],
})
export class IdentityProviderModule {}
@@ -0,0 +1,185 @@
import {
BadRequestException,
Inject,
Injectable,
OnModuleInit,
} from '@nestjs/common';
import { DucClient } from '../duc/client.config';
import { ClientGrpc } from '@nestjs/microservices';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { IdentityProviderProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import { lastValueFrom } from 'rxjs';
import { IdentityProviderRequest } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import { Metadata } from '@grpc/grpc-js';
import { Issuer, generators } from 'openid-client';
import { SsoSignInDto } from './dto/sso-signin.dto';
import { CacheService } from 'src/services/cache.service';
import { CreateIdentityProvider } from './dto/identity-provider.dto';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
@Injectable()
export class IdentityProviderService implements OnModuleInit {
private logger: DadosferaLogger;
private identityProviderService: IdentityProviderProtoService;
constructor(
@Inject(DucClient.name) private readonly grpcClient: ClientGrpc,
private readonly cacheService: CacheService<SsoSignInDto>,
@Inject(DadosferaLogger)
private dadosferaLoggger: DadosferaLogger
) {
this.logger = dadosferaLoggger.logger;
}
onModuleInit() {
this.identityProviderService =
this.grpcClient.getService<IdentityProviderProtoService>(
ProtoServices.IdentityProviderProtoService,
);
}
async create(body: IdentityProviderRequest, metadata: Metadata) {
this.logger.info("Call IdentityProvider GRPC Create")
return await lastValueFrom(
this.identityProviderService.Create(body, metadata),
);
}
async getList(metadata: Metadata) {
this.logger.info("Call IdentityProvider GRPC GetList")
return await lastValueFrom(
this.identityProviderService.GetList({}, metadata),
);
}
async loginIdentityProvider(id: string, callbackUrl: string) {
this.logger.info("Call IdentityProvider GRPC FindIdentityProvider with: " + id);
const idp = await lastValueFrom(
this.identityProviderService.FindIdentityProvider({ id }),
);
this.logger.info("Discovery issueURL: " + idp.issuerUrl)
const issuer = await Issuer.discover(idp.issuerUrl);
const client = new issuer.Client({
client_id: idp.clientId,
client_secret: idp.clientSecret,
redirect_uris: idp.redirectUrls,
response_types: ['code'],
});
this.logger.info("Generate Challenge")
const code_verifier: string = generators.codeVerifier();
const code_challenge: string = generators.codeChallenge(code_verifier);
this.logger.info("Generate State")
const state = generators.state();
this.logger.info("Generate Nonce")
const nonce = generators.nonce();
// Using state because it is returned in the callback
// and we can use it to retrieve the code_verifier and nonce
this.logger.info("Save Login parameters in redis")
await this.cacheService.set(state, {
codeVerifier: code_verifier,
nonce,
id: idp.id,
state,
clientId: idp.clientId,
clientSecret: idp.clientSecret,
issuerUrl: idp.issuerUrl,
redirectUrls: idp.redirectUrls,
});
this.logger.info("Generate Authorization URL")
const url = client.authorizationUrl({
scope: 'openid email',
response_type: 'code',
code_challenge,
code_challenge_method: 'S256',
state,
nonce,
redirect_uri: callbackUrl,
});
const idpUrl = url + '&identity_provider=' + idp.name;
this.logger.info(idpUrl)
return idpUrl;
}
async getTokenByIdp(code: string, state: string, callbackUrl: string) {
this.logger.info("Get login parameters in redis")
const ssoSign = await this.cacheService.get(state);
if (!ssoSign) {
this.logger.info("Login Parameters Not Found")
throw new BadRequestException('SSO sign-in is expired or not found');
}
this.logger.info("Discovery Issue URL: " + ssoSign.issuerUrl)
const issuer = await Issuer.discover(ssoSign.issuerUrl);
const client = new issuer.Client({
client_id: ssoSign.clientId,
client_secret: ssoSign.clientSecret,
redirect_uris: ssoSign.redirectUrls,
});
const params = client.callbackParams(
`${callbackUrl}?code=${code}&state=${state}`,
);
try {
this.logger.info("Get Token Set");
const tokenSet = await client.callback(callbackUrl, params, {
nonce: ssoSign.nonce,
code_verifier: ssoSign.codeVerifier,
state: ssoSign.state
});
this.logger.info("Delete parameters in redis");
await this.cacheService.delete(ssoSign.state);
this.logger.info("Call IdentityProvider GRPC SignInUser");
return await lastValueFrom(
this.identityProviderService.SignInUser({
accessToken: tokenSet.access_token,
idToken: tokenSet.id_token,
refreshToken: tokenSet.refresh_token,
id: ssoSign.id,
}),
);
} catch (error) {
this.logger.error(error);
throw error;
}
}
async deleteIdentityProvider(id: string, metadata: Metadata) {
this.logger.info("Call IdentityProvider GRPC Delete with: " + id)
return await lastValueFrom(
this.identityProviderService.DeleteIdentityProvider({ id }, metadata),
);
}
async updateIdentityProviders(
id: string,
body: CreateIdentityProvider,
metadata: Metadata,
) {
this.logger.info("Call IdentityProvider GRPC Update with: " + id)
return await lastValueFrom(
this.identityProviderService.UpdateIdentityProvider(
{
id,
...body,
},
metadata,
),
);
}
async identityProvidersLinksPerDomain(frontDomain: string) {
this.logger.info("Call IdentityProvider GRPC LinksPerDomain with: " + frontDomain)
return await lastValueFrom(
this.identityProviderService.GetProviderLinksFromDomain({ frontDomain }),
);
}
}
+30 -77
View File
@@ -1,93 +1,46 @@
import { Body, Controller, Inject, Param, Post } from '@nestjs/common';
import { init } from 'mixpanel';
import { Authenticated } from 'src/decorators/authentication.decorator';
import { Body, Controller, Inject, Param, Post, Req } from '@nestjs/common';
import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { RequestUser } from 'src/decorators/user.decorator';
import { MixpanelService } from './mixpanel.service';
import { extractUserFrom } from 'src/authentication/extract-user';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
@ApiInternalOnlyController()
@Authenticated()
@Controller('trackEvent')
export class MixpanelController {
logger: DadosferaLogger;
constructor(
@Inject('MIXPANEL_TOKEN')
private readonly mixpanelToken: string,
) {}
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private mixpanelService: MixpanelService,
) {
this.logger = dadosferaLogger.logger;
}
@Post(':id')
async trackEvent(@Param('id') id, @Body() body, @User() user: RequestUser) {
async trackEvent(
@Param('id') id,
@Body() body,
@Req() request
) {
this.logger.info(`POST Track Event: ${id}`)
delete body.info;
const mixpanel = init(this.mixpanelToken);
const separator = user.username.includes('-') ? '-' : '.';
const removeValues = [
'.dadosferatech.dadosfera',
'.demo.dadosfera',
'.dadosferademo',
'.dadosferarh.dadosfera',
'.dadosferatech.dadosfera2',
'.dadosferatech.dadosfera',
'.dadosfera.fin',
'.dadosferafin.dadosfera',
'.praxio.dadosfera',
'.dadosfera.tech',
'.treinamentos@dadosfera.ai',
'.dadosfera2',
'.treinamentosfera',
'.dadosfera',
];
const anonymousUser = {
username: "anonymous",
customer_name: "anonymous"
} as RequestUser
let username = user.username;
const hasToken = request.headers['authorization'];
removeValues.forEach((value) => {
username = username.replace(value, '');
});
const user = hasToken ? extractUserFrom(hasToken) : anonymousUser;
username = username.split('@')?.[0];
username = username.split('+')?.[0];
let firstName = username
.substring(0, username.indexOf(separator))
.replace('dadosfera', '');
let lastName = username
.substring(username.lastIndexOf(separator) + 1)
.replace('dadosfera', '');
if (!firstName) {
firstName = lastName;
lastName = '';
}
firstName = this.capitalize(firstName);
lastName = this.capitalize(lastName);
await mixpanel.people.set(user.username, {
$first_name: firstName,
$last_name: lastName,
$name: this.getFullName(firstName, lastName),
$email: user.username.includes('@')
? user.username
: user.username + '@dadosfera.ai',
customer_name: user.customer_name,
});
await mixpanel.track(id, {
distinct_id: user.username,
customer: user.customer_name,
env: process.env.ENV,
...body,
});
this.logger.info(`Has user: ${typeof hasToken == "string"}`)
await this.mixpanelService.track(id, user, request, body)
this.logger.info(`Event successful`)
return { id, body, user: user.username };
}
capitalize(sentence: string): string {
if (!sentence) {
return '';
}
return sentence[0].toUpperCase() + sentence.substring(1);
}
getFullName(firstName: string, lastName: string) {
return `${firstName}${lastName ? ' ' + lastName : ''}`;
}
}
+6 -1
View File
@@ -1,6 +1,8 @@
import { Module } from '@nestjs/common';
import { getSecretFromSecretsManager } from 'src/utils/SecretManager';
import { MixpanelController } from './mixpanel.controller';
import { MixpanelService } from './mixpanel.service';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
@Module({
controllers: [MixpanelController],
@@ -8,9 +10,12 @@ import { MixpanelController } from './mixpanel.controller';
{
provide: 'MIXPANEL_TOKEN',
useValue: getSecretFromSecretsManager(
`${process.env.ENV}/root/mixpanel_token`,
`prd/root/mixpanel_token`,
),
},
MixpanelService,
DadosferaLogger
],
exports: [MixpanelService]
})
export class MixpanelModule {}
+118
View File
@@ -0,0 +1,118 @@
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { Inject } from '@nestjs/common';
import { Request } from 'express';
import mixpanel, { init } from 'mixpanel';
import { RequestUser } from 'src/decorators/user.decorator';
export class MixpanelService {
logger: DadosferaLogger;
constructor(
@Inject('MIXPANEL_TOKEN')
private readonly mixpanelToken: string,
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
) {
this.logger = dadosferaLogger.logger;
}
async track(eventName: string, user: RequestUser, request: Request, body: any) {
this.logger.info("track: " + JSON.stringify({
eventName,
...body
}))
const mixpanel = init(this.mixpanelToken);
await this.setPeople(user, mixpanel);
await mixpanel.track(eventName, {
distinct_id: user.username,
customer: user.customer_name,
env: process.env.ENV,
$ip: request.ip,
$os: request.headers['sec-ch-ua-platform'] || '',
$browser: request.headers['user-agent'],
...body,
});
return;
}
async trackShare(request: Request, body: any) {
this.logger.info("trackShare: " + JSON.stringify(body))
const mixpanel = init(this.mixpanelToken);
await mixpanel.track("share_page", {
env: process.env.ENV,
$ip: request.ip,
$os: request.headers['sec-ch-ua-platform'] || '',
$browser: request.headers['user-agent'],
...body,
});
return;
}
private async setPeople(user: RequestUser, mixpanel: mixpanel.Mixpanel) {
const separator = user.username.includes('-') ? '-' : '.';
const removeValues = [
'.dadosferatech.dadosfera',
'.demo.dadosfera',
'.dadosferademo',
'.dadosferarh.dadosfera',
'.dadosferatech.dadosfera2',
'.dadosferatech.dadosfera',
'.dadosfera.fin',
'.dadosferafin.dadosfera',
'.praxio.dadosfera',
'.dadosfera.tech',
'.treinamentos@dadosfera.ai',
'.dadosfera2',
'.treinamentosfera',
'.dadosfera',
];
let username = user.username;
removeValues.forEach((value) => {
username = username.replace(value, '');
});
username = username.split('@')?.[0];
username = username.split('+')?.[0];
let firstName = username
.substring(0, username.indexOf(separator))
.replace('dadosfera', '');
let lastName = username
.substring(username.lastIndexOf(separator) + 1)
.replace('dadosfera', '');
if (!firstName) {
firstName = lastName;
lastName = '';
}
firstName = this.capitalize(firstName);
lastName = this.capitalize(lastName);
await mixpanel.people.set(user.username, {
$first_name: firstName,
$last_name: lastName,
$name: this.getFullName(firstName, lastName),
$email: user.username.includes('@')
? user.username
: user.username + '@dadosfera.ai',
customer_name: user.customer_name,
});
}
private capitalize(sentence: string): string {
if (!sentence) {
return '';
}
return sentence[0].toUpperCase() + sentence.substring(1);
}
private getFullName(firstName: string, lastName: string) {
return `${firstName}${lastName ? ' ' + lastName : ''}`;
}
}
@@ -0,0 +1,6 @@
import { ApiProperty } from "@nestjs/swagger";
export class NetworkPoliciesDTO {
@ApiProperty()
policies: string []
}
@@ -0,0 +1,64 @@
import { Body, Controller, Delete, Get, HttpCode, HttpStatus, Post } from '@nestjs/common';
import { ApiOkResponse } from '@nestjs/swagger';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import {
Authenticated,
RequireAllPermissions,
} from 'src/decorators/authentication.decorator';
import { NetworkPoliciesDTO } from './dto/network-policy.dto';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { NetworkPolicyService } from './network-policy.service';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
@Controller('network-policy')
export class NetworkPolicyController {
constructor(private networkPolicyService: NetworkPolicyService) {}
@Get()
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@HttpCode(HttpStatus.OK)
async getNetworks(
@User() user: RequestUser,
) {
return await this.networkPolicyService.getByCustomer(
user.customer_id
);
}
@Post()
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@ApiOkResponse()
@HttpCode(HttpStatus.CREATED)
async applyNetworkPolicies(
@User() user: RequestUser,
@Body() data: NetworkPoliciesDTO,
) {
const metadata = PackTheMetadata(user);
return await this.networkPolicyService.apply(
data.policies,
user.customer_id,
metadata,
);
}
@Delete()
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@ApiOkResponse()
@HttpCode(HttpStatus.OK)
async removeNetworkPolicies(
@User() user: RequestUser,
@Body() data: NetworkPoliciesDTO,
) {
const metadata = PackTheMetadata(user);
return await this.networkPolicyService.delete(
data.policies,
user.customer_id,
metadata,
);
}
}
@@ -0,0 +1,19 @@
import { Module } from '@nestjs/common';
import { NetworkPolicyController } from './network-policy.controller';
import { NetworkPolicyService } from './network-policy.service';
import { ClientsModule } from '@nestjs/microservices';
import { DucClient } from '../duc/client.config';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
const ducClient = new DucClient();
@Module({
imports: [
ClientsModule.register([
ducClient.providerOptions
]),
],
controllers: [NetworkPolicyController],
providers: [NetworkPolicyService, DadosferaLogger]
})
export class NetworkPolicyModule {}
@@ -0,0 +1,74 @@
import { Inject, Injectable } from '@nestjs/common';
import { DucClient } from '../duc/client.config';
import { ClientGrpc } from '@nestjs/microservices';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { CustomersProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { Metadata } from '@grpc/grpc-js';
import { lastValueFrom } from 'rxjs';
import { NetworkPoliciesDTO } from './dto/network-policy.dto';
@Injectable()
export class NetworkPolicyService {
private customerService: CustomersProtoService;
private logger: DadosferaLogger;
constructor(
@Inject(DucClient.name) private readonly grpcClient: ClientGrpc,
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.customerService = this.grpcClient.getService<CustomersProtoService>(
ProtoServices.CustomersProtoService,
);
}
async getByCustomer(id: string): Promise<NetworkPoliciesDTO> {
const {
customer
} = await lastValueFrom(this.customerService.CustomerFindOneById({
id
}));
return {
policies: customer.networkPolicies
}
}
async apply(
networkPolicies: string[],
customerId: string,
metadata: Metadata,
) {
return await lastValueFrom(
this.customerService.CustomerCreateNetworkPolicy(
{
customerId,
networkPolicies,
},
metadata,
),
);
}
async delete(
networkPolicies: string[],
customerId: string,
metadata: Metadata,
) {
return await lastValueFrom(
this.customerService.CustomerRemoveNetworkPolicy(
{
customerId,
networkPolicies,
},
metadata,
),
);
}
}
+1 -1
View File
@@ -4,7 +4,7 @@ import { AuthGuard } from '@nestjs/passport';
import { ConnectionClientService } from '../connection/client.service';
import jwt from 'jsonwebtoken';
import DadosferaLogger from '@dadosfera/dadosfera-logs/dist';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { ApiInternalOnlyEndpoint } from 'src/decorators/swagger.decorator';
@ApiTags('oauth')
@Controller('oauth')
+40 -26
View File
@@ -1,5 +1,5 @@
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { Body, Controller, Inject, Param, Post, Query, UseFilters, UseGuards } from '@nestjs/common';
import { Body, Controller, ForbiddenException, Header, Headers, HttpCode, HttpException, Inject, Param, Post, Query, Req, Res, UseFilters, UseGuards, UseInterceptors } from '@nestjs/common';
import { ApiCreatedResponse, ApiHeaders, ApiOkResponse, ApiTags } from '@nestjs/swagger';
import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
@@ -8,9 +8,10 @@ import { UsersService } from '../users/users.service';
import { Language } from 'src/decorators/language.decorator';
import { OpenDataService } from './open-data.service';
import { CreateUserOpenDataDTO, WordpressForm } from './dto/wordpres-form';
import { CORSGuard, SetOrigin } from 'src/decorators/set-origin.decorator';
import { Metadata } from '@grpc/grpc-js';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { request } from 'http';
import { Request } from 'express';
@Controller('open-data')
@ApiInternalOnlyController()
@@ -29,27 +30,33 @@ export class OpenDataController {
}
@Post("/sharing-ocean-data")
// @SetOrigin('https://devsbm.dadosfera.io/')
// @UseGuards(CORSGuard)
@HttpCode(200)
@Header('content-type', 'application/json')
@ApiOkResponse()
async createUser(
@Body()
body: WordpressForm,
@Query()
language: string
@Query('language')
language: string,
@Req()
request: Request,
@Headers('origin')
origin: string
) {
this.logger.info('createUser for open data' + JSON.stringify({ body }));
this.logger.info('createUser for open data' + JSON.stringify(request.headers));
// "401573bb-334f-44b2-b30e-88d4cea31ae9"
// const OPENDATA_PUBLIC_USERS_GROUP_ID = process.env.OPEN_GROUP_ID;
// ""f239718a-a271-4ef9-ae7e-02a2f0f3aa6e""
// const OPENDATA_CUSTOMER_ID = process.env.OPEN_CUSTOMER_ID;
const OPENDATA_CUSTOMER_ID = "bb4e9b26-d465-40ff-9345-e768ca69a55e";
// const corslist = ["https://devsbm.dadosfera.io", "https://sharingoceandata.com"];
// if (!corslist.includes(origin)) {
// this.logger.info('block request by cors list: '+ origin);
// throw new ForbiddenException();
// }
const roles = ["4d621501-5ea4-444b-b46e-1a009f3132ff"];
const OPENDATA_CUSTOMER_ID = process.env.OPEN_CUSTOMER_ID;
const OPENDATA_GROUP_ID = process.env.OPEN_GROUP_ID;
const roles = [process.env.OPEN_GROUP_ID];
const metadata = PackTheMetadata({
language,
});
language: language || 'en-us'
});
const data = {}
@@ -59,12 +66,11 @@ export class OpenDataController {
.forEach(key => {
const field = body.fields[key]
data[field.id] = field.value
});
});
} catch (e) {
this.logger.error('user data' + e.message);
this.logger.error('user data ' + e.message);
}
const user: CreateUserOpenDataDTO = {
email: data["email"],
enquiryType: data["enquiry_type"],
@@ -72,16 +78,24 @@ export class OpenDataController {
lastName: data["last_name"],
organization: data["organization"]
}
this.logger.info('user request' + JSON.stringify({ user, roles, customer: OPENDATA_CUSTOMER_ID }));
this.logger.info(`user request to group ${OPENDATA_CUSTOMER_ID} with role ${OPENDATA_GROUP_ID}`);
try {
const response = await this.openDataService.createUser(OPENDATA_CUSTOMER_ID, user, roles, metadata);
this.logger.info('user created with sucessfull data' + JSON.stringify(response));
return "success"
const id = await this.openDataService.createUser(OPENDATA_CUSTOMER_ID, user, roles, metadata);
this.logger.info('user created with id: '+ id);
return {
success: true,
status: 'success',
message: 'user created with succesfull'
}
} catch (e) {
this.logger.error('user data' + e.message);
return e;
this.logger.error('failed with exception: ' + e.message);
return {
success: false,
status: 'failed',
message: e.message
};
}
}
}
+2 -2
View File
@@ -40,10 +40,10 @@ export class OpenDataService implements OnModuleInit {
}
try {
await lastValueFrom(
const { user } = await lastValueFrom(
this.usersClientService.SimpleUserCreate(body, metadata),
);
return "User created";
return user.id;
} catch(err) {
return err;
}
+3 -20
View File
@@ -3,6 +3,7 @@ import { ApiOperation, ApiTags } from '@nestjs/swagger';
import {
AuthenticateCondition,
Authenticated,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { PipelinesService } from './pipelines.service';
@@ -13,26 +14,6 @@ import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
@ApiTags('Pipelines')
@Controller('pipelines')
@Authenticated()
@AuthenticateCondition((req, user) => {
let action;
switch (req.method) {
case 'POST':
action = 'CREATE';
break;
case 'PUT':
action = 'UPDATE';
break;
default:
action = req.method;
}
return user.permissions.includes(
PERMISSIONS_GROUPS.PIPELINE.permissions[action].seqid,
);
})
export class PipelinesController {
logger: DadosferaLogger;
constructor(
@@ -44,6 +25,7 @@ export class PipelinesController {
}
@Post('start/:id')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@ApiOperation({
deprecated: true,
description:
@@ -71,6 +53,7 @@ export class PipelinesController {
description:
'This method is deprecated. Please use route /pipelinesV2/:id/status instead',
})
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineStatus(@Body() body, @Param('id') id: string) {
body.id = id;
+22 -35
View File
@@ -26,13 +26,14 @@ import {
import {
AuthenticateCondition,
RequireAllPermissions,
RequireSomePermission,
} from 'src/decorators/authentication.decorator';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { PipelinesService } from './pipelines.service';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { Messages } from '@dadosfera/protospack-v2/dist/lib/PipelineV2';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/ PackTheMetadata';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { PipelinesService as OldPipelineService } from 'src/modules/pipelines/pipelines.service';
import {
@@ -52,30 +53,6 @@ import { ApiInternalOnlyEndpoint } from 'src/decorators/swagger.decorator';
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@UseFilters(new GrpcToHttpExceptionFilter())
@Controller('pipelinesV2')
@AuthenticateCondition((req, user) => {
let action;
switch (req.method) {
case 'POST':
action = 'CREATE';
break;
case 'PUT':
action = 'UPDATE';
break;
case 'PATCH':
action = 'UPDATE';
break;
default:
action = req.method;
}
return user.permissions.includes(
PERMISSIONS_GROUPS.PIPELINE.permissions[action].seqid,
);
})
export class PipelinesController {
logger: DadosferaLogger;
constructor(
@@ -88,6 +65,7 @@ export class PipelinesController {
}
@Get('monitoring-dashboard')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getMonitoringDashboard(@User() user: RequestUser) {
this.logger.info('PipelinesController - getMonitoringDashboard', { user });
@@ -100,6 +78,7 @@ export class PipelinesController {
}
@Post()
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@ApiCreatedResponse({ type: IPipelineV2 })
async create(
@Language() language: LanguageEnum,
@@ -126,6 +105,7 @@ export class PipelinesController {
}
@Get()
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async findAll(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -150,6 +130,7 @@ export class PipelinesController {
}
@Get('/download-logs')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async downloadLogs(
@User() user: RequestUser,
@Language() language: LanguageEnum,
@@ -180,6 +161,7 @@ export class PipelinesController {
}
@Get(':id/config')
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW,PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineproperties(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@@ -191,6 +173,7 @@ export class PipelinesController {
}
@Get(':id/objects')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineObjects(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@@ -202,16 +185,14 @@ export class PipelinesController {
}
@Get(':id/status')
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineStatus(@Body() body, @Param('id') id: string) {
body.id = id;
this.logger.info(
process.env.DEV_URL + `/pipeline/${id} - ON GET PIPELINE STATUS ROUTE`,
{
user: body.info.user_id,
customer: body.info.customer,
},
);
this.logger.info(`/pipeline/${id} - ON GET PIPELINE STATUS ROUTE`, {
user: body.info.user_id,
customer: body.info.customer,
});
const response = await this.oldPipelinesService.getPipelineStatus(body);
@@ -219,6 +200,7 @@ export class PipelinesController {
}
@Get('/:id')
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async findOne(
@Language() language: LanguageEnum,
@User() user: RequestUser,
@@ -256,10 +238,12 @@ export class PipelinesController {
});
return res;
});
return result;
}
@Patch('/:id')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async update(
@Language() language: LanguageEnum,
@Body() updatePipelineDto,
@@ -299,6 +283,7 @@ export class PipelinesController {
deprecated: true,
description: 'This method is deprecated. Please use PATCH instead',
})
@RequireSomePermission(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW, PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updateDeprecated(
@Language() language: LanguageEnum,
@Body() updatePipelineDto,
@@ -314,6 +299,7 @@ export class PipelinesController {
@Delete(':id')
@ApiNoContentResponse()
@HttpCode(HttpStatus.NO_CONTENT)
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE)
async delete(@Param('id') id: string, @User() user: RequestUser) {
this.logger.info('PipelinesController - delete', { user });
const metadata = PackTheMetadata({
@@ -327,7 +313,7 @@ export class PipelinesController {
@ApiInternalOnlyEndpoint()
@Post('/init-upload')
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@RequireAllPermissions(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW)
async initUploadFile(
@User() user: RequestUser,
@Body() body: IInitUploadCSVFile,
@@ -359,7 +345,7 @@ export class PipelinesController {
@ApiInternalOnlyEndpoint()
@Post('/complete-upload')
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@RequireAllPermissions(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW)
async completeUploadFile(
@User() user: RequestUser,
@Body() body: ICompleteUploadCSVFile,
@@ -377,7 +363,7 @@ export class PipelinesController {
@ApiInternalOnlyEndpoint()
@Post('/file')
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
@RequireAllPermissions(PERMISSIONS_GROUPS.IMPORT_FILES.permissions.VIEW)
async uploadedFile(
@User() user: RequestUser,
@Body() body: ICreatePipelineCSVFile,
@@ -427,6 +413,7 @@ export class PipelinesController {
@ApiInternalOnlyEndpoint()
@Post('start/:id')
@RequireSomePermission(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
async activate(@Param('id') id: string, @Body() body) {
const { info } = body;
+1 -1
View File
@@ -328,7 +328,7 @@ export class PipelinesService implements OnModuleInit {
const res = await lastValueFrom(
this.pipelineReadService.PipelineV2GetDashboardUrl(
{
dashboard_id: '83',
dashboard_id: '95',
exp: '15m',
metabase_customer_name: 'dadosferatech',
},
@@ -0,0 +1,11 @@
export const PLATFORM_API_CONFIG = {
getUrl: (): string => {
const url = process.env.PLATFORM_API_URL;
if (!url) {
throw new Error('PLATFORM_API_URL environment variable is not set');
}
return url;
},
region: process.env.AWS_REGION || 'us-east-1',
timeout: parseInt(process.env.PLATFORM_API_TIMEOUT || '30000', 10),
};
@@ -0,0 +1,977 @@
import {
Controller,
Get,
Post,
Put,
Patch,
Delete,
Param,
Body,
Query,
Inject,
BadRequestException,
} from '@nestjs/common';
import { ApiTags, ApiOperation } from '@nestjs/swagger';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import {
Authenticated,
RequireAllPermissions,
} from '../../decorators/authentication.decorator';
import { User, RequestUser } from '../../decorators/user.decorator';
import { PlatformApiService } from './platform-api.service';
import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
import { ElasticsearchService } from '../../services/elasticsearch';
import { DynamoDBService, ReferenceColumn } from '../../services/dynamodb';
import { CustomersService } from '../customers/customers.service';
import { validateCronAgainstScheduleLimit } from '../../utils/cron-validation';
@ApiTags('Platform API')
@Controller('platform')
export class PlatformApiController {
private logger: any;
constructor(
private readonly platformApiService: PlatformApiService,
private readonly elasticsearchService: ElasticsearchService,
private readonly dynamoDBService: DynamoDBService,
private readonly customersService: CustomersService,
@Inject(DadosferaLogger) dadosferaLogger: DadosferaLogger,
) {
this.logger = dadosferaLogger.logger;
}
/**
* Normalize pipeline ID to match Platform-API format.
* Platform-API replaces '-' with '_' in pipeline IDs.
*/
private normalizePipelineId(id: string): string {
return id?.replace(/-/g, '_') || '';
}
/**
* Denormalize ID back to UUID format (replace _ with -).
* Used when we receive a normalized ID but need the original UUID.
*/
private denormalizeId(id: string): string {
return id?.replace(/_/g, '-') || '';
}
/**
* Normalize job ID to match Platform-API format.
* Platform-API replaces '-' with '_' in job IDs.
*
* Example: "2ccf5481-59f5-4036-8a94-7d5f28f4f899-0" -> "2ccf5481_59f5_4036_8a94_7d5f28f4f899_0"
*/
private normalizeJobId(jobId: string): string {
return jobId?.replace(/-/g, '_') || '';
}
/**
* Extract the pipeline ID (base UUID) from a job ID.
* Job IDs have format "uuid-suffix" where suffix is the job index (e.g., "0", "1").
* Handles both hyphenated and underscored formats, always returns hyphenated UUID for ES.
*
* Examples:
* - "2ccf5481-59f5-4036-8a94-7d5f28f4f899-0" -> "2ccf5481-59f5-4036-8a94-7d5f28f4f899"
* - "2ccf5481_59f5_4036_8a94_7d5f28f4f899_0" -> "2ccf5481-59f5-4036-8a94-7d5f28f4f899"
*/
private extractPipelineIdFromJobId(jobId: string): string {
if (!jobId) return '';
// Determine the separator used in the jobId
const hasUnderscores = jobId.includes('_');
const separator = hasUnderscores ? '_' : '-';
const parts = jobId.split(separator);
// UUID has 5 parts (8-4-4-4-12), job suffix is the 6th part
if (parts.length >= 6) {
// Always return hyphenated format for Elasticsearch lookup
return parts.slice(0, 5).join('-');
}
// If no suffix found, return the ID in hyphenated format
return hasUnderscores ? jobId.replace(/_/g, '-') : jobId;
}
private readonly VALID_CONNECTORS = ['jdbc', 'singer', 's3'];
private readonly MAX_MEMORY_MB = 12000; // 12GB maximum memory per pipeline/job
/**
* Validate that connector is provided and is a valid type.
*/
private validateConnector(connector: string | undefined): void {
if (!connector || !this.VALID_CONNECTORS.includes(connector)) {
throw new BadRequestException(
`connector is required in job input and must be one of: ${this.VALID_CONNECTORS.join(', ')}`,
);
}
}
/**
* Validate memory allocation against maximum limit.
*/
private validateMemory(memoryMb: number): void {
if (memoryMb > this.MAX_MEMORY_MB) {
throw new BadRequestException(
`Memory limit exceeded. Maximum allowed: ${this.MAX_MEMORY_MB}MB (12GB)`,
);
}
}
/**
* Validate cron expression against customer's schedule limit.
* Fetches current scheduleLimit from DUC to ensure up-to-date configuration.
*/
private async validateScheduleLimit(cron: string, customerId: string): Promise<void> {
if (!cron) return;
const { customer } = await this.customersService.getCustomer(customerId);
const scheduleLimit = customer?.scheduleLimit || 'day';
const result = validateCronAgainstScheduleLimit(cron, scheduleLimit);
if (!result.valid) {
throw new BadRequestException(result.message);
}
}
/**
* Map connector type to DynamoDB type.
* jdbc -> 'database', singer -> 'application', s3 -> 'file'
*/
private mapConnectorToDynamoType(connector: string): string {
switch (connector) {
case 'jdbc':
return 'database';
case 'singer':
return 'application';
case 's3':
return 'file';
default:
return connector;
}
}
/**
* Extract and transform tables from jobs for DynamoDB input.
* Maps connector-specific fields to a common table format.
*
* - JDBC: load_type, table_name, column_include_list (columns), incremental_column_name/type (reference_column object)
* - Singer: type maps replication_method (FULL_TABLE -> full_load, INCREMENTAL -> incremental), no columns
* - S3: same mapping as Singer, no columns
*/
private extractTablesFromJobs(jobs: any[], connector: string): Array<{
name: string;
type: string;
columns?: string[];
reference_column?: ReferenceColumn;
}> {
if (!jobs || jobs.length === 0) return [];
const tables: Array<{
name: string;
type: string;
columns?: string[];
reference_column?: ReferenceColumn;
}> = [];
for (const job of jobs) {
const input = job.input;
if (!input) continue;
if (connector === 'jdbc') {
// JDBC: table_name, load_type, column_include_list, incremental_column_name/type
const table: {
name: string;
type: string;
columns?: string[];
reference_column?: ReferenceColumn;
} = {
name: input.table_name || '',
type: input.load_type || 'full_load',
};
if (input.column_include_list && input.column_include_list.length > 0) {
table.columns = input.column_include_list;
}
if (input.incremental_column_name) {
// reference_column is stored as an object with name and type
table.reference_column = {
name: input.incremental_column_name,
type: input.incremental_column_type || 'unknown',
};
}
tables.push(table);
} else if (connector === 'singer' || connector === 's3') {
// Singer/S3: replication_method -> type mapping, no columns
let type = 'full_load';
if (input.replication_method === 'INCREMENTAL') {
type = 'incremental';
} else if (input.replication_method === 'FULL_TABLE') {
type = 'full_load';
}
tables.push({
name: input.table_name || '',
type,
});
}
}
return tables;
}
/**
* Build properties object for Elasticsearch based on connector type.
* Different connectors have different property structures.
*
* Note: In pi-factory flow, properties come pre-built from frontend.
* In Maestro proxy flow, we reconstruct from job input fields.
*/
private buildPipelineProperties(jobInput: any): Record<string, any> {
if (!jobInput) return {};
const connector = jobInput.connector;
const properties: Record<string, any> = {};
// Determine credentials_type
if (jobInput.auth_parameters?.credentials_type) {
properties.credentials_type = jobInput.auth_parameters.credentials_type;
} else {
// Default based on connector type
// S3 connector typically uses iam_user, others use basic_auth
properties.credentials_type = connector === 's3' ? 'iam_user' : 'basic_auth';
}
if (connector === 'jdbc') {
// JDBC connectors: schema comes from table_schema
if (jobInput.table_schema) {
properties.schema = jobInput.table_schema;
}
} else if (connector === 'singer') {
// Singer connectors: merge config fields (dates, selected_competitions, etc.)
if (jobInput.config) {
Object.assign(properties, jobInput.config);
}
} else if (connector === 's3') {
// S3 connector
if (jobInput.engine) properties.engine = jobInput.engine;
if (jobInput.source_bucket) properties.source_bucket = jobInput.source_bucket;
if (jobInput.source_prefix) properties.source_prefix = jobInput.source_prefix;
if (jobInput.file_format_params) properties.file_format_params = jobInput.file_format_params;
}
return properties;
}
/**
* Sync job input changes to DynamoDB for a specific connector type.
* Extracts pipeline ID from job ID, fetches ES document to find input ID,
* then updates the table entry in DynamoDB.
*
* Job ID transformations:
* - Raw format (from endpoint): "2ccf5481-59f5-4036-8a94-7d5f28f4f899-0"
* - Platform API format: "2ccf5481_59f5_4036_8a94_7d5f28f4f899_0" (underscores)
* - Elasticsearch pipeline ID: "2ccf5481-59f5-4036-8a94-7d5f28f4f899" (UUID only, hyphens)
*
* @param connectorType - The connector type ('jdbc', 'singer', 's3') for the Platform API endpoint
*/
private async syncJobInputToDynamoDB(
jobId: string,
body: any,
user: RequestUser,
connectorType: 'jdbc' | 'singer' | 's3',
): Promise<void> {
try {
// Normalize job ID for Platform API GET (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
// Get job details using connector-specific endpoint to find table_name
const jobResult = await this.platformApiService.proxy(
'GET',
`/jobs/${connectorType}/${normalizedJobId}`,
user,
);
// Extract the pipeline ID (base UUID) from the raw job ID for ES lookup
const esPipelineId = this.extractPipelineIdFromJobId(jobId);
const tableName = body.table_name || jobResult.source_config?.table_name;
if (!esPipelineId || !tableName) {
this.logger.warn('Cannot sync job input: missing pipeline_id or table_name', {
jobId,
esPipelineId,
tableName,
});
return;
}
// Get pipeline from ES to find input ID (stored in config.tables)
const pipeline = await this.elasticsearchService.getPipeline(
user.customer_name,
esPipelineId,
);
const inputId = pipeline?.config?.tables;
if (!inputId) {
this.logger.warn('Cannot sync job input: no input ID in ES', {
jobId,
esPipelineId,
});
return;
}
// Build changes for DynamoDB table entry
// reference_column is stored as an object with name and type
const changes: {
type?: string;
columns?: string[];
reference_column?: ReferenceColumn | null;
} = {};
if ('target_load_type' in body) {
changes.type = body.target_load_type;
}
if ('column_include_list' in body) {
changes.columns = body.column_include_list;
}
if ('incremental_column_name' in body) {
// reference_column is stored as an object with name and type
if (body.incremental_column_name) {
changes.reference_column = {
name: body.incremental_column_name,
type: body.incremental_column_type || 'unknown',
};
} else {
changes.reference_column = null;
}
}
// Update DynamoDB if there are changes
if (Object.keys(changes).length > 0) {
await this.dynamoDBService.updateInputTable(
user.customer_id,
inputId,
tableName,
changes,
);
}
} catch (error) {
this.logger.error('Failed to sync job input to DynamoDB', {
jobId,
connectorType,
error: error.message,
});
// Don't throw - Platform API update succeeded, just log the sync error
}
}
/**
* Sync sync-mode changes to DynamoDB for JDBC connectors.
* Always passes both target_load_type and incremental_column_name to ensure proper sync.
*/
private async syncJdbcSyncModeToDynamoDB(
jobId: string,
body: any,
user: RequestUser,
): Promise<void> {
// JDBC sync mode uses target_load_type field
const changes: any = {};
if ('target_load_type' in body) {
changes.target_load_type = body.target_load_type;
}
// Handle incremental_column_name:
// - If provided in body, use that value
// - If changing to full_load, explicitly clear it
if ('incremental_column_name' in body) {
changes.incremental_column_name = body.incremental_column_name;
changes.incremental_column_type = body.incremental_column_type;
} else if (body.target_load_type === 'full_load') {
// Changing to full_load without specifying incremental_column - clear it
changes.incremental_column_name = null;
}
await this.syncJobInputToDynamoDB(jobId, changes, user, 'jdbc');
}
/**
* Sync sync-mode changes to DynamoDB for Singer connectors.
*/
private async syncSingerSyncModeToDynamoDB(
jobId: string,
body: any,
user: RequestUser,
): Promise<void> {
// Singer sync mode uses replication_method field
// Map to DynamoDB type: FULL_TABLE -> full_load, INCREMENTAL -> incremental
if ('replication_method' in body) {
const type = body.replication_method === 'INCREMENTAL' ? 'incremental' : 'full_load';
await this.syncJobInputToDynamoDB(jobId, { load_type: type }, user, 'singer');
}
}
// ==================== PIPELINE ROUTES ====================
@Post('pipeline')
@ApiOperation({ summary: 'Create a new pipeline' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.CREATE)
async createPipeline(@Body() body: any, @User() user: RequestUser) {
// Validate that pipeline has jobs
if (!body.jobs || body.jobs.length === 0) {
throw new BadRequestException('Pipeline must have at least one job');
}
// Validate connector before proceeding
const firstJob = body.jobs[0]?.input;
this.validateConnector(firstJob?.connector);
// Validate cron against customer's schedule limit
await this.validateScheduleLimit(body.cron, user.customer_id);
// Inject customer_id (actually customer_name) and normalized ID into body for Platform-API
// Note: Platform-API was created before customer_id existed, so it expects customer_name in the customer_id field
const enrichedBody = {
...body,
id: this.normalizePipelineId(body.id),
customer_id: user.customer_name,
};
const result = await this.platformApiService.proxy('POST', '/pipeline', user, enrichedBody);
// Sync to Elasticsearch and DynamoDB
try {
const plugin = firstJob?.plugin;
const connectorType = firstJob?.connector;
const connectionId = firstJob?.auth_parameters?.config_id;
// Build properties based on connector type
const properties = this.buildPipelineProperties(firstJob);
const connector = plugin
? await this.elasticsearchService.getConnectorByPlugin(plugin)
: null;
// Extract tables from jobs and create DynamoDB input
const tables = this.extractTablesFromJobs(body.jobs, connectorType);
let inputId: string | undefined;
if (tables.length > 0) {
const inputDoc = await this.dynamoDBService.createInput(
user.customer_id,
user.user_id,
{
name: body.name,
description: body.description,
plugin: plugin || '',
type: this.mapConnectorToDynamoType(connectorType),
tables,
},
);
inputId = inputDoc.id;
this.logger.info('Created DynamoDB input for tables config', {
inputId,
pipelineId: body.id,
tablesCount: tables.length,
});
}
const pipelineType = this.mapConnectorToDynamoType(connectorType);
this.logger.info('Syncing pipeline to Elasticsearch', {
customerName: user.customer_name,
pipelineId: body.id,
plugin,
connector: connectorType,
type: pipelineType,
properties,
inputId,
});
// Keep original UUID format for Elasticsearch (not normalized)
await this.elasticsearchService.createPipeline(
user.customer_name,
body.id,
{
name: body.name,
description: body.description,
user_id: user.user_id,
username: user.username,
customer_id: user.customer_id,
plugin,
connection_id: connectionId,
cron: body.cron,
tables: inputId,
properties,
type: pipelineType,
},
connector,
);
} catch (error) {
this.logger.error('Failed to sync pipeline creation to Elasticsearch/DynamoDB', {
pipelineId: body.id,
customerName: user.customer_name,
error: error.message,
errorName: error.name,
});
}
return result;
}
@Get('pipelines')
@ApiOperation({ summary: 'List all pipelines for customer' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelines(
@User() user: RequestUser,
@Query() query: Record<string, string>,
) {
return this.platformApiService.proxy(
'GET',
'/pipelines',
user,
undefined,
query,
);
}
@Get('pipeline/:pipelineId')
@ApiOperation({ summary: 'Get pipeline by ID' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipeline(
@Param('pipelineId') pipelineId: string,
@User() user: RequestUser,
) {
const normalizedId = this.normalizePipelineId(pipelineId);
return this.platformApiService.proxy('GET', `/pipeline/${normalizedId}`, user);
}
@Patch('pipeline/:pipelineId')
@ApiOperation({ summary: 'Update pipeline by ID' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updatePipeline(
@Param('pipelineId') pipelineId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Validate cron against customer's schedule limit if cron is being updated
if (body.cron) {
await this.validateScheduleLimit(body.cron, user.customer_id);
}
const normalizedId = this.normalizePipelineId(pipelineId);
const result = await this.platformApiService.proxy(
'PATCH',
`/pipeline/${normalizedId}`,
user,
body,
);
// Sync to Elasticsearch (use original UUID, not normalized)
// Only pass fields that are explicitly provided in the request body
try {
const esChanges: {
name?: string;
description?: string;
cron?: string;
status?: string;
} = {};
if ('name' in body) esChanges.name = body.name;
if ('description' in body) esChanges.description = body.description;
if ('cron' in body) esChanges.cron = body.cron;
if ('status' in body) esChanges.status = body.status;
await this.elasticsearchService.updatePipeline(
user.customer_name,
pipelineId,
esChanges,
);
} catch (error) {
this.logger.error('Failed to sync pipeline update to Elasticsearch', {
pipelineId,
error: error.message,
});
}
return result;
}
@Delete('pipeline/:pipelineId')
@ApiOperation({ summary: 'Delete pipeline by ID' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE)
async deletePipeline(
@Param('pipelineId') pipelineId: string,
@User() user: RequestUser,
) {
const normalizedId = this.normalizePipelineId(pipelineId);
const result = await this.platformApiService.proxy(
'DELETE',
`/pipeline/${normalizedId}`,
user,
);
// Sync to Elasticsearch (use original UUID, not normalized)
try {
await this.elasticsearchService.deletePipeline(
user.customer_name,
pipelineId,
);
} catch (error) {
this.logger.error('Failed to sync pipeline deletion to Elasticsearch', {
pipelineId,
error: error.message,
});
}
return result;
}
@Post('pipeline/execute')
@ApiOperation({ summary: 'Execute a pipeline' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async executePipeline(@Body() body: any, @User() user: RequestUser) {
// Inject customer_id (actually customer_name) into body for Platform-API
// Note: Platform-API was created before customer_id existed, so it expects customer_name in the customer_id field
const enrichedBody = {
...body,
customer_id: user.customer_name,
};
return this.platformApiService.proxy('POST', '/pipeline/execute', user, enrichedBody);
}
@Post('pipeline/pause')
@ApiOperation({ summary: 'Pause a pipeline' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async pausePipeline(@Body() body: any, @User() user: RequestUser) {
// Inject customer_id (actually customer_name) into body for Platform-API
// Note: Platform-API was created before customer_id existed, so it expects customer_name in the customer_id field
const enrichedBody = {
...body,
customer_id: user.customer_name,
};
return this.platformApiService.proxy('POST', '/pipeline/pause', user, enrichedBody);
}
@Post('pipeline/unpause')
@ApiOperation({ summary: 'Unpause a pipeline' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async unpausePipeline(@Body() body: any, @User() user: RequestUser) {
// Inject customer_id (actually customer_name) into body for Platform-API
// Note: Platform-API was created before customer_id existed, so it expects customer_name in the customer_id field
const enrichedBody = {
...body,
customer_id: user.customer_name,
};
return this.platformApiService.proxy('POST', '/pipeline/unpause', user, enrichedBody);
}
@Put('pipeline/:pipelineId/memory')
@ApiOperation({ summary: 'Update pipeline memory configuration' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updatePipelineMemory(
@Param('pipelineId') pipelineId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Validate memory limit
if (body.amount) {
this.validateMemory(body.amount);
}
return this.platformApiService.proxy(
'PUT',
`/pipeline/${pipelineId}/memory`,
user,
body,
);
}
// ==================== PIPELINE METADATA ROUTES ====================
@Put('pipeline/:pipelineId/metadata')
@ApiOperation({ summary: 'Update pipeline metadata' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updatePipelineMetadata(
@Param('pipelineId') pipelineId: string,
@Body() body: any,
@User() user: RequestUser,
) {
return this.platformApiService.proxy(
'PUT',
`/pipeline/${pipelineId}/metadata`,
user,
body,
);
}
@Get('pipelines/metadata')
@ApiOperation({ summary: 'Get all pipelines metadata' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelinesMetadata(
@User() user: RequestUser,
@Query() query: Record<string, string>,
) {
return this.platformApiService.proxy(
'GET',
'/pipelines/metadata',
user,
undefined,
query,
);
}
// ==================== PIPELINE RUN ROUTES ====================
@Get('pipeline/:pipelineId/pipeline_run')
@ApiOperation({ summary: 'Get pipeline runs for a pipeline' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineRuns(
@Param('pipelineId') pipelineId: string,
@User() user: RequestUser,
@Query() query: Record<string, string>,
) {
const normalizedId = this.normalizePipelineId(pipelineId);
return this.platformApiService.proxy(
'GET',
`/pipeline/${normalizedId}/pipeline_run`,
user,
undefined,
query,
);
}
@Get('pipeline/:pipelineId/pipeline_run/:runId')
@ApiOperation({ summary: 'Get specific pipeline run' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineRun(
@Param('pipelineId') pipelineId: string,
@Param('runId') runId: string,
@User() user: RequestUser,
) {
const normalizedPipelineId = this.normalizePipelineId(pipelineId);
const normalizedRunId = this.normalizePipelineId(runId);
return this.platformApiService.proxy(
'GET',
`/pipeline/${normalizedPipelineId}/pipeline_run/${normalizedRunId}`,
user,
);
}
@Get('pipeline/pipeline_run/:runId/logs')
@ApiOperation({ summary: 'Get pipeline run logs' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getPipelineRunLogs(
@Param('runId') runId: string,
@User() user: RequestUser,
@Query() query: Record<string, string>,
) {
const normalizedRunId = this.normalizePipelineId(runId);
return this.platformApiService.proxy(
'GET',
`/pipeline/pipeline_run/${normalizedRunId}/logs`,
user,
undefined,
query,
);
}
// ==================== JOBS - COLUMN EDITING ROUTES ====================
@Put('jobs/:jobId/input')
@ApiOperation({ summary: 'Update job input columns' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updateJobInput(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
const result = await this.platformApiService.proxy(
'PUT',
`/jobs/${normalizedJobId}/input`,
user,
body,
);
// Sync to DynamoDB if connector type is provided
const connectorType = body.connector as 'jdbc' | 'singer' | 's3' | undefined;
if (connectorType && this.VALID_CONNECTORS.includes(connectorType)) {
await this.syncJobInputToDynamoDB(jobId, body, user, connectorType);
}
return result;
}
@Patch('jobs/:jobId/input')
@ApiOperation({ summary: 'Partial update job input columns' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async patchJobInput(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
const result = await this.platformApiService.proxy(
'PATCH',
`/jobs/${normalizedJobId}/input`,
user,
body,
);
// Sync to DynamoDB if connector type is provided
const connectorType = body.connector as 'jdbc' | 'singer' | 's3' | undefined;
if (connectorType && this.VALID_CONNECTORS.includes(connectorType)) {
await this.syncJobInputToDynamoDB(jobId, body, user, connectorType);
}
return result;
}
@Put('jobs/:jobId/memory')
@ApiOperation({ summary: 'Update job memory configuration' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updateJobMemory(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Validate memory limit
if (body.amount) {
this.validateMemory(body.amount);
}
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
return this.platformApiService.proxy(
'PUT',
`/jobs/${normalizedJobId}/memory`,
user,
body,
);
}
@Post('jobs/:jobId/reset-state')
@ApiOperation({ summary: 'Reset job state' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async resetJobState(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
return this.platformApiService.proxy(
'POST',
`/jobs/${normalizedJobId}/reset-state`,
user,
body,
);
}
// ==================== JOBS - JDBC SYNC MODE ROUTES ====================
@Get('jobs/jdbc/:jobId')
@ApiOperation({ summary: 'Get JDBC job details' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getJdbcJob(@Param('jobId') jobId: string, @User() user: RequestUser) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
return this.platformApiService.proxy('GET', `/jobs/jdbc/${normalizedJobId}`, user);
}
@Post('jobs/jdbc/:jobId/sync-mode')
@ApiOperation({ summary: 'Update JDBC job sync mode' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updateJdbcSyncMode(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
const result = await this.platformApiService.proxy(
'POST',
`/jobs/jdbc/${normalizedJobId}/sync-mode`,
user,
body,
);
// Sync to DynamoDB (pass raw jobId for pipeline extraction)
await this.syncJdbcSyncModeToDynamoDB(jobId, body, user);
return result;
}
@Get('jobs/jdbc/configs/allowed_datatypes')
@ApiOperation({ summary: 'Get allowed datatypes for JDBC' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getJdbcAllowedDatatypes(@User() user: RequestUser) {
return this.platformApiService.proxy(
'GET',
'/jobs/jdbc/configs/allowed_datatypes',
user,
);
}
// ==================== JOBS - SINGER REPLICATION ROUTES ====================
@Get('jobs/singer/:jobId')
@ApiOperation({ summary: 'Get Singer job details' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getSingerJob(@Param('jobId') jobId: string, @User() user: RequestUser) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
return this.platformApiService.proxy('GET', `/jobs/singer/${normalizedJobId}`, user);
}
@Post('jobs/singer/:jobId/sync-mode')
@ApiOperation({ summary: 'Update Singer job sync mode' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE)
async updateSingerSyncMode(
@Param('jobId') jobId: string,
@Body() body: any,
@User() user: RequestUser,
) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
const result = await this.platformApiService.proxy(
'POST',
`/jobs/singer/${normalizedJobId}/sync-mode`,
user,
body,
);
// Sync to DynamoDB (pass raw jobId for pipeline extraction)
await this.syncSingerSyncModeToDynamoDB(jobId, body, user);
return result;
}
// ==================== JOBS - S3 ROUTES ====================
@Get('jobs/s3/:jobId')
@ApiOperation({ summary: 'Get S3 job details' })
@RequireAllPermissions(PERMISSIONS_GROUPS.PIPELINE.permissions.GET)
async getS3Job(@Param('jobId') jobId: string, @User() user: RequestUser) {
// Normalize job ID for Platform API (replace - with _)
const normalizedJobId = this.normalizeJobId(jobId);
return this.platformApiService.proxy('GET', `/jobs/s3/${normalizedJobId}`, user);
}
// ==================== HEALTH ROUTE ====================
@Get('health')
@ApiOperation({ summary: 'Platform API health check' })
@Authenticated()
async healthCheck(@User() user: RequestUser) {
return this.platformApiService.proxy('GET', '/health', user);
}
}
@@ -0,0 +1,17 @@
import { Module } from '@nestjs/common';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { PlatformApiController } from './platform-api.controller';
import { PlatformApiService } from './platform-api.service';
import { ElasticsearchModule } from '../../services/elasticsearch';
import { DynamoDBModule } from '../../services/dynamodb';
import { CustomersModule } from '../customers/customers.module';
@Module({
imports: [ElasticsearchModule, DynamoDBModule, CustomersModule],
controllers: [PlatformApiController],
providers: [PlatformApiService, DadosferaLogger],
exports: [PlatformApiService],
})
export class PlatformApiModule {}
@@ -0,0 +1,123 @@
import { Injectable, Inject, HttpException } from '@nestjs/common';
import { SignatureV4 } from '@aws-sdk/signature-v4';
import { Sha256 } from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import axios, { AxiosResponse, Method } from 'axios';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { RequestUser } from '../../decorators/user.decorator';
import { PLATFORM_API_CONFIG } from './platform-api.config';
@Injectable()
export class PlatformApiService {
private signer: SignatureV4;
private logger: any;
constructor(
@Inject(DadosferaLogger) dadosferaLogger: DadosferaLogger,
) {
this.logger = dadosferaLogger.logger;
this.signer = new SignatureV4({
service: 'execute-api',
region: PLATFORM_API_CONFIG.region,
credentials: defaultProvider(),
sha256: Sha256,
});
}
async proxy(
method: string,
path: string,
user: RequestUser,
body?: any,
query?: Record<string, string>,
): Promise<any> {
const baseUrl = PLATFORM_API_CONFIG.getUrl();
const url = new URL(`${baseUrl}${path}`);
// Add query params
if (query) {
Object.entries(query).forEach(([key, value]) => {
if (value !== undefined && value !== null) {
url.searchParams.set(key, String(value));
}
});
}
const headers: Record<string, string> = {
host: url.hostname,
'content-type': 'application/json',
// Forward user context headers
// Note: platform-api expects customer_name in the 'customer_id' header (contract inconsistency)
'customer_id': user.customer_name || '',
'customer_name': user.customer_name || '',
'x-user-id': user.user_id || '',
'x-username': user.username || '',
'x-customer-tier': user.customer_tier || '',
'x-customer-id': user.customer_id || '',
};
const requestToSign = {
method: method.toUpperCase(),
protocol: url.protocol,
hostname: url.hostname,
port: url.port ? parseInt(url.port, 10) : undefined,
path: url.pathname + url.search,
headers,
body: body ? JSON.stringify(body) : undefined,
};
this.logger.info('Proxying request to platform-api', {
method: method.toUpperCase(),
path,
customer_id: user.customer_id,
user_id: user.user_id,
});
try {
// Sign with IAM v4
const signedRequest = await this.signer.sign(requestToSign);
const response: AxiosResponse = await axios({
method: method as Method,
url: url.href,
headers: signedRequest.headers as Record<string, string>,
data: body,
timeout: PLATFORM_API_CONFIG.timeout,
validateStatus: () => true, // Don't throw on non-2xx
});
// Propagate non-2xx responses as HttpExceptions
if (response.status >= 400) {
throw new HttpException(response.data, response.status);
}
return response.data;
} catch (error) {
this.logger.error('Platform API proxy error', {
error: error.message,
status: error.response?.status,
path,
method: method.toUpperCase(),
});
if (error instanceof HttpException) {
throw error;
}
if (error.response) {
throw new HttpException(error.response.data, error.response.status);
}
if (error.code === 'ECONNREFUSED') {
throw new HttpException('Platform API service unavailable', 503);
}
if (error.code === 'ETIMEDOUT' || error.code === 'ECONNABORTED') {
throw new HttpException('Platform API request timeout', 504);
}
throw new HttpException('Internal server error', 500);
}
}
}
@@ -0,0 +1,10 @@
export class CreateShareMetadataDto {
assetId: string;
proposedId: string;
type: string;
}
export class UpdateShareMetadataDto {
shareId: string;
type: string;
}
@@ -0,0 +1,31 @@
import { Controller, Get, Body, Param, Put, Patch } from '@nestjs/common';
import { ShareMetadataService } from './share-metadata.service';
import { Authenticated } from 'src/decorators/authentication.decorator';
import { RequestUser, User } from 'src/decorators/user.decorator';
import { PackTheMetadata } from 'src/utils/PackTheMetadata';
import { CreateShareMetadataDto, UpdateShareMetadataDto } from './dto/create-share-metadata.dto';
@Controller('share')
@Authenticated()
export class ShareMetadataController {
constructor(private readonly shareMetadataService: ShareMetadataService) {}
@Put('/')
findOrcreate(@Body() data: CreateShareMetadataDto, @User() user: RequestUser) {
const metadata = PackTheMetadata(user);
return this.shareMetadataService.create(data, metadata);
}
@Patch('/')
updateShareType(@Body() data: UpdateShareMetadataDto, @User() user: RequestUser) {
const metadata = PackTheMetadata(user);
return this.shareMetadataService.updateShare(data, metadata);
}
@Get('/:id')
get(@Param('id') id: string, @User() user: RequestUser) {
const metadata = PackTheMetadata(user);
return this.shareMetadataService.get(id, metadata);
}
}
@@ -0,0 +1,16 @@
import { Module } from '@nestjs/common';
import { ShareMetadataService } from './share-metadata.service';
import { ShareMetadataController } from './share-metadata.controller';
import { DucClient } from '../duc/client.config';
import { ClientsModule } from '@nestjs/microservices';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
const client = new DucClient();
@Module({
imports: [ClientsModule.register([client.providerOptions])],
controllers: [ShareMetadataController],
providers: [ShareMetadataService, DadosferaLogger],
exports: [ShareMetadataService]
})
export class ShareMetadataModule {}
@@ -0,0 +1,62 @@
import { Inject, Injectable, OnModuleInit } from '@nestjs/common';
import { Metadata } from '@grpc/grpc-js';
import { ClientGrpc } from '@nestjs/microservices';
import { DucClient } from '../duc/client.config';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { ShareMetadataProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import { lastValueFrom } from 'rxjs';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { CreateShareMetadataDto, UpdateShareMetadataDto } from './dto/create-share-metadata.dto';
@Injectable()
export class ShareMetadataService implements OnModuleInit {
ducService: ShareMetadataProtoService;
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
@Inject(DucClient.name) private readonly grpcClient: ClientGrpc,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.ducService =this.grpcClient.getService<ShareMetadataProtoService>(
ProtoServices.ShareMetadataProtoService,
);
}
create(createShareDto: CreateShareMetadataDto, metadata: Metadata) {
return lastValueFrom(
this.ducService.FindOrCreateShareMetadata(
{
assetId: createShareDto.assetId,
proposedId: createShareDto.proposedId,
type: createShareDto.type
},
metadata,
),
);
}
updateShare(updateShare: UpdateShareMetadataDto, metadata: Metadata) {
return lastValueFrom(
this.ducService.ChangeShareMetadataType(
updateShare,
metadata,
),
);
}
get(id: string, metadata: Metadata) {
return lastValueFrom(
this.ducService.GetShareMetadata(
{
id,
},
metadata,
),
);
}
}
+47
View File
@@ -0,0 +1,47 @@
import { Link, Theme } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/entities';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
export class CustomerLink implements Link {
@ApiProperty()
href: string;
@ApiProperty()
name: string;
@ApiProperty()
description: string;
@ApiPropertyOptional()
iconSrc: string;
}
export class CustomerLinkRequest {
@ApiProperty({ type: [CustomerLink] })
links: CustomerLink[];
}
export class CustomerLinksResponse {
@ApiProperty({ type: [CustomerLink] })
links: CustomerLink[];
}
export class CustomerTheme implements Theme {
@ApiProperty()
backgroundColor: string;
@ApiProperty()
textColor: string;
@ApiProperty()
logo: string;
@ApiProperty()
logoLogin: string;
}
export class CustomerThemeResponse {
@ApiProperty()
theme: CustomerTheme;
}
export class CustomerThemeRequest {
@ApiProperty()
displayName: string;
@ApiProperty()
backgroundColor: string;
@ApiProperty()
textColor: string;
}
+147
View File
@@ -0,0 +1,147 @@
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import {
Body,
Controller,
Get,
HttpException,
HttpStatus,
Inject,
Param,
Post,
Put,
Query,
UploadedFiles,
UseFilters,
UseInterceptors,
HttpCode
} from '@nestjs/common';
import { ApiOkResponse, ApiTags } from '@nestjs/swagger';
import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
import {
Authenticated,
RequireAllPermissions,
} from 'src/decorators/authentication.decorator';
import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
import { CustomerThemeRequest, CustomerThemeResponse } from './dtos/customers';
import ErrorCodes from 'src/utils/errorCodes';
import { AnyFilesInterceptor } from '@nestjs/platform-express';
import { ThemeService } from './theme.service';
@ApiTags('Theme')
@Controller('customers')
@UseFilters(GrpcToHttpExceptionFilter)
export class ThemeController {
logger: DadosferaLogger;
constructor(
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
private themeService: ThemeService,
) {
this.logger = dadosferaLogger.logger;
}
@Post('/:id/theme')
@Authenticated()
@RequireAllPermissions(PERMISSIONS_GROUPS.USERS.permissions.ADMIN)
@ApiOkResponse({ type: CustomerThemeResponse })
@UseInterceptors(AnyFilesInterceptor())
@HttpCode(HttpStatus.OK)
async saveCustomertheme(
@Param('id') id: string,
@Body() data: CustomerThemeRequest,
@UploadedFiles() files: Array<Express.Multer.File>
) {
this.logger.info('saveCustomertheme' + JSON.stringify({
id,
}));
const logo = files.find(file => file.fieldname === 'logo');
const logoLogin = files.find(file => file.fieldname === 'logoLogin');
this.validFileSize(logo);
this.validFileSize(logoLogin);
this.validMimeType(logo);
this.validMimeType(logoLogin);
try {
const theme = await this.themeService.createThemeByCustomer(id, {
...data,
logo,
logoLogin
});
this.logger.info('saveCustomertheme' + JSON.stringify(theme));
return theme;
} catch (err) {
if (err.details === ErrorCodes.CUSTOMER.NOT_FOUND) {
this.logger.error('Error - saveCustomertheme - Expect CUSTOMER.NOT_FOUND');
throw new HttpException(err.details, HttpStatus.NOT_FOUND);
} else {
this.logger.error('Error - saveCustomertheme Unknown Error:' + err?.message);
return { theme: null };
};
}
}
private validFileSize(file: Express.Multer.File) {
const maxFileSize = 10 * 1024 * 1024; // 10MB
if (file && file.size > maxFileSize) {
throw new HttpException(`O Arquivo ${file.filename} possui mais de 10MB`, HttpStatus.BAD_REQUEST);
}
}
private validMimeType(file: Express.Multer.File) {
const mimeTypesValid = ['image/jpeg', 'image/jpg', 'image/png'];
if (file && !mimeTypesValid.includes(file.mimetype)) {
throw new HttpException(`O Arquivo ${file.fieldname} deve ser jpeg, jpg, ou png`, HttpStatus.BAD_REQUEST);
}
}
@Get('/:id/theme')
@ApiOkResponse({ type: CustomerThemeResponse })
async getCustomerTheme(@Param('id') id: string) {
this.logger.info('getCustomerTheme with id' + id);
try {
const data = await this.themeService.getThemeByCustomer(id);
this.logger.info('Success - getCustomerTheme'+ JSON.stringify(data));
if (data?.theme) return data;
return { theme: null };
}catch (err) {
if (err.details === ErrorCodes.CUSTOMER.NOT_FOUND) {
this.logger.error('Error - getCustomerTheme - Expect CUSTOMER.NOT_FOUND');
throw new HttpException(err.details, HttpStatus.NOT_FOUND);
} else {
this.logger.error('Error - getCustomerTheme Unknown Error:' + err?.message);
return { theme: null };
};
}
}
@Post('/:id/theme/reset')
@ApiOkResponse({ type: CustomerThemeResponse })
async resetTheme(@Param('id') id: string) {
this.logger.info('getCustomerTheme with id' + id);
try {
await this.themeService.resetTheme(id);
return { theme: null };
}catch (err) {
if (err.details === ErrorCodes.CUSTOMER.NOT_FOUND) {
this.logger.error('Error - getCustomerTheme - Expect CUSTOMER.NOT_FOUND');
throw new HttpException(err.details, HttpStatus.NOT_FOUND);
} else {
this.logger.error('Error - getCustomerTheme Unknown Error:' + err?.message);
return { theme: null };
};
}
}
}
+20
View File
@@ -0,0 +1,20 @@
import { Module } from '@nestjs/common';
import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
import { ClientsModule } from '@nestjs/microservices';
import { ThemeController } from './theme.controller';
import { ThemeService } from './theme.service';
import { DucClient } from '../duc/client.config';
const ducClient = new DucClient();
@Module({
imports: [
ClientsModule.register([
ducClient.providerOptions,
]),
],
controllers: [ThemeController],
providers: [ThemeService, DadosferaLogger],
exports: [ThemeService],
})
export class ThemeModule {}
+170
View File
@@ -0,0 +1,170 @@
import {
OnModuleInit,
Inject,
Injectable,
HttpException,
HttpStatus,
InternalServerErrorException,
} from '@nestjs/common';
import { firstValueFrom, lastValueFrom, ReplaySubject } from 'rxjs';
import { Link } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/entities';
import { DucClient } from '../duc/client.config';
import { ClientGrpc } from '@nestjs/microservices';
import { ProtoServices } from '@dadosfera/protospack-v2/dist/lib/Duc';
import { CustomerUpdateRequest } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import { ThemeProtoService } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/write-service';
import { ThemeRequest } from '@dadosfera/protospack-v2/dist/lib/Duc/interfaces/messages';
import { CustomerThemeRequest, CustomerThemeResponse } from './dtos/customers';
import DadosferaLogger from '@dadosfera/dadosfera-logs';
import { resolve } from 'path';
import { Readable } from 'stream';
type Files = {
logo: Express.Multer.File,
logoLogin: Express.Multer.File,
}
@Injectable()
export class ThemeService implements OnModuleInit {
private themeService: ThemeProtoService;
private logger: DadosferaLogger;
constructor(
@Inject(DucClient.name) private readonly grpcClient: ClientGrpc,
@Inject(DadosferaLogger)
dadosferaLogger: DadosferaLogger,
) {
this.logger = dadosferaLogger.logger;
}
onModuleInit() {
this.themeService = this.grpcClient.getService<ThemeProtoService>(
ProtoServices.ThemeProtoService,
);
}
async resetTheme(id: string) {
const { theme } = await firstValueFrom(
this.themeService.ResetCustomerTheme({
id
}),
);
return {
theme
}
}
async createThemeByCustomer(id: string, theme: CustomerThemeRequest & Files) {
if (!id) {
this.logger.error('Error - saveCustomertheme - not found id:' + id);
throw new HttpException(null, HttpStatus.BAD_REQUEST);
}
const customerThemeRequest$ = new ReplaySubject<ThemeRequest>();
customerThemeRequest$.next({
customerId: id,
displayName: theme.displayName,
backgroundColor: theme.backgroundColor,
textColor: theme.textColor,
isMetadata: true,
filename: '',
chunk: Buffer.alloc(0)
})
if(theme.logo) {
await this.sendFile(theme.logo, customerThemeRequest$);
}
if(theme.logoLogin) {
await this.sendFile(theme.logoLogin, customerThemeRequest$);
}
customerThemeRequest$.complete();
const stream = this.themeService.CustomerCreateTheme(customerThemeRequest$);
return lastValueFrom(stream);
}
async getThemeByCustomer(id: string): Promise<CustomerThemeResponse> {
if (!id) {
this.logger.error('Error - getCustomerTheme - not found id:' + id);
throw new HttpException(null, HttpStatus.BAD_REQUEST);
}
const { theme } = await firstValueFrom(
this.themeService.CustomerGetTheme({
id
}),
);
return {
theme
}
}
private async sendFile(file: Express.Multer.File, stream$: ReplaySubject<ThemeRequest>) {
const chunkSize = 4 * 1024 * 1024;
const bufferStream = new CustomBufferStream(file.buffer, chunkSize);
const parseMimitypeForExtension = {
'image/jpeg': '.jpeg',
'image/jpg': '.jpg',
'image/png': '.png',
'image/svg+xml': '.svg',
}
const extension = parseMimitypeForExtension[file.mimetype];
return new Promise((resolve, reject) => {
bufferStream.on('data', (chunk) => {
const filename = file.fieldname.concat(extension);
stream$.next({
customerId: '',
displayName: '',
backgroundColor: '',
textColor: '',
isMetadata: false,
filename: filename,
chunk: chunk
});
});
bufferStream.on('end', () => {
resolve(file.filename)
});
bufferStream.on('error', (err) => {
reject(err);
});
});
}
}
class CustomBufferStream extends Readable {
buffer: Buffer;
offset: number;
chunkSize: number;
constructor(buffer: Buffer, chunkSize: number) {
super({ highWaterMark: chunkSize }); // Configura o tamanho do chunk
this.buffer = buffer;
this.offset = 0;
this.chunkSize = chunkSize;
}
_read() {
if (this.offset < this.buffer.length) {
const end = Math.min(this.offset + this.chunkSize, this.buffer.length);
const copiedBuf = Uint8Array.prototype.slice.call(this.buffer);
const chunk = copiedBuf.slice(this.offset, end);
this.offset = end;
this.push(chunk);
} else {
this.push(null);
}
}
}

Some files were not shown because too many files have changed in this diff Show More