Merge pull request #514 from dadosfera/feat/migration-nimbus-data-beta

Feat/migration nimbus data beta
FEAT: add data_preview field to data asset creation in CatalogService
2026-09-01 12:18:15 +00:00 · 2026-08-25 17:27:45 -03:00 · 2026-08-25 17:14:22 -03:00 · 2026-08-25 17:05:34 -03:00 · 2026-08-25 14:06:25 -03:00 · 2026-08-25 12:08:47 -03:00
17 changed files with 681 additions and 15 deletions
@@ -900,6 +900,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -949,6 +952,9 @@
            "connections"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -997,6 +1003,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -1033,6 +1042,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -1066,6 +1078,9 @@
            "connections"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2403,6 +2418,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2486,6 +2504,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2525,6 +2546,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2564,6 +2588,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2601,6 +2628,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2640,6 +2670,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2681,6 +2714,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -2722,6 +2758,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3071,6 +3110,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3120,6 +3162,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3190,6 +3235,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3230,6 +3278,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3270,6 +3321,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3310,6 +3364,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3350,6 +3407,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3397,6 +3457,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3442,6 +3505,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3490,6 +3556,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3528,6 +3597,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3586,6 +3658,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3634,6 +3709,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3674,6 +3752,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3716,6 +3797,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3765,6 +3849,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3814,6 +3901,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -3861,6 +3951,9 @@
            "PipelinesV2"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4031,6 +4124,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4056,6 +4152,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4092,6 +4191,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4126,6 +4228,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4160,6 +4265,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4187,6 +4295,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4214,6 +4325,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4241,6 +4355,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4277,6 +4394,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4313,6 +4433,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4340,6 +4463,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4367,6 +4493,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4404,6 +4533,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4440,6 +4572,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4484,6 +4619,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4520,6 +4658,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4564,6 +4705,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4620,6 +4764,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4656,6 +4803,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4690,6 +4840,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4726,6 +4879,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4762,6 +4918,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4799,6 +4958,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4826,6 +4988,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -4853,6 +5018,9 @@
            "Platform API"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5264,6 +5432,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5307,6 +5478,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5416,6 +5590,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5448,6 +5625,9 @@
            "Catalog"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5490,6 +5670,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5625,6 +5808,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5683,6 +5869,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5731,6 +5920,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5820,6 +6012,60 @@
          "tags": [
            "Catalog"
          ],
+          "security": [
+            {
+              "access-token": []
+            },
+            {
+              "access-token": []
+            },
+            {
+              "access-token": []
+            }
+          ]
+        },
+        "patch": {
+          "operationId": "CatalogController_updateColumnsMetadata",
+          "parameters": [
+            {
+              "name": "dadosfera-lang",
+              "in": "header",
+              "required": false,
+              "schema": {
+                "enum": [
+                  "pt-br",
+                  "en-us"
+                ],
+                "type": "string"
+              }
+            },
+            {
+              "name": "id",
+              "required": true,
+              "in": "path",
+              "schema": {
+                "type": "string"
+              }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/UpdateColumnsMetadataRequest"
+                }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": ""
+            }
+          },
+          "tags": [
+            "Catalog"
+          ],
          "security": [
            {
              "access-token": []
@@ -5874,6 +6120,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5932,6 +6181,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -5988,6 +6240,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6048,6 +6303,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6088,6 +6346,9 @@
            "Catalog"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6128,6 +6389,9 @@
            "Catalog"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6188,6 +6452,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6246,6 +6513,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -6622,6 +6892,9 @@
            {
              "access-token": []
            },
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -7075,6 +7348,9 @@
            "Connection Test"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -7111,6 +7387,9 @@
            "Connection Test"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -7147,6 +7426,9 @@
            "Connection Test"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -7183,6 +7465,9 @@
            "Connection Test"
          ],
          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -7219,6 +7504,97 @@
            "Connection Test"
          ],
          "security": [
+            {
+              "access-token": []
+            },
+            {
+              "access-token": []
+            }
+          ]
+        }
+      },
+      "/connection-test/refresh-catalog": {
+        "post": {
+          "operationId": "ConnectionTestController_refreshCatalog",
+          "parameters": [],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/RefreshCatalogReq"
+                }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "",
+              "content": {
+                "application/json": {
+                  "schema": {
+                    "$ref": "#/components/schemas/RefreshCatalogRes"
+                  }
+                }
+              }
+            },
+            "202": {
+              "description": "",
+              "content": {
+                "application/json": {
+                  "schema": {
+                    "$ref": "#/components/schemas/RefreshCatalogRes"
+                  }
+                }
+              }
+            }
+          },
+          "tags": [
+            "Connection Test"
+          ],
+          "security": [
+            {
+              "access-token": []
+            },
+            {
+              "access-token": []
+            }
+          ]
+        }
+      },
+      "/connection-test/refresh-catalog/status": {
+        "post": {
+          "operationId": "ConnectionTestController_refreshCatalogStatus",
+          "parameters": [],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/RefreshCatalogStatusReq"
+                }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "",
+              "content": {
+                "application/json": {
+                  "schema": {
+                    "$ref": "#/components/schemas/RefreshCatalogRes"
+                  }
+                }
+              }
+            }
+          },
+          "tags": [
+            "Connection Test"
+          ],
+          "security": [
+            {
+              "access-token": []
+            },
            {
              "access-token": []
            }
@@ -10985,6 +11361,35 @@
            "columns_metadata"
          ]
        },
+        "ColumnDescriptionDto": {
+          "type": "object",
+          "properties": {
+            "column_name": {
+              "type": "string"
+            },
+            "description": {
+              "type": "string"
+            }
+          },
+          "required": [
+            "column_name",
+            "description"
+          ]
+        },
+        "UpdateColumnsMetadataRequest": {
+          "type": "object",
+          "properties": {
+            "columns": {
+              "type": "array",
+              "items": {
+                "$ref": "#/components/schemas/ColumnDescriptionDto"
+              }
+            }
+          },
+          "required": [
+            "columns"
+          ]
+        },
        "IData": {
          "type": "object",
          "properties": {
@@ -11581,11 +11986,15 @@
            },
            "type": {
              "type": "string"
+            },
+            "is_primary_key": {
+              "type": "boolean"
            }
          },
          "required": [
            "name",
-            "type"
+            "type",
+            "is_primary_key"
          ]
        },
        "TableMetadataDto": {
@@ -11631,6 +12040,79 @@
            "tables_metadata"
          ]
        },
+        "RefreshCatalogReq": {
+          "type": "object",
+          "properties": {
+            "connection_id": {
+              "type": "string"
+            },
+            "plugin": {
+              "type": "string",
+              "enum": [
+                "oracle",
+                "mysql",
+                "postgresql",
+                "sqlserver"
+              ]
+            }
+          },
+          "required": [
+            "connection_id",
+            "plugin"
+          ]
+        },
+        "RefreshCatalogRes": {
+          "type": "object",
+          "properties": {
+            "operation_result": {
+              "type": "boolean"
+            },
+            "status": {
+              "type": "string"
+            },
+            "session_id": {
+              "type": "string"
+            },
+            "date": {
+              "type": "string"
+            }
+          },
+          "required": [
+            "operation_result",
+            "status",
+            "session_id",
+            "date"
+          ]
+        },
+        "RefreshCatalogStatusReq": {
+          "type": "object",
+          "properties": {
+            "connection_id": {
+              "type": "string"
+            },
+            "plugin": {
+              "type": "string",
+              "enum": [
+                "oracle",
+                "mysql",
+                "postgresql",
+                "sqlserver"
+              ]
+            },
+            "session_id": {
+              "type": "string"
+            },
+            "date": {
+              "type": "string"
+            }
+          },
+          "required": [
+            "connection_id",
+            "plugin",
+            "session_id",
+            "date"
+          ]
+        },
        "INote": {
          "type": "object",
          "properties": {
@@ -16,7 +16,7 @@
        "@aws-sdk/lib-dynamodb": "^3.414.0",
        "@aws-sdk/signature-v4": "^3.370.0",
        "@dadosfera/dadosfera-logs": "^1.0.0-beta.4",
-        "@dadosfera/protospack-v2": "^3.40.0-beta.14",
+        "@dadosfera/protospack-v2": "^3.40.0-beta.19",
        "@grpc/grpc-js": "^1.9.3",
        "@grpc/proto-loader": "^0.7.9",
        "@nestjs/cli": "^9.5.0",
@@ -1735,9 +1735,10 @@
      }
    },
    "node_modules/@dadosfera/protospack-v2": {
-      "version": "3.40.0-beta.14",
-      "resolved": "https://dadosfera-611330257153.d.codeartifact.us-east-1.amazonaws.com/npm/dadosfera-npm/@dadosfera/protospack-v2/-/protospack-v2-3.40.0-beta.14.tgz",
-      "integrity": "sha512-pv3pxq0x1XcBgf3ajD6QOFRLOduh8iEozKFA3AKlIW4gid+gT4iL0GcU2M+O7h0QFeO4JIzRZe/nEMN82nqk7A==",
+      "version": "3.40.0-beta.19",
+      "resolved": "https://dadosfera-611330257153.d.codeartifact.us-east-1.amazonaws.com/npm/dadosfera-npm/@dadosfera/protospack-v2/-/protospack-v2-3.40.0-beta.19.tgz",
+      "integrity": "sha512-RT7BYWHD2aD945FXzDXHa/xM+sD2B2Ypepy/W/WS+RwKYTSMCfuUXGzK2lXcd9HP9+U+Zjfh90Lkktl++8enjg==",
+      "license": "ISC",
      "dependencies": {
        "@grpc/grpc-js": "^1.9.3",
        "rxjs": "^7.5.5"
@@ -34,7 +34,7 @@
    "@aws-sdk/lib-dynamodb": "^3.414.0",
    "@aws-sdk/signature-v4": "^3.370.0",
    "@dadosfera/dadosfera-logs": "^1.0.0-beta.4",
-    "@dadosfera/protospack-v2": "^3.40.0-beta.14",
+    "@dadosfera/protospack-v2": "^3.40.0-beta.19",
    "@grpc/grpc-js": "^1.9.3",
    "@grpc/proto-loader": "^0.7.9",
    "@nestjs/cli": "^9.5.0",
@@ -722,6 +722,8 @@ export const DADOSFERA_MODULES_KEYS = {
  PII: 'pii',
  EMBED: 'embedded-analytics',
  EMBED_ASSIGNED: 'embed-assigned',
+  CATALOG: 'catalog',
+  COLLECT: 'collect',
 }

 export const DADOSFERA_MODULES: Array<DadosferaModule> = [
@@ -37,6 +37,7 @@ import {
  RequireAllPermissions,
 } from 'src/decorators/authentication.decorator';
 import { AuthClientService } from './auth.service';
+import { UserDTO } from './dtos/login';
 import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
 import { GrpcToHttpExceptionFilter } from '../../error/grpc-to-http-exception.filter';
 import { RequestUser, User } from 'src/decorators/user.decorator';
@@ -486,7 +487,7 @@ export class AuthController {
      this.logger.info('Authenticating via X-Api-key header');
      const { api_key } = await this.apiKeyService.get(apiKey);

-      const userDto = {
+      const userDto: UserDTO = {
        id: api_key.user_id,
        name: api_key.username,
        email: api_key.username,
@@ -494,7 +495,8 @@ export class AuthController {
          id: api_key.customer_id,
          name: api_key.customer_name,
          tier: api_key.customer_tier,
-        }
+        },
+        permissions: [],
      };

      return res.status(200).json(userDto);
@@ -447,6 +447,9 @@ export class AuthClientService implements OnModuleInit {
        name: payload.customer_name,
        tier: payload.customer_tier,
      },
+      // Raw permission seqids from the JWT. Consumers own the seqid->meaning
+      // mapping (e.g. Orchest's auth-server); Maestro reports them as-is.
+      permissions: payload.permissions ?? [],
    };

    return userDto;
@@ -152,5 +152,6 @@ export type UserDTO = {
    id: string,
    name: string,
    tier: string,
-  }
+  },
+  permissions: number[],
 }
@@ -9,6 +9,7 @@ import {
  Inject,
  NotFoundException,
  Param,
+  Patch,
  Post,
  Put,
  Query,
@@ -51,6 +52,7 @@ import {
  IUpdateDataRequest,
  TriggerCatalogReq,
  TriggerCatalogRes,
+  UpdateColumnsMetadataRequest,
 } from './dtos';
 import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
 import { Language } from 'src/decorators/language.decorator';
@@ -85,6 +87,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async searchCatalog(
    @User() user: RequestUser,
    @Query() query: ICatalogAllRequest,
@@ -124,6 +129,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async dowloadAsserts(
    @User() user: RequestUser,
    @Query() query: ICatalogAllRequest,
@@ -167,6 +175,9 @@ export class CatalogController {

  @ApiInternalOnlyEndpoint()
  @Get('data-asset')
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async findByPipelineAndObject(@User() user: RequestUser, @Query() query) {
    const { username, user_id, customer_id, customer_name, permissions } = user;
    const { pipeline, object } = query;
@@ -225,6 +236,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async findAllTags(@Body() body) {
    this.logger.info(`/catalog - ON FIND ALL TAGS ROUTE`, {
      user: body.info.user_id,
@@ -293,6 +307,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async getDataAsset(
    @User() user: RequestUser,
    @Param('id') id: string,
@@ -404,6 +421,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async getDataAssetColumnsMetadata(
    @User() user: RequestUser,
    @Language() language: LanguageEnum,
@@ -430,11 +450,50 @@ export class CatalogController {
    return { columns_metadata };
  }

+  @Patch('data-asset/:id/columns-metadata')
+  @RequireSomePermission(
+    PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
+    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
+  )
+  async updateColumnsMetadata(
+    @User() user: RequestUser,
+    @Language() language: LanguageEnum,
+    @Param('id') id: string,
+    @Body(new ValidationPipe()) body: UpdateColumnsMetadataRequest,
+  ): Promise<{ success: boolean }> {
+    const { customer_name, customer_id, user_id, username } = user;
+
+    this.logger.info(`/catalog - update columns metadata`, {
+      user_id,
+      customer_name,
+      columns_count: body.columns.length,
+    });
+
+    const metadata = PackTheMetadata({
+      customer_name,
+      customer_id,
+      user_id,
+      username,
+      language,
+    });
+
+    await this.catalogService.updateColumnsDescriptions(
+      id,
+      body.columns,
+      metadata,
+    );
+
+    return { success: true };
+  }
+
  @Get('data-asset/:id/preview')
  @RequireSomePermission(
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async getDataAssetPreview(
    @User() user: RequestUser,
    @Language() language: LanguageEnum,
@@ -466,6 +525,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.GET,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async getDataAssetDocs(
    @User() user: RequestUser,
    @Language() language: LanguageEnum,
@@ -497,6 +559,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async updateDataAsset(
    @User() user: RequestUser,
    @Language() language: LanguageEnum,
@@ -532,6 +597,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.CERTIFY,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async updateDataAssetCertificationStatus(
    @User() user: RequestUser,
    @Language() language: LanguageEnum,
@@ -559,6 +627,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async manageDataAssetDocs(
    @User() user: RequestUser,
    @Headers() headers,
@@ -596,6 +667,9 @@ export class CatalogController {

  @ApiInternalOnlyEndpoint()
  @Put('data-asset/:id/manage-permissions')
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async manageDataAssetPermissions(
    @Param('id') id: string,
    @User() user: RequestUser,
@@ -618,6 +692,9 @@ export class CatalogController {

  @ApiInternalOnlyEndpoint()
  @Put('data-asset/:id/revoke-permissions')
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async revokeDataAssetPermissions(
    @Param('id') id: string,
    @User() user: RequestUser,
@@ -643,6 +720,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.CREATE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async createDataAsset(
    @User() user: RequestUser,
    @Body() body: ICreateDataAsset,
@@ -667,6 +747,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async commentOnDataAsset(
    @Param('id') id: string,
    @User() user: RequestUser,
@@ -693,6 +776,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.DELETE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async deleteDataAsset(@Param('id') id: string, @User() user: RequestUser) {
    const { customer_id, customer_name, user_id, username } = user;
    const metadata = PackTheMetadata({
@@ -714,6 +800,9 @@ export class CatalogController {
    PERMISSIONS_GROUPS.CATALOG.permissions.UPDATE,
    PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async deleteComment(
    @Param('id') id: string,
    @User() user: RequestUser,
@@ -867,6 +956,9 @@ export class CatalogController {

  @Get('nimbus-dashboards')
  @RequireAllPermissions(PERMISSIONS_GROUPS.CATALOG.permissions.DATA_MANAGER)
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.CATALOG
+  )
  async getNimbusDashboards(
    @User() user: RequestUser,
    @Body() body: GetNimbusDashboardsRequest,
@@ -467,6 +467,16 @@ class CatalogService implements OnModuleInit {
    return result;
  }

+  async updateColumnsDescriptions(
+    id: string,
+    columns: { column_name: string; description: string }[],
+    metadata: Metadata,
+  ) {
+    await lastValueFrom(
+      this.catalogWriteService.UpdateColumnDescriptions({ id, columns }, metadata),
+    );
+  }
+
  async createDataDocs(body: CreateDataDocsDTO, metadata: Metadata) {
    if (body.asset_type === 'table' || body.asset_type === 'view') {
      return this.createDataDocsViaNimbus(body);
@@ -832,6 +842,8 @@ class CatalogService implements OnModuleInit {
              data_asset_id: table_metadata_id.toString(),
              customer_name: customer_name,
              data_asset_type: 'dataset',
+              column_metadata: [],
+              data_preview: '',
            },
          ],
        },
@@ -1,10 +1,13 @@
 import { ApiProperty, ApiPropertyOptional, PickType } from '@nestjs/swagger';
 import {
+  ArrayNotEmpty,
+  IsArray,
  IsEnum,
  IsNotEmpty,
-  IsOptional,
  IsString,
+  ValidateNested,
 } from 'class-validator';
+import { Type } from 'class-transformer';
 import { CreateDataAssetRequest } from '@dadosfera/protospack-v2/dist/lib/Catalog/interfaces/messages';

 export enum DataAssetShareType {
@@ -247,6 +250,26 @@ export class IUpdateCertificationStatusRequest {
  certification_status: CertificationStatus;
 }

+export class ColumnDescriptionDto {
+  @ApiProperty()
+  @IsString()
+  @IsNotEmpty()
+  column_name: string;
+
+  @ApiProperty()
+  @IsString()
+  description: string;
+}
+
+export class UpdateColumnsMetadataRequest {
+  @ApiProperty({ type: [ColumnDescriptionDto] })
+  @IsArray()
+  @ArrayNotEmpty()
+  @ValidateNested({ each: true })
+  @Type(() => ColumnDescriptionDto)
+  columns: ColumnDescriptionDto[];
+}
+
 export class ICreateDataAsset implements CreateDataAssetRequest {
  @ApiProperty()
  display_name: string;
@@ -27,15 +27,19 @@ import {
  RefreshCatalogStatusReq,
 } from './dto/connection-test';
 import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
-import { Authenticated } from 'src/decorators/authentication.decorator';
+import { Authenticated, RequireModule } from 'src/decorators/authentication.decorator';
 import { GrpcToHttpExceptionFilter } from 'src/error/grpc-to-http-exception.filter';
 import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
+import { DADOSFERA_MODULES_KEYS } from 'src/authentication/permissions.enum';

@ApiInternalOnlyController()
@ApiTags('Connection Test')
@Controller('connection-test')
@UseFilters(new GrpcToHttpExceptionFilter())
@Authenticated()
+@RequireModule(
+  DADOSFERA_MODULES_KEYS.COLLECT
+)
 export class ConnectionTestController {
  logger: any;
  constructor(
@@ -16,8 +16,9 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
 import {
  Authenticated,
  RequireAllPermissions,
+  RequireModule,
 } from 'src/decorators/authentication.decorator';
-import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
+import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
 import { RequestUser, User } from 'src/decorators/user.decorator';
 import { ValidationPipe } from '../../pipes/object-validation.pipe';
 import {
@@ -39,6 +40,9 @@ const connectionPermissions = PERMISSIONS_GROUPS.CONNECTION.permissions;
@ApiTags('connections')
@Authenticated()
@Controller('connections')
+@RequireModule(
+  DADOSFERA_MODULES_KEYS.COLLECT
+)
 export class ConnectionController {
  logger: any;
  constructor(
@@ -25,9 +25,10 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
 import {
  Authenticated,
  RequireAllPermissions,
+  RequireModule,
  RequireSomePermission,
 } from 'src/decorators/authentication.decorator';
-import { PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
+import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from 'src/authentication/permissions.enum';
 import { Language } from 'src/decorators/language.decorator';
 import { LanguageEnum } from 'src/utils/languages.enum';
 import { ApiInternalOnlyController } from 'src/decorators/swagger.decorator';
@@ -99,6 +100,9 @@ export class ConnectorController {
    PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
    PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async getAllConnectors(
    @Language() language: LanguageEnum,
    @Query() queries: GetAllDto,
@@ -131,6 +135,9 @@ export class ConnectorController {
    PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
    PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async getConnectorsTags() {
    return await this.connectorClientService.getConnectorsTags();
  }
@@ -143,6 +150,9 @@ export class ConnectorController {
    PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
    PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async getConnector(
    @Language() language: LanguageEnum,
    @Param('plugin') plugin: string,
@@ -171,6 +181,9 @@ export class ConnectorController {
    PERMISSIONS_GROUPS.PIPELINE.permissions.UPDATE,
    PERMISSIONS_GROUPS.PIPELINE.permissions.DELETE,
  )
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async getConnectorDetails(
    @Language() language: LanguageEnum,
    @Param('plugin') plugin: string,
@@ -193,6 +206,9 @@ export class ConnectorController {
  @Put('/:plugin')
  @RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
  @ApiConsumes('multipart/form-data')
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async updateConnector(
    @Param('plugin') plugin: string,
    @Body() body: UpdateDto,
@@ -214,6 +230,9 @@ export class ConnectorController {

  @Put('/:plugin/add-tag')
  @RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async addTagOnConnector(
    @Param('plugin') plugin: string,
    @Body() body: AddTagDto,
@@ -241,6 +260,9 @@ export class ConnectorController {

  @Put('/:plugin/remove-tag')
  @RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.UPDATE)
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async removeTagOnConnector(
    @Param('plugin') plugin: string,
    @Body() body: RemoveTagDto,
@@ -269,6 +291,9 @@ export class ConnectorController {

  @Delete('/:plugin')
  @RequireAllPermissions(PERMISSIONS_GROUPS.CONNECTORS.permissions.DELETE)
+  @RequireModule(
+    DADOSFERA_MODULES_KEYS.COLLECT
+  )
  async deleteConnector(
    @Param('plugin') plugin: string,
    @Query('version') version: string,
@@ -25,9 +25,10 @@ import {
 } from '@nestjs/swagger';
 import {
  RequireAllPermissions,
+  RequireModule,
  RequireSomePermission,
 } from 'src/decorators/authentication.decorator';
-import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
+import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
 import { PipelinesService } from './pipelines.service';
 import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
 import { Messages } from '@dadosfera/protospack-v2/dist/lib/PipelineV2';
@@ -57,6 +58,9 @@ type PipelineTablesConfig = { input_id?: string; tables: PipelineTable[] };
@ApiHeaders([{ name: 'dadosfera-lang', enum: LanguageEnum, required: false }])
@UseFilters(new GrpcToHttpExceptionFilter())
@Controller('pipelinesV2')
+@RequireModule(
+  DADOSFERA_MODULES_KEYS.COLLECT
+)
 export class PipelinesController {
  logger: DadosferaLogger;
  constructor(
@@ -20,10 +20,11 @@ import { DadosferaLogger } from '@dadosfera/dadosfera-logs';
 import {
  Authenticated,
  RequireAllPermissions,
+  RequireModule,
 } from '../../decorators/authentication.decorator';
 import { User, RequestUser } from '../../decorators/user.decorator';
 import { PlatformApiService } from './platform-api.service';
-import { PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
+import { DADOSFERA_MODULES_KEYS, PERMISSIONS_GROUPS } from '../../authentication/permissions.enum';
 import { ElasticsearchService } from '../../services/elasticsearch';
 import { DynamoDBService, ReferenceColumn } from '../../services/dynamodb';
 import { CustomersService } from '../customers/customers.service';
@@ -49,6 +50,7 @@ type RenameTablesBody = {

@ApiTags('Platform API')
@Controller('platform')
+@RequireModule(DADOSFERA_MODULES_KEYS.COLLECT)
 export class PlatformApiController {
  private logger: any;

@@ -315,6 +315,14 @@ export function EnrichErrorCode(code: string) {
          'Tente realizar a ação novamente. Caso o erro persista, entre em contato com o suporte',
        code,
      };
+    case ErrorCodes.CATALOG.COLUMN_NOT_FOUND:
+      return {
+        statusCode: HttpStatus.NOT_FOUND,
+        error: 'Coluna não encontrada',
+        message:
+          'Uma ou mais colunas informadas não existem neste ativo. Verifique os nomes e tente novamente.',
+        code,
+      };
    case ErrorCodes.CATALOG.PREVIEW_TOO_BIG:
      return {
        statusCode: HttpStatus.INTERNAL_SERVER_ERROR,
@@ -74,6 +74,7 @@ const CATALOG = {
  DATA_ASSET_NOT_FOUND: 'CATALOG.DATA_ASSET_NOT_FOUND',
  PREVIEW_TOO_BIG: 'CATALOG.PREVIEW_TOO_BIG',
  METADATA_TOO_BIG: 'CATALOG.METADATA_TOO_BIG',
+  COLUMN_NOT_FOUND: 'CATALOG.COLUMN_NOT_FOUND',
 };
 const IDENTITY_PROVIDER = {
  INVALID_RESPONSE: 'IDENTITY_PROVIDER.INVALID_RESPONSE',
Author	SHA1	Message	Date
vinicius gadea	c1f5c013ee	Merge pull request #514 from dadosfera/feat/migration-nimbus-data-beta Feat/migration nimbus data beta	2026-08-25 17:27:45 -03:00
viniciusgadea	ba609a9494	FEAT: add data_preview field to data asset creation in CatalogService	2026-08-25 17:14:22 -03:00
viniciusgadea	8916dfc46c	FEAT: update @dadosfera/protospack-v2 dependency to version 3.40.0-beta.19	2026-08-25 17:05:34 -03:00
viniciusgadea	cf4e0aea22	FEAT: update protospack-v2 dependency to version 3.40.0-beta.18 and refactor UpdateDataAsset method in CatalogService	2026-08-25 14:06:25 -03:00
viniciusgadea	9d57e69e74	Merge remote-tracking branch 'origin/beta' into feat/migration-nimbus-data-beta # Conflicts: # docsfera.json # src/modules/catalog/dtos/index.ts	2026-08-25 12:08:47 -03:00
viniciusgadea	4db15fceab	FEAT: add request body and DTOs for updating columns metadata in catalog	2026-08-25 09:36:02 -03:00
viniciusgadea	a4f4335e43	FEAT: update columns metadata endpoint to accept batch updates and add error handling for missing columns	2026-08-25 09:05:46 -03:00
RafaelandWOZCODE	bf0314f5b1	FIX: trigger release for /auth/me permission seqids (PR #513 ) PR #513 merged to beta but no semantic-release ran: its commits used conventional-commits prefixes (feat(auth):, fix(auth):) which the .releaserc.json eslint preset does not recognise, so commit-analyzer found no release-worthy change. This empty FIX: commit matches the eslint preset's releaseRules (tag FIX -> patch) to cut a beta release that includes the /auth/me permission-seqids change, so stg can deploy it. Co-Authored-By: WOZCODE <contact@withwoz.com>	2026-08-24 19:18:59 -03:00
Rafael Santana	c97180ac95	Merge pull request #513 from dadosfera/feat/auth-me-orchest-identity feat(auth): return the user's permission seqids from /auth/me	2026-08-24 17:00:15 -03:00
RafaelandWOZCODE	cac36f2c60	refactor(auth): /auth/me returns raw permission seqids Return payload.permissions verbatim (numeric seqids) instead of translating them to claim strings. Consumers own the seqid->meaning mapping. Drops permission-claims.ts entirely; UserDTO.permissions is now number[]. Co-Authored-By: WOZCODE <contact@withwoz.com>	2026-08-24 16:56:23 -03:00
RafaelandWOZCODE	cd21fd0b7b	refactor(auth): /auth/me returns permissions only (drop roles/modules) Keep Maestro a pure identity provider: /auth/me exposes the user's permission claim strings and nothing consumer-specific. Consumers derive whatever meaning they need (roles, module access, groups) from the claim vocabulary — claims are already namespaced group:action. - UserDTO: drop roles[]/modules[], keep permissions[]. - Helper shrinks to a generic seqid->claim translation (orchest-identity.ts -> permission-claims.ts, translateSeqidsToClaims). - api-key branch: permissions: [] only. The roles/modules derivation moves entirely to the consumer (Orchest's auth-server adapter). Co-Authored-By: WOZCODE <contact@withwoz.com>	2026-08-24 16:30:33 -03:00
RafaelandWOZCODE	a5a685ee3f	fix(auth): derive Orchest identity from numeric seqids (JWT carries seqids not claim strings) The JWT `permissions` claim is an array of numeric seqids at runtime (see authentication.guard.ts / authentication.decorator.ts), not claim strings. deriveOrchestIdentity previously matched claim strings against this numeric array, so roles[]/modules[] were always empty for every real user. - deriveOrchestIdentity now takes number[] \| undefined and matches seqids sourced from PERMISSIONS_GROUPS (permissions.enum.ts) instead of hand-copied literals. - permissions is translated back to claim strings via a full seqid->claim catalog built once from PERMISSIONS_GROUPS; unknown seqids are dropped (auth-server ignores permissions[] in v1). - auth.controller.ts's api-key branch literal is now annotated `: UserDTO` so tsc enforces the three fields there. - Both spec files re-fixtured with numeric seqid inputs, including a mixed admin+module case and an exact claim-string translation assertion. Co-Authored-By: WOZCODE <contact@withwoz.com>	2026-08-24 16:03:20 -03:00
Rafael	a16fefe691	feat(auth): return permissions/roles/modules from /auth/me (all branches)	2026-08-24 15:45:09 -03:00
Rafael	9c57485031	feat(auth): pure helper deriving Orchest identity from permissions	2026-08-24 15:39:40 -03:00
viniciusgadea	6efc25ad5e	FEAT: add endpoint to update column description in catalog	2026-08-20 08:26:56 -03:00
marcos.rodrigues	7051b21d86	Merge pull request #509 from dadosfera/release/2026-08-12 FIX: require collect module in endpoints	2026-08-12 15:06:04 -03:00
marcos.rodrigues	0c4888ccdb	FIX: require collect module in endpoints	2026-08-12 15:02:37 -03:00
marcos.rodrigues	e1b0e88bd8	Merge pull request #507 from dadosfera/bugfix/catalog-module-main Bugfix/catalog module main	2026-08-12 09:59:08 -03:00
marcos.rodrigues	cef1184908	Merge branch 'beta' into bugfix/catalog-module-main	2026-08-12 09:58:58 -03:00
marcos.rodrigues	31dda867d1	FIX: require collect module in endpoints	2026-08-12 09:57:23 -03:00
marcos.rodrigues	8a92da470c	Merge pull request #506 from dadosfera/bugfix/catalog-module-main FIX: skip nimbus update when customer haven't catalog module	2026-08-07 16:27:19 -03:00
marcos.rodrigues	b89909ad66	FIX: skip nimbus update when customer haven't catalog module	2026-08-07 14:48:46 -03:00
marcos.rodrigues	2bb280e8de	Merge pull request #505 from dadosfera/bugfix/catalog-module Bugfix/catalog module	2026-08-07 12:26:21 -03:00