Files
DLE/docs-en/tables-system.md

1640 lines
36 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Electronic Tables System in DLE
> **Temporary document for internal analysis**
---
## 📋 Table of Contents
1. [System Overview](#system-overview)
2. [Database Architecture](#database-architecture)
3. [Field Types](#field-types)
4. [Functional Capabilities](#functional-capabilities)
5. [Relations Between Tables](#relations-between-tables)
6. [AI Integration (RAG)](#ai-integration-rag)
7. [API Reference](#api-reference)
8. [Usage Examples](#usage-examples)
9. [Security](#security)
---
## System Overview
### What is it?
The electronic tables system in DLE is a **full-featured database with graphical interface**, similar to **Notion Database** or **Airtable**, built into the application.
### Key Features
```
┌─────────────────────────────────────────────────────────┐
│ DLE Electronic Tables │
├─────────────────────────────────────────────────────────┤
│ │
│ ✅ 6 field types (text, number, relation, lookup, etc.)│
│ ✅ Relations between tables (1:1, 1:N, N:N) │
│ ✅ Lookup and data substitution │
│ ✅ Filtering and sorting │
│ ✅ Real-time updates (WebSocket) │
│ ✅ AI integration (RAG for search) │
│ ✅ Encryption of all data (AES-256) │
│ ✅ Placeholders for API access │
│ ✅ Cascading deletion │
│ ✅ Bulk operations │
│ │
└─────────────────────────────────────────────────────────┘
```
### Differences from Excel/Google Sheets
| Feature | Excel/Sheets | DLE Tables |
|---------|--------------|------------|
| **Data typing** | Weak | Strict (6 types) |
| **Relations between tables** | No | Yes (relation, lookup) |
| **AI search** | No | Yes (RAG, vector search) |
| **Real-time updates** | Partial | Full (WebSocket) |
| **Encryption** | No | AES-256 for all data |
| **API access** | Limited | Full REST API |
| **Access rights** | Basic | Detailed (by roles) |
---
## Database Architecture
### Table Schema (PostgreSQL)
```sql
┌──────────────────────────────────────────────────────────┐
user_tables
├──────────────────────────────────────────────────────────┤
id SERIAL PRIMARY KEY
name_encrypted TEXT NOT NULL
description_encrypted TEXT
is_rag_source_id INTEGER (link to is_rag_source)
created_at TIMESTAMP
updated_at TIMESTAMP
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
user_columns
├──────────────────────────────────────────────────────────┤
id SERIAL PRIMARY KEY
table_id INTEGER user_tables(id)
name_encrypted TEXT NOT NULL
type_encrypted TEXT NOT NULL
placeholder_encrypted TEXT (for API)
placeholder TEXT (unencrypted)
options JSONB (settings)
order INTEGER (display order)
created_at TIMESTAMP
updated_at TIMESTAMP
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
user_rows
├──────────────────────────────────────────────────────────┤
id SERIAL PRIMARY KEY
table_id INTEGER user_tables(id)
order INTEGER (row order)
created_at TIMESTAMP
updated_at TIMESTAMP
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
user_cell_values
├──────────────────────────────────────────────────────────┤
id SERIAL PRIMARY KEY
row_id INTEGER user_rows(id)
column_id INTEGER user_columns(id)
value_encrypted TEXT (encrypted value)
created_at TIMESTAMP
updated_at TIMESTAMP
UNIQUE(row_id, column_id) One cell = one value
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
user_table_relations
├──────────────────────────────────────────────────────────┤
id SERIAL PRIMARY KEY
from_row_id INTEGER user_rows(id)
column_id INTEGER user_columns(id)
to_table_id INTEGER user_tables(id)
to_row_id INTEGER user_rows(id)
created_at TIMESTAMP
updated_at TIMESTAMP
└──────────────────────────────────────────────────────────┘
```
### Cascading Deletion
```
Table deletion (user_tables)
├── Deletes all columns (user_columns)
├── Deletes all rows (user_rows)
│ └── Deletes all cell values (user_cell_values)
└── Deletes all relations (user_table_relations)
```
**Important**: Uses `ON DELETE CASCADE` for automatic cleanup.
### Indexes for Performance
```sql
-- Indexes on relations (user_table_relations)
CREATE INDEX idx_user_table_relations_from_row ON user_table_relations(from_row_id);
CREATE INDEX idx_user_table_relations_column ON user_table_relations(column_id);
CREATE INDEX idx_user_table_relations_to_table ON user_table_relations(to_table_id);
CREATE INDEX idx_user_table_relations_to_row ON user_table_relations(to_row_id);
```
**Effect**: Fast filtering and search by related tables.
---
## Field Types
### 1. Text
**Description**: Regular text field
```json
{
"type": "text",
"options": null
}
```
**Usage**:
- Names
- Descriptions
- Email
- URL
- Any text
### 2. Number
**Description**: Numeric field
```json
{
"type": "number",
"options": null
}
```
**Usage**:
- Prices
- Quantities
- Ratings
- Percentages
### 3. Multiselect
**Description**: Multiple value selection from list
```json
{
"type": "multiselect",
"options": {
"choices": ["Option 1", "Option 2", "Option 3"]
}
}
```
**Usage**:
- Tags
- Categories
- Statuses
- Skills
### 4. Multiselect-Relation
**Description**: Multiple row selection from another table
```json
{
"type": "multiselect-relation",
"options": {
"relatedTableId": 5,
"relatedColumnId": 12
}
}
```
**Usage**:
- Relation Contacts → Tags (N:N)
- Relation Tasks → Assignees (N:N)
- Relation Products → Categories (N:N)
**Storage**: In `user_table_relations` table, multiple records created:
```
from_row_id=100, column_id=3, to_table_id=5, to_row_id=20
from_row_id=100, column_id=3, to_table_id=5, to_row_id=21
from_row_id=100, column_id=3, to_table_id=5, to_row_id=22
```
### 5. Relation
**Description**: Relation with one row from another table (1:1 or 1:N)
```json
{
"type": "relation",
"options": {
"relatedTableId": 3,
"relatedColumnId": 8
}
}
```
**Usage**:
- Task → Project (N:1)
- Contact → Company (N:1)
- Order → Client (N:1)
**Storage**: In `user_table_relations`, one record created:
```
from_row_id=50, column_id=2, to_table_id=3, to_row_id=15
```
### 6. Lookup
**Description**: Automatic value substitution from related table
```json
{
"type": "lookup",
"options": {
"relatedTableId": 4,
"relatedColumnId": 10,
"lookupColumnId": 11 // Which field to substitute
}
}
```
**Example**:
```
Table "Orders"
├── order_id (text)
├── product (relation → Products)
└── product_price (lookup → Products.price)
When you select product, price is automatically substituted!
```
**Usage**:
- Prices from catalog
- Email from contacts
- Statuses from related tasks
---
## Functional Capabilities
### 1. CRUD Operations
#### Create Table
```javascript
// Frontend
await tablesService.createTable({
name: "Contacts",
description: "Customer database",
isRagSourceId: 2 // Source for AI
});
// Backend: POST /tables
// Encrypts name and description with AES-256
```
#### Add Column
```javascript
await tablesService.addColumn(tableId, {
name: "Email",
type: "text",
order: 2,
purpose: "contact" // For special fields
});
// Backend: POST /tables/:id/columns
// Generates unique placeholder: "email", "email_1", ...
```
#### Add Row
```javascript
await tablesService.addRow(tableId);
// Backend: POST /tables/:id/rows
// Automatically indexes in vector store for AI
```
#### Update Cell (Upsert)
```javascript
await tablesService.saveCell({
row_id: 123,
column_id: 5,
value: "new@email.com"
});
// Backend: POST /tables/cell
// INSERT ... ON CONFLICT ... DO UPDATE
// Automatically updates vector store
```
#### Delete Row
```javascript
await tablesService.deleteRow(rowId);
// Backend: DELETE /tables/row/:rowId
// Rebuilds vector store (rebuild)
```
#### Delete Column
```javascript
await tablesService.deleteColumn(columnId);
// Backend: DELETE /tables/column/:columnId
// Cascading deletion:
// 1. All relations (user_table_relations)
// 2. All cell values (user_cell_values)
// 3. Column itself
```
#### Delete Table
```javascript
await tablesService.deleteTable(tableId);
// Backend: DELETE /tables/:id
// Required: req.session.userAccessLevel?.hasAccess
// Cascading deletion of all related data
```
### 2. Data Filtering
#### By Product
```javascript
GET /tables/5/rows?product=Premium
// Backend filters rows:
filtered = rows.filter(r => r.product === 'Premium');
```
#### By Tags
```javascript
GET /tables/5/rows?tags=VIP,B2B
// Backend filters rows with any of tags:
filtered = rows.filter(r =>
r.userTags.includes('VIP') || r.userTags.includes('B2B')
);
```
#### By Relations
```javascript
GET /tables/5/rows?relation_12=45,46
// Filter rows related to to_row_id = 45 or 46
// through column column_id = 12
```
#### By Multiselect
```javascript
GET /tables/5/rows?multiselect_8=10,11,12
// All selected values must be present
```
### 3. Placeholder System
**Automatic generation**:
```javascript
// Function: generatePlaceholder(name, existingPlaceholders)
"Customer Name" "customer_name"
"Email" "email"
"Email" (2nd time) "email_1"
"123" "column" (fallback)
"Price-$" "price"
```
**Transliteration**:
```javascript
const cyrillicToLatinMap = {
а: 'a', б: 'b', в: 'v', г: 'g', д: 'd',
е: 'e', ё: 'e', ж: 'zh', з: 'z', и: 'i',
// ... full map
};
```
**Usage**:
```javascript
// API access to data via placeholder
GET /tables/5/data?fields=email,phone,customer_name
```
### 4. Row Order
```javascript
// Change row order (drag-n-drop)
await tablesService.updateRowsOrder(tableId, [
{ rowId: 100, order: 0 },
{ rowId: 101, order: 1 },
{ rowId: 102, order: 2 }
]);
// Backend: PATCH /tables/:id/rows/order
// Updates "order" field for each row
```
### 5. Real-Time Updates (WebSocket)
```javascript
// Backend sends notifications on changes
broadcastTableUpdate(tableId); // Table update
broadcastTableRelationsUpdate(); // Relations update
broadcastTagsUpdate(null, rowId); // Tags update
// Frontend subscribes to events
socket.on('table-update', (data) => {
if (data.tableId === currentTableId) {
reloadTableData();
}
});
```
### 6. Bulk Operations
```javascript
// Select multiple rows (checkbox)
const selectedRows = [100, 101, 102];
// Bulk deletion
for (const rowId of selectedRows) {
await tablesService.deleteRow(rowId);
}
// After deletion: automatic rebuild vector store
```
---
## Relations Between Tables
### Relation Types
#### 1. One-to-Many (N:1) - Relation
**Example**: Tasks → Projects
```
Table "Tasks" Table "Projects"
├── task_1 → project_id=5 ├── project_5 (Website)
├── task_2 → project_id=5 └── project_6 (API)
└── task_3 → project_id=6
```
**Storage**:
```sql
user_table_relations
├── from_row_id=task_1, column_id=3, to_table_id=2, to_row_id=project_5
├── from_row_id=task_2, column_id=3, to_table_id=2, to_row_id=project_5
└── from_row_id=task_3, column_id=3, to_table_id=2, to_row_id=project_6
```
#### 2. Many-to-Many (N:N) - Multiselect-Relation
**Example**: Contacts → Tags
```
Table "Contacts" Table "Tags"
├── contact_1 → [VIP, B2B] ├── tag_1 (VIP)
├── contact_2 → [VIP] ├── tag_2 (B2B)
└── contact_3 → [B2B, Local] └── tag_3 (Local)
```
**Storage**:
```sql
user_table_relations
├── from_row_id=contact_1, column_id=5, to_table_id=3, to_row_id=tag_1
├── from_row_id=contact_1, column_id=5, to_table_id=3, to_row_id=tag_2
├── from_row_id=contact_2, column_id=5, to_table_id=3, to_row_id=tag_1
├── from_row_id=contact_3, column_id=5, to_table_id=3, to_row_id=tag_2
└── from_row_id=contact_3, column_id=5, to_table_id=3, to_row_id=tag_3
```
#### 3. Lookup (Substitution)
**Example**: Orders → Product Price
```
Table "Orders"
├── order_id (text)
├── product (relation → Products)
└── price (lookup → Products.price)
Table "Products"
├── product_name (text)
└── price (number)
```
**How it works**:
1. Select `product = "Laptop"` (relation to product)
2. `price` automatically substituted from `Products.price`
3. If product price changes, lookup updates
### API for Working with Relations
```javascript
// Get all row relations
GET /tables/:tableId/row/:rowId/relations
// Add relation
POST /tables/:tableId/row/:rowId/relations
Body: {
column_id: 12,
to_table_id: 5,
to_row_id: 45
}
// Add multiple relations (multiselect)
POST /tables/:tableId/row/:rowId/relations
Body: {
column_id: 12,
to_table_id: 5,
to_row_ids: [45, 46, 47]
}
// Delete relation
DELETE /tables/:tableId/row/:rowId/relations/:relationId
```
---
## AI Integration (RAG)
### Vector Search
Tables are used as **knowledge base for AI assistant**.
#### Automatic Indexing
**On row creation/modification**:
```javascript
// Backend: POST /tables/:id/rows
const rows = await getTableRows(tableId);
const upsertRows = rows
.filter(r => r.row_id && r.text)
.map(r => ({
row_id: r.row_id,
text: r.text, // Question (question column)
metadata: {
answer: r.answer, // Answer (answer column)
product: r.product, // Product filter
userTags: r.userTags, // Tags filter
priority: r.priority // Priority
}
}));
if (upsertRows.length > 0) {
await vectorSearchClient.upsert(tableId, upsertRows);
}
```
**On row deletion**:
```javascript
// Backend: DELETE /tables/row/:rowId
const rows = await getTableRows(tableId);
const rebuildRows = /* ... */;
if (rebuildRows.length > 0) {
await vectorSearchClient.rebuild(tableId, rebuildRows);
}
```
#### Special Fields for RAG
```javascript
// Columns with purpose
{
"type": "text",
"options": {
"purpose": "question" // Question for AI
}
}
{
"type": "text",
"options": {
"purpose": "answer" // AI Answer
}
}
{
"type": "multiselect",
"options": {
"purpose": "product" // Product filter
}
}
{
"type": "multiselect",
"options": {
"purpose": "userTags" // Tags filter
}
}
```
#### Manual Index Rebuild
```javascript
// Frontend (admins only)
await tablesService.rebuildIndex(tableId);
// Backend: POST /tables/:id/rebuild-index
// Required: req.session.userAccessLevel?.hasAccess
const { questionCol, answerCol } = await getQuestionAnswerColumnIds(tableId);
const rows = await getRowsWithQA(tableId, questionCol, answerCol);
if (rows.length > 0) {
await vectorSearchClient.rebuild(tableId, rows);
}
```
#### How AI Uses Tables
```
1. User asks AI question:
"How to return product?"
2. AI does vector search:
vectorSearch.search(tableId, query, top_k=3)
3. Finds similar questions in table:
- row_id: 123
- text: "How to process product return?"
- score: -250 (close to threshold 300)
- metadata: { answer: "Return within 14 days..." }
4. AI returns answer from metadata.answer
5. If not found (score > 300):
AI generates answer via LLM (Ollama)
```
#### Filtering by Products and Tags
```javascript
// Search only by product "Premium"
const results = await vectorSearch.search(tableId, query, 3);
const filtered = results.filter(r => r.metadata.product === 'Premium');
// Search only by tags "VIP" or "B2B"
const filtered = results.filter(r =>
r.metadata.userTags.includes('VIP') ||
r.metadata.userTags.includes('B2B')
);
```
---
## API Reference
### Tables
#### GET /tables
Get list of all tables
**Response**:
```json
[
{
"id": 1,
"name": "Contacts",
"description": "Customer database",
"is_rag_source_id": 2,
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-01-15T10:00:00Z"
}
]
```
#### POST /tables
Create new table
**Request**:
```json
{
"name": "Contacts",
"description": "Customer database",
"isRagSourceId": 2
}
```
**Response**: Created table object
#### GET /tables/:id
Get table structure and data
**Response**:
```json
{
"name": "Contacts",
"description": "Customer database",
"columns": [
{
"id": 1,
"table_id": 1,
"name": "Email",
"type": "text",
"placeholder": "email",
"options": null,
"order": 0
}
],
"rows": [
{
"id": 100,
"table_id": 1,
"order": 0,
"created_at": "2025-01-15T10:00:00Z"
}
],
"cellValues": [
{
"id": 500,
"row_id": 100,
"column_id": 1,
"value": "user@example.com"
}
]
}
```
#### PATCH /tables/:id
Update table metadata
**Request**:
```json
{
"name": "Customers",
"description": "Updated description"
}
```
#### DELETE /tables/:id
Delete table (admins only)
**Requirements**: `req.session.userAccessLevel?.hasAccess === true`
### Columns
#### POST /tables/:id/columns
Add column
**Request**:
```json
{
"name": "Email",
"type": "text",
"order": 2,
"purpose": "contact"
}
```
#### PATCH /tables/column/:columnId
Update column
**Request**:
```json
{
"name": "New name",
"type": "text",
"order": 5
}
```
#### DELETE /tables/column/:columnId
Delete column (cascading deletion of all values)
### Rows
#### POST /tables/:id/rows
Add row
**Response**: Created row object
#### DELETE /tables/row/:rowId
Delete row
#### PATCH /tables/:id/rows/order
Change row order
**Request**:
```json
{
"order": [
{ "rowId": 100, "order": 0 },
{ "rowId": 101, "order": 1 }
]
}
```
### Cells
#### POST /tables/cell
Create or update cell value (upsert)
**Request**:
```json
{
"row_id": 100,
"column_id": 5,
"value": "new@email.com"
}
```
**Logic**:
```sql
INSERT INTO user_cell_values (row_id, column_id, value_encrypted)
VALUES ($1, $2, encrypt_text($3, $4))
ON CONFLICT (row_id, column_id)
DO UPDATE SET value_encrypted = encrypt_text($3, $4), updated_at = NOW()
```
### Filtering
#### GET /tables/:id/rows
Get filtered rows
**Parameters**:
```
?product=Premium // Filter by product
&tags=VIP,B2B // Filter by tags
&relation_12=45,46 // Filter by relation (column_id=12)
&multiselect_8=10,11 // Filter by multiselect (column_id=8)
&lookup_15=100 // Filter by lookup (column_id=15)
```
### RAG Index
#### POST /tables/:id/rebuild-index
Rebuild vector index (admins only)
**Requirements**: `req.session.userAccessLevel?.hasAccess === true`
**Response**:
```json
{
"success": true,
"count": 150
}
```
### Relations
#### GET /tables/:tableId/row/:rowId/relations
Get all row relations
**Response**:
```json
[
{
"id": 1000,
"from_row_id": 100,
"column_id": 12,
"to_table_id": 5,
"to_row_id": 45
}
]
```
#### POST /tables/:tableId/row/:rowId/relations
Add relation or relations
**Single relation**:
```json
{
"column_id": 12,
"to_table_id": 5,
"to_row_id": 45
}
```
**Multiple relations** (multiselect):
```json
{
"column_id": 12,
"to_table_id": 5,
"to_row_ids": [45, 46, 47]
}
```
**Logic**:
- Deletes old relations for column_id
- Adds new relations
#### DELETE /tables/:tableId/row/:rowId/relations/:relationId
Delete relation
### Placeholders
#### GET /tables/:id/placeholders
Get placeholders for table columns
**Response**:
```json
[
{
"id": 1,
"name": "Email",
"placeholder": "email"
},
{
"id": 2,
"name": "Customer Name",
"placeholder": "customer_name"
}
]
```
#### GET /tables/placeholders/all
Get all placeholders across all tables
**Response**:
```json
[
{
"column_id": 1,
"column_name": "Email",
"placeholder": "email",
"table_id": 1,
"table_name": "Contacts"
}
]
```
---
## Usage Examples
### Example 1: FAQ Knowledge Base for AI
#### Create Table
```javascript
const table = await tablesService.createTable({
name: "FAQ",
description: "Frequently asked questions for AI",
isRagSourceId: 2 // RAG source
});
```
#### Add Columns
```javascript
// Question (for vector search)
await tablesService.addColumn(table.id, {
name: "Question",
type: "text",
order: 0,
purpose: "question"
});
// Answer (for AI)
await tablesService.addColumn(table.id, {
name: "Answer",
type: "text",
order: 1,
purpose: "answer"
});
// Product (for filtering)
await tablesService.addColumn(table.id, {
name: "Product",
type: "multiselect",
order: 2,
purpose: "product",
options: {
choices: ["Basic", "Premium", "Enterprise"]
}
});
// Tags (for filtering)
await tablesService.addColumn(table.id, {
name: "Tags",
type: "multiselect",
order: 3,
purpose: "userTags",
options: {
choices: ["Payment", "Delivery", "Return", "Warranty"]
}
});
```
#### Add Data
```javascript
// Add row
const row = await tablesService.addRow(table.id);
// Fill cells
await tablesService.saveCell({
row_id: row.id,
column_id: 1, // Question
value: "How to return product?"
});
await tablesService.saveCell({
row_id: row.id,
column_id: 2, // Answer
value: "Product return is possible within 14 days of purchase..."
});
// Automatically indexed in vector store!
```
#### Search via AI
```javascript
// User asks AI
const userQuestion = "can I return my purchase?";
// AI does vector search
const results = await vectorSearch.search(table.id, userQuestion, 3);
// Finds similar question "How to return product?" (score: -200)
// Returns answer from metadata.answer
```
### Example 2: CRM System
#### Structure
```javascript
// Table "Companies"
const companies = await tablesService.createTable({
name: "Companies",
description: "Company database"
});
await tablesService.addColumn(companies.id, {
name: "Name",
type: "text",
order: 0
});
await tablesService.addColumn(companies.id, {
name: "Website",
type: "text",
order: 1
});
await tablesService.addColumn(companies.id, {
name: "Industry",
type: "multiselect",
order: 2,
options: { choices: ["IT", "Finance", "Retail", "Manufacturing"] }
});
// Table "Contacts"
const contacts = await tablesService.createTable({
name: "Contacts",
description: "Contact database"
});
await tablesService.addColumn(contacts.id, {
name: "Name",
type: "text",
order: 0
});
await tablesService.addColumn(contacts.id, {
name: "Email",
type: "text",
order: 1
});
// Relation: Contact → Company
await tablesService.addColumn(contacts.id, {
name: "Company",
type: "relation",
order: 2,
options: {
relatedTableId: companies.id,
relatedColumnId: 1 // Company name
}
});
// Lookup: Company Website
await tablesService.addColumn(contacts.id, {
name: "Company Website",
type: "lookup",
order: 3,
options: {
relatedTableId: companies.id,
relatedColumnId: 2, // Relation through "Company"
lookupColumnId: 2 // Substitute "Website"
}
});
```
#### Usage
```javascript
// Add company
const company = await tablesService.addRow(companies.id);
await tablesService.saveCell({
row_id: company.id,
column_id: 1,
value: "Microsoft"
});
await tablesService.saveCell({
row_id: company.id,
column_id: 2,
value: "https://microsoft.com"
});
// Add contact
const contact = await tablesService.addRow(contacts.id);
await tablesService.saveCell({
row_id: contact.id,
column_id: 1,
value: "John Doe"
});
// Link contact to company
await api.post(`/tables/${contacts.id}/row/${contact.id}/relations`, {
column_id: 3, // "Company"
to_table_id: companies.id,
to_row_id: company.id
});
// Lookup automatically substitutes "https://microsoft.com"!
```
### Example 3: Task Management
#### Structure
```javascript
// Table "Projects"
const projects = await tablesService.createTable({
name: "Projects",
description: "Active projects"
});
await tablesService.addColumn(projects.id, {
name: "Name",
type: "text",
order: 0
});
await tablesService.addColumn(projects.id, {
name: "Status",
type: "multiselect",
order: 1,
options: { choices: ["Planning", "In Progress", "Completed"] }
});
// Table "Tasks"
const tasks = await tablesService.createTable({
name: "Tasks",
description: "Project tasks"
});
await tablesService.addColumn(tasks.id, {
name: "Name",
type: "text",
order: 0
});
await tablesService.addColumn(tasks.id, {
name: "Project",
type: "relation",
order: 1,
options: {
relatedTableId: projects.id,
relatedColumnId: 1
}
});
await tablesService.addColumn(tasks.id, {
name: "Priority",
type: "number",
order: 2
});
await tablesService.addColumn(tasks.id, {
name: "Status",
type: "multiselect",
order: 3,
options: { choices: ["To Do", "In Progress", "Review", "Done"] }
});
```
#### Filter Tasks by Project
```javascript
// Get all tasks for project with ID = 5
const tasks = await api.get(`/tables/${tasks.id}/rows?relation_2=5`);
// Get tasks with priority > 5
const highPriority = tasks.filter(task => {
const priority = cellValues.find(
cell => cell.row_id === task.id && cell.column_id === 3
)?.value;
return parseInt(priority) > 5;
});
```
---
## Security
### Data Encryption
**All sensitive data encrypted with AES-256**:
```javascript
// Encrypted:
name_encrypted // Table name
description_encrypted // Description
value_encrypted // Cell values
placeholder_encrypted // Placeholders
// NOT encrypted (for indexes and performance):
placeholder // Unencrypted placeholder
options // JSONB settings
order // Order
```
**Encryption functions in PostgreSQL**:
```sql
-- Encryption
encrypt_text(plain_text, encryption_key)
-- Decryption
decrypt_text(encrypted_text, encryption_key)
-- Usage example
INSERT INTO user_tables (name_encrypted)
VALUES (encrypt_text('Contacts', $1));
SELECT decrypt_text(name_encrypted, $1) as name
FROM user_tables;
```
### Access Rights
```javascript
// View: all authorized users
GET /tables
GET /tables/:id
GET /tables/:id/rows
// Editing: users with rights
if (!canEditData) {
return res.status(403).json({ error: 'Access denied' });
}
POST /tables/:id/columns
POST /tables/:id/rows
POST /tables/cell
PATCH /tables/column/:columnId
// Deletion: administrators only
if (!req.session.userAccessLevel?.hasAccess) {
return res.status(403).json({ error: 'Administrators only' });
}
DELETE /tables/:id
DELETE /tables/column/:columnId
DELETE /tables/row/:rowId
POST /tables/:id/rebuild-index
```
### Token-Based Rights Verification
```javascript
// Backend checks token balance
const address = req.session.address;
const dleContract = new ethers.Contract(dleAddress, dleAbi, provider);
const balance = await dleContract.balanceOf(address);
if (balance === 0n) {
return res.status(403).json({
error: 'Access denied: no tokens'
});
}
// Determine access level
const accessLevel = determineAccessLevel(balance);
req.session.userAccessLevel = accessLevel;
```
### SQL Injection Protection
**Parameterized queries**:
```javascript
// ✅ Safe (parameters)
await db.getQuery()(
'SELECT * FROM user_tables WHERE id = $1',
[tableId]
);
// ❌ DANGEROUS (concatenation)
await db.getQuery()(
`SELECT * FROM user_tables WHERE id = ${tableId}`
);
```
### Input Validation
```javascript
// Type check
if (typeof name !== 'string') {
return res.status(400).json({ error: 'Invalid name' });
}
// Existence check
const exists = await db.getQuery()(
'SELECT id FROM user_tables WHERE id = $1',
[tableId]
);
if (!exists.rows[0]) {
return res.status(404).json({ error: 'Table not found' });
}
// Uniqueness check (placeholder)
const duplicate = await db.getQuery()(
'SELECT id FROM user_columns WHERE placeholder = $1 AND id != $2',
[placeholder, columnId]
);
if (duplicate.rows.length > 0) {
placeholder = generateUniquePlaceholder();
}
```
### Cascading Deletion (Protection from Orphaned Data)
```sql
-- All relations with ON DELETE CASCADE
CREATE TABLE user_columns (
table_id INTEGER NOT NULL
REFERENCES user_tables(id) ON DELETE CASCADE
);
CREATE TABLE user_rows (
table_id INTEGER NOT NULL
REFERENCES user_tables(id) ON DELETE CASCADE
);
CREATE TABLE user_cell_values (
row_id INTEGER NOT NULL
REFERENCES user_rows(id) ON DELETE CASCADE,
column_id INTEGER NOT NULL
REFERENCES user_columns(id) ON DELETE CASCADE
);
-- Result: table deletion automatically deletes EVERYTHING
```
### Rate Limiting
```javascript
// Can be added in backend/routes/tables.js
const rateLimit = require('express-rate-limit');
const tablesLimiter = rateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes
max: 100, // 100 requests
message: 'Too many requests to tables'
});
router.use(tablesLimiter);
```
---
## Performance
### Optimizations
#### 1. Parallel Queries
```javascript
// Instead of sequential queries:
const tableMeta = await db.query('SELECT ...');
const columns = await db.query('SELECT ...');
const rows = await db.query('SELECT ...');
const cellValues = await db.query('SELECT ...');
// Use parallel:
const [tableMeta, columns, rows, cellValues] = await Promise.all([
db.query('SELECT ...'),
db.query('SELECT ...'),
db.query('SELECT ...'),
db.query('SELECT ...')
]);
// Speedup: 4x
```
#### 2. Indexes on Relations
```sql
CREATE INDEX idx_user_table_relations_from_row
ON user_table_relations(from_row_id);
CREATE INDEX idx_user_table_relations_to_table
ON user_table_relations(to_table_id);
-- Result: fast filtering by relations
```
#### 3. UNIQUE Constraint
```sql
CREATE TABLE user_cell_values (
...
UNIQUE(row_id, column_id)
);
-- Advantages:
-- 1. Prevents duplicate cells
-- 2. Speeds up upsert (ON CONFLICT)
-- 3. Automatic index
```
#### 4. WebSocket Instead of Polling
```javascript
// ❌ Polling (slow)
setInterval(async () => {
const data = await fetchTableData();
updateUI(data);
}, 5000);
// ✅ WebSocket (instant)
socket.on('table-update', (data) => {
if (data.tableId === currentTableId) {
updateUI(data);
}
});
// Result: real-time updates, no server load
```
#### 5. Caching
```javascript
// Backend can add cache for frequently requested tables
const NodeCache = require('node-cache');
const tableCache = new NodeCache({ stdTTL: 300 }); // 5 minutes
router.get('/:id', async (req, res) => {
const cacheKey = `table_${req.params.id}`;
const cached = tableCache.get(cacheKey);
if (cached) {
return res.json(cached);
}
const data = await fetchTableData(req.params.id);
tableCache.set(cacheKey, data);
res.json(data);
});
```
### Metrics
**Typical response times**:
```
GET /tables → 50-100ms (all tables)
GET /tables/:id → 150-300ms (with data, Promise.all)
POST /tables/cell → 100-200ms (upsert + vector update)
DELETE /tables/row/:id → 200-400ms (deletion + rebuild vector)
POST /tables/:id/rebuild-index → 1-5s (depends on size)
```
**Optimal table sizes**:
```
Rows: up to 10,000 → Excellent
Rows: 10,000-50,000 → Good
Rows: >50,000 → Need additional optimizations (pagination, lazy load)
```
---
## Limitations and Future Improvements
### Current Limitations
1. **No pagination**: All rows loaded at once
- For large tables (>1000 rows) may be slow
2. **No formulas**: Cannot create calculated fields
- Workaround: use lookup
3. **No grouping**: Cannot group rows
- Workaround: filtering on frontend
4. **No change history**: Not tracked who and when changed
- Can add audit trail
5. **Limited sorting**: Only through order field
- No column sorting on backend
### Possible Improvements
```javascript
// 1. Pagination
GET /tables/:id/rows?page=1&limit=50
// 2. Sorting
GET /tables/:id/rows?sort_by=column_id&order=asc
// 3. Formulas
{
"type": "formula",
"options": {
"formula": "{{price}} * {{quantity}}"
}
}
// 4. Change history
CREATE TABLE user_cell_history (
id SERIAL PRIMARY KEY,
cell_id INTEGER REFERENCES user_cell_values(id),
old_value TEXT,
new_value TEXT,
changed_by INTEGER,
changed_at TIMESTAMP
);
// 5. Export/import
POST /tables/:id/export CSV/Excel
POST /tables/:id/import CSV/Excel
// 6. Table templates
POST /tables/templates/crm Create CRM from template
POST /tables/templates/tasks Create Kanban from template
```
---
## Conclusion
The electronic tables system in DLE is a **powerful tool** for managing structured data with:
**Flexible structure** (6 field types)
**Relations between tables** (relation, lookup)
**AI integration** (RAG, vector search)
**Real-time updates** (WebSocket)
**Security** (AES-256, access rights)
**Performance** (indexes, parallel queries)
This is **not just Excel**, but a **full-featured database** with convenient interface and AI assistant!
---
**© 2024-2025 Tarabanov Alexander Viktorovich. All rights reserved.**
**Document version**: 1.0.0
**Creation date**: October 25, 2025
**Status**: Temporary (for internal use)