User Quota Service

Data flow diagram

Miro Board

• Quota source information (REST API, NATS STREAM EVENTS). Receive updates on the user's data usage. Sent by telemetry service. Needed for updates of values in Redis (incremental).
• ClickHouse. Need for aggregation of events (SUM) to get values of used resources
• Redis. External cache required to speed up PBAC Agent requests.
• PBAC Agent.

Quota List

Name	Description
asset_used_size	size in bytes
gpu_used	time in number (second)
cpu_used	time in number (second)
vm_used	time in number (count)
data_uploading	size in bytes
data_downloading	size in bytes

REST API

All calls are made with a JWT token

Returns a list of the user's quotas

URL: GET /api/v1/quota/{$user_id}

RESULT:

{
  "asset_used_size": "size in bytes",
  "gpu_used": "time in number (second)",
  "cpu_used": "time in number (second)",
  "vm_used": "time in number (count)",
  "data_uploading": "size in bytes",
  "data_downloading": "size in bytes"
}

Update user quota

URL: PATCH /api/v1/quota/{user_id}

BODY:

{
 "asset_used_size": "increment size in bytes",
  "gpu_used": "increment time in number (second)",
  "cpu_used": "increment time in number (second)",
  "vm_used": "increment time in number (count)",
  "data_uploading": "increment size in bytes",
  "data_downloading": "increment size in bytes"
}

RESULT: 201 NO CONTENT

Reset redis cache for user

URL: DELETE /api/v1/quota/{user_id}

RESULT: 201 NO CONTENT

Redis data schema

Need to store data in Redis hash

The key is formed as:

• "u:{user_id}" in month quota UTC

User fields:

• asset_used_size: size in bytes
• gpu_used: time in number (second)
• cpu_used: time in number (second)
• vm_used: time in number (count)
• data_uploading: size in bytes
• data_downloading: size in bytes

Memory deletion time 2 hours

NATS Data Schema

Streaming channel type: JetStream

Stream name: quotas

Subject: quota.update

BODY

message UserQuotaUpdate {
  string user_id = 1;
  int64 asset_used_size = 2;
  int64 gpu_used = 3;
  int64 cpu_used = 4;
  int64 vm_used = 5;
  int64 data_uploading = 6;
  int64 data_downloading = 7;
}

Requests

Get Asset Size

SELECT e.used_space_size as asset_size
    FROM events e
    INNER JOIN (
        SELECT max(created) AS max_created
        FROM events
        WHERE user_id = {$user_id}
          AND event_type IN('assets.access.delete', 'assets.access.upload')
          AND used_space_size IS NOT NULL
    ) m ON e.created = m.max_created
    WHERE e.user_id = {$user_id}
      AND e.event_type IN('assets.access.delete', 'assets.access.upload')
      AND e.used_space_size IS NOT NULL

Get GPU used

SELECT e.gpu_avr_used
    FROM events e
    INNER JOIN (
        SELECT max(created) AS max_created
        FROM events
        WHERE user_id = {$user_id}
          AND event_type IN('xumi.generation.finish')
          AND gpu_avr_used IS NOT NULL
    ) m ON e.created = m.max_created
    WHERE e.user_id = {$user_id}
      AND e.event_type IN('xumi.generation.finish')
      AND e.gpu_avr_used IS NOT NULL

Get CPU used

SELECT e.cpu_avr_used
    FROM events e
    INNER JOIN (
        SELECT max(created) AS max_created
        FROM events
        WHERE user_id = {$user_id}
          AND event_type IN('xumi.generation.finish')
          AND cpu_avr_used IS NOT NULL
    ) m ON e.created = m.max_created
    WHERE e.user_id = {$user_id}
      AND e.event_type IN('xumi.generation.finish')
      AND e.cpu_avr_used IS NOT NULL

Get VM used

SELECT  e.used_vm_count
  FROM events e
  INNER JOIN (
      SELECT max(created) AS max_created
      FROM events
      WHERE user_id = {$user_id}
        AND event_type IN('assets.access.delete')
        AND used_vm_count IS NOT NULL
  ) m ON e.created = m.max_created
  WHERE e.user_id = {$user_id}
    AND e.event_type IN('assets.access.delete')
    AND e.used_vm_count IS NOT NULL

Get data uploading

SELECT SUM(object_size) as total_upload
FROM events
WHERE user_id = {$user_id}
  AND event_type IN ('assets.access.upload')
  AND object_size IS NOT NULL
  AND created >= toStartOfMonth(today())

Get data downloading

SELECT SUM(object_size) as total_download
FROM events
WHERE user_id = {$user_id}
  AND event_type IN ('assets.access.download')
  AND object_size IS NOT NULL
  AND created >= toStartOfMonth(today())