docs:add local format design doc - #22
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: aoiasd The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
|
||
| The **local_format** feature addresses this by: | ||
|
|
||
| - Allowing users to specify `local_format=vortex` per field at collection creation time. |
There was a problem hiding this comment.
this should definitely not be bound to collection schema?
There was a problem hiding this comment.
we should be able to change loading local format through milvus config
There was a problem hiding this comment.
For fields can use different formats to adapt to various situations.
|
|
||
| ### 2. Column Group Splitting | ||
|
|
||
| Fields with `local_format=vortex` are split into a separate column group from default (row) format fields. This is handled by `LocalFormatPolicy` in the column group splitting pipeline: |
There was a problem hiding this comment.
Do you mean splited in s3 or locally on QN? I assume local format should not affect remote formate?
There was a problem hiding this comment.
Splited in S3. When we write binlog. Means field 'localformat=vortex' should not in same column group with field 'localformat=row'.
| DefaultPolicies execution order: | ||
| 1. SystemColumnPolicy — split system fields (RowID, Timestamp) + PK | ||
| 2. AvgSizePolicy — split large fields by avg size | ||
| 3. SelectedDataTypePolicy — split vector / text fields (each gets own group) | ||
| 4. LocalFormatPolicy — split vortex fields into separate group | ||
| 5. RemanentShortPolicy — merge remaining short fields |
There was a problem hiding this comment.
is this something we already have?
There was a problem hiding this comment.
is this something we already have?
Yes, Only add LocalFormatPolicy here.
|
|
||
| **VortexGroupChunkTranslator construction:** | ||
|
|
||
| 1. Download vortex files from object storage into Arrow Buffers (in-memory) |
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
3 participants
relate: milvus-io/milvus#47557