feat: add NULL support for nullable fields and default values

[MrLoh]

Implement nullable field and default_value support matching Milvus Server behavior, fixing timeouts when inserting None into nullable scalar fields. Also fixes the CI build pipeline and e2e test suite which had been broken, and adds a devcontainer for local development.

Dev environment

The C++ build requires Conan 1.x, GCC 12, Rust, and specific system libraries (openblas, libatomic) that don't work with modern macOS toolchains. Previously there was no documented way to build locally — only CI could produce wheels. This made iterating on C++ changes impractical, which was a blocker for the nullable feature work.

Adds .devcontainer/ (AlmaLinux 8, multi-arch) with the full toolchain pre-installed so you can build and test inside the container on any host. Rewrites CONTRIBUTING.md with setup instructions.

Infrastructure fixes

The CI build and e2e tests had been broken for a while — the build image was stale and the test suite was pinned to an old pymilvus version that silently downgraded the freshly built wheel. These had to be fixed first to get a working feedback loop for the nullable feature.

CI / build

• Restore --update to conan install while forcing source builds for bison/flex/libtool/openssl on Linux to avoid GLIBC_2.29+ incompatibilities in pre-built Conan binaries.
• Install missing deps in ci/entrypoint.sh (openblas-devel, libatomic-static, perl-IPC-Cmd/perl-Digest-SHA) until the base image is rebuilt.
• Move milvus.patch application before add_subdirectory(knowhere) in the root CMakeLists.txt so the knowhere source-list patch takes effect.

Test fixes

• Bump pymilvus from 2.4.4 to 2.5.4 — the old version pulled milvus-lite<2.5.0 from PyPI, so e2e tests never ran against the CI-built wheel.
• Remove dead pymilvus.Milvus legacy client usage that crashes on pymilvus 2.5+.
• Fix test_milvus_client_collection_self_creation_default: correct DataType.Array enum casing, uppercase metric_type, add missing args, and align assertions with actual schema field names.

Pre-existing bug fixes

• Binary protobuf truncation in AddSchema/AddIndex (.c_str() truncates at embedded null bytes).
• BFloat16Vector field count reading float16_vector() instead of bfloat16_vector().

Nullable field support

Removes the explicit "MilvusLite doesn't support nullable" block and implements the full nullable/default-value data path so that nullable=True fields and default_value behave identically to Milvus Server.

Data path

PyMilvus sends nullable data in a compacted wire format where the payload only contains non-null values and a separate valid_data bitmask tracks nullability per row. Segcore expects uncompacted data where the payload length matches num_rows. The core change is a decompaction step in the insert path (DecompactFieldData) that expands compacted data, applies configured default values for omitted fields, and propagates valid_data through all downstream operations (index reordering, slicing, storage round-trips). Non-nullable fields with default_value get filled without valid_data to avoid falsely marking them as nullable downstream.

Expression parsing

is null / is not null filter expressions are handled by extending the ANTLR grammar with ISNULL and ISNOTNULL lexer tokens and parser productions (ported from upstream Milvus Server's Plan.g4), with corresponding visitor methods that build NullExpr protobufs. All expression parsing stays in a single ANTLR pipeline — no regex pre-processing or manual operator-precedence logic.

Closes #246

[MrLoh]

@junjiejiangjjj any chance you could take a look at this. I could also separate this into 2 PRs: one that improves the local dev setup and fixes CI and one that adds the actual Null support feature, if that would be easier.

[MrLoh]

@junjiejiangjjj what do I have to do to get any kind of traction on this? We are happy customers of milvus cloud, but having milvus lite for local research experiments is essential for us and the missing null support is a huge pain. While I'm not an expert in this codebase, I put serious effort into making this PR and I'd at least love some feedback.

[junjiejiangjjj]

Hi @MrLoh, thanks for the contribution and sorry for the delay.

The project is currently being rewritten in Python, so we’re unlikely to continue evolving the current codebase in this direction. Nullable support is planned in the rewritten version.

Thanks again for the effort and follow-up.