# Changelog Please mark all change in change log and use the issue from GitHub # Milvus 0.7.0 (TBD) ## Bug - \#715 - Milvus crash when searching and building index simultaneously using SQ8H - \#744 - Don't return partition table for show_tables - \#770 - Server unittest run failed on low-end server - \#805 - IVFTest.gpu_seal_test unittest failed - \#831 - Judge branch error in CommonUtil.cpp - \#977 - Server crash when create tables concurrently - \#990 - check gpu resources setting when assign repeated value - \#995 - table count set to 0 if no tables found - \#1010 - improve error message when offset or page_size is equal 0 - \#1022 - check if partition name is legal - \#1028 - check if table exists when show partitions - \#1029 - check if table exists when try to delete partition - \#1066 - optimize http insert and search speed - \#1067 - Add binary vectors support in http server - \#1075 - improve error message when page size or offset is illegal - \#1082 - check page_size or offset value to avoid float - \#1115 - http server support load table into memory - \#1211 - Server down caused by searching with index_type: HNSW - \#1240 - Update license declaration - \#1298 - Unittest failed when on CPU2GPU case ## Feature - \#216 - Add CLI to get server info - \#343 - Add Opentracing - \#665 - Support get/set config via CLI - \#759 - Put C++ sdk out of milvus/core - \#766 - If partition tag is similar, wrong partition is searched - \#771 - Add server build commit info interface - \#788 - Add web server into server module - \#813 - Add push mode for prometheus monitor - \#815 - Support MinIO storage - \#823 - Support binary vector tanimoto/jaccard/hamming metric - \#853 - Support HNSW - \#910 - Change Milvus c++ standard to c++17 ## Improvement - \#738 - Use Openblas / lapack from apt install - \#758 - Enhance config description - \#791 - Remove Arrow - \#834 - add cpu mode for built-in Faiss - \#848 - Add ready-to-use config files to the Milvus repo for enhanced user experince - \#860 - Remove redundant checks in CacheMgr's constructor - \#908 - Move "primary_path" and "secondary_path" to storage config - \#931 - Remove "collector" from config - \#966 - Update NOTICE.md - \#1002 - Rename minio to s3 in Storage Config section - \#1078 - Move 'insert_buffer_size' to Cache Config section - \#1105 - Error message is not clear when creating IVFSQ8H index without gpu resources - \#740, #849, #878, #972, #1033, #1161, #1173, #1199, #1190, #1223, #1222, #1257, #1264, #1269, #1164, #1304, #1324 - Various fixes and improvements for Milvus documentation. - \#1320 - Remove debug logging from faiss ## Task # Milvus 0.6.0 (2019-12-07) ## Bug - \#228 - memory usage increased slowly during searching vectors - \#246 - Exclude src/external folder from code coverage for jenkin ci - \#248 - Reside src/external in thirdparty - \#316 - Some files not merged after vectors added - \#327 - Search does not use GPU when index type is FLAT - \#331 - Add exception handle when search fail - \#340 - Test cases run failed on 0.6.0 - \#353 - Rename config.h.in to version.h.in - \#374 - sdk_simple return empty result - \#377 - Create partition success if tag name only contains spaces - \#397 - sdk_simple return incorrect result - \#399 - Create partition should be failed if partition tag existed - \#412 - Message returned is confused when partition created with null partition name - \#416 - Drop the same partition success repeatally - \#440 - Query API in customization still uses old version - \#440 - Server cannot startup with gpu_resource_config.enable=false in GPU version - \#458 - Index data is not compatible between 0.5 and 0.6 - \#465 - Server hang caused by searching with nsg index - \#485 - Increase code coverage rate - \#486 - gpu no usage during index building - \#497 - CPU-version search performance decreased - \#504 - The code coverage rate of core/src/scheduler/optimizer is too low - \#509 - IVF_PQ index build trapped into dead loop caused by invalid params - \#513 - Unittest DELETE_BY_RANGE sometimes failed - \#523 - Erase file data from cache once the file is marked as deleted - \#527 - faiss benchmark not compatible with faiss 1.6.0 - \#530 - BuildIndex stop when do build index and search simultaneously - \#532 - assigin value to `table_name` from confest shell - \#533 - NSG build failed with MetricType Inner Product - \#543 - client raise exception in shards when search results is empty - \#545 - Avoid dead circle of build index thread when error occurs - \#547 - NSG build failed using GPU-edition if set gpu_enable false - \#548 - NSG search accuracy is too low - \#552 - Server down during building index_type: IVF_PQ using GPU-edition - \#561 - Milvus server should report exception/error message or terminate on mysql metadata backend error - \#579 - Build index hang in GPU version when gpu_resources disabled - \#596 - Frequently insert operation cost too much disk space - \#599 - Build index log is incorrect - \#602 - Optimizer specify wrong gpu_id - \#606 - No log generated during building index with CPU - \#616 - IP search metric_type is not supported by IVF_PQ index - \#631 - FAISS isn't compiled with O3 option - \#636 - [CPU] Create index PQ should be failed if table metric type set Inner Product - \#649 - Typo "partiton" should be "partition" - \#654 - Random crash when frequently insert vector one by one - \#658 - Milvus error out when building SQ8H index without GPU resources - \#668 - Update badge of README - \#670 - Random failure of unittest db_test::SEARCH_TEST - \#674 - Server down in stability test - \#696 - Metric_type changed from IP to L2 - \#705 - Fix search SQ8H crash without GPU resource ## Feature - \#12 - Pure CPU version for Milvus - \#77 - Support table partition - \#127 - Support new Index type IVFPQ - \#226 - Experimental shards middleware for Milvus - \#227 - Support new index types SPTAG-KDT and SPTAG-BKT - \#346 - Support build index with multiple gpu - \#420 - Update shards merge part to match v0.5.3 - \#488 - Add log in scheduler/optimizer - \#502 - C++ SDK support IVFPQ and SPTAG - \#560 - Add version in server config file - \#605 - Print more messages when server start - \#644 - Add a new rpc command to get milvus build version whether cpu or gpu - \#709 - Show last commit id when server start ## Improvement - \#255 - Add ivfsq8 test report detailed version - \#260 - C++ SDK README - \#266 - Rpc request source code refactor - \#274 - Logger the time cost during preloading data - \#275 - Rename C++ SDK IndexType - \#284 - Change C++ SDK to shared library - \#306 - Use int64 for all config integer - \#310 - Add Q&A for 'protocol https not supported or disable in libcurl' issue - \#314 - add Find FAISS in CMake - \#322 - Add option to enable / disable prometheus - \#354 - Build migration scripts into milvus docker image - \#358 - Add more information in build.sh and install.md - \#404 - Add virtual method Init() in Pass abstract class - \#409 - Add a Fallback pass in optimizer - \#433 - C++ SDK query result is not easy to use - \#449 - Add ShowPartitions example for C++ SDK - \#470 - Small raw files should not be build index - \#584 - Intergrate internal FAISS - \#611 - Remove MILVUS_CPU_VERSION - \#634 - FAISS GPU version is compiled with O0 - \#737 - Refactor server module to separate Grpc from server handler and scheduler ## Task # Milvus 0.5.3 (2019-11-13) ## Bug - \#258 - Bytes type in proto cause big-endian/little-endian problem ## Feature ## Improvement - \#204 - improve grpc performance in search - \#207 - Add more unittest for config set/get - \#208 - optimize unittest to support run single test more easily - \#284 - Change C++ SDK to shared library - \#260 - C++ SDK README ## Task # Milvus 0.5.2 (2019-11-07) ## Bug - \#194 - Search faild: message="Table file doesn't exist" ## Feature ## Improvement - \#190 - Update default config:use_blas_threshold to 1100 and server version printout to 0.5.2 ## Task # Milvus 0.5.1 (2019-11-04) ## Bug - \#134 - JFrog cache error - \#161 - Search IVFSQHybrid crash on gpu - \#169 - IVF_FLAT search out of memory ## Feature - \#90 - The server start error messages could be improved to enhance user experience - \#104 - test_scheduler core dump - \#115 - Using new structure for tasktable - \#139 - New config option use_gpu_threshold - \#146 - Add only GPU and only CPU version for IVF_SQ8 and IVF_FLAT - \#164 - Add CPU version for building index ## Improvement - \#64 - Improvement dump function in scheduler - \#80 - Print version information into log during server start - \#82 - Move easyloggingpp into "external" directory - \#92 - Speed up CMake build process - \#96 - Remove .a file in milvus/lib for docker-version - \#118 - Using shared_ptr instead of weak_ptr to avoid performance loss - \#122 - Add unique id for Job - \#130 - Set task state MOVED after resource copy it completed - \#149 - Improve large query optimizer pass - \#156 - Not return error when search_resources and index_build_device set cpu - \#159 - Change the configuration name from 'use_gpu_threshold' to 'gpu_search_threshold' - \#168 - Improve result reduce - \#175 - add invalid config unittest ## Task # Milvus 0.5.0 (2019-10-21) ## Bug - MS-568 - Fix gpuresource free error - MS-572 - Milvus crash when get SIGINT - MS-577 - Unittest Query randomly hung - MS-587 - Count get wrong result after adding vectors and index built immediately - MS-599 - Search wrong result when table created with metric_type: IP - MS-601 - Docker logs error caused by get CPUTemperature error - MS-605 - Server going down during searching vectors - MS-620 - Get table row counts display wrong error code - MS-622 - Delete vectors should be failed if date range is invalid - MS-624 - Search vectors failed if time ranges long enough - MS-637 - Out of memory when load too many tasks - MS-639 - SQ8H index created failed and server hang - MS-640 - Cache object size calculate incorrect - MS-641 - Segment fault(signal 11) in PickToLoad - MS-644 - Search crashed with index-type: flat - MS-647 - grafana display average cpu-temp - MS-652 - IVFSQH quantization double free - MS-650 - SQ8H index create issue - MS-653 - When config check fail, Milvus close without message - MS-654 - Describe index timeout when building index - MS-658 - Fix SQ8 Hybrid can't search - MS-665 - IVF_SQ8H search crash when no GPU resource in search_resources - \#9 - Change default gpu_cache_capacity to 4 - \#20 - C++ sdk example get grpc error - \#23 - Add unittest to improve code coverage - \#31 - make clang-format failed after run build.sh -l - \#39 - Create SQ8H index hang if using github server version - \#30 - Some troubleshoot messages in Milvus do not provide enough information - \#48 - Config unittest failed - \#59 - Topk result is incorrect for small dataset ## Improvement - MS-552 - Add and change the easylogging library - MS-553 - Refine cache code - MS-555 - Remove old scheduler - MS-556 - Add Job Definition in Scheduler - MS-557 - Merge Log.h - MS-558 - Refine status code - MS-562 - Add JobMgr and TaskCreator in Scheduler - MS-566 - Refactor cmake - MS-574 - Milvus configuration refactor - MS-578 - Make sure milvus5.0 don't crack 0.3.1 data - MS-585 - Update namespace in scheduler - MS-606 - Speed up result reduce - MS-608 - Update TODO names - MS-609 - Update task construct function - MS-611 - Add resources validity check in ResourceMgr - MS-619 - Add optimizer class in scheduler - MS-626 - Refactor DataObj to support cache any type data - MS-648 - Improve unittest - MS-655 - Upgrade SPTAG - \#42 - Put union of index_build_device and search resources to gpu_pool - \#67 - Avoid linking targets multiple times in cmake ## Feature - MS-614 - Preload table at startup - MS-627 - Integrate new index: IVFSQHybrid - MS-631 - IVFSQ8H Index support - MS-636 - Add optimizer in scheduler for FAISS_IVFSQ8H ## Task - MS-554 - Change license to Apache 2.0 - MS-561 - Add contributing guidelines, code of conduct and README docs - MS-567 - Add NOTICE.md - MS-569 - Complete the NOTICE.md - MS-575 - Add Clang-format & Clang-tidy & Cpplint - MS-586 - Remove BUILD_FAISS_WITH_MKL option - MS-590 - Refine cmake code to support cpplint - MS-600 - Reconstruct unittest code - MS-602 - Remove zilliz namespace - MS-610 - Change error code base value from hex to decimal - MS-624 - Re-organize project directory for open-source - MS-635 - Add compile option to support customized faiss - MS-660 - add ubuntu_build_deps.sh - \#18 - Add all test cases # Milvus 0.4.0 (2019-09-12) ## Bug - MS-119 - The problem of combining the log files - MS-121 - The problem that user can't change the time zone - MS-411 - Fix metric unittest linking error - MS-412 - Fix gpu cache logical error - MS-416 - ExecutionEngineImpl::GpuCache has not return value cause crash - MS-417 - YAML sequence load disable cause scheduler startup failed - MS-413 - Create index failed and server exited - MS-427 - Describe index error after drop index - MS-432 - Search vectors params nprobe need to check max number - MS-431 - Search vectors params nprobe: 0/-1, expected result: raise exception - MS-331 - Crate Table : when table exists, error code is META_FAILED(code=15) rather than ILLEGAL TABLE NAME(code=9)) - MS-430 - Search no result if index created with FLAT - MS-443 - Create index hang again - MS-436 - Delete vectors failed if index created with index_type: IVF_FLAT/IVF_SQ8 - MS-449 - Add vectors twice success, once with ids, the other no ids - MS-450 - server hang after run stop_server.sh - MS-458 - Keep building index for one file when no gpu resource - MS-461 - Mysql meta unittest failed - MS-462 - Run milvus server twices, should display error - MS-463 - Search timeout - MS-467 - mysql db test failed - MS-470 - Drop index success, which table not created - MS-471 - code coverage run failed - MS-492 - Drop index failed if index have been created with index_type: FLAT - MS-493 - Knowhere unittest crash - MS-453 - GPU search error when nprobe set more than 1024 - MS-474 - Create index hang if use branch-0.3.1 server config - MS-510 - unittest out of memory and crashed - MS-507 - Dataset 10m-512, index type sq8,performance in-normal when set CPU_CACHE to 16 or 64 - MS-543 - SearchTask fail without exception - MS-582 - grafana displays changes frequently ## Improvement - MS-327 - Clean code for milvus - MS-336 - Scheduler interface - MS-344 - Add TaskTable Test - MS-345 - Add Node Test - MS-346 - Add some implementation of scheduler to solve compile error - MS-348 - Add ResourceFactory Test - MS-350 - Remove knowhere submodule - MS-354 - Add task class and interface in scheduler - MS-355 - Add copy interface in ExcutionEngine - MS-357 - Add minimum schedule function - MS-359 - Add cost test in new scheduler - MS-361 - Add event in resource - MS-364 - Modify tasktableitem in tasktable - MS-365 - Use tasktableitemptr instead in event - MS-366 - Implement TaskTable - MS-368 - Implement cost.cpp - MS-371 - Add TaskTableUpdatedEvent - MS-373 - Add resource test - MS-374 - Add action definition - MS-375 - Add Dump implementation for Event - MS-376 - Add loader and executor enable flag in Resource avoid diskresource execute task - MS-377 - Improve process thread trigger in ResourceMgr, Scheduler and TaskTable - MS-378 - Debug and Update normal_test in scheduler unittest - MS-379 - Add Dump implementation in Resource - MS-380 - Update resource loader and executor, work util all finished - MS-383 - Modify condition variable usage in scheduler - MS-384 - Add global instance of ResourceMgr and Scheduler - MS-389 - Add clone interface in Task - MS-390 - Update resource construct function - MS-391 - Add PushTaskToNeighbourHasExecutor action - MS-394 - Update scheduler unittest - MS-400 - Add timestamp record in task state change function - MS-402 - Add dump implementation for TaskTableItem - MS-406 - Add table flag for meta - MS-403 - Add GpuCacheMgr - MS-404 - Release index after search task done avoid memory increment continues - MS-405 - Add delete task support - MS-407 - Reconstruct MetricsCollector - MS-408 - Add device_id in resource construct function - MS-409 - Using new scheduler - MS-413 - Remove thrift dependency - MS-410 - Add resource config comment - MS-414 - Add TaskType in Scheduler::Task - MS-415 - Add command tasktable to dump all tasktables - MS-418 - Update server_config.template file, set CPU compute only default - MS-419 - Move index_file_size from IndexParam to TableSchema - MS-421 - Add TaskLabel in scheduler - MS-422 - Support DeleteTask in Multi-GpuResource case - MS-428 - Add PushTaskByDataLocality in scheduler - MS-440 - Add DumpTaskTables in sdk - MS-442 - Merge Knowhere - MS-445 - Rename CopyCompleted to LoadCompleted - MS-451 - Update server_config.template file, set GPU compute default - MS-455 - Distribute tasks by minimal cost in scheduler - MS-460 - Put transport speed as weight when choosing neighbour to execute task - MS-459 - Add cache for pick function in tasktable - MS-476 - Improve search performance - MS-482 - Change search stream transport to unary in grpc - MS-487 - Define metric type in CreateTable - MS-488 - Improve code format in scheduler - MS-495 - cmake: integrated knowhere - MS-496 - Change the top_k limitation from 1024 to 2048 - MS-502 - Update tasktable_test in scheduler - MS-504 - Update node_test in scheduler - MS-505 - Install core unit test and add to coverage - MS-508 - Update normal_test in scheduler - MS-532 - Add grpc server unittest - MS-511 - Update resource_test in scheduler - MS-517 - Update resource_mgr_test in scheduler - MS-518 - Add schedinst_test in scheduler - MS-519 - Add event_test in scheduler - MS-520 - Update resource_test in scheduler - MS-524 - Add some unittest in event_test and resource_test - MS-525 - Disable parallel reduce in SearchTask - MS-527 - Update scheduler_test and enable it - MS-528 - Hide some config used future - MS-530 - Add unittest for SearchTask->Load - MS-531 - Disable next version code - MS-533 - Update resource_test to cover dump function - MS-523 - Config file validation - MS-539 - Remove old task code - MS-546 - Add simple mode resource_config - MS-570 - Add prometheus docker-compose file - MS-576 - Scheduler refactor - MS-592 - Change showtables stream transport to unary ## Feature - MS-343 - Implement ResourceMgr - MS-338 - NewAPI: refine code to support CreateIndex - MS-339 - NewAPI: refine code to support DropIndex - MS-340 - NewAPI: implement DescribeIndex ## Task - MS-297 - disable mysql unit test # Milvus 0.3.1 (2019-07-10) ## Bug - MS-148 - Disable cleanup if mode is read only - MS-149 - Fixed searching only one index file issue in distributed mode - MS-153 - Fix c_str error when connecting to MySQL - MS-157 - Fix changelog - MS-190 - Use env variable to switch mem manager and fix cmake - MS-217 - Fix SQ8 row count bug - MS-224 - Return AlreadyExist status in MySQLMetaImpl::CreateTable if table already exists - MS-232 - Add MySQLMetaImpl::UpdateTableFilesToIndex and set maximum_memory to default if config value = 0 - MS-233 - Remove mem manager log - MS-230 - Change parameter name: Maximum_memory to insert_buffer_size - MS-234 - Some case cause background merge thread stop - MS-235 - Some test cases random fail - MS-236 - Add MySQLMetaImpl::HasNonIndexFiles - MS-257 - Update bzip2 download url - MS-288 - Update compile scripts - MS-330 - Stability test failed caused by server core dumped - MS-347 - Build index hangs again - MS-382 - fix MySQLMetaImpl::CleanUpFilesWithTTL unknown column bug ## Improvement - MS-156 - Add unittest for merge result functions - MS-152 - Delete assert in MySQLMetaImpl and change MySQLConnectionPool impl - MS-204 - Support multi db_path - MS-206 - Support SQ8 index type - MS-208 - Add buildinde interface for C++ SDK - MS-212 - Support Inner product metric type - MS-241 - Build Faiss with MKL if using Intel CPU; else build with OpenBlas - MS-242 - Clean up cmake and change MAKE_BUILD_ARGS to be user defined variable - MS-245 - Improve search result transfer performance - MS-248 - Support AddVector/SearchVector profiling - MS-256 - Add more cache config - MS-260 - Refine log - MS-249 - Check machine hardware during initialize - MS-261 - Update faiss version to 1.5.3 and add BUILD_FAISS_WITH_MKL as an option - MS-266 - Improve topk reduce time by using multi-threads - MS-275 - Avoid sqlite logic error excetion - MS-278 - add IndexStatsHelper - MS-313 - add GRPC - MS-325 - add grpc status return for C++ sdk and modify some format - MS-278 - Add IndexStatsHelper - MS-312 - Set openmp thread number by config - MS-305 - Add CPU core percent metric - MS-310 - Add milvus CPU utilization ratio and CPU/GPU temperature metrics - MS-324 - Show error when there is not enough gpu memory to build index - MS-328 - Check metric type on server start - MS-332 - Set grpc and thrift server run concurrently - MS-352 - Add hybrid index ## Feature - MS-180 - Add new mem manager - MS-195 - Add nlist and use_blas_threshold conf - MS-137 - Integrate knowhere ## Task - MS-125 - Create 0.3.1 release branch - MS-306 - Optimize build efficiency # Milvus 0.3.0 (2019-06-30) ## Bug - MS-104 - Fix unittest lcov execution error - MS-102 - Fix build script file condition error - MS-80 - Fix server hang issue - MS-89 - Fix compile failed, libgpufaiss.a link missing - MS-90 - Fix arch match incorrect on ARM - MS-99 - Fix compilation bug - MS-110 - Avoid huge file size ## Improvement - MS-82 - Update server startup welcome message - MS-83 - Update vecwise to Milvus - MS-77 - Performance issue of post-search action - MS-22 - Enhancement for MemVector size control - MS-92 - Unify behavior of debug and release build - MS-98 - Install all unit test to installation directory - MS-115 - Change is_startup of metric_config switch from true to on - MS-122 - Archive criteria config - MS-124 - HasTable interface - MS-126 - Add more error code - MS-128 - Change default db path ## Feature - MS-57 - Implement index load/search pipeline - MS-56 - Add version information when server is started - MS-64 - Different table can have different index type - MS-52 - Return search score - MS-66 - Support time range query - MS-68 - Remove rocksdb from third-party - MS-70 - cmake: remove redundant libs in src - MS-71 - cmake: fix faiss dependency - MS-72 - cmake: change prometheus source to git - MS-73 - cmake: delete civetweb - MS-65 - Implement GetTableRowCount interface - MS-45 - Implement DeleteTable interface - MS-75 - cmake: change faiss version to 1.5.2; add CUDA gencode - MS-81 - fix faiss ptx issue; change cuda gencode - MS-84 - cmake: add arrow, jemalloc and jsoncons third party; default build option OFF - MS-85 - add NetIO metric - MS-96 - add new query interface for specified files - MS-97 - Add S3 SDK for MinIO Storage - MS-105 - Add MySQL - MS-130 - Add prometheus_test - MS-144 - Add nprobe config - MS-147 - Enable IVF - MS-130 - Add prometheus_test ## Task - MS-74 - Change README.md in cpp - MS-88 - Add support for arm architecture # Milvus 0.2.0 (2019-05-31) ## Bug - MS-32 - Fix thrift error - MS-34 - Fix prometheus-cpp thirdparty - MS-67 - Fix license check bug - MS-76 - Fix pipeline crash bug - MS-100 - cmake: fix AWS build issue - MS-101 - change AWS build type to Release ## Improvement - MS-20 - Clean Code Part 1 ## Feature - MS-5 - Implement Auto Archive Feature - MS-6 - Implement SDK interface part 1 - MS-16 - Implement metrics without prometheus - MS-21 - Implement SDK interface part 2 - MS-26 - cmake. Add thirdparty packages - MS-31 - cmake: add prometheus - MS-33 - cmake: add -j4 to make third party packages build faster - MS-27 - support gpu config and disable license build config in cmake - MS-47 - Add query vps metrics - MS-37 - Add query, cache usage, disk write speed and file data size metrics - MS-30 - Use faiss v1.5.2 - MS-54 - cmake: Change Thrift third party URL to github.com - MS-69 - prometheus: add all proposed metrics ## Task - MS-1 - Add CHANGELOG.md - MS-4 - Refactor the vecwise_engine code structure - MS-62 - Search range to all if no date specified