# Changelog Please mark all changes in change log and use the issue from GitHub # Milvus 0.11.0 (TBD) ## Bug - \#2532 Fix Milvus docker image report illegal instruction - \#2551 Fix test_hybrid_db and test_rpc error - \#2582 CreateHybridIndex.cpp compile error - \#2693 Collection create success if no dimension value provided - \#2694 Collection create success if an invalid field name provided - \#2695 The number of fields should be limited - \#2696 Check the validity of the parameters of creating collection: segment_size - \#2697 Index can not be created - \#2698 Count entities got wrong result with binary vectors - \#2728 Index type name should returned if index type is not supported - \#2731 No entity returned with `get_entity_by_id` - \#2732 Server destroyed after `delete by id` - \#2733 The max value of top-k should be limited - \#2763 Unexpected error when insert binary entities - \#2765 Server crashed when calling `get_entity_by_id` - \#2783 Wrong result returned if searching with tags - \#2790 Distances returned by calling `search` is inaccurate - \#2818 Wrong result returned by `get_entity_by_id` - \#2823 Server crashed during inserting, and can not restart - \#2845 Server crashed after calling `delete_entity_by_id` - \#2852 Fix Prometheus rebuild problem. - \#2869 Create index failed with binary vectors - \#2893 Insert binary data failed - \#2957 There is no exisitence check of annoy search parameter - \#3406 No debug and info log generated after server started - \#3407 Server get stuck if create index after entities inserted with no manual flush - \#3446 ListIDInSegment get wrong result when delete more than one ids - \#3448 RHNSWFlatTest failed - \#3453 Fix server crashed during stability test - \#3482 Server crashed during adding entities - \#3490 Fix ut DBTest.DeleteEntitiesTest assert fail - \#3511 SearchTask::nq() should not return 0 - \#3514 Search failed with regex tag name ## Feature - \#2319 Redo metadata to support MVCC - \#2509 Count up query statistics for debug ease - \#2572 Support structured data index - \#2585 Support IVF_PQ on GPU with using metric_type IP - \#2689 Construct Knowhere Index Without Data - \#2940 Add option to build.sh for cuda arch - \#3132 Refine the implementation of hnsw in faiss and add support for hnsw-flat, hnsw-pq and hnsw-sq8 ## Improvement - \#2543 Remove secondary_path related code - \#2544 Optimize unittest build - \#2561 Clean util dependencies with other modules - \#2568 Upgrade thirdparty oatpp to v1.1.0 - \#2612 Move all APIs in utils into namespace milvus - \#2675 Print out system memory size when report invalid cpu cache size - \#2686 Remove dependency on sqlite_orm - \#2841 Replace IndexType/EngineType/MetricType - \#2858 Unify index name in db - \#2884 Using BlockingQueue in JobMgr - \#3220 Enable -Werror to improve code quality - \#3449 Upgrade master version to v0.11.0 - \#3465 Optimize gc event executor ## Task # Milvus 0.10.2 (2020-08-15) ## Bug - \#2890 Fix the index size caculation in cache - \#2952 Fix the result merging of IVF_PQ IP - \#2975 Fix config UT failed - \#3012 If the cache is too small, queries using multiple GPUs will cause to crash - \#3133 Reverse query result in mishards if metric type is IP ## Feature ## Improvement - \#2653 Improve IVF search performance when NQ and nProbe are both large - \#2828 Let Faiss not to compile half float by default ## Task # Milvus 0.10.1 (2020-07-20) ## Bug - \#2487 Enlarge timeout value for creating collection - \#2487 HotFix release lock failed on NAS - \#2557 Fix random crash of INSERT_DUPLICATE_ID case - \#2578 Result count doesn't match target vectors count - \#2585 Support IVF_PQ IP on GPU - \#2598 Fix Milvus docker image report illegal instruction - \#2617 Fix HNSW and RNSG index files size - \#2637 Suit the range of HNSW parameters - \#2642 Create index failed and server crashed - \#2649 Search parameter of annoy has conflict with document - \#2690 Remove body parser in show-partitions endpoints - \#2692 Milvus hangs during multi-thread concurrent search - \#2739 Fix mishards start failed - \#2752 Milvus formats vectors data to double-precision and return to http client - \#2767 Fix a bug of getting wrong nprobe limitation in knowhere on GPU version - \#2768 After building the index, the number of vectors increases - \#2774 Server down during loading data - \#2776 Fix too many data copies during creating IVF index - \#2813 To implemente RNSG IP ## Feature ## Improvement - \#2932 Upgrade mishards for milvus 0.10.1 ## Task # Milvus 0.10.0 (2020-6-15) ## Bug - \#2367 Fix inconsistent reading and writing when using mishards - \#2368 Make read node detect delete behavior - \#2373 Build index for small segment waste time on waiting background index thread finish - \#2394 Drop collection timeout if too many partitions created on collection - \#2549 Launch server fail using demo config - \#2564 cache.cache_size range check error ## Feature - \#2363 Update branch version - \#2510 Upgrade Milvus config ## Improvement - \#2381 Upgrade FAISS to 1.6.3 - \#2429 Fix Milvus 0.9.1 performance degrade issue - \#2441 Improve Knowhere code coverage - \#2466 Optimize k-selection implemention of faiss gpu version - \#2489 Add exception throw on mysql meta error - \#2495 Add creating lock file failure reason. - \#2516 Improve unit test coverage - \#2548 Upgrade mishards for milvus v0.10.0 # Milvus 0.9.1 (2020-05-29) ## Bug - \#2366 Reduce SQL execution times for collection contains lot of partitions - \#2378 Duplicate data after server restart - \#2395 Fix large nq cudaMalloc error - \#2399 The nlist set by the user may not take effect - \#2403 MySQL max_idle_time is 10 by default - \#2450 The deleted vectors may be found on GPU - \#2456 openblas library install failed ## Feature ## Improvement - \#2307 Disable index SPTAG by default - \#2353 Remove log_config from code and scripts - \#2370 Clean compile warning - \#2410 Logging build index progress - \#2461 Upgrade mishards for milvus 0.9.1 # Milvus 0.9.0 (2020-05-15) ## Bug - \#1705 Limit the insert data batch size - \#1776 Error out when index SQ8H run in CPU mode - \#1925 To flush all collections, flush cannot work - \#1929 Skip MySQL meta schema field width check - \#1946 Fix load index file CPU2GPU fail during searching - \#1955 Switch create_index operation to background once client break connection - \#1997 Index file missed after compact - \#2002 Remove log error msg `Attributes is null` - \#2073 Fix CheckDBConfigBackendUrl error message - \#2076 CheckMetricConfigAddress error message - \#2120 Fix Search expected failed if search params set invalid - \#2121 Allow regex match partition tag when search - \#2128 Check has_partition params - \#2131 Distance/ID returned is not correct if searching with duplicate ids - \#2141 Fix server start failed if wal directory exist - \#2169 Fix SingleIndexTest.IVFSQHybrid unittest - \#2194 Fix get collection info failed - \#2196 Fix server start failed if wal is disabled - \#2203 0.8.0 id=-1 is returned when total count < topk - \#2228 Fix show partitions failed in http module - \#2231 Use server_config to define hard-delete delay time for segment files - \#2261 Re-define result returned by has_collection if collection in delete state - \#2264 Milvus opened too many files when the metric_config.enable_monitor=true - \#2266 Server hang when using multi-clients to query different collections - \#2280 has_partition should return true for `_default` ## Feature - \#1751 Add api SearchByID - \#1752 Add api GetVectorsByID - \#1962 Add api HasPartition - \#1965 FAISS/NSG/HNSW/ANNOY use unified distance calculation algorithm - \#2054 Check if CPU instruction sets are illegal - \#2057 Add a config parameter to switch off http server - \#2059 Add lock file avoid multiple instances modifying data at the same time - \#2064 Warn when use SQLite as metadata management - \#2111 Check GPU environment before start server - \#2206 Log file rotating - \#2240 Obtain running rpc requests information - \#2268 Intelligently detect openblas library in system to avoid installing from source code every time - \#2283 Suspend the building tasks when any query comand arrives. - \#2417 Support Structured Index Based on Sort ## Improvement - \#221 Refactor LOG macro - \#833 Catch exception in RolloutHandler and output in stderr - \#1796 Compile Openblas with source code to improve the performance - \#1942 Background merge file strategy - \#2039 Support Milvus run on SSE CPUs - \#2149 Merge server_cpu_config.template and server_gpu_config.template - \#2153 Upgrade thirdparty oatpp to v1.0.0 - \#2167 Merge log_config.conf with server_config.yaml - \#2173 Check storage permission - \#2178 Using elkan K-Means to improve IVF - \#2185 Change id to string format in http module - \#2186 Update endpoints in http module - \#2190 Fix memory usage is twice of index size when using GPU searching - \#2248 Use hostname and port as instance label of metrics - \#2252 Upgrade mishards APIs and requirements - \#2256 k-means clustering algorithm use only Euclidean distance metric - \#2300 Upgrade mishrads configuration to version 0.4 - \#2311 Update mishards methods - \#2330 Change url for behavior 'get_entities_by_id' - \#2347 Update http document for v0.9.0 - \#2358 Upgrade mishards for v0.9.0 ## Task # Milvus 0.8.0 (2020-04-15) ## Bug - \#1276 SQLite throw exception after create 50000+ partitions in a table - \#1762 Server is not forbidden to create new partition which tag is `_default` - \#1789 Fix multi-client search cause server crash - \#1832 Fix crash in tracing module - \#1873 Fix index file serialize to incorrect path - \#1881 Fix bad alloc when index files lost - \#1883 Fix inserted vectors becomes all zero when index_file_size >= 2GB - \#1901 Search failed with flat index - \#1903 Fix invalid annoy result - \#1910 C++ SDK GetIDsInSegment could not work for large dataset ## Feature - \#261 Integrate ANNOY into Milvus - \#1655 GPU index support delete vectors - \#1660 IVF PQ CPU support deleted vectors searching - \#1661 HNSW support deleted vectors searching - \#1825 Add annoy index type in C++ sdk - \#1849 NSG support deleted vectors searching - \#1893 Log config information and device information ## Improvement - \#1627 Move read/write index APIs into codec - \#1784 Add Substructure and Superstructure in http module - \#1858 Disable S3 build - \#1882 Add index annoy into http module - \#1885 Optimize knowhere unittest - \#1886 Refactor log on search and insert request - \#1897 Heap pop and push can be realized by heap_swap_top - \#1921 Use TimeRecorder instead of chrono - \#1928 Fix too many data and uid copies when loading files - \#1930 Upgrade mishards to v0.8.0 ## Task # Milvus 0.7.1 (2020-03-29) ## Bug - \#1301 Data in WAL may be accidentally inserted into a new table with the same name. - \#1634 Fix search demo bug in HTTP doc - \#1635 Vectors can be returned by searching after vectors deleted if `cache_insert_data` set true - \#1648 The cache cannot be used all when the vector type is binary - \#1651 Check validity of dimension when collection metric type is binary one - \#1663 PQ index parameter 'm' validation - \#1686 API search_in_files cannot work correctly when vectors is stored in certain non-default partition - \#1689 Fix SQ8H search fail on SIFT-1B dataset - \#1667 Create index failed with type: rnsg if metric_type is IP - \#1708 NSG search crashed - \#1724 Remove unused unittests - \#1728 Optimize request handler to combine similar query - \#1734 Opentracing for combined search request - \#1735 Fix search out of memory with ivf_flat - \#1747 Expected error status if search with partition_tag not existed - \#1756 Fix memory exhausted during searching - \#1781 Fix search hang with SQ8H - \#1812 Fix incorrect request method in search example in http readme - \#1818 Duplicate data generated after restart milvus server ## Feature - \#1603 BinaryFlat add 2 Metric: Substructure and Superstructure ## Improvement - \#267 Improve search performance: reduce delay - \#342 Knowhere and Wrapper refactor - \#1537 Optimize raw vector and uids read/write - \#1546 Move Config.cpp to config directory - \#1547 Rename storage/file to storage/disk and rename classes - \#1548 Move store/Directory to storage/Operation and add FSHandler - \#1572 Optimize config cpu/gpu cache_capacity setter - \#1619 Improve compact performance - \#1649 Fix Milvus crash on old CPU - \#1653 IndexFlat (SSE) and IndexBinaryFlat performance improvement for small NQ - \#1678 Remove CUSTOMIZATION macro - \#1698 Upgrade mishards to v0.7.0 - \#1719 Improve Milvus log - \#1754 Optimize behavior to get file ids from metadata in mishards - \#1799 Update docker images to 0.7.1 in mishards ## Task # Milvus 0.7.0 (2020-03-11) ## Bug - \#715 Milvus crash when searching and building index simultaneously using SQ8H - \#744 Don't return partition table for show_tables - \#770 Server unittest run failed on low-end server - \#805 IVFTest.gpu_seal_test unittest failed - \#831 Judge branch error in CommonUtil.cpp - \#977 Server crash when create tables concurrently - \#990 Check gpu resources setting when assign repeated value - \#995 Table count set to 0 if no tables found - \#1010 Improve error message when offset or page_size is equal 0 - \#1022 Check if partition name is valid - \#1028 Check if table exists when show partitions - \#1029 Check if table exists when try to delete partition - \#1066 Optimize http insert and search speed - \#1022 Check if partition name is legal - \#1028 Check if table exists when show partitions - \#1029 Check if table exists when try to delete partition - \#1066 Optimize http insert and search speed - \#1067 Add binary vectors support in http server - \#1075 Improve error message when page size or offset is illegal - \#1082 Check page_size or offset value to avoid float - \#1115 Http server support load table into memory - \#1152 Error log output continuously after server start - \#1211 Server down caused by searching with index_type: HNSW - \#1240 Update license declaration - \#1298 Unit test failed when on CPU2GPU case - \#1359 Negative distance value returned when searching with HNSW index type - \#1429 Server crashed when searching vectors with GPU - \#1476 Fix vectors results bug when getting vectors from segments - \#1484 Index type changed to IDMAP after compacted - \#1491 Server crashed during adding vectors - \#1499 Fix duplicated ID number issue - \#1504 Avoid possible race condition between delete and search - \#1507 set_config for insert_buffer_size is wrong - \#1510 Add set interfaces for WAL configurations - \#1511 Fix big integer cannot pass to server correctly - \#1517 Result is not correct when search vectors in multi partition, index type is RNSG - \#1518 Table count did not match after deleting vectors and compact - \#1521 Make cache_insert_data take effect in-service - \#1525 Add setter API for config preload_table - \#1529 Fix server crash when cache_insert_data enabled - \#1530 Set table file with correct engine type in meta - \#1532 Search with ivf_flat failed with open-dataset: sift-256-hamming - \#1535 Degradation searching performance with metric_type: binary_idmap - \#1549 Fix server/wal config setting bug - \#1556 Index file not created after table and index created - \#1560 Search crashed with Super-high dimensional binary vector - \#1564 Too low recall for glove-200-angular, ivf_pq index - \#1571 Meta engine type become IDMAP after dropping index for BINARY table - \#1574 Set all existing bitset in cache when applying deletes - \#1577 Row count incorrect if delete vectors then create index - \#1580 Old segment folder not removed after merge/compact if create_index is called before adding data - \#1590 Server down caused by failure to write file during concurrent mixed operations - \#1598 Server down during mixed operations - \#1601 External link bug in HTTP doc - \#1609 Refine Compact function - \#1808 Building index params check for Annoy - \#1852 Search index type failed with reason `failed to load index file` ## Feature - \#216 Add CLI to get server info - \#343 Add Opentracing - \#665 Support get/set config via CLI - \#759 Put C++ sdk out of milvus/core - \#766 If partition tag is similar, wrong partition is searched - \#771 Add server build commit info interface - \#788 Add web server into server module - \#813 Add push mode for prometheus monitor - \#815 Support MinIO storage - \#823 Support binary vector tanimoto/jaccard/hamming metric - \#830 Support WAL(write-ahead logging) - \#853 Support HNSW - \#861 Support DeleteById / SearchByID / GetVectorById / Flush - \#910 Change Milvus c++ standard to c++17 - \#1122 Support AVX-512 in FAISS - \#1204 Add api to get table data information - \#1250 Support CPU profiling - \#1302 Get all record IDs in a segment by given a segment id - \#1461 Add crud APIs and segments APIs into http module - \#1463 Update config version to 0.2 - \#1531 Remove S3 related config ## Improvement - \#738 Use Openblas / lapack from apt install - \#758 Enhance config description - \#791 Remove Arrow - \#834 Add cpu mode for built-in Faiss - \#848 Add ready-to-use config files to the Milvus repo for enhanced user experince - \#860 Remove redundant checks in CacheMgr's constructor - \#908 Move "primary_path" and "secondary_path" to storage config - \#931 Remove "collector" from config - \#966 Update NOTICE.md - \#1002 Rename minio to s3 in Storage Config section - \#1078 Move 'insert_buffer_size' to Cache Config section - \#1105 Error message is not clear when creating IVFSQ8H index without gpu resources - \#740, #849, #878, #972, #1033, #1161, #1173, #1199, #1190, #1223, #1222, #1257, #1264, #1269, #1164, #1303, #1304, #1324, #1388, #1459 Various fixes and improvements for Milvus documentation. - \#1297 Hide partition_name parameter, avid user directly access partition table - \#1234 Do S3 server validation check when Milvus startup - \#1263 Allow system conf modifiable and some take effect directly - \#1310 Add default partition tag for a table - \#1320 Remove debug logging from faiss - \#1426 Support to configure whether to enabled autoflush and the autoflush interval - \#1444 Improve delete - \#1448 General proto api for NNS libraries - \#1480 Add return code for AVX512 selection - \#1524 Update config "preload_table" description - \#1544 Update resources name in HTTP module - \#1567 Update yaml config description ## Task - \#1327 Exclude third-party code from codebeat - \#1331 Exclude third-party code from codacy # Milvus 0.6.0 (2019-12-07) ## Bug - \#228 Memory usage increased slowly during searching vectors - \#246 Exclude src/external folder from code coverage for jenkin ci - \#248 Reside src/external in thirdparty - \#316 Some files not merged after vectors added - \#327 Search does not use GPU when index type is FLAT - \#331 Add exception handle when search fail - \#340 Test cases run failed on 0.6.0 - \#353 Rename config.h.in to version.h.in - \#374 sdk_simple return empty result - \#377 Create partition success if tag name only contains spaces - \#397 sdk_simple return incorrect result - \#399 Create partition should be failed if partition tag existed - \#412 Message returned is confused when partition created with null partition name - \#416 Drop the same partition success repeatally - \#440 Query API in customization still uses old version - \#440 Server cannot startup with gpu_resource_config.enable=false in GPU version - \#458 Index data is not compatible between 0.5 and 0.6 - \#465 Server hang caused by searching with nsg index - \#485 Increase code coverage rate - \#486 gpu no usage during index building - \#497 CPU-version search performance decreased - \#504 The code coverage rate of core/src/scheduler/optimizer is too low - \#509 IVF_PQ index build trapped into dead loop caused by invalid params - \#513 Unittest DELETE_BY_RANGE sometimes failed - \#523 Erase file data from cache once the file is marked as deleted - \#527 faiss benchmark not compatible with faiss 1.6.0 - \#530 BuildIndex stop when do build index and search simultaneously - \#532 Assigin value to `table_name` from confest shell - \#533 NSG build failed with MetricType Inner Product - \#543 client raise exception in shards when search results is empty - \#545 Avoid dead circle of build index thread when error occurs - \#547 NSG build failed using GPU-edition if set gpu_enable false - \#548 NSG search accuracy is too low - \#552 Server down during building index_type: IVF_PQ using GPU-edition - \#561 Milvus server should report exception/error message or terminate on mysql metadata backend error - \#579 Build index hang in GPU version when gpu_resources disabled - \#596 Frequently insert operation cost too much disk space - \#599 Build index log is incorrect - \#602 Optimizer specify wrong gpu_id - \#606 No log generated during building index with CPU - \#616 IP search metric_type is not supported by IVF_PQ index - \#631 FAISS isn't compiled with O3 option - \#636 (CPU) Create index PQ should be failed if table metric type set Inner Product - \#649 Typo "partiton" should be "partition" - \#654 Random crash when frequently insert vector one by one - \#658 Milvus error out when building SQ8H index without GPU resources - \#668 Update badge of README - \#670 Random failure of unittest db_test::SEARCH_TEST - \#674 Server down in stability test - \#696 Metric_type changed from IP to L2 - \#705 Fix search SQ8H crash without GPU resource ## Feature - \#12 Pure CPU version for Milvus - \#77 Support table partition - \#127 Support new Index type IVFPQ - \#226 Experimental shards middleware for Milvus - \#227 Support new index types SPTAG-KDT and SPTAG-BKT - \#346 Support build index with multiple gpu - \#420 Update shards merge part to match v0.5.3 - \#488 Add log in scheduler/optimizer - \#502 C++ SDK support IVFPQ and SPTAG - \#560 Add version in server config file - \#605 Print more messages when server start - \#644 Add a new rpc command to get milvus build version whether cpu or gpu - \#709 Show last commit id when server start ## Improvement - \#255 Add ivfsq8 test report detailed version - \#260 C++ SDK README - \#266 RPC request source code refactor - \#274 Logger the time cost during preloading data - \#275 Rename C++ SDK IndexType - \#284 Change C++ SDK to shared library - \#306 Use int64 for all config integer - \#310 Add Q&A for 'protocol https not supported or disable in libcurl' issue - \#314 add Find FAISS in CMake - \#322 Add option to enable / disable prometheus - \#354 Build migration scripts into milvus docker image - \#358 Add more information in build.sh and install.md - \#404 Add virtual method Init() in Pass abstract class - \#409 Add a Fallback pass in optimizer - \#433 C++ SDK query result is not easy to use - \#449 Add ShowPartitions example for C++ SDK - \#470 Small raw files should not be build index - \#584 Intergrate internal FAISS - \#611 Remove MILVUS_CPU_VERSION - \#634 FAISS GPU version is compiled with O0 - \#737 Refactor server module to separate Grpc from server handler and scheduler ## Task # Milvus 0.5.3 (2019-11-13) ## Bug - \#258 Bytes type in proto cause big-endian/little-endian problem ## Feature ## Improvement - \#204 improve grpc performance in search - \#207 Add more unittest for config set/get - \#208 Optimize unittest to support run single test more easily - \#284 Change C++ SDK to shared library - \#260 C++ SDK README ## Task # Milvus 0.5.2 (2019-11-07) ## Bug - \#194 Search faild: message="Table file doesn't exist" ## Feature ## Improvement - \#190 Update default config:use_blas_threshold to 1100 and server version printout to 0.5.2 ## Task # Milvus 0.5.1 (2019-11-04) ## Bug - \#134 JFrog cache error - \#161 Search IVFSQHybrid crash on gpu - \#169 IVF_FLAT search out of memory ## Feature - \#90 The server start error messages could be improved to enhance user experience - \#104 test_scheduler core dump - \#115 Using new structure for tasktable - \#139 New config option use_gpu_threshold - \#146 Add only GPU and only CPU version for IVF_SQ8 and IVF_FLAT - \#164 Add CPU version for building index ## Improvement - \#64 Improvement dump function in scheduler - \#80 Print version information into log during server start - \#82 Move easyloggingpp into "external" directory - \#92 Speed up CMake build process - \#96 Remove .a file in milvus/lib for docker-version - \#118 Using shared_ptr instead of weak_ptr to avoid performance loss - \#122 Add unique id for Job - \#130 Set task state MOVED after resource copy it completed - \#149 Improve large query optimizer pass - \#156 Not return error when search_resources and index_build_device set cpu - \#159 Change the configuration name from 'use_gpu_threshold' to 'gpu_search_threshold' - \#168 Improve result reduce - \#175 add invalid config unittest ## Task # Milvus 0.5.0 (2019-10-21) ## Bug - MS-568 Fix gpuresource free error - MS-572 Milvus crash when get SIGINT - MS-577 Unittest Query randomly hung - MS-587 Count get wrong result after adding vectors and index built immediately - MS-599 Search wrong result when table created with metric_type: IP - MS-601 Docker logs error caused by get CPUTemperature error - MS-605 Server going down during searching vectors - MS-620 Get table row counts display wrong error code - MS-622 Delete vectors should be failed if date range is invalid - MS-624 Search vectors failed if time ranges long enough - MS-637 Out of memory when load too many tasks - MS-639 SQ8H index created failed and server hang - MS-640 Cache object size calculate incorrect - MS-641 Segment fault(signal 11) in PickToLoad - MS-644 Search crashed with index-type: flat - MS-647 grafana display average cpu-temp - MS-652 IVFSQH quantization double free - MS-650 SQ8H index create issue - MS-653 When config check fail, Milvus close without message - MS-654 Describe index timeout when building index - MS-658 Fix SQ8 Hybrid can't search - MS-665 IVF_SQ8H search crash when no GPU resource in search_resources - \#9 Change default gpu_cache_capacity to 4 - \#20 C++ sdk example get grpc error - \#23 Add unittest to improve code coverage - \#31 make clang-format failed after run build.sh -l - \#39 Create SQ8H index hang if using github server version - \#30 Some troubleshoot messages in Milvus do not provide enough information - \#48 Config unittest failed - \#59 Topk result is incorrect for small dataset ## Improvement - MS-552 Add and change the easylogging library - MS-553 Refine cache code - MS-555 Remove old scheduler - MS-556 Add Job Definition in Scheduler - MS-557 Merge Log.h - MS-558 Refine status code - MS-562 Add JobMgr and TaskCreator in Scheduler - MS-566 Refactor cmake - MS-574 Milvus configuration refactor - MS-578 Make sure milvus5.0 don't crack 0.3.1 data - MS-585 Update namespace in scheduler - MS-606 Speed up result reduce - MS-608 Update TODO names - MS-609 Update task construct function - MS-611 Add resources validity check in ResourceMgr - MS-619 Add optimizer class in scheduler - MS-626 Refactor DataObj to support cache any type data - MS-648 Improve unittest - MS-655 Upgrade SPTAG - \#42 Put union of index_build_device and search resources to gpu_pool - \#67 Avoid linking targets multiple times in cmake ## Feature - MS-614 Preload table at startup - MS-627 Integrate new index: IVFSQHybrid - MS-631 IVFSQ8H Index support - MS-636 Add optimizer in scheduler for FAISS_IVFSQ8H ## Task - MS-554 Change license to Apache 2.0 - MS-561 Add contributing guidelines, code of conduct and README docs - MS-567 Add NOTICE.md - MS-569 Complete the NOTICE.md - MS-575 Add Clang-format & Clang-tidy & Cpplint - MS-586 Remove BUILD_FAISS_WITH_MKL option - MS-590 Refine cmake code to support cpplint - MS-600 Reconstruct unittest code - MS-602 Remove zilliz namespace - MS-610 Change error code base value from hex to decimal - MS-624 Re-organize project directory for open-source - MS-635 Add compile option to support customized faiss - MS-660 add ubuntu_build_deps.sh - \#18 Add all test cases # Milvus 0.4.0 (2019-09-12) ## Bug - MS-119 The problem of combining the log files - MS-121 The problem that user can't change the time zone - MS-411 Fix metric unittest linking error - MS-412 Fix gpu cache logical error - MS-416 ExecutionEngineImpl::GpuCache has not return value cause crash - MS-417 YAML sequence load disable cause scheduler startup failed - MS-413 Create index failed and server exited - MS-427 Describe index error after drop index - MS-432 Search vectors params nprobe need to check max number - MS-431 Search vectors params nprobe: 0/-1, expected result: raise exception - MS-331 Crate Table : when table exists, error code is META_FAILED(code=15) rather than ILLEGAL TABLE NAME(code=9)) - MS-430 Search no result if index created with FLAT - MS-443 Create index hang again - MS-436 Delete vectors failed if index created with index_type: IVF_FLAT/IVF_SQ8 - MS-449 Add vectors twice success, once with ids, the other no ids - MS-450 server hang after run stop_server.sh - MS-458 Keep building index for one file when no gpu resource - MS-461 Mysql meta unittest failed - MS-462 Run milvus server twices, should display error - MS-463 Search timeout - MS-467 mysql db test failed - MS-470 Drop index success, which table not created - MS-471 code coverage run failed - MS-492 Drop index failed if index have been created with index_type: FLAT - MS-493 Knowhere unittest crash - MS-453 GPU search error when nprobe set more than 1024 - MS-474 Create index hang if use branch-0.3.1 server config - MS-510 unittest out of memory and crashed - MS-507 Dataset 10m-512, index type sq8，performance in-normal when set CPU_CACHE to 16 or 64 - MS-543 SearchTask fail without exception - MS-582 grafana displays changes frequently ## Improvement - MS-327 Clean code for milvus - MS-336 Scheduler interface - MS-344 Add TaskTable Test - MS-345 Add Node Test - MS-346 Add some implementation of scheduler to solve compile error - MS-348 Add ResourceFactory Test - MS-350 Remove knowhere submodule - MS-354 Add task class and interface in scheduler - MS-355 Add copy interface in ExcutionEngine - MS-357 Add minimum schedule function - MS-359 Add cost test in new scheduler - MS-361 Add event in resource - MS-364 Modify tasktableitem in tasktable - MS-365 Use tasktableitemptr instead in event - MS-366 Implement TaskTable - MS-368 Implement cost.cpp - MS-371 Add TaskTableUpdatedEvent - MS-373 Add resource test - MS-374 Add action definition - MS-375 Add Dump implementation for Event - MS-376 Add loader and executor enable flag in Resource avoid diskresource execute task - MS-377 Improve process thread trigger in ResourceMgr, Scheduler and TaskTable - MS-378 Debug and Update normal_test in scheduler unittest - MS-379 Add Dump implementation in Resource - MS-380 Update resource loader and executor, work util all finished - MS-383 Modify condition variable usage in scheduler - MS-384 Add global instance of ResourceMgr and Scheduler - MS-389 Add clone interface in Task - MS-390 Update resource construct function - MS-391 Add PushTaskToNeighbourHasExecutor action - MS-394 Update scheduler unittest - MS-400 Add timestamp record in task state change function - MS-402 Add dump implementation for TaskTableItem - MS-406 Add table flag for meta - MS-403 Add GpuCacheMgr - MS-404 Release index after search task done avoid memory increment continues - MS-405 Add delete task support - MS-407 Reconstruct MetricsCollector - MS-408 Add device_id in resource construct function - MS-409 Using new scheduler - MS-413 Remove thrift dependency - MS-410 Add resource config comment - MS-414 Add TaskType in Scheduler::Task - MS-415 Add command tasktable to dump all tasktables - MS-418 Update server_config.template file, set CPU compute only default - MS-419 Move index_file_size from IndexParam to TableSchema - MS-421 Add TaskLabel in scheduler - MS-422 Support DeleteTask in Multi-GpuResource case - MS-428 Add PushTaskByDataLocality in scheduler - MS-440 Add DumpTaskTables in sdk - MS-442 Merge Knowhere - MS-445 Rename CopyCompleted to LoadCompleted - MS-451 Update server_config.template file, set GPU compute default - MS-455 Distribute tasks by minimal cost in scheduler - MS-460 Put transport speed as weight when choosing neighbour to execute task - MS-459 Add cache for pick function in tasktable - MS-476 Improve search performance - MS-482 Change search stream transport to unary in grpc - MS-487 Define metric type in CreateTable - MS-488 Improve code format in scheduler - MS-495 cmake: integrated knowhere - MS-496 Change the top_k limitation from 1024 to 2048 - MS-502 Update tasktable_test in scheduler - MS-504 Update node_test in scheduler - MS-505 Install core unit test and add to coverage - MS-508 Update normal_test in scheduler - MS-532 Add grpc server unittest - MS-511 Update resource_test in scheduler - MS-517 Update resource_mgr_test in scheduler - MS-518 Add schedinst_test in scheduler - MS-519 Add event_test in scheduler - MS-520 Update resource_test in scheduler - MS-524 Add some unittest in event_test and resource_test - MS-525 Disable parallel reduce in SearchTask - MS-527 Update scheduler_test and enable it - MS-528 Hide some config used future - MS-530 Add unittest for SearchTask->Load - MS-531 Disable next version code - MS-533 Update resource_test to cover dump function - MS-523 Config file validation - MS-539 Remove old task code - MS-546 Add simple mode resource_config - MS-570 Add prometheus docker-compose file - MS-576 Scheduler refactor - MS-592 Change showtables stream transport to unary ## Feature - MS-343 Implement ResourceMgr - MS-338 NewAPI: refine code to support CreateIndex - MS-339 NewAPI: refine code to support DropIndex - MS-340 NewAPI: implement DescribeIndex ## Task - MS-297 disable mysql unit test # Milvus 0.3.1 (2019-07-10) ## Bug - MS-148 Disable cleanup if mode is read only - MS-149 Fixed searching only one index file issue in distributed mode - MS-153 Fix c_str error when connecting to MySQL - MS-157 Fix changelog - MS-190 Use env variable to switch mem manager and fix cmake - MS-217 Fix SQ8 row count bug - MS-224 Return AlreadyExist status in MySQLMetaImpl::CreateTable if table already exists - MS-232 Add MySQLMetaImpl::UpdateTableFilesToIndex and set maximum_memory to default if config value = 0 - MS-233 Remove mem manager log - MS-230 Change parameter name: Maximum_memory to insert_buffer_size - MS-234 Some case cause background merge thread stop - MS-235 Some test cases random fail - MS-236 Add MySQLMetaImpl::HasNonIndexFiles - MS-257 Update bzip2 download url - MS-288 Update compile scripts - MS-330 Stability test failed caused by server core dumped - MS-347 Build index hangs again - MS-382 fix MySQLMetaImpl::CleanUpFilesWithTTL unknown column bug ## Improvement - MS-156 Add unittest for merge result functions - MS-152 Delete assert in MySQLMetaImpl and change MySQLConnectionPool impl - MS-204 Support multi db_path - MS-206 Support SQ8 index type - MS-208 Add buildinde interface for C++ SDK - MS-212 Support Inner product metric type - MS-241 Build Faiss with MKL if using Intel CPU; else build with OpenBlas - MS-242 Clean up cmake and change MAKE_BUILD_ARGS to be user defined variable - MS-245 Improve search result transfer performance - MS-248 Support AddVector/SearchVector profiling - MS-256 Add more cache config - MS-260 Refine log - MS-249 Check machine hardware during initialize - MS-261 Update faiss version to 1.5.3 and add BUILD_FAISS_WITH_MKL as an option - MS-266 Improve topk reduce time by using multi-threads - MS-275 Avoid sqlite logic error excetion - MS-278 add IndexStatsHelper - MS-313 add GRPC - MS-325 add grpc status return for C++ sdk and modify some format - MS-278 Add IndexStatsHelper - MS-312 Set openmp thread number by config - MS-305 Add CPU core percent metric - MS-310 Add milvus CPU utilization ratio and CPU/GPU temperature metrics - MS-324 Show error when there is not enough gpu memory to build index - MS-328 Check metric type on server start - MS-332 Set grpc and thrift server run concurrently - MS-352 Add hybrid index ## Feature - MS-180 Add new mem manager - MS-195 Add nlist and use_blas_threshold conf - MS-137 Integrate knowhere ## Task - MS-125 Create 0.3.1 release branch - MS-306 Optimize build efficiency # Milvus 0.3.0 (2019-06-30) ## Bug - MS-104 Fix unittest lcov execution error - MS-102 Fix build script file condition error - MS-80 Fix server hang issue - MS-89 Fix compile failed, libgpufaiss.a link missing - MS-90 Fix arch match incorrect on ARM - MS-99 Fix compilation bug - MS-110 Avoid huge file size ## Improvement - MS-82 Update server startup welcome message - MS-83 Update vecwise to Milvus - MS-77 Performance issue of post-search action - MS-22 Enhancement for MemVector size control - MS-92 Unify behavior of debug and release build - MS-98 Install all unit test to installation directory - MS-115 Change is_startup of metric_config switch from true to on - MS-122 Archive criteria config - MS-124 HasTable interface - MS-126 Add more error code - MS-128 Change default db path ## Feature - MS-57 Implement index load/search pipeline - MS-56 Add version information when server is started - MS-64 Different table can have different index type - MS-52 Return search score - MS-66 Support time range query - MS-68 Remove rocksdb from third-party - MS-70 cmake: remove redundant libs in src - MS-71 cmake: fix faiss dependency - MS-72 cmake: change prometheus source to git - MS-73 cmake: delete civetweb - MS-65 Implement GetTableRowCount interface - MS-45 Implement DeleteTable interface - MS-75 cmake: change faiss version to 1.5.2; add CUDA gencode - MS-81 Fix faiss ptx issue; change cuda gencode - MS-84 cmake: add arrow, jemalloc and jsoncons third party; default build option OFF - MS-85 add NetIO metric - MS-96 add new query interface for specified files - MS-97 Add S3 SDK for MinIO Storage - MS-105 Add MySQL - MS-130 Add prometheus_test - MS-144 Add nprobe config - MS-147 Enable IVF - MS-130 Add prometheus_test ## Task - MS-74 Change README.md in cpp - MS-88 Add support for arm architecture # Milvus 0.2.0 (2019-05-31) ## Bug - MS-32 Fix thrift error - MS-34 Fix prometheus-cpp thirdparty - MS-67 Fix license check bug - MS-76 Fix pipeline crash bug - MS-100 CMake: fix AWS build issue - MS-101 Change AWS build type to Release ## Improvement - MS-20 Clean Code Part 1 ## Feature - MS-5 Implement Auto Archive Feature - MS-6 Implement SDK interface part 1 - MS-16 Implement metrics without prometheus - MS-21 Implement SDK interface part 2 - MS-26 CMake. Add thirdparty packages - MS-31 CMake: add prometheus - MS-33 CMake: add -j4 to make third party packages build faster - MS-27 Support gpu config and disable license build config in cmake - MS-47 Add query vps metrics - MS-37 Add query, cache usage, disk write speed and file data size metrics - MS-30 Use faiss v1.5.2 - MS-54 CMake: Change Thrift third party URL to github.com - MS-69 Prometheus: add all proposed metrics ## Task - MS-1 Add CHANGELOG.md - MS-4 Refactor the vecwise_engine code structure - MS-62 Search range to all if no date specified