38 Commits

Author SHA1 Message Date
John Sully
848057ff19 RocksDB Read Performance Improvements
Former-commit-id: 80cca4869888e048e10e11f1f20796c482c3e5b3
2020-08-09 23:36:20 +00:00
John Sully
2c9c4a98ea Latency normalization with SCAN
Former-commit-id: 237f25d854e70d4d7a3095fdf56aaa80770e492e
2020-08-04 04:37:16 +00:00
John Sully
414b3102ca Merge branch 'flash_cache' into keydbpro
Former-commit-id: 2a721ef645921d62b39f1374c0a3f5c92b00fae5
2020-07-13 03:53:34 +00:00
John Sully
d67265de66 Implement storage key cache, and writeback memory model
Former-commit-id: 732bd9c153459f1174475ad67de36c399ddbe359
2020-07-11 21:23:48 +00:00
John Sully
2cc88a96ab Merge branch 'keydbpro' into PRO_RELEASE_6
Former-commit-id: 243dcb3853cc965109cb24a940229db7844cdd11
2020-07-10 04:11:57 +00:00
John Sully
2192c97d35 MVCC scan support filtering by type on the async thread
Former-commit-id: 14f8c0ff686b93976eead5fa6bf526c2eecb5ae0
2020-07-10 03:43:56 +00:00
John Sully
3692771457 Fix issue where SCAN misses elements while snapshot is in flight
Former-commit-id: ce005d748ebf0e116d674a96f74d698d17394010
2020-07-10 01:43:51 +00:00
John Sully
4ec5c3eb31 Fix issue where we fail to return all data
Former-commit-id: 7341c9ed14d13386a045afd6deda9c0db422cd5f
2020-07-10 00:18:10 +00:00
John Sully
29e4973d7b Initial scan implementation (not tested)
Former-commit-id: 0f3911d56e6878d91b6e3cacc7d637934ffe6099
2020-07-09 12:57:35 -04:00
John Sully
29faa06028 Fix assert false alarm when freeTombstoneObjects is called
Former-commit-id: e21a011072505cfa0a37afefb94f7270e80625c5
2020-06-09 20:59:09 -04:00
John Sully
324e8e01f2 COW pointer is not thread safe so we need a global expire lock
Former-commit-id: ac12eef75301564d920e622a08e4f586486fd395
2020-06-09 19:58:42 -04:00
John Sully
848cd33d51 We need to search snapshots before deciding to not save a key
Former-commit-id: 357b09c500eecfe61970e121644a7200ac2387b9
2020-06-05 00:39:58 -04:00
John Sully
8888498bfd Make snapshot completion faster and add latency monitor
Former-commit-id: 8063be6ee70a652c22c3263dccf318366e208891
2020-06-04 01:07:14 -04:00
John Sully
cb4bb9d7eb Consolidate snapshots should not consider keys in bulk storage
Former-commit-id: e02a8f503d45daf63150c60a8a8c3751b9b8581e
2020-06-04 00:26:51 -04:00
John Sully
4cf14b6e49 Fix crash where tombstone is evicted
Former-commit-id: 1f6ff7cc0fc0173610d77a69076eeaf478c3b701
2020-06-03 23:30:35 -04:00
John Sully
a3b80c293b Be *much* more aggressive flushing memory
Former-commit-id: f0bdc4fb5fce02d79c1aa2bcf384aa06580ff9e1
2020-05-07 23:07:31 -04:00
John Sully
ad8a61697b Fix OOM errors during forkless bgsave
Former-commit-id: c31c64b13409c741e8d52ad06add78300c39fce2
2020-03-23 23:12:10 -04:00
John Sully
3610783851 Optimize bgsave snapshotting
Former-commit-id: 15b4bf003537fe6fd926146f46d38203683846be
2020-02-02 23:42:44 -05:00
John Sully
78da9b3af1 endSnapshotAsync implementation (fails tests w/ multithreading)
Former-commit-id: b401e9fcae40cf1d4e4c1584f57e760d9adf36a9
2020-02-01 22:28:24 -05:00
John Sully
40fc4eefbb use dictMerge() in endSnapshot for speed and cleaner code
Former-commit-id: ed8e10234e4499506f2557b0858dba86c30c9dc0
2020-02-01 21:08:26 -05:00
John Sully
e3e64034e2 snapshot optimizations
Former-commit-id: 642f67376a480497554f2891ebfbe7639583c6f1
2020-01-31 17:32:48 -05:00
John Sully
3186cb683f Merge KeyDB 6
Former-commit-id: 9129b98c12b0da856bbda3eb68b4ddc7530693c9
2020-01-29 12:55:23 -05:00
John Sully
49c2b7b77f Fix snapshot consolidation bugs and reduce log noise
Former-commit-id: 7f3cb2981529277d3f02dceb2f60e7aa8385847d
2020-01-12 01:22:44 -05:00
John Sully
ce0fde973a Add support for storing expirations in FLASH
Former-commit-id: 1dca07bd564042fce1b01d275641f35b918ae557
2020-01-03 15:53:36 -05:00
John Sully
4fd552d8b6 Add more test code, and fix bugs uncovered
Former-commit-id: 5362fa4b62f89cbc1e92e01c73a45c4e3718708b
2019-12-23 17:17:41 -05:00
John Sully
448b1093cd Evicted flash no longer keeps the key around
Former-commit-id: 908b303d1a8dd032c092f0bc035361a09c0291a5
2019-12-20 17:45:07 -05:00
John Sully
5449859ede Perf fixes for expire snapshots
Former-commit-id: da805e4442815c89e85ca7b9fc855dd11ef2a6c8
2019-12-17 17:39:04 -05:00
John Sully
b8015a4c2f complete snapshot handling of expires, may still be perf issues
Former-commit-id: 7c6cf4ba4696ba8a7b4069491e8ad0a583958bdc
2019-12-16 22:07:53 -05:00
John Sully
2e05aa89d0 snapshots no longer have dangling keys in the expire set
Former-commit-id: 939b6b61084fbec7b7bd0ada49f6853e334f87dd
2019-12-16 21:14:16 -05:00
John Sully
0b24112fe2 Merge the expire datastructs on endsnapshot
Former-commit-id: 968681ac37d1505a7b38628425f60e08b2cd41c4
2019-12-16 20:13:19 -05:00
John Sully
fce67c145f A bunch of bug fixes
Former-commit-id: 228339586a19874f869cf45dc3834a270f99768a
2019-12-10 01:01:45 -05:00
John Sully
38f7cca61a Add support for multiple databases with rocksdb
Former-commit-id: cfa3b760b00776876134692d42e25a60fcd8dea9
2019-12-09 20:45:58 -05:00
John Sully
11f710c5fc Fixup IStorage uses
Former-commit-id: 5ea0ce143a79365fb3903c6fc7caeb1c9760b0cc
2019-12-06 17:43:28 -05:00
John Sully
5f1331cc3c It is permissible to call dictFind() while iterating
Former-commit-id: 43a598c145d5ea9f2c0b2a16f93f5a8cbb724e7e
2019-11-30 00:03:34 -05:00
John Sully
6533428887 Reduce snapshot window to 1 second instead of 10
Former-commit-id: 4f91f0503fe4eea1c20b977a5858ac345c3e7b9b
2019-11-29 22:21:49 -05:00
John Sully
5fd848eedc Nested snapshot garbage collection. Works but huge fragmentation
Former-commit-id: 82b2a3f3dbf8f864d9157655b5422c69845c4019
2019-11-28 19:00:51 -05:00
John Sully
65a3485dfb Ensure size() count is accurate, and fix find_threadsafe which didn't check children
Former-commit-id: ffa44b636afd4fe907c79fe7deebac3d1e091ee9
2019-11-25 17:50:40 -05:00
John Sully
6ae66162c5 Move snapshot code into its own file
Former-commit-id: 4afd0a6ce650cc93a73ea3e4bb632226087dde37
2019-11-24 17:59:02 -05:00