72 Commits

Author SHA1 Message Date
John Sully
098684bf8e Track key count accurately before we've committed to the database. Also use this to optimize DB lookups when all keys are in memory
Former-commit-id: 78c07f0276109a085761b7044d4edc63af26d3da
2019-12-24 00:03:38 -05:00
John Sully
7da74195a8 Perf remove extra lookup in rocksdb
Former-commit-id: 8074472c7a25572a53f1166911920c2cb168c141
2019-12-23 23:32:04 -05:00
John Sully
054bbe9baa Fix issue where database size is off when setting key with expire
Former-commit-id: 7796918bfb4a98bc056b3b8f4065f1416da8d89a
2019-12-23 22:07:33 -05:00
John Sully
4fd552d8b6 Add more test code, and fix bugs uncovered
Former-commit-id: 5362fa4b62f89cbc1e92e01c73a45c4e3718708b
2019-12-23 17:17:41 -05:00
John Sully
59f7bd49c9 Add additional tests
Former-commit-id: e7d2c78e776b8262fd54caeb7b6bd210182e9bdb
2019-12-22 19:16:30 -05:00
John Sully
ca3c4474c2 Fix FLASH eviction issue, the algo relies on correct key count
Former-commit-id: dc260a7f44fe197b9b0022d044d559be88bf91be
2019-12-22 18:38:10 -05:00
John Sully
1e49dfe8e7 Optimization fails because dbAdd() will ensure the key
Former-commit-id: 1448b555c49fb5ba4a573a97681f7843ba02141f
2019-12-22 17:49:54 -05:00
John Sully
7c8dda47cb Fix issue where DELETE always returns 1 when using FLASH
Former-commit-id: 0574db59fdc740255388ae77d2ece8622898bb49
2019-12-22 17:30:15 -05:00
John Sully
448b1093cd Evicted flash no longer keeps the key around
Former-commit-id: 908b303d1a8dd032c092f0bc035361a09c0291a5
2019-12-20 17:45:07 -05:00
John Sully
439362992f FLASH storage no longer keeps evicted keys in memory
Former-commit-id: 65c2b47c895d57231bbd3f451ecf1f6d9871f5f7
2019-12-19 17:30:31 -05:00
John Sully
67e429e898 Threading fixes
Former-commit-id: 4f1cec6abd72d6e215880f2ce7812e88509cd218
2019-12-16 22:08:18 -05:00
John Sully
2e05aa89d0 snapshots no longer have dangling keys in the expire set
Former-commit-id: 939b6b61084fbec7b7bd0ada49f6853e334f87dd
2019-12-16 21:14:16 -05:00
John Sully
c4b9d485a1 Expireset should do a deep hash
Former-commit-id: cdcdf57dd1419ec09eab6579caf5e5ff9a5a242c
2019-12-16 19:06:07 -05:00
John Sully
fce67c145f A bunch of bug fixes
Former-commit-id: 228339586a19874f869cf45dc3834a270f99768a
2019-12-10 01:01:45 -05:00
John Sully
38f7cca61a Add support for multiple databases with rocksdb
Former-commit-id: cfa3b760b00776876134692d42e25a60fcd8dea9
2019-12-09 20:45:58 -05:00
John Sully
cb136d2ac5 Flash running well
Former-commit-id: 9cf393eb9fb69bcc7550cd8b83e1568b3f360310
2019-12-08 20:06:22 -05:00
John Sully
992784c21d modify how the database array is stored so swaps are easier
Former-commit-id: 48fd6ec5ca69dd505d0e757862c09a898c19cf22
2019-12-08 16:49:59 -05:00
John Sully
86cd47fe80 Prevent extra copy of key
Former-commit-id: 9c3ef97039e59c6b820f71a90f8a9fb4eae8a7b1
2019-12-08 16:02:24 -05:00
John Sully
1e92c7b117 Optimize remote storage by bulk saving keys after processing operations but before sending responses to clients
Former-commit-id: 63a6dc2a29680d5d3f3f245168ca7de5d6bee1eb
2019-12-08 10:56:05 -05:00
John Sully
130589d8d1 Initial RocksDB integration
Former-commit-id: 0de9e5b692c02e779e538ddd0a56d10215e501bb
2019-12-06 20:39:32 -05:00
John Sully
11f710c5fc Fixup IStorage uses
Former-commit-id: 5ea0ce143a79365fb3903c6fc7caeb1c9760b0cc
2019-12-06 17:43:28 -05:00
John Sully
5fd848eedc Nested snapshot garbage collection. Works but huge fragmentation
Former-commit-id: 82b2a3f3dbf8f864d9157655b5422c69845c4019
2019-11-28 19:00:51 -05:00
John Sully
65a3485dfb Ensure size() count is accurate, and fix find_threadsafe which didn't check children
Former-commit-id: ffa44b636afd4fe907c79fe7deebac3d1e091ee9
2019-11-25 17:50:40 -05:00
John Sully
6ae66162c5 Move snapshot code into its own file
Former-commit-id: 4afd0a6ce650cc93a73ea3e4bb632226087dde37
2019-11-24 17:59:02 -05:00
John Sully
551b30fb77 Restrict the snapshot API
Former-commit-id: cbafb88f267a9480cdbde6a88e9e1992e8a85971
2019-11-24 17:53:06 -05:00
John Sully
9a75557651 Get rid of key only iterate, the threadsafe is just as fast and key only users can ignore the robj
Former-commit-id: 29689a5ed6141c5b2019d41ccd4036e6ddd1a121
2019-11-24 15:45:17 -05:00
John Sully
0a123e145e Fix crash on shutdown
Former-commit-id: 6dc55cc0f838b3c82afb233985429b4e7fc5dfb1
2019-11-24 14:24:41 -05:00
John Sully
da4adb261b Fix multithreading data races
Former-commit-id: 80f6e5818fd575cb08a5f620c35eed1cd862eb57
2019-11-24 13:44:43 -05:00
John Sully
5f64e227a4 Fix warning
Former-commit-id: 1318b87013c61f7149127eae8e30caf887a6db7b
2019-11-23 19:19:54 -05:00
John Sully
57595d8e47 Add support for nesting snapshots
Former-commit-id: 5e6c37f388233cc3f2d204ad17ac2897d3333a21
2019-11-23 19:02:50 -05:00
John Sully
fda0f82d64 Add support for nested snapshots
Former-commit-id: 43456c7807152062d59b2a90597b6204f637f5cd
2019-11-22 20:53:36 -05:00
John Sully
b057c1f8fb Merge branch 'unstable' into advanced_db
Former-commit-id: 00ad497fb3f409dbfcaa62fa9ea5d793263eb13d
2019-11-21 20:21:48 -05:00
John Sully
051bde5d3d Fix issue #107, active replicas do their own expires
Former-commit-id: 8e4f323439df29a5e8c0de9db7a848291721fd07
2019-11-20 19:44:31 -05:00
John Sully
065ee57c8d Run KEYS async
Former-commit-id: 214a54a815ff66ef4a1526da90296787240ecdf9
2019-11-19 22:12:52 -05:00
John Sully
408a28bd64 Multiple threads should be able to get the same snapshot provided its not too old
Former-commit-id: 054331098ee18dfb1887fd2b0a67688ef894823e
2019-11-19 17:40:35 -05:00
John Sully
80f9b812ca Ensure iterating the keyspace still works during background save
Former-commit-id: 23d7a8a293f253262801bc6ce88ad85e4e38f59a
2019-11-19 15:47:06 -05:00
John Sully
a60925c949 Final design of forkless background save. expires NYI
Former-commit-id: e2dc24b441bf52b181c820c853e0bc7524254f3f
2019-11-18 19:47:12 -05:00
John Sully
a9f20f0a38 Merge branch 'unstable' into advanced_db
Former-commit-id: 7530d93cd5c4a9fc4e6466abcf96edbdd8daea9e
2019-11-18 15:22:17 -05:00
John Sully
b39e6c9497 Remove unlocked KEYS command support, we will do this with snapshotting
Former-commit-id: 4396682c07d4df3fdca01d1299ad171e310a9fc7
2019-10-24 20:18:11 -04:00
John Sully
0e0e9e73f9 Forkless background save. NOT OPTIMIZED
Former-commit-id: bd9d8e05b0430efd226be773c0530828f1f6b428
2019-10-20 23:54:05 -04:00
John Sully
0c7f279b2c Factor out dependencies on rdb_child_pid which will be going away
Former-commit-id: 2ab4c6b99e3f3cb9f216bd81e78914a6fcd394f6
2019-10-17 23:10:45 -04:00
John Sully
48787c8a03 Dependencies for forkless RDB saving
Former-commit-id: c12be594754ac8079f467e0fa9f8580bd65e3a55
2019-10-17 22:53:50 -04:00
John Sully
6ee58a3b21 Ensure RDB saving only uses data from the persistent data struct, and cleanup inheritance
Former-commit-id: fcdda67d87b3076023f459532c75392b72768a51
2019-10-17 17:20:47 -04:00
John Sully
cec6f0e934 Cleanup inheritance
Former-commit-id: e8debc83e8271cf8d76c5084923cf6e1ea7af3b4
2019-10-16 13:45:36 -04:00
John Sully
a2bd45dbd3 Add the IStorage interface and wire it up
Former-commit-id: 898efbfc0c7038818083ea29fdd63cafa47721fb
2019-10-13 15:24:08 -04:00
John Sully
f9f4c719d4 Isolate the persistent parts of the database from the runtime parts
Former-commit-id: 8d63f5b560d067a634f2ef671962be8815c9e23c
2019-10-13 15:24:08 -04:00
John Sully
b4a34cd600 Hide the database dict
Former-commit-id: 024365521c2b74275839562ba9326b6ea26d3836
2019-10-13 15:13:01 -04:00
John Sully
e57d2e24c9 Update more slave instances to use replica (Issue #75)
Former-commit-id: 252725d50fc9d4ff2b6e9246a36c38176d61beae
2019-10-13 12:29:20 -04:00
John Sully
0a00341a80 Implement TTL and PERSIST commands for subkeys. Also ensure expiremember replaces any old subkey expiries
Former-commit-id: 16f96efbe7c6c27f2a79f5b472447407f905df15
2019-10-13 12:03:25 -04:00
John Sully
bd4ed482c4 KEYS now only blocks one database
Former-commit-id: 18d42a5c353f76533a0ccc4ded24ed089cedacc8
2019-09-12 18:51:10 -04:00