21057 Commits

Author SHA1 Message Date
antirez
8e8f8189eb HyperLogLog invalid representation error code set to INVALIDOBJ. 2014-04-16 09:10:30 +02:00
antirez
e105f2a645 PFDEBUG TODENSE added.
Converts HyperLogLogs from sparse to dense. Used for testing.
2014-04-16 09:05:42 +02:00
antirez
0bbdaca6a0 PFDEBUG TODENSE added.
Converts HyperLogLogs from sparse to dense. Used for testing.
2014-04-16 09:05:42 +02:00
antirez
1dca87d69c User-defined switch point between sparse-dense HLL encodings. 2014-04-15 17:46:51 +02:00
antirez
402110f9fd User-defined switch point between sparse-dense HLL encodings. 2014-04-15 17:46:51 +02:00
antirez
2736ec0d0f PFSELFTEST improved with sparse encoding checks. 2014-04-15 10:10:38 +02:00
antirez
d541f65d66 PFSELFTEST improved with sparse encoding checks. 2014-04-15 10:10:38 +02:00
antirez
c5d86e9db9 PFDEBUG ENCODING added. 2014-04-14 19:35:00 +02:00
antirez
dde8dff73f PFDEBUG ENCODING added. 2014-04-14 19:35:00 +02:00
antirez
ddc1186dbf Set HLL_SPARSE_MAX to 3000.
After running a few benchmarks, 3000 looks like a reasonable value to
keep HLLs with a few thousand elements small while the CPU cost is
still not huge.

This covers all the cases where the dense representation would use N
orders of magnitude more space, like in the case of many HLLs with
carinality of a few tens or hundreds.

It is not impossible that in the future this gets user configurable,
however it is easy to pick an unreasoable value just looking at savings
in the space dimension without checking what happens in the time
dimension.
2014-04-14 16:15:55 +02:00
antirez
54f0156e8c Set HLL_SPARSE_MAX to 3000.
After running a few benchmarks, 3000 looks like a reasonable value to
keep HLLs with a few thousand elements small while the CPU cost is
still not huge.

This covers all the cases where the dense representation would use N
orders of magnitude more space, like in the case of many HLLs with
carinality of a few tens or hundreds.

It is not impossible that in the future this gets user configurable,
however it is easy to pick an unreasoable value just looking at savings
in the space dimension without checking what happens in the time
dimension.
2014-04-14 16:15:55 +02:00
antirez
bff89bd0a3 Error message for invalid HLL objects unified. 2014-04-14 16:11:54 +02:00
antirez
848d0461f9 Error message for invalid HLL objects unified. 2014-04-14 16:11:54 +02:00
antirez
cd73060972 PFMERGE fixed to work with sparse encoding. 2014-04-14 16:09:32 +02:00
antirez
81ceef7d22 PFMERGE fixed to work with sparse encoding. 2014-04-14 16:09:32 +02:00
antirez
f5fa2985f8 Mark PFDEBUG as write command in the commands table.
It is safer since it is able to have side effects.
2014-04-14 15:57:50 +02:00
antirez
9df77fc0c4 Mark PFDEBUG as write command in the commands table.
It is safer since it is able to have side effects.
2014-04-14 15:57:50 +02:00
antirez
4a43c113c5 Correctly replicate PFDEBUG GETREG.
Even if it is a debugging command, make sure that when it forces a
change in encoding, the command is propagated.
2014-04-14 15:57:19 +02:00
antirez
3bc35f9ce9 Correctly replicate PFDEBUG GETREG.
Even if it is a debugging command, make sure that when it forces a
change in encoding, the command is propagated.
2014-04-14 15:57:19 +02:00
antirez
1e7f95441f Added assertion in hllSparseAdd() when promotion to dense occurs.
If we converted to dense, a register must be updated in the dense
representation.
2014-04-14 15:55:21 +02:00
antirez
ba0afb4566 Added assertion in hllSparseAdd() when promotion to dense occurs.
If we converted to dense, a register must be updated in the dense
representation.
2014-04-14 15:55:21 +02:00
antirez
f009069d7c hllSparseAdd(): speed optimization.
Mostly by reordering opcodes check conditional by frequency of opcodes
in larger sparse-encoded HLLs.
2014-04-14 15:42:05 +02:00
antirez
e9cd51c7eb hllSparseAdd(): speed optimization.
Mostly by reordering opcodes check conditional by frequency of opcodes
in larger sparse-encoded HLLs.
2014-04-14 15:42:05 +02:00
antirez
d2174e6c9b Detect corrupted sparse HLLs in hllSparseSum(). 2014-04-14 15:20:26 +02:00
antirez
681bf7468b Detect corrupted sparse HLLs in hllSparseSum(). 2014-04-14 15:20:26 +02:00
antirez
a84b91d052 hllSparseAdd(): faster code removing conditional.
Bottleneck found profiling. Big run time improvement found when testing
after the change.
2014-04-14 12:58:46 +02:00
antirez
db40da0a47 hllSparseAdd(): faster code removing conditional.
Bottleneck found profiling. Big run time improvement found when testing
after the change.
2014-04-14 12:58:46 +02:00
antirez
6c0f0eb21f Comment typo in hllSparseAdd(). first -> fits. 2014-04-14 12:12:53 +02:00
antirez
4e0a99ba51 Comment typo in hllSparseAdd(). first -> fits. 2014-04-14 12:12:53 +02:00
antirez
8670ab5e11 Merge adjacent VAL opcodes in hllSparseAdd().
As more values are added splitting ZERO or XZERO opcodes, try to merge
adjacent VAL opcodes if they have the same value.
2014-04-14 12:11:39 +02:00
antirez
5532b5308a Merge adjacent VAL opcodes in hllSparseAdd().
As more values are added splitting ZERO or XZERO opcodes, try to merge
adjacent VAL opcodes if they have the same value.
2014-04-14 12:11:39 +02:00
antirez
cba3a04160 More robust HLL_SPARSE macros protecting 'p' with parens.
Now the macros will work with arguments such as "ptr+1".
2014-04-14 11:49:53 +02:00
antirez
837ca39081 More robust HLL_SPARSE macros protecting 'p' with parens.
Now the macros will work with arguments such as "ptr+1".
2014-04-14 11:49:53 +02:00
antirez
d15cd39717 hllSparseAdd() opcode seek stop condition fixed. 2014-04-14 11:04:11 +02:00
antirez
142d133c8a hllSparseAdd() opcode seek stop condition fixed. 2014-04-14 11:04:11 +02:00
antirez
c9ee98b388 Fixed error message generation in PFDEBUG GETREG.
Bulk length for registers was emitted too early, so if there was a bug
the reply looked like a long array with just one element, blocking the
client as result.
2014-04-14 10:25:19 +02:00
antirez
1ee18db922 Fixed error message generation in PFDEBUG GETREG.
Bulk length for registers was emitted too early, so if there was a bug
the reply looked like a long array with just one element, blocking the
client as result.
2014-04-14 10:25:19 +02:00
antirez
70a3bcf3a3 Fixed memmove() count in hllSparseAdd(). 2014-04-14 09:40:07 +02:00
antirez
82c31f750d Fixed memmove() count in hllSparseAdd(). 2014-04-14 09:40:07 +02:00
antirez
ac3655f2a1 hllSparseAdd(): more correct dense conversion conditional.
We want to promote if the total string size exceeds the resulting size
after the upgrade.
2014-04-14 09:36:32 +02:00
antirez
3b20003503 hllSparseAdd(): more correct dense conversion conditional.
We want to promote if the total string size exceeds the resulting size
after the upgrade.
2014-04-14 09:36:32 +02:00
antirez
4fb59e800d hllSparseToDense(): sanity check added.
The function checks if all the HLL_REGISTERS were processed during the
convertion from sparse to dense encoding, returning REDIS_OK or
REDIS_ERR to signal a corruption problem.

A bug in PFDEBUG GETREG was fixed: when the object is converted to the
dense representation we need to reassign the new pointer to the header
structure pointer.
2014-04-14 09:27:01 +02:00
antirez
b7571b7453 hllSparseToDense(): sanity check added.
The function checks if all the HLL_REGISTERS were processed during the
convertion from sparse to dense encoding, returning REDIS_OK or
REDIS_ERR to signal a corruption problem.

A bug in PFDEBUG GETREG was fixed: when the object is converted to the
dense representation we need to reassign the new pointer to the header
structure pointer.
2014-04-14 09:27:01 +02:00
antirez
2800054f34 PFDEBUG DECODE added.
Provides a human readable description of the opcodes composing a
run-length encoded HLL (sparse encoding).
The command is only useful for debugging / development tasks.
2014-04-14 09:00:53 +02:00
antirez
f9dc3cb04d PFDEBUG DECODE added.
Provides a human readable description of the opcodes composing a
run-length encoded HLL (sparse encoding).
The command is only useful for debugging / development tasks.
2014-04-14 09:00:53 +02:00
antirez
62106ad517 PFDEBUG added, PFGETREG removed.
PFDEBUG will be the interface to do debugging tasks with a key
containing an HLL object.
2014-04-13 23:01:21 +02:00
antirez
261da523e8 PFDEBUG added, PFGETREG removed.
PFDEBUG will be the interface to do debugging tasks with a key
containing an HLL object.
2014-04-13 23:01:21 +02:00
antirez
5349296f29 hllSparseToDense API changed to take ref to object.
The new API takes directly the object doing everything needed to
turn it into a dense representation, including setting the new
representation as object->ptr.
2014-04-13 22:59:27 +02:00
antirez
e8e717e145 hllSparseToDense API changed to take ref to object.
The new API takes directly the object doing everything needed to
turn it into a dense representation, including setting the new
representation as object->ptr.
2014-04-13 22:59:27 +02:00
antirez
fd412fcb55 hllSparseAdd() sanity check for span != 0 added. 2014-04-13 10:19:12 +02:00