27289 Commits

Author SHA1 Message Date
antirez
f72612ad7e Use endian neutral hash function for HyperLogLog.
We need to be sure that you can save a dataset in a Redis instance,
reload it in a different architecture, and continue to count in the same
HyperLogLog structure.

So 32 and 64 bit, little or bit endian, must all guarantee to output the
same hash for the same element.
2014-03-30 00:55:49 +01:00
antirez
aaf6db459b Use endian neutral hash function for HyperLogLog.
We need to be sure that you can save a dataset in a Redis instance,
reload it in a different architecture, and continue to count in the same
HyperLogLog structure.

So 32 and 64 bit, little or bit endian, must all guarantee to output the
same hash for the same element.
2014-03-30 00:55:49 +01:00
antirez
e67d55d949 HyperLogLog internal representation modified.
The new representation is more obvious, starting from the LSB of the
first byte and using bits going to MSB, and passing to next byte as
needed.

There was also a subtle error: first two bits were unused, everything
was carried over on the right of two bits, even if it worked because of
the code requirement of always having a byte more at the end.

During the rewrite the code was made safer trying to avoid undefined
behavior due to shifting an uint8_t for more than 8 bits.
2014-03-29 16:04:27 +01:00
antirez
4628ac0065 HyperLogLog internal representation modified.
The new representation is more obvious, starting from the LSB of the
first byte and using bits going to MSB, and passing to next byte as
needed.

There was also a subtle error: first two bits were unused, everything
was carried over on the right of two bits, even if it worked because of
the code requirement of always having a byte more at the end.

During the rewrite the code was made safer trying to avoid undefined
behavior due to shifting an uint8_t for more than 8 bits.
2014-03-29 16:04:27 +01:00
antirez
105ead7ea6 Remove a few useless operations from hllCount() fast path. 2014-03-29 12:17:56 +01:00
antirez
5317a582cf Remove a few useless operations from hllCount() fast path. 2014-03-29 12:17:56 +01:00
antirez
7c8277cc75 HLLCOUNT 3x faster taking fast path for default params. 2014-03-29 12:12:44 +01:00
antirez
3ed947fb30 HLLCOUNT 3x faster taking fast path for default params. 2014-03-29 12:12:44 +01:00
antirez
0c280053dd Use processor base types in HLL_(GET|SET)_REGISTER.
This speedups the macros by a noticeable factor.
2014-03-29 08:37:01 +01:00
antirez
28dce36f76 Use processor base types in HLL_(GET|SET)_REGISTER.
This speedups the macros by a noticeable factor.
2014-03-29 08:37:01 +01:00
antirez
4983350c6c HyperLogLog: use precomputed table for 2^(-M[i]). 2014-03-28 22:49:24 +01:00
antirez
ac8fbe8829 HyperLogLog: use precomputed table for 2^(-M[i]). 2014-03-28 22:49:24 +01:00
antirez
74547ebed9 hll-err.rb: speedup using pipelining. 2014-03-28 22:25:30 +01:00
antirez
fdf81b2d35 hll-err.rb: speedup using pipelining. 2014-03-28 22:25:30 +01:00
antirez
d622c7d0aa hll-err.rb added to test error rate of Redis HyperLogLog. 2014-03-28 18:35:21 +01:00
antirez
f29123364b hll-err.rb added to test error rate of Redis HyperLogLog. 2014-03-28 18:35:21 +01:00
antirez
e6874512b9 HyperLogLog algorithm fixed in two ways.
There was an error in the computation of 2^register, and the sequence of
zeroes computed after the hashing did not included the "1".
2014-03-28 18:24:05 +01:00
antirez
f90a4af3d7 HyperLogLog algorithm fixed in two ways.
There was an error in the computation of 2^register, and the sequence of
zeroes computed after the hashing did not included the "1".
2014-03-28 18:24:05 +01:00
antirez
cde7093b10 HLLCOUNT implemented. 2014-03-28 17:37:18 +01:00
antirez
ded86076b3 HLLCOUNT implemented. 2014-03-28 17:37:18 +01:00
antirez
abb0b21919 HLLADD implemented. 2014-03-28 16:24:35 +01:00
antirez
156929ee97 HLLADD implemented. 2014-03-28 16:24:35 +01:00
antirez
55c7347125 hllAdd() low level HyperLogLog "add" implemented. 2014-03-28 14:42:30 +01:00
antirez
5660ff1cc1 hllAdd() low level HyperLogLog "add" implemented. 2014-03-28 14:42:30 +01:00
antirez
1954abf924 HyperLogLog: redefine constants using "P". 2014-03-28 14:09:28 +01:00
antirez
e3234116ad HyperLogLog: redefine constants using "P". 2014-03-28 14:09:28 +01:00
antirez
d1e473cacf HLL_SET_REGISTER fixed.
There was an error in the first version of the macro.
Now the HLLSELFTEST test reports success.
2014-03-28 13:56:07 +01:00
antirez
e73839e7d5 HLL_SET_REGISTER fixed.
There was an error in the first version of the macro.
Now the HLLSELFTEST test reports success.
2014-03-28 13:56:07 +01:00
antirez
2084bf44b6 Use REDIS_HLL_REGISTER_MAX when possible. 2014-03-28 12:16:39 +01:00
antirez
f22397dd7f Use REDIS_HLL_REGISTER_MAX when possible. 2014-03-28 12:16:39 +01:00
antirez
c359fd89a5 HLL_(SET|GET)_REGISTER types fixed. 2014-03-28 12:15:46 +01:00
antirez
1c88c5941b HLL_(SET|GET)_REGISTER types fixed. 2014-03-28 12:15:46 +01:00
antirez
61f3239044 HLLSELFTEST command implemented.
To test the bitfield array of counters set/get macros from the Redis Tcl
suite is hard, so a specialized command that is able to test the
internals was developed.
2014-03-28 12:11:55 +01:00
antirez
552eb5407a HLLSELFTEST command implemented.
To test the bitfield array of counters set/get macros from the Redis Tcl
suite is hard, so a specialized command that is able to test the
internals was developed.
2014-03-28 12:11:55 +01:00
antirez
5669bf9643 HyperLogLog: initial sketch of registers access. 2014-03-28 11:18:48 +01:00
antirez
0609380603 HyperLogLog: initial sketch of registers access. 2014-03-28 11:18:48 +01:00
antirez
0f816d9867 Cluster: last_vote_epoch -> lastVoteEpoch.
Use cammel case for epochs that are persisted on disk.
2014-03-27 15:01:24 +01:00
antirez
8f52173b2c Cluster: last_vote_epoch -> lastVoteEpoch.
Use cammel case for epochs that are persisted on disk.
2014-03-27 15:01:24 +01:00
antirez
befba3795d Cluster: save/restore vars that must persist after recovery.
This fixes issue #1479.
2014-03-27 14:56:29 +01:00
antirez
7fb14b73ba Cluster: save/restore vars that must persist after recovery.
This fixes issue #1479.
2014-03-27 14:56:29 +01:00
antirez
d28faade17 Cluster: handshake "already known" error logged to VERBOSE.
This is not really an error but something that always happens for
example when creating a new cluster, or if the sysadmin rejoins manually
a node that is already known.

Since useless logs don't help, moved to VERBOSE level.
2014-03-26 16:35:38 +01:00
antirez
6dd2dbbd36 Cluster: handshake "already known" error logged to VERBOSE.
This is not really an error but something that always happens for
example when creating a new cluster, or if the sysadmin rejoins manually
a node that is already known.

Since useless logs don't help, moved to VERBOSE level.
2014-03-26 16:35:38 +01:00
antirez
52dddf2503 Cluster: clusterHandleConfigEpochCollision() fixed.
New config epochs must always be obtained incrementing the currentEpoch,
that is itself guaranteed to be >= the max configEpoch currently known
to the node.
2014-03-26 12:31:28 +01:00
antirez
3cf6f1f54f Cluster: clusterHandleConfigEpochCollision() fixed.
New config epochs must always be obtained incrementing the currentEpoch,
that is itself guaranteed to be >= the max configEpoch currently known
to the node.
2014-03-26 12:31:28 +01:00
antirez
5c2bc5207a Cluster: better logging for clusterUpdateSlotsConfigWith(). 2014-03-26 12:09:38 +01:00
antirez
80d4c52cdf Cluster: better logging for clusterUpdateSlotsConfigWith(). 2014-03-26 12:09:38 +01:00
antirez
242646544a Cluster: CLUSTER SETSLOT implementation comment updated.
Update the comment since the implementation details changed.
2014-03-25 17:50:46 +01:00
antirez
eb746ec408 Cluster: CLUSTER SETSLOT implementation comment updated.
Update the comment since the implementation details changed.
2014-03-25 17:50:46 +01:00
antirez
41fb27d1d1 Cluster: redis-trib cluster allocation more even across nodes.
redis-trib used to allocate slots not considering fractions of nodes
when computing the slots_per_node amount. So the fractional part was
carried over till the end of the allocation, where the last node
received a few more slots than any other (or a lot more if the cluster
was composed of many nodes).

The computation was changed to allocate slots more evenly when they are
not exactly divisible for the number of masters we have.
2014-03-25 17:44:39 +01:00
antirez
0064b1a583 Cluster: redis-trib cluster allocation more even across nodes.
redis-trib used to allocate slots not considering fractions of nodes
when computing the slots_per_node amount. So the fractional part was
carried over till the end of the allocation, where the last node
received a few more slots than any other (or a lot more if the cluster
was composed of many nodes).

The computation was changed to allocate slots more evenly when they are
not exactly divisible for the number of masters we have.
2014-03-25 17:44:39 +01:00