53 Commits

Author SHA1 Message Date
Salvatore Sanfilippo
d5fd9f031d Fix BITPOS unaligned memory access. 2017-02-23 22:38:44 +08:00
Salvatore Sanfilippo
98ef4372c9 ARM: Avoid fast path for BITOP.
GCC will produce certain unaligned multi load-store instructions
that will be trapped by the Linux kernel since ARM v6 cannot
handle them with unaligned addresses. Better to use the slower
but safer implementation instead of generating the exception which
should be anyway very slow.
2017-02-19 15:07:08 +00:00
sunhe
c7fc514f03 bitops.c/bitfieldCommand: update higest_write_offset with check 2016-10-22 01:54:46 +08:00
antirez
e096bb424a Minor aesthetic fixes to PR #3264.
Comment format fixed + local var modified from camel case to underscore
separators as Redis code base normally does (camel case is mostly used
for global symbols like structure names, function names, global vars,
...).
2016-06-16 12:54:33 +02:00
Salvatore Sanfilippo
955efc65ca Merge pull request #3264 from oranagra/bitfield_fix2
fix crash in BITFIELD GET on non existing key or wrong type see #3259
2016-06-16 12:52:36 +02:00
antirez
5cd8e6832f GETRANGE: return empty string with negative, inverted start/end. 2016-06-15 12:48:58 +02:00
antirez
185ec271cf Remove additional round brackets from fix for #3282. 2016-06-15 12:16:39 +02:00
Salvatore Sanfilippo
f476cc6e50 Merge pull request #3282 from wenduo/unstable
bitcount bug:return non-zero value when start > end (both negative)
2016-06-15 12:15:34 +02:00
Pierre Chapuis
f0d0231473 fix some compiler warnings 2016-06-05 16:48:45 +02:00
antirez
58b3a27b6d Avoid undefined behavior in BITFIELD implementation.
Probably there is no compiler that will actaully break the code or raise
a signal for unsigned -> signed overflowing conversion, still it was
apparently possible to write it in a more correct way.

All tests passing.
2016-05-31 11:52:07 +02:00
wenduo
386f7f5ae0 bitcount bug:return non-zero value when start > end (both negative) 2016-05-30 16:21:08 +08:00
oranagra
170f9dbc22 check WRONGTYPE in BITFIELD before looping on the operations.
optimization: lookup key only once, and grow at once to the max need
fixes #3259 and #3221, and also an early return if wrongtype is discovered by SET
2016-05-24 23:31:36 +03:00
oranagra
63b1eb3729 fix crash in BITFIELD GET on non existing key or wrong type see #3259
this was a bug in the recent refactoring: cbcee71ccba6766d58d729070ede73d808296193
2016-05-24 14:52:43 +03:00
antirez
cbcee71ccb Code to access object string bytes repeated 3x refactored into 1 function. 2016-05-18 15:35:19 +02:00
oranagra
d69bf84ecf fix crash in BITFIELD GET when key is integer encoded 2016-05-10 11:19:45 +03:00
antirez
53ad30831f BITFIELD: Farest bit set is offset+bits-1. Off by one error fixed. 2016-03-02 16:20:28 +01:00
antirez
bb561c71a7 More BITFIELD fixes. Overflow conditional simplified.
See issue #3114.
2016-03-02 15:13:45 +01:00
Sun He
524b28f864 bitops/bitfield: fix length, overflow condition and *sign 2016-03-02 18:11:30 +08:00
antirez
78a32d7f89 BITFIELD: refactoring & fix of retval on FAIL. 2016-02-29 09:08:46 +01:00
antirez
9d34efb3f5 BITFIELD: Fix #<index> form parsing. 2016-02-26 15:53:29 +01:00
antirez
b722e12f77 BITFIELD: Support #<index> offsets form. 2016-02-26 15:16:24 +01:00
antirez
5b3ab9d132 BITFIELD command initial implementation.
The new bitfield command is an extension to the Redis bit operations,
where not just single bit operations are performed, but the array of
bits composing a string, can be addressed at random, not aligned
offsets, with any width unsigned and signed integers like u8, s5, u10
(up to 64 bit signed integers and 63 bit unsigned integers).

The BITFIELD command supports subcommands that can SET, GET, or INCRBY
those arbitrary bit counters, with multiple overflow semantics.

Trivial and credits:

A similar command was imagined a few times in the past, but for
some reason looked a bit far fetched or not well specified.
Finally the command was proposed again in a clear form by
Yoav Steinberg from Redis Labs, that proposed a set of commands on
arbitrary sized integers stored at bit offsets.

Starting from this proposal I wrote an initial specification of a single
command with sub-commands similar to what Yoav envisioned, using short
names for types definitions, and adding control on the overflow.

This commit is the resulting implementation.

Examples:

    BITFIELD mykey OVERFLOW wrap INCRBY i2 10 -1 GET i2 10
2016-02-26 15:00:19 +01:00
antirez
c15cac0d77 RDMF: More consistent define names. 2015-07-27 14:37:58 +02:00
antirez
8a893fa4cf RDMF: REDIS_OK REDIS_ERR -> C_OK C_ERR. 2015-07-26 23:17:55 +02:00
antirez
62b27ebc2a RDMF: OBJ_ macros for object related stuff. 2015-07-26 15:28:00 +02:00
antirez
fa26d3dd63 RDMF: use client instead of redisClient, like Disque. 2015-07-26 15:20:52 +02:00
antirez
6a424b5e36 RDMF (Redis/Disque merge friendlyness) refactoring WIP 1. 2015-07-26 15:17:18 +02:00
Matt Stancliff
101f0b44a5 Bitops: Stop overallocating storage space on set
Previously the string was created empty then re-sized
to fit the offset, but sds resize causes the sds to
over-allocate by at least 1 MB (which is a lot when
you are operating at bit-level access).

This also improves the speed of initial sets by 2% to 6%
based on quick testing.

Patch logic provided by @oranagra

Fixes #1918
2014-12-11 10:54:21 -05:00
Sun He
e03fbdd000 bitops.c/bitopCommand: skip short minlen for FAST PATH 2014-12-03 10:07:58 +08:00
Sun He
38ce5147a5 bitops.c/redisPopcount: little optimization in loop 2014-12-02 14:46:15 +08:00
antirez
2e94ffb1d1 Remove warnings and improve integer sign correctness. 2014-08-13 11:44:38 +02:00
antirez
6028bc5501 String value unsharing refactored into proper function.
All the Redis functions that need to modify the string value of a key in
a destructive way (APPEND, SETBIT, SETRANGE, ...) require to make the
object unshared (if refcount > 1) and encoded in raw format (if encoding
is not already REDIS_ENCODING_RAW).

This was cut & pasted many times in multiple places of the code. This
commit puts the small logic needed into a function called
dbUnshareStringValue().
2014-03-30 18:32:17 +02:00
antirez
6fec8e9300 warnigns -> warnings in redisBitpos(). 2014-02-27 13:17:23 +01:00
antirez
a47d73261b More consistent BITPOS behavior with bit=0 and ranges.
With the new behavior it is possible to specify just the start in the
range (the end will be assumed to be the first byte), or it is possible
to specify both start and end.

This is useful to change the behavior of the command when looking for
zeros inside a string.

1) If the user specifies both start and end, and no 0 is found inside
   the range, the command returns -1.

2) If instead no range is specified, or just the start is given, even
   if in the actual string no 0 bit is found, the command returns the
   first bit on the right after the end of the string.

So for example if the string stored at key foo is "\xff\xff":

    BITPOS foo (returns 16)
    BITPOS foo 0 -1 (returns -1)
    BITPOS foo 0 (returns 16)

The idea is that when no end is given the user is just looking for the
first bit that is zero and can be set to 1 with SETBIT, as it is
"available". Instead when a specific range is given, we just look for a
zero within the boundaries of the range.
2014-02-27 12:53:03 +01:00
antirez
eeb949a94f Initial implementation of BITPOS.
It appears to work but more stress testing, and both unit tests and
fuzzy testing, is needed in order to ensure the implementation is sane.
2014-02-27 12:44:27 +01:00
antirez
fd9c925022 Fix misaligned word access in redisPopcount(). 2014-02-27 09:46:20 +01:00
antirez
aa32f92338 Introduction of a new string encoding: EMBSTR
Previously two string encodings were used for string objects:

1) REDIS_ENCODING_RAW: a string object with obj->ptr pointing to an sds
stirng.

2) REDIS_ENCODING_INT: a string object where the obj->ptr void pointer
is casted to a long.

This commit introduces a experimental new encoding called
REDIS_ENCODING_EMBSTR that implements an object represented by an sds
string that is not modifiable but allocated in the same memory chunk as
the robj structure itself.

The chunk looks like the following:

+--------------+-----------+------------+--------+----+
| robj data... | robj->ptr | sds header | string | \0 |
+--------------+-----+-----+------------+--------+----+
                     |                       ^
                     +-----------------------+

The robj->ptr points to the contiguous sds string data, so the object
can be manipulated with the same functions used to manipulate plan
string objects, however we need just on malloc and one free in order to
allocate or release this kind of objects. Moreover it has better cache
locality.

This new allocation strategy should benefit both the memory usage and
the performances. A performance gain between 60 and 70% was observed
during micro-benchmarks, however there is more work to do to evaluate
the performance impact and the memory usage behavior.
2013-07-22 10:31:38 +02:00
antirez
1065918c36 function renamed: popcount_binary -> redisPopcount. 2013-06-26 15:19:06 +02:00
YAMAMOTO Takashi
ed23c6ff59 rename popcount to popcount_binary to avoid a conflict with NetBSD libc
NetBSD-current's libc has a function named popcount.
hiding these extensions using feature macros is not possible because
redis uses other extensions covered by the same feature macro.
eg. inet_aton
2013-05-17 17:21:28 +09:00
antirez
e6b3fd7a1c Revert "use long long instead of size_t make it more safe"
This reverts commit 47f730ba26cffd207169b960f51f0de4406c82e5.

After further analysis, it is very unlikely that we'll raise the
string size limit to > 512MB, and at the same time such big strings
will be used in 32 bit systems.

Better to revert to size_t so that 32 bit processors will not be
forced to use a 64 bit counter in normal operations, that is currently
completely useless.
2013-05-08 10:01:27 +02:00
Jiahao Huang
47f730ba26 use long long instead of size_t make it more safe 2013-05-07 23:37:22 +08:00
Jiahao Huang
69227d8b76 in 32bit machine, popcount don't work with a input string length up to 512 MB,
bitcount commant may return negtive integer with string length more than 256 MB
2013-05-07 19:55:57 +08:00
antirez
f28d386cc5 Keyspace events: it is now possible to select subclasses of events.
When keyspace events are enabled, the overhead is not sever but
noticeable, so this commit introduces the ability to select subclasses
of events in order to avoid to generate events the user is not
interested in.

The events can be selected using redis.conf or CONFIG SET / GET.
2013-01-28 13:15:12 +01:00
antirez
f2d1105618 Keyspace events added for more commands. 2013-01-28 13:14:56 +01:00
guiquanz
df7a5b7157 Fixed many typos. 2013-01-19 10:59:44 +01:00
antirez
a32d1ddff6 BSD license added to every C source and header file. 2012-11-08 18:31:32 +01:00
Haruto Otake
d60cc951f1 BITCOUNT: fix segmentation fault.
remove unsafe and unnecessary cast.
until now, this cast may lead segmentation fault when end > UINT_MAX

setbit foo 0 1
bitcount  0 4294967295
=> ok
bitcount  0 4294967296
=> cause segmentation fault.

Note by @antirez: the commit was modified a bit to also change the
string length type to long, since it's guaranteed to be at max 512 MB in
size, so we can work with the same type across all the code path.

A regression test was also added.
2012-09-05 16:19:04 +02:00
antirez
3fbd3b28a0 Don't assume that "char" is signed.
For the C standard char can be either signed or unsigned, it's up to the
compiler, but Redis assumed that it was signed in a few places.

The practical effect of this patch is that now Redis 2.6 will run
correctly in every system where char is unsigned, notably the RaspBerry
PI and other ARM systems with GCC.

Thanks to Georgi Marinov (@eesn on twitter) that reported the problem
and allowed me to use his RaspBerry via SSH to trace and fix the issue!
2012-07-18 12:04:58 +02:00
antirez
d13ed62ce7 BITOP bug when called against non existing keys fixed.
In the issue #529 an user reported a bug that can be triggered with the
following code:

flushdb
set a
"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
bitop or x a b

The bug was introduced with the speed optimization in commit 8bbc076
that specializes every BITOP operation loop up to the minimum length of
the input strings.

However the computation of the minimum length contained an error when a
non existing key was present in the input, after a key that was non zero
length.

This commit fixes the bug and adds a regression test for it.
2012-05-31 21:52:47 +02:00
antirez
9aff93dc1b BITOP command 10x speed improvement.
This commit adds a fast-path to the BITOP that can be used for all the
bytes from 0 to the minimal length of the string, and if there are
at max 16 input keys.

Often the intersected bitmaps are roughly the same size, so this
optimization can provide a 10x speed boost to most real world usages
of the command.

Bytes are processed four full words at a time, in loops specialized
for the specific BITOP sub-command, without the need to check for
length issues with the inputs (since we run this algorithm only as far
as there is data from all the keys at the same time).

The remaining part of the string is intersected in the usual way using
the slow but generic algorith.

It is possible to do better than this with inputs that are not roughly
the same size, sorting the input keys by length, by initializing the
result string in a smarter way, and noticing that the final part of the
output string composed of only data from the longest string does not
need any proecessing since AND, OR and XOR against an empty string does
not alter the output (zero in the first case, and the original string in
the other two cases).

More implementations will be implemented later likely, but this should
be enough to release Redis 2.6-RC4 with bitops merged in.

Note: this commit also adds better testing for BITOP NOT command, that
is currently the faster and hard to optimize further since it just
flips the bits of a single input string.
2012-05-24 15:20:20 +02:00