464 Commits

Author SHA1 Message Date
Oran Agra
28cf208bf3 slave buffers were wasteful and incorrectly counted causing eviction
A) slave buffers didn't count internal fragmentation and sds unused space,
   this caused them to induce eviction although we didn't mean for it.

B) slave buffers were consuming about twice the memory of what they actually needed.
- this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
  but networking.c not storing more than 16k (partially fixed recently in 237a38737).
- besides it wasn't able to store half of the new string into one buffer and the
  other half into the next (so the above mentioned fix helped mainly for small items).
- lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
  consumed but not used.

C) inefficient performance due to starting from a small string and reallocing many times.

what i changed:
- creating dedicated buffers for reply list, counting their size with zmalloc_size
- when creating a new reply node from, preallocate it to at least 16k.
- when appending a new reply to the buffer, first fill all the unused space of the
  previous node before starting a new one.

other changes:
- expose mem_not_counted_for_evict info field for the benefit of the test suite
- add a test to make sure slave buffers are counted correctly and that they don't cause eviction
2018-07-16 16:43:42 +03:00
antirez
b01cb491a8 Test: XDEL fuzz testing. Remove and check stage. 2018-07-13 17:58:17 +02:00
antirez
6ddca524c4 Test: XDEL fuzz testing, stream creation. 2018-07-13 17:47:26 +02:00
antirez
ee176661ca Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-13 17:41:10 +02:00
antirez
9b799a052d Test: XDEL basic test. 2018-07-13 17:40:48 +02:00
WuYunlong
1362d2d348 Add test in slowlog.tcl 2018-07-13 17:51:06 +08:00
antirez
fadce22ae9 Add regression test for #5111. 2018-07-12 13:35:17 +02:00
Jack Drogon
bae1d36e5d Fix typo 2018-07-03 18:19:46 +02:00
chendianqiang
602d502e5c Merge branch 'unstable' into pending-querybuf 2018-07-03 10:07:26 +08:00
chendianqiang
117a76ac63 limit the size of pending-querybuf in masterclient 2018-07-01 14:43:53 +08:00
Oran Agra
dcd3529acf minor fix in creating a stream NACK for rdb and defrag tests 2018-06-27 15:34:17 +03:00
Oran Agra
160802b06f add active defrag support for streams 2018-06-27 15:00:41 +03:00
shenlongxing
dbd1e254bd fix typo 2018-06-21 22:08:09 +08:00
Guy Benoish
4330bc3e3a Enhance RESTORE with RDBv9 new features
RESTORE now supports:
1. Setting LRU/LFU
2. Absolute-time TTL

Other related changes:
1. RDB loading will not override LRU bits when RDB file
   does not contain the LRU opcode.
2. RDB loading will not set LRU/LFU bits if the server's
   maxmemory-policy does not match.
2018-06-20 15:11:08 +07:00
antirez
030543ce25 Fix SCAN bug regression test, avoiding empty SREM call. 2018-06-14 12:21:58 +02:00
antirez
e937fff4ab Regression test for issue #5006. 2018-06-12 13:13:35 +02:00
antirez
0a4201713a Improved regression test for #4906.
Removing the fix about 50% of the times the test will not be able to
pass cleanly. It's very hard to write a test that will always fail, or
actually, it is possible but then it's likely that it will consistently
pass if we change some random bit, so better to use randomization here.
2018-06-11 13:10:06 +02:00
antirez
6bef2759bb Regression test for the dictScan() issue #4906. 2018-06-11 12:51:26 +02:00
antirez
2c08c2d425 ZPOP: invert score-ele to match ZRANGE WITHSCORES order. 2018-06-05 17:06:25 +02:00
antirez
5b69ad6e6d Streams: fix test ID format. 2018-05-25 16:57:08 +02:00
antirez
9108bbab07 Make active defragmentation tests optional.
They failed when active defrag could not be activated because the
Jemalloc version does not include the additional APIs.
2018-05-24 18:04:21 +02:00
Oran Agra
0bd6e341f5 Active defrag fixes for 32bit builds
problems fixed:
* failing to read fragmentation information from jemalloc
* overflow in jemalloc fragmentation hint to the defragger
* test suite not triggering eviction after population
2018-05-17 09:52:00 +03:00
antirez
d0326f472e ZPOP: fix the tests according to new non-blocking "count" argument. 2018-05-11 18:07:10 +02:00
antirez
cbbeef707e ZPOP: renaming to have explicit MIN/MAX score idea.
This commit also adds a top comment about a subtle behavior of mixing
blocking operations of different types in the same key.
2018-05-11 17:31:53 +02:00
Itamar Haber
52c9565dda Implements [B]Z[REV]POP and the respective unit tests
An implementation of the
[Ze POP Redis Module](https://github.com/itamarhaber/zpop) as core
Redis commands.

Fixes #1861.
2018-04-30 02:10:42 +03:00
antirez
81383a5b45 AOF: run tests with preamble off when it makes sense. 2018-03-25 13:03:38 +02:00
Salvatore Sanfilippo
d799fdf733 Merge pull request #4691 from oranagra/active_defrag_v2
Active defrag v2
2018-03-22 09:16:32 +01:00
antirez
72282c440b CG: test XACK remaining features. 2018-03-15 12:54:10 +01:00
antirez
c72389ef0f CG: test XACK ability to remove items from the PELs. 2018-03-15 12:54:10 +01:00
antirez
862c29fc1b CG: test XPENDING ability to return pending items. 2018-03-15 12:54:10 +01:00
antirez
0a5272cea8 CG: test XGROUPREAD abilities. 2018-03-15 12:54:10 +01:00
antirez
8d3c0e4186 CG: More specific duplicated group error. 2018-03-15 12:54:10 +01:00
Oran Agra
5805907825 Adding real allocator fragmentation to INFO and MEMORY command + active defrag test
other fixes / improvements:
- LUA script memory isn't taken from zmalloc (taken from libc malloc)
  so it can cause high fragmentation ratio to be displayed (which is false)
- there was a problem with "fragmentation" info being calculated from
  RSS and used_memory sampled at different times (now sampling them together)

other details:
- adding a few more allocator info fields to INFO and MEMORY commands
- improve defrag test to measure defrag latency of big keys
- increasing the accuracy of the defrag test (by looking at real grag info)
  this way we can use an even lower threshold and still avoid false positives
- keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
- add these the MEMORY DOCTOR command
- deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
- reduce sampling rate of the rss and allocator info
2018-03-12 15:08:52 +02:00
antirez
f33db219a6 Test: MIGRATE AUTH test added.
Related to #2507.
2018-01-09 18:49:19 +01:00
antirez
b404a1717f Streams: add some initial test for XREVRANGE. 2017-12-01 10:24:25 +01:00
antirez
0e6b2095ec Streams: fix XREAD test broken after previous tests improvements.
10% of times the data is not just "item 0" but there is also the
"otherfield" part. Use [lrange] to avoid the issue.
This commit fixes #4416.
2017-12-01 10:24:24 +01:00
antirez
6808443d33 Streams: move ID ms/seq separator from '.' to '-'
After checking with the community via Twitter (here:
https://twitter.com/antirez/status/915130876861788161) the verdict was to
use ":". However I later realized, after users lamented the fact that
it's hard to copy IDs just with double click, that this was the reason
why I moved to "." in the first instance. Fortunately "-", that was the
other option with most votes, also gets selected with double click on
most terminal applications on Linux and MacOS.

So my reasoning was:

1) We can't retain "." because it's actually confusing to newcomers, it
looks like a floating number, people may be tricked into thinking they
can order IDs numerically as floats.

2) Moving to a double-click-to-select format is much better. People will
work with such IDs for long time when coding / debugging. Why making now
a choice that will impact this for the next years?

The only other viable option was "-", and that's what I did. Thanks.
2017-12-01 10:24:24 +01:00
antirez
b9184aea54 Streams: add XADD + MAXLEN test. 2017-12-01 10:24:24 +01:00
antirez
82a49ea37c Streams: modify tests to stress compression. 2017-12-01 10:24:24 +01:00
antirez
6bfd24f6b1 Streams: tests for blocking and non-blocking XREAD. 2017-12-01 10:24:24 +01:00
antirez
8d446360f0 Streams: XRANGE fuzz testing. 2017-12-01 10:24:24 +01:00
antirez
6f55ebcf57 Streams: more advanced XADD and XRANGE tests. 2017-12-01 10:24:24 +01:00
antirez
08b179a719 Streams: basic XADD tests. 2017-12-01 10:24:24 +01:00
antirez
1dc952210a Test: regression test for latency expire events logging bug.
Regression for #4452.
2017-11-24 18:33:31 +01:00
Oran Agra
43a17cc78d fix string to double conversion, stopped parsing on \0 even if the string has more data.
getLongLongFromObject calls string2ll which has this line:
/* Return if not all bytes were used. */
so if you pass an sds with 3 characters "1\01" it will fail.

but getLongDoubleFromObject calls strtold, and considers it ok if eptr[0]==`\0`
i.e. if the end of the string found by strtold ends with null terminator

127.0.0.1:6379> set a 1
OK
127.0.0.1:6379> setrange a 2 2
(integer) 3
127.0.0.1:6379> get a
"1\x002"
127.0.0.1:6379> incrbyfloat a 2
"3"
127.0.0.1:6379> get a
"3"
2017-11-23 17:15:27 +02:00
antirez
55d48534bb Regression test for issue #4391. 2017-10-30 13:45:46 +01:00
antirez
041d783b19 Fix GEORADIUS edge case with huge radius.
This commit closes issue #3698, at least for now, since the root cause
was not fixed: the bounding box function, for huge radiuses, does not
return a correct bounding box, there are points still within the radius
that are left outside.

So when using GEORADIUS queries with radiuses in the order of 5000 km or
more, it was possible to see, at the edge of the area, certain points
not correctly reported.

Because the bounding box for now was used just as an optimization, and
such huge radiuses are not common, for now the optimization is just
switched off when the radius is near such magnitude.

Three test cases found by the Continuous Integration test were added, so
that we can easily trigger the bug again, both for regression testing
and in order to properly fix it as some point in the future.
2017-07-03 19:38:31 +02:00
xuzhou
3436bcf681 Fix set with ex/px option when propagated to aof 2017-06-16 17:51:38 +08:00
antirez
08536da477 SLOWLOG: log offending client address and name. 2017-06-15 12:57:54 +02:00
antirez
ad75edaff9 Defrag: test currently disabled, too many false positives.
Related to #3786.
2017-04-22 15:59:57 +02:00