23b016d192
This commit fixes an issue where Tile38 was using lots of extra memory to track objects that are marked to expire. This was creating problems with applications that set big TTLs. How it worked before: Every collection had a unique hashmap that stores expiration timestamps for every object in that collection. Along with the hashmaps, there's also one big server-wide list that gets appended every time a new SET+EX is performed. From a background routine, this list is looped over at least 10 times per second and is randomly searched for potential candidates that might need expiring. The routine then removes those entries from the list and tests if the objects matching the entries have actually expired. If so, these objects are deleted them from the database. When at least 25% of the 20 candidates are deleted the loop is immediately continued, otherwise the loop backs off with a 100ms pause. Why this was a problem. The list grows one entry for every SET+EX. When TTLs are long, like 24-hours or more, it would take at least that much time before the entry is removed. So for databased that have objects that use TTLs and are updated often this could lead to a very large list. How it was fixed. The list was removed and the hashmap is now search randomly. This required a new hashmap implementation, as the built-in Go map does not provide an operation for randomly geting entries. The chosen implementation is a robinhood-hash because it provides open-addressing, which makes for simple random bucket selections. Issue #502
47 lines
1.4 KiB
Go
47 lines
1.4 KiB
Go
// +build !appengine
|
|
|
|
// This file encapsulates usage of unsafe.
|
|
// xxhash_safe.go contains the safe implementations.
|
|
|
|
package xxhash
|
|
|
|
import (
|
|
"reflect"
|
|
"unsafe"
|
|
)
|
|
|
|
// Notes:
|
|
//
|
|
// See https://groups.google.com/d/msg/golang-nuts/dcjzJy-bSpw/tcZYBzQqAQAJ
|
|
// for some discussion about these unsafe conversions.
|
|
//
|
|
// In the future it's possible that compiler optimizations will make these
|
|
// unsafe operations unnecessary: https://golang.org/issue/2205.
|
|
//
|
|
// Both of these wrapper functions still incur function call overhead since they
|
|
// will not be inlined. We could write Go/asm copies of Sum64 and Digest.Write
|
|
// for strings to squeeze out a bit more speed. Mid-stack inlining should
|
|
// eventually fix this.
|
|
|
|
// Sum64String computes the 64-bit xxHash digest of s.
|
|
// It may be faster than Sum64([]byte(s)) by avoiding a copy.
|
|
func Sum64String(s string) uint64 {
|
|
var b []byte
|
|
bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
|
|
bh.Data = (*reflect.StringHeader)(unsafe.Pointer(&s)).Data
|
|
bh.Len = len(s)
|
|
bh.Cap = len(s)
|
|
return Sum64(b)
|
|
}
|
|
|
|
// WriteString adds more data to d. It always returns len(s), nil.
|
|
// It may be faster than Write([]byte(s)) by avoiding a copy.
|
|
func (d *Digest) WriteString(s string) (n int, err error) {
|
|
var b []byte
|
|
bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
|
|
bh.Data = (*reflect.StringHeader)(unsafe.Pointer(&s)).Data
|
|
bh.Len = len(s)
|
|
bh.Cap = len(s)
|
|
return d.Write(b)
|
|
}
|