lsm-tree-0.1.0.0: Log-structured merge-trees
Safe HaskellSafe-Inferred
LanguageGHC2021

Data.BloomFilter.Blocked

Description

A fast, space efficient Bloom filter implementation. A Bloom filter is a set-like data structure that provides a probabilistic membership test.

  • Queries do not give false negatives. When an element is added to a filter, a subsequent membership test will definitely return True.
  • False positives are possible. If an element has not been added to a filter, a membership test may nevertheless indicate that the element is present.
Synopsis

Types

type Hash = Word64 Source #

A hash value is 64 bits wide.

class Hashable a Source #

The class of types that can be converted to a hash value.

The instances are meant to be stable, the hash values can be persisted.

Minimal complete definition

hashSalt64

Instances

Instances details
Hashable ByteArray Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable Word32 Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable Word64 Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable ByteString Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable ByteString Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable () Source # 
Instance details

Defined in Data.BloomFilter.Hash

Methods

hashSalt64 :: Word64 -> () -> Word64 Source #

Hashable Char Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable Int Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable Word Source # 
Instance details

Defined in Data.BloomFilter.Hash

Hashable a => Hashable [a] Source # 
Instance details

Defined in Data.BloomFilter.Hash

Methods

hashSalt64 :: Word64 -> [a] -> Word64 Source #

(Hashable a, Hashable b) => Hashable (a, b) Source # 
Instance details

Defined in Data.BloomFilter.Hash

Methods

hashSalt64 :: Word64 -> (a, b) -> Word64 Source #

Immutable Bloom filters

data Bloom a Source #

An immutable Bloom filter.

Instances

Instances details
Show (Bloom a) Source # 
Instance details

Defined in Data.BloomFilter.Blocked.Internal

Methods

showsPrec :: Int -> Bloom a -> ShowS #

show :: Bloom a -> String #

showList :: [Bloom a] -> ShowS #

NFData (Bloom a) Source # 
Instance details

Defined in Data.BloomFilter.Blocked.Internal

Methods

rnf :: Bloom a -> () #

Eq (Bloom a) Source # 
Instance details

Defined in Data.BloomFilter.Blocked.Internal

Methods

(==) :: Bloom a -> Bloom a -> Bool #

(/=) :: Bloom a -> Bloom a -> Bool #

Creation

create Source #

Arguments

:: BloomSize 
-> (forall s. MBloom s a -> ST s ())

setup function

-> Bloom a 

Create an immutable Bloom filter, using the given setup function which executes in the ST monad.

Example:

filter = create (sizeForBits 16 2) $ mf -> do
           insert mf "foo"
           insert mf "bar"
 

Note that the result of the setup function is not used.

unfold Source #

Arguments

:: forall a b. Hashable a 
=> BloomSize 
-> (b -> Maybe (a, b))

seeding function

-> b

initial seed

-> Bloom a 

Build an immutable Bloom filter from a seed value. The seeding function populates the filter as follows.

  • If it returns Nothing, it is finished producing values to insert into the filter.
  • If it returns Just (a,b), a is added to the filter and b is used as a new seed.

fromList Source #

Arguments

:: (Foldable t, Hashable a) 
=> BloomPolicy 
-> t a

values to populate with

-> Bloom a 

Create a Bloom filter, populating it from a sequence of values.

For example

filt = fromList (policyForBits 10) ["foo", "bar", "quux"]

(De)Serialisation

formatVersion :: Int Source #

The version of the format used by serialise and deserialise. The format number will change when there is an incompatible change in the library, such that deserialising and using the filter will not work. This can include more than just changes to the serialised format, for example changes to hash functions or how the hash is mapped to bits.

Note that the format produced does not include this version. Version checking is the responsibility of the user of the library.

The library guarantes that the format version value for the classic (Data.BloomFilter.Classic) and blocked (Data.BloomFilter.Blocked) implementation will not overlap with each other or any previous value used by either implementation. So switching between the two implementations will always be detectable and unambigious.

History:

  • Version 1000: original blocked implementation

serialise :: Bloom a -> (BloomSize, ByteArray, Int, Int) Source #

Serialise the bloom filter to a BloomSize (which is needed to deserialise) and a ByteArray along with the offset and length containing the filter's bit data.

See also formatVersion for compatibility advice.

deserialise :: PrimMonad m => BloomSize -> (MutableByteArray (PrimState m) -> Int -> Int -> m ()) -> m (Bloom a) Source #

Sizes

data BloomSize Source #

Parameters for constructing a Bloom filter.

Constructors

BloomSize 

Fields

  • sizeBits :: !Int

    The requested number of bits in filter. The actual size will be rounded up to the nearest 512.

  • sizeHashes :: !Int

    The number of hash functions to use.

Instances

Instances details
Show BloomSize Source # 
Instance details

Defined in Data.BloomFilter.Classic.Calc

data BloomPolicy Source #

A policy on intended bloom filter size -- independent of the number of elements.

We can decide a policy based on:

  1. a target false positive rate (FPR) using policyForFPR
  2. a number of bits per entry using policyForBits

A policy can be turned into a BloomSize given a target NumEntries using sizeForPolicy.

Either way we define the policy, we can inspect the result to see:

  1. The bits per entry policyBits. This will determine the size of the bloom filter in bits. In general the bits per entry can be fractional. The final bloom filter size in will be rounded to a whole number of bits.
  2. The number of hashes policyHashes.
  3. The expected FPR for the policy using policyFPR.

Constructors

BloomPolicy 

Instances

Instances details
Show BloomPolicy Source # 
Instance details

Defined in Data.BloomFilter.Classic.Calc

Accessors

size :: Bloom a -> BloomSize Source #

Return the size of the Bloom filter.

elem :: Hashable a => a -> Bloom a -> Bool Source #

Query an immutable Bloom filter for membership. If the value is present, return True. If the value is not present, there is still some possibility that True will be returned.

notElem :: Hashable a => a -> Bloom a -> Bool Source #

Query an immutable Bloom filter for non-membership. If the value is present, return False. If the value is not present, there is still some possibility that False will be returned.

(?) :: Hashable a => Bloom a -> a -> Bool Source #

Same as elem but with the opposite argument order:

x `elem` bfilter

versus

bfilter ? x

Mutable Bloom filters

data MBloom s a Source #

A mutable Bloom filter, for use within the ST monad.

Instances

Instances details
Show (MBloom s a) Source # 
Instance details

Defined in Data.BloomFilter.Blocked.Internal

Methods

showsPrec :: Int -> MBloom s a -> ShowS #

show :: MBloom s a -> String #

showList :: [MBloom s a] -> ShowS #

NFData (MBloom s a) Source # 
Instance details

Defined in Data.BloomFilter.Blocked.Internal

Methods

rnf :: MBloom s a -> () #

new :: BloomSize -> ST s (MBloom s a) Source #

Create a new mutable Bloom filter.

The filter size is capped at maxSizeBits.

insert :: Hashable a => MBloom s a -> a -> ST s () Source #

Insert a value into a mutable Bloom filter. Afterwards, a membership query for the same value is guaranteed to return True.

insertMany Source #

Arguments

:: forall a s. Hashable a 
=> MBloom s a 
-> (Int -> ST s a)

Action to lookup elements, indexed 0..n-1

-> Int

n, number of elements to insert

-> ST s () 

A bulk insert of many elements.

This is somewhat faster than repeated insertion using insert. It uses memory prefetching to improve the utilisation of memory bandwidth. This has greatest benefit for large filters (that do not fit in L3 cache) and for inserting many elements, e.g. > 10.

To get best performance, you probably want to specialise this function to the Hashable instance and to the lookup action. It is marked INLINABLE to help with this.

Conversion

freeze :: MBloom s a -> ST s (Bloom a) Source #

Create an immutable Bloom filter from a mutable one. The mutable filter may be modified afterwards.

thaw :: Bloom a -> ST s (MBloom s a) Source #

Copy an immutable Bloom filter to create a mutable one. There is no non-copying equivalent.

unsafeFreeze :: MBloom s a -> ST s (Bloom a) Source #

Create an immutable Bloom filter from a mutable one without copying. The mutable filter must not be modified afterwards. For a safer creation interface, use freeze or create.

Low level variants

data Hashes a Source #

A small family of hashes, for probing bits in a (blocked) bloom filter.

hashes :: Hashable a => a -> Hashes a Source #

insertHashes :: forall s a. MBloom s a -> Hashes a -> ST s () Source #

elemHashes :: Bloom a -> Hashes a -> Bool Source #

Query an immutable Bloom filter for membership using already constructed Hash value.

Prefetching

prefetchInsert :: MBloom s a -> Hashes a -> ST s () Source #

prefetchElem :: Bloom a -> Hashes a -> ST s () Source #