lsm-tree-0.1.0.0: Log-structured merge-trees
Safe HaskellSafe-Inferred
LanguageGHC2021

Database.LSMTree.Internal.IncomingRun

Synopsis

Documentation

data IncomingRun m h Source #

An incoming run is either a single run, or a merge.

Credits and credit tracking

With scheduled merges, each update (e.g., insert) on a table contributes to the progression of ongoing merges in the levels structure. This ensures that merges are finished in time before a new merge has to be started. The points in the evolution of the levels structure where new merges are started are known: a flush of a full write buffer will create a new run on the first level, and after sufficient flushes (e.g., 4) we will start at least one new merge on the second level. This may cascade down to lower levels depending on how full the levels are. As such, we have a well-defined measure to determine when merges should be finished: it only depends on the maximum size of the write buffer!

The simplest solution to making sure merges are done in time is to step them to completion immediately when started. This does not, however, spread out work over time nicely. Instead, we schedule merge work based on how many updates are made on the table, taking care to ensure that the merge is finished just in time before the next flush comes around, and not too early.

The progression is tracked using nominal credits. Each individual update contributes a single credit to each level, since each level contains precisely one ongoing merge. Contributing a credit does not, however, translate directly to performing one unit of merging work:

  • The amount of work to do for one credit is adjusted depending on the actual size of the merge we are doing. Last-level merges, for example, can have larger inputs, and therefore we have to do a little more work for each credit. Or input runs involved in a merge can be less than maximal size for the level, and so there may be less merging work to do. As such, we scale NominalCredits to MergeCredits, and then supply the MergeCredits to the MergingRun.
  • Supplying MergeCredits to a MergingRun does not necessarily directly translate into performing merging work. Merge credits are accumulated until they go over a threshold, after which a batch of merge work will be performed. Configuring this threshold should allow a good balance between spreading out I/O and achieving good (concurrent) performance.

Merging runs can be shared across tables, which means that multiple threads can contribute to the same merge concurrently. Incoming runs however are not shared between tables. As such the tracking of NominalCredits does not need to use any concurrency precautions.

newtype NominalDebt Source #

Total merge debt to complete the merge in an incoming run.

This corresponds to the number (worst case, minimum number) of update operations inserted into the table, before we will expect the merge to complete.

Constructors

NominalDebt Int 

newtype NominalCredits Source #

Merge credits that get supplied to a table's levels.

This corresponds to the number of update operations inserted into the table.

Constructors

NominalCredits Int 

Instances

Instances details
NFData NominalCredits Source # 
Instance details

Defined in Database.LSMTree.Internal.IncomingRun

Methods

rnf :: NominalCredits -> () #

Eq NominalCredits Source # 
Instance details

Defined in Database.LSMTree.Internal.IncomingRun

DecodeVersioned NominalCredits Source # 
Instance details

Defined in Database.LSMTree.Internal.Snapshot.Codec

Encode NominalCredits Source # 
Instance details

Defined in Database.LSMTree.Internal.Snapshot.Codec

Prim NominalCredits Source # 
Instance details

Defined in Database.LSMTree.Internal.IncomingRun

supplyCreditsIncomingRun :: (MonadSTM m, MonadST m, MonadMVar m, MonadMask m) => TableConfig -> LevelNo -> IncomingRun m h -> NominalCredits -> m () Source #

Supply a given number of nominal credits to the merge in an incoming run. This is a relative addition of credits, not a new absolute total value.

immediatelyCompleteIncomingRun :: (MonadSTM m, MonadST m, MonadMVar m, MonadMask m) => TableConfig -> LevelNo -> IncomingRun m h -> m (Ref (Run m h)) Source #

Supply enough credits to complete the merge now.