Source File
options.go
Belonging Package
github.com/dgraph-io/badger/v4
/** SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.* SPDX-License-Identifier: Apache-2.0*/package badgerimport ()// Note: If you add a new option X make sure you also add a WithX method on Options.// Options are params for creating DB object.//// This package provides DefaultOptions which contains options that should// work for most applications. Consider using that as a starting point before// customizing it for your own needs.//// Each option X is documented on the WithX method.type Options struct {testOnlyOptions// Required options.Dir stringValueDir string// Usually modified options.SyncWrites boolNumVersionsToKeep intReadOnly boolLogger LoggerCompression options.CompressionTypeInMemory boolMetricsEnabled bool// Sets the Stream.numGo fieldNumGoroutines int// Fine tuning options.MemTableSize int64BaseTableSize int64BaseLevelSize int64LevelSizeMultiplier intTableSizeMultiplier intMaxLevels intVLogPercentile float64ValueThreshold int64NumMemtables int// Changing BlockSize across DB runs will not break badger. The block size is// read from the block index stored at the end of the table.BlockSize intBloomFalsePositive float64BlockCacheSize int64IndexCacheSize int64NumLevelZeroTables intNumLevelZeroTablesStall intValueLogFileSize int64ValueLogMaxEntries uint32NumCompactors intCompactL0OnClose boolLmaxCompaction boolZSTDCompressionLevel int// When set, checksum will be validated for each entry read from the value log file.VerifyValueChecksum bool// Encryption related options.EncryptionKey []byte // encryption keyEncryptionKeyRotationDuration time.Duration // key rotation duration// BypassLockGuard will bypass the lock guard on badger. Bypassing lock// guard can cause data corruption if multiple badger instances are using// the same directory. Use this options with caution.BypassLockGuard bool// ChecksumVerificationMode decides when db should verify checksums for SSTable blocks.ChecksumVerificationMode options.ChecksumVerificationMode// DetectConflicts determines whether the transactions would be checked for// conflicts. The transactions can be processed at a higher rate when// conflict detection is disabled.DetectConflicts bool// NamespaceOffset specifies the offset from where the next 8 bytes contains the namespace.NamespaceOffset int// Magic version used by the application using badger to ensure that it doesn't open the DB// with incompatible data format.ExternalMagicVersion uint16// Transaction start and commit timestamps are managed by end-user.// This is only useful for databases built on top of Badger (like Dgraph).// Not recommended for most users.managedTxns bool// 4. Flags for testing purposes// ------------------------------maxBatchCount int64 // max entries in batchmaxBatchSize int64 // max batch size in bytesmaxValueThreshold float64}// DefaultOptions sets a list of recommended options for good performance.// Feel free to modify these to suit your needs with the WithX methods.func ( string) Options {return Options{Dir: ,ValueDir: ,MemTableSize: 64 << 20,BaseTableSize: 2 << 20,BaseLevelSize: 10 << 20,TableSizeMultiplier: 2,LevelSizeMultiplier: 10,MaxLevels: 7,NumGoroutines: 8,MetricsEnabled: true,NumCompactors: 4, // Run at least 2 compactors. Zero-th compactor prioritizes L0.NumLevelZeroTables: 5,NumLevelZeroTablesStall: 15,NumMemtables: 5,BloomFalsePositive: 0.01,BlockSize: 4 * 1024,SyncWrites: false,NumVersionsToKeep: 1,CompactL0OnClose: false,VerifyValueChecksum: false,Compression: options.Snappy,BlockCacheSize: 256 << 20,IndexCacheSize: 0,// The following benchmarks were done on a 4 KB block size (default block size). The// compression is ratio supposed to increase with increasing compression level but since the// input for compression algorithm is small (4 KB), we don't get significant benefit at// level 3.// NOTE: The benchmarks are with DataDog ZSTD that requires CGO. Hence, no longer valid.// no_compression-16 10 502848865 ns/op 165.46 MB/s -// zstd_compression/level_1-16 7 739037966 ns/op 112.58 MB/s 2.93// zstd_compression/level_3-16 7 756950250 ns/op 109.91 MB/s 2.72// zstd_compression/level_15-16 1 11135686219 ns/op 7.47 MB/s 4.38// Benchmark code can be found in table/builder_test.go fileZSTDCompressionLevel: 1,// (2^30 - 1)*2 when mmapping < 2^31 - 1, max int32.// -1 so 2*ValueLogFileSize won't overflow on 32-bit systems.ValueLogFileSize: 1<<30 - 1,ValueLogMaxEntries: 1000000,VLogPercentile: 0.0,ValueThreshold: maxValueThreshold,Logger: defaultLogger(INFO),EncryptionKey: []byte{},EncryptionKeyRotationDuration: 10 * 24 * time.Hour, // Default 10 days.DetectConflicts: true,NamespaceOffset: -1,}}func buildTableOptions( *DB) table.Options {:= .opt, := .registry.LatestDataKey()y.Check()return table.Options{ReadOnly: .ReadOnly,MetricsEnabled: .opt.MetricsEnabled,TableSize: uint64(.BaseTableSize),BlockSize: .BlockSize,BloomFalsePositive: .BloomFalsePositive,ChkMode: .ChecksumVerificationMode,Compression: .Compression,ZSTDCompressionLevel: .ZSTDCompressionLevel,BlockCache: .blockCache,IndexCache: .indexCache,AllocPool: .allocPool,DataKey: ,}}const (maxValueThreshold = (1 << 20) // 1 MB)// LSMOnlyOptions follows from DefaultOptions, but sets a higher ValueThreshold// so values would be collocated with the LSM tree, with value log largely acting// as a write-ahead log only. These options would reduce the disk usage of value// log, and make Badger act more like a typical LSM tree.func ( string) Options {// Let's not set any other options, because they can cause issues with the// size of key-value a user can pass to Badger. For e.g., if we set// ValueLogFileSize to 64MB, a user can't pass a value more than that.// Setting it to ValueLogMaxEntries to 1000, can generate too many files.// These options are better configured on a usage basis, than broadly here.// The ValueThreshold is the most important setting a user needs to do to// achieve a heavier usage of LSM tree.// NOTE: If a user does not want to set 64KB as the ValueThreshold because// of performance reasons, 1KB would be a good option too, allowing// values smaller than 1KB to be collocated with the keys in the LSM tree.return DefaultOptions().WithValueThreshold(maxValueThreshold /* 1 MB */)}// parseCompression returns badger.compressionType and compression level given compression string// of format compression-type:compression-levelfunc parseCompression( string) (options.CompressionType, int, error) {:= strings.Split(, ":"):= [0]:= 3var errorif len() == 2 {, = strconv.Atoi([1])y.Check()if <= 0 {return 0, 0,fmt.Errorf("ERROR: compression level(%v) must be greater than zero", )}} else if len() > 2 {return 0, 0, fmt.Errorf("ERROR: Invalid badger.compression argument")}switch {case "zstd":return options.ZSTD, , nilcase "snappy":return options.Snappy, 0, nilcase "none":return options.None, 0, nil}return 0, 0, fmt.Errorf("ERROR: compression type (%s) invalid", )}// generateSuperFlag generates an identical SuperFlag string from the provided Options.func generateSuperFlag( Options) string {:= "":= reflect.ValueOf(&).Elem():= .Type()for := 0; < .NumField(); ++ {if := .Field(); .CanInterface() {:= strings.ToLower(.Field().Name):= .Field().Kind()switch {case reflect.Bool:+= + "="+= fmt.Sprintf("%v; ", .Bool())case reflect.Int, reflect.Int64:+= + "="+= fmt.Sprintf("%v; ", .Int())case reflect.Uint32, reflect.Uint64:+= + "="+= fmt.Sprintf("%v; ", .Uint())case reflect.Float64:+= + "="+= fmt.Sprintf("%v; ", .Float())case reflect.String:+= + "="+= fmt.Sprintf("%v; ", .String())default:continue}}}return}// FromSuperFlag fills Options fields for each flag within the superflag. For// example, replacing the default Options.NumGoroutines://// options := FromSuperFlag("numgoroutines=4", DefaultOptions(""))//// It's important to note that if you pass an empty Options struct, FromSuperFlag// will not fill it with default values. FromSuperFlag only writes to the fields// present within the superflag string (case insensitive).//// It specially handles compression subflag.// Valid options are {none,snappy,zstd:<level>}// Example: compression=zstd:3;// Unsupported: Options.Logger, Options.EncryptionKeyfunc ( Options) ( string) Options {// currentOptions act as a default value for the options superflag.:= generateSuperFlag()+= "compression=;":= z.NewSuperFlag().MergeAndCheckDefault():= reflect.ValueOf(&).Elem():= .Type()for := 0; < .NumField(); ++ {// only iterate over exported fieldsif := .Field(); .CanInterface() {// z.SuperFlag stores keys as lowercase, keep everything case// insensitive:= strings.ToLower(.Field().Name)if == "compression" {// We will specially handle this later. Skip it here.continue}:= .Field().Kind()switch {case reflect.Bool:.SetBool(.GetBool())case reflect.Int, reflect.Int64:.SetInt(.GetInt64())case reflect.Uint32, reflect.Uint64:.SetUint(.GetUint64())case reflect.Float64:.SetFloat(.GetFloat64())case reflect.String:.SetString(.GetString())}}}// Only update the options for special flags that were present in the input superflag.:= z.NewSuperFlag()if .Has("compression") {, , := parseCompression(.GetString("compression"))switch {case nil:.Compression =.ZSTDCompressionLevel =default:= options.CompressionType(.GetUint32("compression"))y.AssertTruef( <= 2, "ERROR: Invalid format or compression type. Got: %s",.GetString("compression")).Compression =}}return}// WithDir returns a new Options value with Dir set to the given value.//// Dir is the path of the directory where key data will be stored in.// If it doesn't exist, Badger will try to create it for you.// This is set automatically to be the path given to `DefaultOptions`.func ( Options) ( string) Options {.Dir =return}// WithValueDir returns a new Options value with ValueDir set to the given value.//// ValueDir is the path of the directory where value data will be stored in.// If it doesn't exist, Badger will try to create it for you.// This is set automatically to be the path given to `DefaultOptions`.func ( Options) ( string) Options {.ValueDir =return}// WithSyncWrites returns a new Options value with SyncWrites set to the given value.//// Badger does all writes via mmap. So, all writes can survive process crashes or k8s environments// with SyncWrites set to false.//// When set to true, Badger would call an additional msync after writes to flush mmap buffer over to// disk to survive hard reboots. Most users of Badger should not need to do this.//// The default value of SyncWrites is false.func ( Options) ( bool) Options {.SyncWrites =return}// WithNumVersionsToKeep returns a new Options value with NumVersionsToKeep set to the given value.//// NumVersionsToKeep sets how many versions to keep per key at most.//// The default value of NumVersionsToKeep is 1.func ( Options) ( int) Options {.NumVersionsToKeep =return}// WithNumGoroutines sets the number of goroutines to be used in Stream.//// The default value of NumGoroutines is 8.func ( Options) ( int) Options {.NumGoroutines =return}// WithReadOnly returns a new Options value with ReadOnly set to the given value.//// When ReadOnly is true the DB will be opened on read-only mode.// Multiple processes can open the same Badger DB.// Note: if the DB being opened had crashed before and has vlog data to be replayed,// ReadOnly will cause Open to fail with an appropriate message.//// The default value of ReadOnly is false.func ( Options) ( bool) Options {.ReadOnly =return}// WithMetricsEnabled returns a new Options value with MetricsEnabled set to the given value.//// When MetricsEnabled is set to false, then the DB will be opened and no badger metrics// will be logged. Metrics are defined in metric.go file.//// This flag is useful for use cases like in Dgraph where we open temporary badger instances to// index data. In those cases we don't want badger metrics to be polluted with the noise from// those temporary instances.//// Default value is set to truefunc ( Options) ( bool) Options {.MetricsEnabled =return}// WithLogger returns a new Options value with Logger set to the given value.//// Logger provides a way to configure what logger each value of badger.DB uses.//// The default value of Logger writes to stderr using the log package from the Go standard library.func ( Options) ( Logger) Options {.Logger =return}// WithLoggingLevel returns a new Options value with logging level of the// default logger set to the given value.// LoggingLevel sets the level of logging. It should be one of DEBUG, INFO,// WARNING or ERROR levels.//// The default value of LoggingLevel is INFO.func ( Options) ( loggingLevel) Options {.Logger = defaultLogger()return}// WithBaseTableSize returns a new Options value with BaseTableSize set to the given value.//// BaseTableSize sets the maximum size in bytes for LSM table or file in the base level.//// The default value of BaseTableSize is 2MB.func ( Options) ( int64) Options {.BaseTableSize =return}// WithLevelSizeMultiplier returns a new Options value with LevelSizeMultiplier set to the given// value.//// LevelSizeMultiplier sets the ratio between the maximum sizes of contiguous levels in the LSM.// Once a level grows to be larger than this ratio allowed, the compaction process will be// triggered.//// The default value of LevelSizeMultiplier is 10.func ( Options) ( int) Options {.LevelSizeMultiplier =return}// WithMaxLevels returns a new Options value with MaxLevels set to the given value.//// Maximum number of levels of compaction allowed in the LSM.//// The default value of MaxLevels is 7.func ( Options) ( int) Options {.MaxLevels =return}// WithValueThreshold returns a new Options value with ValueThreshold set to the given value.//// ValueThreshold sets the threshold used to decide whether a value is stored directly in the LSM// tree or separately in the log value files.//// The default value of ValueThreshold is 1 MB, and LSMOnlyOptions sets it to maxValueThreshold// which is set to 1 MB too.func ( Options) ( int64) Options {.ValueThreshold =return}// WithVLogPercentile returns a new Options value with ValLogPercentile set to given value.//// VLogPercentile with 0.0 means no dynamic thresholding is enabled.// MinThreshold value will always act as the value threshold.//// VLogPercentile with value 0.99 means 99 percentile of value will be put in LSM tree// and only 1 percent in vlog. The value threshold will be dynamically updated within the range of// [ValueThreshold, Options.maxValueThreshold]//// # Say VLogPercentile with 1.0 means threshold will eventually set to Options.maxValueThreshold//// The default value of VLogPercentile is 0.0.func ( Options) ( float64) Options {.VLogPercentile =return}// WithNumMemtables returns a new Options value with NumMemtables set to the given value.//// NumMemtables sets the maximum number of tables to keep in memory before stalling.//// The default value of NumMemtables is 5.func ( Options) ( int) Options {.NumMemtables =return}// WithMemTableSize returns a new Options value with MemTableSize set to the given value.//// MemTableSize sets the maximum size in bytes for memtable table.//// The default value of MemTableSize is 64MB.func ( Options) ( int64) Options {.MemTableSize =return}// WithBloomFalsePositive returns a new Options value with BloomFalsePositive set// to the given value.//// BloomFalsePositive sets the false positive probability of the bloom filter in any SSTable.// Before reading a key from table, the bloom filter is checked for key existence.// BloomFalsePositive might impact read performance of DB. Lower BloomFalsePositive value might// consume more memory.//// The default value of BloomFalsePositive is 0.01.//// Setting this to 0 disables the bloom filter completely.func ( Options) ( float64) Options {.BloomFalsePositive =return}// WithBlockSize returns a new Options value with BlockSize set to the given value.//// BlockSize sets the size of any block in SSTable. SSTable is divided into multiple blocks// internally. Each block is compressed using prefix diff encoding.//// The default value of BlockSize is 4KB.func ( Options) ( int) Options {.BlockSize =return}// WithNumLevelZeroTables sets the maximum number of Level 0 tables before compaction starts.//// The default value of NumLevelZeroTables is 5.func ( Options) ( int) Options {.NumLevelZeroTables =return}// WithNumLevelZeroTablesStall sets the number of Level 0 tables that once reached causes the DB to// stall until compaction succeeds.//// The default value of NumLevelZeroTablesStall is 15.func ( Options) ( int) Options {.NumLevelZeroTablesStall =return}// WithBaseLevelSize sets the maximum size target for the base level.//// The default value is 10MB.func ( Options) ( int64) Options {.BaseLevelSize =return}// WithValueLogFileSize sets the maximum size of a single value log file.//// The default value of ValueLogFileSize is 1GB.func ( Options) ( int64) Options {.ValueLogFileSize =return}// WithValueLogMaxEntries sets the maximum number of entries a value log file// can hold approximately. A actual size limit of a value log file is the// minimum of ValueLogFileSize and ValueLogMaxEntries.//// The default value of ValueLogMaxEntries is one million (1000000).func ( Options) ( uint32) Options {.ValueLogMaxEntries =return}// WithNumCompactors sets the number of compaction workers to run concurrently. Setting this to// zero stops compactions, which could eventually cause writes to block forever.//// The default value of NumCompactors is 4. One is dedicated just for L0 and L1.func ( Options) ( int) Options {.NumCompactors =return}// WithCompactL0OnClose determines whether Level 0 should be compacted before closing the DB. This// ensures that both reads and writes are efficient when the DB is opened later.//// The default value of CompactL0OnClose is false.func ( Options) ( bool) Options {.CompactL0OnClose =return}// WithEncryptionKey is used to encrypt the data with AES. Type of AES is used based on the key// size. For example 16 bytes will use AES-128. 24 bytes will use AES-192. 32 bytes will// use AES-256.func ( Options) ( []byte) Options {.EncryptionKey =return}// WithEncryptionKeyRotationDuration returns new Options value with the duration set to// the given value.//// Key Registry will use this duration to create new keys. If the previous generated// key exceed the given duration. Then the key registry will create new key.// The default value is set to 10 days.func ( Options) ( time.Duration) Options {.EncryptionKeyRotationDuration =return}// WithCompression is used to enable or disable compression. When compression is enabled, every// block will be compressed using the specified algorithm. This option doesn't affect existing// tables. Only the newly created tables will be compressed.//// The default compression algorithm used is snappy. Compression is enabled by default.func ( Options) ( options.CompressionType) Options {.Compression =return}// WithVerifyValueChecksum is used to set VerifyValueChecksum. When VerifyValueChecksum is set to// true, checksum will be verified for every entry read from the value log. If the value is stored// in SST (value size less than value threshold) then the checksum validation will not be done.//// The default value of VerifyValueChecksum is False.func ( Options) ( bool) Options {.VerifyValueChecksum =return}// WithChecksumVerificationMode returns a new Options value with ChecksumVerificationMode set to// the given value.//// ChecksumVerificationMode indicates when the db should verify checksums for SSTable blocks.//// The default value of VerifyValueChecksum is options.NoVerification.func ( Options) ( options.ChecksumVerificationMode) Options {.ChecksumVerificationMode =return}// WithBlockCacheSize returns a new Options value with BlockCacheSize set to the given value.//// This value specifies how much data cache should hold in memory. A small size// of cache means lower memory consumption and lookups/iterations would take// longer. It is recommended to use a cache if you're using compression or encryption.// If compression and encryption both are disabled, adding a cache will lead to// unnecessary overhead which will affect the read performance. Setting size to// zero disables the cache altogether.//// Default value of BlockCacheSize is 256 MB.func ( Options) ( int64) Options {.BlockCacheSize =return}// WithInMemory returns a new Options value with Inmemory mode set to the given value.//// When badger is running in InMemory mode, everything is stored in memory. No value/sst files are// created. In case of a crash all data will be lost.func ( Options) ( bool) Options {.InMemory =return}// WithZSTDCompressionLevel returns a new Options value with ZSTDCompressionLevel set// to the given value.//// The ZSTD compression algorithm supports 20 compression levels. The higher the compression// level, the better is the compression ratio but lower is the performance. Lower levels// have better performance and higher levels have better compression ratios.// We recommend using level 1 ZSTD Compression Level. Any level higher than 1 seems to// deteriorate badger's performance.// The following benchmarks were done on a 4 KB block size (default block size). The compression is// ratio supposed to increase with increasing compression level but since the input for compression// algorithm is small (4 KB), we don't get significant benefit at level 3. It is advised to write// your own benchmarks before choosing a compression algorithm or level.//// NOTE: The benchmarks are with DataDog ZSTD that requires CGO. Hence, no longer valid.// no_compression-16 10 502848865 ns/op 165.46 MB/s -// zstd_compression/level_1-16 7 739037966 ns/op 112.58 MB/s 2.93// zstd_compression/level_3-16 7 756950250 ns/op 109.91 MB/s 2.72// zstd_compression/level_15-16 1 11135686219 ns/op 7.47 MB/s 4.38// Benchmark code can be found in table/builder_test.go filefunc ( Options) ( int) Options {.ZSTDCompressionLevel =return}// WithBypassLockGuard returns a new Options value with BypassLockGuard// set to the given value.//// When BypassLockGuard option is set, badger will not acquire a lock on the// directory. This could lead to data corruption if multiple badger instances// write to the same data directory. Use this option with caution.//// The default value of BypassLockGuard is false.func ( Options) ( bool) Options {.BypassLockGuard =return}// WithIndexCacheSize returns a new Options value with IndexCacheSize set to// the given value.//// This value specifies how much memory should be used by table indices. These// indices include the block offsets and the bloomfilters. Badger uses bloom// filters to speed up lookups. Each table has its own bloom// filter and each bloom filter is approximately of 5 MB.//// Zero value for IndexCacheSize means all the indices will be kept in// memory and the cache is disabled.//// The default value of IndexCacheSize is 0 which means all indices are kept in// memory.func ( Options) ( int64) Options {.IndexCacheSize =return}// WithDetectConflicts returns a new Options value with DetectConflicts set to the given value.//// Detect conflicts options determines if the transactions would be checked for// conflicts before committing them. When this option is set to false// (detectConflicts=false) badger can process transactions at a higher rate.// Setting this options to false might be useful when the user application// deals with conflict detection and resolution.//// The default value of Detect conflicts is True.func ( Options) ( bool) Options {.DetectConflicts =return}// WithNamespaceOffset returns a new Options value with NamespaceOffset set to the given value. DB// will expect the namespace in each key at the 8 bytes starting from NamespaceOffset. A negative// value means that namespace is not stored in the key.//// The default value for NamespaceOffset is -1.func ( Options) ( int) Options {.NamespaceOffset =return}// WithExternalMagic returns a new Options value with ExternalMagicVersion set to the given value.// The DB would fail to start if either the internal or the external magic number fails validated.func ( Options) ( uint16) Options {.ExternalMagicVersion =return}func ( Options) () int {var int// opt.SyncWrites would be using msync to sync. All writes go through mmap.if .ReadOnly {|= os.O_RDONLY} else {|= os.O_RDWR}return}
![]() |
The pages are generated with Golds v0.8.4. (GOOS=linux GOARCH=amd64) Golds is a Go 101 project developed by Tapir Liu. PR and bug reports are welcome and can be submitted to the issue list. Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds. |