Analyzer analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
{
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
});
For more examples, see the [Common](../analysis-common/overview.html): Analyzers for indexing content in different languages and domains. [ICU](../icu/Lucene.Net.Analysis.Icu.html): Exposes functionality from ICU to Apache Lucene. [Kuromoji](../analysis-kuromoji/Lucene.Net.Analysis.Ja.html): Morphological analyzer for Japanese text. [Morfologik](../analysis-morfologik/Lucene.Net.Analysis.Morfologik.html): Dictionary-driven lemmatization for the Polish language. [OpenNLP](../analysis-opennlp/Lucene.Net.Analysis.OpenNlp.html): Analysis integration with Apache OpenNLP. [Phonetic](../analysis-phonetic/Lucene.Net.Analysis.Phonetic.html): Analysis for indexing phonetic signatures (for sounds-alike search). [Smart Chinese](../analysis-smartcn/Lucene.Net.Analysis.Cn.Smart.html): Analyzer for Simplified Chinese, which indexes words. [Stempel](../analysis-stempel/Lucene.Net.Analysis.Stempel.html): Algorithmic Stemmer for the Polish Language.
var analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
{
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
});
var analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
{
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
}, reuseStrategy);
var analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
{
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
}, initReader: (fieldName, reader) =>
{
return new HTMLStripCharFilter(reader);
});
var analyzer = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
{
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
}, initReader: (fieldName, reader) =>
{
return new HTMLStripCharFilter(reader);
}, reuseStrategy);
IndexableFieldType fieldType = new IndexableFieldType(TextField.TYPE_NOT_STORED)
{
OmitNorms = true,
IndexOptions = IndexOptions.DOCS_ONLY
};
Field field = new Field(name, new NumericTokenStream(precisionStep).SetInt32Value(value), fieldType);
document.Add(field);
NumericTokenStream stream = new NumericTokenStream(precisionStep);
IndexableFieldType fieldType = new IndexableFieldType(TextField.TYPE_NOT_STORED)
{
OmitNorms = true,
IndexOptions = IndexOptions.DOCS_ONLY
};
Field field = new Field(name, stream, fieldType);
Document document = new Document();
document.Add(field);
for(all documents)
{
stream.SetInt32Value(value)
writer.AddDocument(document);
}
new Field(name, new NumericTokenStream(precisionStep).SetInt64Value(value)) new Field(name, new NumericTokenStream(precisionStep).SetInt32Value(value)) new Field(name, new NumericTokenStream(precisionStep).SetDoubleValue(value)) new Field(name, new NumericTokenStream(precisionStep).SetSingleValue(value)) Copying text from a string (type is reset to if not specified): return reusableToken.Reinit(string, startOffset, endOffset[, type]);Copying some text from a string (type is reset to if not specified): return reusableToken.Reinit(string, 0, string.Length, startOffset, endOffset[, type]);Copying text from char[] buffer (type is reset to if not specified): return reusableToken.Reinit(buffer, 0, buffer.Length, startOffset, endOffset[, type]);Copying some text from a char[] buffer (type is reset to if not specified): return reusableToken.Reinit(buffer, start, end - start, startOffset, endOffset[, type]);Copying from one one to another (type is reset to if not specified): return reusableToken.Reinit(source.Buffer, 0, source.Length, source.StartOffset, source.EndOffset[, source.Type]);
initializes all of the fields to default values. this was changed in contrast to Lucene 2.4, but should affect no one. Because s can be chained, one cannot assume that the 's current type is correct. The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them. When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.
EndOffset -
obj.SetLength(30).Append("hey you");
EndOffset - Set it to zero to put multiple terms in the same position. this is useful if, e.g., a word has multiple stems. Searches for phrases including either stem will match. In this case, all but the first stem's increment should be set to zero: the increment of the first instance should be one. Repeating a token with an increment of zero can also be used to boost the scores of matches on that token. Set it to values greater than one to inhibit exact phrase matches. If, for example, one does not want phrases to match across removed stop words, then one could build a stop word filter that removes stop words and also sets the increment to the number of stop words removed before each non-stop word. Then exact phrase queries will only match when the terms occur with no intervening stop words.
TermToBytesRefAttribute termAtt = tokenStream.GetAttribute<TermToBytesRefAttribute>;
BytesRef bytes = termAtt.BytesRef;
while (tokenStream.IncrementToken()
{
// you must call termAtt.FillBytesRef() before doing something with the bytes.
// this encodes the term value (internally it might be a char[], etc) into the bytes.
int hashCode = termAtt.FillBytesRef();
if (IsInteresting(bytes))
{
// because the bytes are reused by the attribute (like ICharTermAttribute's char[] buffer),
// you should make a copy if you need persistent access to the bytes, otherwise they will
// be rewritten across calls to IncrementToken()
DoSomethingWith(new BytesRef(bytes));
}
}
...
@lucene.experimental this is a very expert API, please use
, a whose input is a ; and , a whose input is another .
Instantiation of / s which add/get attributes to/from the . The consumer calls . The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access. The consumer calls until it returns false consuming the attributes after each call. The consumer calls so that any end-of-stream operations can be performed. The consumer calls to release any resource when finished using the .
.tim: Term Dictionary .tip: Term Index
Term Dictionary
TermsDict (.tim) --> Header, PostingsHeader, NodeBlockNumBlocks, FieldSummary, DirOffset, Footer NodeBlock --> (OuterNode | InnerNode) OuterNode --> EntryCount, SuffixLength, ByteSuffixLength, StatsLength, < TermStats >EntryCount, MetaLength, <TermMetadata>EntryCount InnerNode --> EntryCount, SuffixLength[,Sub?], ByteSuffixLength, StatsLength, < TermStats ? >EntryCount, MetaLength, <TermMetadata ? >EntryCount TermStats --> DocFreq, TotalTermFreq FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumTotalTermFreq?, SumDocFreq, DocCount>NumFields Header --> CodecHeader ( DirOffset --> Uint64 ( ) EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount --> VInt ( _ TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> VLong ( ) Footer --> CodecFooter ( )
Header is a CodecHeader ( ) storing the version information for the BlockTree implementation. DirOffset is a pointer to the FieldSummary section. DocFreq is the count of documents which contain the term. TotalTermFreq is the total number of occurrences of the term. this is encoded as the difference between the total number of occurrences and the DocFreq. FieldNumber is the fields number from . (.fnm) NumTerms is the number of unique terms for the field. RootCode points to the root block for the field. SumDocFreq is the total number of postings, the number of term-document pairs across the entire field. DocCount is the number of documents that have at least one posting for this field. PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files). For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding and TermMetadata are omitted
Term Index
TermsIndex (.tip) --> Header, FSTIndexNumFields <IndexStartFP>NumFields, DirOffset, Footer Header --> CodecHeader ( ) DirOffset --> Uint64 ( )IndexStartFP --> VLong ( )FSTIndex --> Footer --> CodecFooter (
The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST. DirOffset is a pointer to the start of the IndexStartFPs for all fields It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.
Subclass this class. Subclass , override the method, and add the line base.ScanForCodecs(typeof(YourCodec).Assembly) . If you have any codec classes in your assembly that are not meant for reading, you can add theto them so they are ignored by the scan. set the new by calling at application startup.
Magic --> Uint32 ( ). this identifies the start of the header. It is always . CodecName --> String ( ). this is a string to identify this file. Version --> Uint32 ( ). Records the version of the file.
Magic --> Uint32 ( ). this identifies the start of the footer. It is always . AlgorithmID --> Uint32 ( ). this indicates the checksum algorithm used. Currently this is always 0, for zlib-crc32. Checksum --> Uint32 ( ). The actual checksum value for all previous bytes in the stream, including the bytes from Magic and AlgorithmID.
DocValuesConsumer is created by or . , , or are called for each Numeric, Binary, or Sorted docvalues field. The API is a "pull" rather than "push", and the implementation is free to iterate over the values multiple times ( ). After all fields are added, the consumer is d.
Subclass this class. Subclass , override the method, and add the line base.ScanForDocValuesFormats(typeof(YourDocValuesFormat).Assembly) . If you have any format classes in your assembly that are not meant for reading, you can add theto them so they are ignored by the scan. Set the new by calling at application startup.
FieldsConsumer is created by . For each field, is called, returning a for the field. After all fields are added, the consumer is d.
public sealed class CustomCodec : FilterCodec
{
public CustomCodec()
: base("CustomCodec", new Lucene46Codec())
{
}
public override LiveDocsFormat LiveDocsFormat
{
get { return new CustomLiveDocsFormat(); }
}
}
a count() method, which efficiently computes the number of one bits; optimized read from and write to disk; inlinable get() method; store and load, as bit set or d-gaps, depending on sparseness;
.dv.cfs : compound container () .dv.cfe : compound entries ()
<segment>_<fieldNumber>.dat : data values<segment>_<fieldNumber>.idx : index into the .dat for DEREF types
.dat --> Header, PackedType, MinValue, DefaultValue, PackedStream .dat --> Header, ValueSize, Byte ( ) maxdoc .dat --> Header, ValueSize, Short ( ) maxdoc .dat --> Header, ValueSize, Int32 ( ) maxdoc .dat --> Header, ValueSize, Int64 ( ) maxdoc .dat --> Header, ValueSize, Float32maxdoc .dat --> Header, ValueSize, Float64maxdoc .dat --> Header, ValueSize, (Byte ( ) * ValueSize)maxdoc .idx --> Header, TotalBytes, Addresses .dat --> Header, (Byte ( ) * variable ValueSize)maxdoc .idx --> Header, NumValues, Addresses .dat --> Header, ValueSize, (Byte ( ) * ValueSize)NumValues .idx --> Header, TotalVarBytes, Addresses .dat --> Header, (LengthPrefix + Byte ( ) * variable ValueSize)NumValues .idx --> Header, NumValues, Ordinals .dat --> Header, ValueSize, (Byte ( ) * ValueSize)NumValues .idx --> Header, TotalVarBytes, Addresses, Ordinals .dat --> Header, (Byte ( ) * variable ValueSize)NumValues
Header --> CodecHeader ( ) PackedType --> Byte ( ) MaxAddress, MinValue, DefaultValue --> Int64 ( ) PackedStream, Addresses, Ordinals --> ValueSize, NumValues --> Int32 ( ) Float32 --> 32-bit float encoded with then written as Int32 ( ) Float64 --> 64-bit float encoded with then written as Int64 ( ) TotalBytes --> VLong ( ) TotalVarBytes --> Int64 ( ) LengthPrefix --> Length of the data value as VInt ( ) (maximum of 2 bytes)
PackedType is a 0 when compressed, 1 when the stream is written as 64-bit integers. Addresses stores pointers to the actual byte location (indexed by docid). In the VAR_STRAIGHT case, each entry can have a different length, so to determine the length, docid+1 is retrieved. A sentinel address is written at the end for the VAR_STRAIGHT case, so the Addresses stream contains maxdoc+1 indices. For the deduplicated VAR_DEREF case, each length is encoded as a prefix to the data itself as a VInt ( ) (maximum of 2 bytes). Ordinals stores the term ID in sorted order (indexed by docid). In the FIXED_SORTED case, the address into the .dat can be computed from the ordinal as Header+ValueSize+(ordinal*ValueSize) because the byte length is fixed. In the VAR_SORTED case, there is double indirection (docid -> ordinal -> address), but an additional sentinel ordinal+address is always written (so there are NumValues+1 ordinals). To determine the length, ord+1's address is looked up as well.in contrast to other straight variants uses a .idx file to improve lookup perfromance. In contrast toit doesn't apply deduplication of the document values.
Binary doc values can be at most in length.
Header --> CodecHeader ( ) FieldsCount --> VInt ( ) FieldName --> String ( ) FieldBits, DocValuesBits --> Byte ( ) FieldNumber --> VInt ( ) Attributes --> IDictionary<String,String> ( )
FieldsCount: the number of fields in this file. FieldName: name of the field as a UTF-8 String. FieldNumber: the field's number. Note that unlike previous versions of Lucene, the fields are not numbered implicitly by their order in the file, instead explicitly. FieldBits: a byte containing field options. The low-order bit is one for indexed fields, and zero for non-indexed fields. The second lowest-order bit is one for fields that have term vectors stored, and zero for fields without term vectors. If the third lowest order-bit is set (0x4), offsets are stored into the postings list in addition to positions. Fourth bit is unused. If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field. If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field. If the seventh lowest-order bit is set (0x40), term frequencies and positions omitted for the indexed field. If the eighth lowest-order bit is set (0x80), positions are omitted for the indexed field.
DocValuesBits: a byte containing per-document value types. The type recorded as two four-bit integers, with the high-order bits representing norms options, and the low-order bits representingoptions. Each four-bit integer can be decoded as such: 0: no DocValues for this field. 1: variable-width signed integers. ( ) 2: 32-bit floating point values. ( ) 3: 64-bit floating point values. ( ) 4: fixed-length byte array values. ( ) 5: fixed-length dereferenced byte array values. ( ) 6: variable-length byte array values. ( ) 7: variable-length dereferenced byte array values. ( ) 8: 16-bit signed integers. ( ) 9: 32-bit signed integers. ( ) 10: 64-bit signed integers. ( ) 11: 8-bit signed integers. ( ) 12: fixed-length sorted byte array values. ( ) 13: variable-length sorted byte array values. ( )
Attributes: a key-value map of codec-private attributes.
Format,ByteSize,BitCount --> Uint32 ( ) Bits --> < Byte ( ) > ByteCount DGaps --> <DGap,NonOnesByte> NonzeroBytesCount DGap --> VInt ( ) NonOnesByte --> Byte( ) Header --> CodecHeader ( )
.nrm.cfs : compound container () .nrm.cfe : compound entries ()
.tim: Term Dictionary .tip: Term Index .frq: Frequencies .prx: Positions
Term Dictionary
Postings Metadata --> Header, SkipInterval, MaxSkipLevels, SkipMinimum Term Metadata --> FreqDelta, SkipDelta?, ProxDelta? Header --> CodecHeader ( ) SkipInterval,MaxSkipLevels,SkipMinimum --> Uint32 ( ) SkipDelta,FreqDelta,ProxDelta --> VLong ( )
Header is a CodecHeader ( ) storing the version information for the postings. SkipInterval is the fraction of TermDocs stored in skip tables. It is used to accelerate . Larger values result in smaller indexes, greater acceleration, but fewer accelerable cases, while smaller values result in bigger indexes, less acceleration (in case of a small value for MaxSkipLevels) and more accelerable cases. MaxSkipLevels is the max. number of skip levels stored for each term in the .frq file. A low value results in smaller indexes but less acceleration, a larger value results in slightly larger indexes but greater acceleration. See format of .frq file for more information about skip levels. SkipMinimum is the minimum document frequency a term must have in order to write any skip data at all. FreqDelta determines the position of this term's TermFreqs within the .frq file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the block). ProxDelta determines the position of this term's TermPositions within the .prx file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the block. For fields that omit position data, this will be 0 since prox information is not stored. SkipDelta determines the position of this term's SkipData within the .frq file. In particular, it is the number of bytes after TermFreqs that the SkipData starts. In other words, it is the length of the TermFreq data. SkipDelta is only stored if DocFreq is not smaller than SkipMinimum.
Term Index
Frequencies
FreqFile (.frq) --> Header, <TermFreqs, SkipData?> TermCount Header --> CodecHeader ( ) TermFreqs --> <TermFreq> DocFreq TermFreq --> DocDelta[, Freq?] SkipData --> <<SkipLevelLength, SkipLevel> NumSkipLevels-1, SkipLevel> <SkipDatum> SkipLevel --> <SkipDatum> DocFreq/(SkipInterval^(Level + 1)) SkipDatum --> DocSkip,PayloadLength?,OffsetLength?,FreqSkip,ProxSkip,SkipChildLevelPointer? DocDelta,Freq,DocSkip,PayloadLength,OffsetLength,FreqSkip,ProxSkip --> VInt ( ) SkipChildLevelPointer --> VLong ( )
Positions
ProxFile (.prx) --> Header, <TermPositions> TermCount Header --> CodecHeader ( ) TermPositions --> <Positions> DocFreq Positions --> <PositionDelta,PayloadLength?,OffsetDelta?,OffsetLength?,PayloadData?> Freq PositionDelta,OffsetDelta,OffsetLength,PayloadLength --> VInt ( ) PayloadData --> byte ( ) PayloadLength
.si: Header, SegVersion, SegSize, IsCompoundFile, Diagnostics, Attributes, Files
Header --> CodecHeader ( ) SegSize --> Int32 ( ) SegVersion --> String ( ) Files --> ISet<String> ( ) Diagnostics, Attributes --> IDictionary<String,String> ( ) IsCompoundFile --> Int8 ( )
SegVersion is the code version that created the segment. SegSize is the number of documents contained in the segment index. IsCompoundFile records whether the segment is written as a compound file or not. If this is -1, the segment is not a compound file. If it is 1, the segment is a compound file. Checksum contains the CRC32 checksum of all bytes in the segments_N file up until the checksum. This is used to verify integrity of the file on opening the index. The Diagnostics Map is privately written by , as a debugging aid, for each segment it creates. It includes metadata like the current Lucene version, OS, .NET/Java version, why the segment was created (merge, flush, addIndexes), etc. Attributes: a key-value map of codec-private attributes. Files is a list of files referred to by this segment.
The field index, or .fdx file.This is used to find the location within the field data file of the fields of a particular document. Because it contains fixed-length data, this file may be easily randomly accessed. The position of document n 's field data is the Uint64 ( ) at n*8 in this file. This contains, for each document, a pointer to its field data, as follows: FieldIndex (.fdx) --> <Header>, <FieldValuesPosition> SegSize Header --> CodecHeader ( ) FieldValuesPosition --> Uint64 ( )
The field data, or .fdt file.This contains the stored fields of each document, as follows: FieldData (.fdt) --> <Header>, <DocFieldData> SegSize Header --> CodecHeader ( ) DocFieldData --> FieldCount, <FieldNum, Bits, Value> FieldCount FieldCount --> VInt ( ) FieldNum --> VInt ( ) Bits --> Byte ( ) low order bit reserved. second bit is one for fields containing binary data third bit reserved. 4th to 6th bit (mask: 0x7<<3) define the type of a numeric field: all bits in mask are cleared if no numeric field at all 1<<3: Value is Int 2<<3: Value is Long 3<<3: Value is Int as Float (as of 4<<3: Value is Long as Double (as of
Value --> String | BinaryValue | Int | Long (depending on Bits) BinaryValue --> ValueSize, < Byte ( ) >^ValueSize ValueSize --> VInt ( )
The Document Index or .tvx file. For each document, this stores the offset into the document data (.tvd) and field data (.tvf) files. DocumentIndex (.tvx) --> Header,<DocumentPosition,FieldPosition> NumDocs Header --> CodecHeader ( ) DocumentPosition --> UInt64 ( ) (offset in the .tvd file) FieldPosition --> UInt64 ( ) (offset in the .tvf file)
The Document or .tvd file. This contains, for each document, the number of fields, a list of the fields with term vector info and finally a list of pointers to the field information in the .tvf (Term Vector Fields) file. The .tvd file is used to map out the fields that have term vectors stored and where the field information is in the .tvf file. Document (.tvd) --> Header,<NumFields, FieldNums, FieldPositions> NumDocs Header --> CodecHeader ( ) NumFields --> VInt ( ) FieldNums --> <FieldNumDelta> NumFields FieldNumDelta --> VInt ( ) FieldPositions --> <FieldPositionDelta> NumFields-1 FieldPositionDelta --> VLong ( )
The Field or .tvf file. This file contains, for each field that has a term vector stored, a list of the terms, their frequencies and, optionally, position, offset, and payload information. Field (.tvf) --> Header,<NumTerms, Flags, TermFreqs> NumFields Header --> CodecHeader ( ) NumTerms --> VInt ( ) Flags --> Byte ( ) TermFreqs --> <TermText, TermFreq, Positions?, PayloadData?, Offsets?> NumTerms TermText --> <PrefixLength, Suffix> PrefixLength --> VInt ( ) Suffix --> String ( ) TermFreq --> VInt ( ) Positions --> <PositionDelta PayloadLength?>TermFreq PositionDelta --> VInt ( ) PayloadLength --> VInt ( ) PayloadData --> Byte ( ) NumPayloadBytes Offsets --> <VInt ( ), VInt ( ) >TermFreq
Notes: Flags byte stores whether this term vector has position, offset, payload. information stored. Term byte prefixes are shared. The PrefixLength is the number of initial bytes from the previous term which must be pre-pended to a term's suffix in order to form the term's bytes. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y". PositionDelta is, if payloads are disabled for the term's field, the difference between the position of the current occurrence in the document and the previous occurrence (or zero, if this is the first occurrence in this document). If payloads are enabled for the term's field, then PositionDelta/2 is the difference between the current and the previous position. If payloads are enabled and PositionDelta is odd, then PayloadLength is stored, indicating the length of the payload at the current term position. PayloadData is metadata associated with a term position. If PayloadLength is stored at the current position, then it indicates the length of this payload. If PayloadLength is not stored, then this payload has the same length as the payload at the previous position. PayloadData encodes the concatenated bytes for all of a terms occurrences. Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.
Packed Blocks and VInt Blocks: In packed blocks, integers are encoded with the same bit width packed format ( ): the block size (i.e. number of integers inside block) is fixed (currently 128). Additionally blocks that are all the same value are encoded in an optimized way. In VInt blocks, integers are encoded as VInt ( ): the block size is variable. Block structure: When the postings are long enough, Lucene41PostingsFormat will try to encode most integer data as a packed block. Take a term with 259 documents as an example, the first 256 document ids are encoded as two packed blocks, while the remaining 3 are encoded as one VInt block. Different kinds of data are always encoded separately into different packed blocks, but may possibly be interleaved into the same VInt block. This strategy is applied to pairs: <document number, frequency>, <position, payload length>, <position, offset start, offset length>, and <position, payload length, offsetstart, offset length>. Skipdata settings: The structure of skip table is quite similar to previous version of Lucene. Skip interval is the same as block size, and each skip entry points to the beginning of each block. However, for the first block, skip data is omitted. Positions, Payloads, and Offsets: A position is an integer indicating where the term occurs within one document. A payload is a blob of metadata associated with current position. An offset is a pair of integers indicating the tokenized start/end offsets for given term in current position: it is essentially a specialized payload. When payloads and offsets are not omitted, numPositions==numPayloads==numOffsets (assuming a null payload contributes one count). As mentioned in block structure, it is possible to encode these three either combined or separately. In all cases, payloads and offsets are stored together. When encoded as a packed block, position data is separated out as .pos, while payloads and offsets are encoded in .pay (payload metadata will also be stored directly in .pay). When encoded as VInt blocks, all these three are stored interleaved into the .pos (so is payload metadata). With this strategy, the majority of payload and offset data will be outside .pos file. So for queries that require only position data, running on a full index with payloads and offsets, this reduces disk pre-fetches.
.tim : Term Dictionary.tip : Term Index.doc : Frequencies and Skip Data.pos : Positions.pay : Payloads and Offsets
-
Term Dictionary
The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and pointers to the frequencies, positions, payload and skip data in the .doc, .pos, and .pay files. See for more details on the format. NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the PostingsHeader and TermMetadata sections described here: PostingsHeader --> Header, PackedBlockSize TermMetadata --> (DocFPDelta|SingletonDocID), PosFPDelta?, PosVIntBlockFPDelta?, PayFPDelta?, SkipFPDelta? Header, --> CodecHeader ( ) PackedBlockSize, SingletonDocID --> VInt ( ) DocFPDelta, PosFPDelta, PayFPDelta, PosVIntBlockFPDelta, SkipFPDelta --> VLong ( ) Footer --> CodecFooter ( )
Notes: Header is a CodecHeader ( ) storing the version information for the postings. PackedBlockSize is the fixed block size for packed blocks. In packed block, bit width is determined by the largest integer. Smaller block size result in smaller variance among width of integers hence smaller indexes. Larger block size result in more efficient bulk i/o hence better acceleration. This value should always be a multiple of 64, currently fixed as 128 as a tradeoff. It is also the skip interval used to accelerate . DocFPDelta determines the position of this term's TermFreqs within the .doc file. In particular, it is the difference of file offset between this term's data and previous term's data (or zero, for the first term in the block).On disk it is stored as the difference from previous value in sequence. PosFPDelta determines the position of this term's TermPositions within the .pos file. While PayFPDelta determines the position of this term's <TermPayloads, TermOffsets?> within the .pay file. Similar to DocFPDelta, it is the difference between two file positions (or neglected, for fields that omit payloads and offsets). PosVIntBlockFPDelta determines the position of this term's last TermPosition in last pos packed block within the .pos file. It is synonym for PayVIntBlockFPDelta or OffsetVIntBlockFPDelta. This is actually used to indicate whether it is necessary to load following payloads and offsets from .pos instead of .pay. Every time a new block of positions are to be loaded, the PostingsReader will use this value to check whether current block is packed format or VInt. When packed format, payloads and offsets are fetched from .pay, otherwise from .pos. (this value is neglected when total number of positions i.e. totalTermFreq is less or equal to PackedBlockSize). SkipFPDelta determines the position of this term's SkipData within the .doc file. In particular, it is the length of the TermFreq data. SkipDelta is only stored if DocFreq is not smaller than SkipMinimum (i.e. 128 in Lucene41PostingsFormat). SingletonDocID is an optimization when a term only appears in one document. In this case, instead of writing a file pointer to the .doc file (DocFPDelta), and then a VIntBlock at that location, the single document ID is written to the term dictionary.
-
Term Index
The .tip file contains an index into the term dictionary, so that it can be accessed randomly. See for more details on the format.
-
Frequencies and Skip Data
The .doc file contains the lists of documents which contain each term, along with the frequency of the term in that document (except when frequencies are omitted: ). It also saves skip data to the beginning of each packed or VInt block, when the length of document list is larger than packed block size. docFile(.doc) --> Header, <TermFreqs, SkipData?>TermCount, Footer Header --> CodecHeader ( ) TermFreqs --> <PackedBlock> PackedDocBlockNum, VIntBlock? PackedBlock --> PackedDocDeltaBlock, PackedFreqBlock? VIntBlock --> <DocDelta[, Freq?]>DocFreq-PackedBlockSize*PackedDocBlockNum SkipData --> <<SkipLevelLength, SkipLevel> NumSkipLevels-1, SkipLevel>, SkipDatum? SkipLevel --> <SkipDatum> TrimmedDocFreq/(PackedBlockSize^(Level + 1)) SkipDatum --> DocSkip, DocFPSkip, <PosFPSkip, PosBlockOffset, PayLength?, PayFPSkip?>?, SkipChildLevelPointer? PackedDocDeltaBlock, PackedFreqBlock --> PackedInts ( ) DocDelta, Freq, DocSkip, DocFPSkip, PosFPSkip, PosBlockOffset, PayByteUpto, PayFPSkip --> VInt ( ) SkipChildLevelPointer --> VLong ( ) Footer --> CodecFooter ( )
Notes: PackedDocDeltaBlock is theoretically generated from two steps: Calculate the difference between each document number and previous one, and get a d-gaps list (for the first document, use absolute value); For those d-gaps from first one to PackedDocBlockNum*PackedBlockSizeth, separately encode as packed blocks.
VIntBlock stores remaining d-gaps (along with frequencies when possible) with a format that encodes DocDelta and Freq: DocDelta: if frequencies are indexed, this determines both the document number and the frequency. In particular, DocDelta/2 is the difference between this document number and the previous document number (or zero when this is the first document in a TermFreqs). When DocDelta is odd, the frequency is one. When DocDelta is even, the frequency is read as another VInt. If frequencies are omitted, DocDelta contains the gap (not multiplied by 2) between document numbers and no frequency information is stored. For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven, with frequencies indexed, would be the following sequence of VInts: 15, 8, 3 If frequencies were omitted ( ) it would be this sequence of VInts instead: 7,4 PackedDocBlockNum is the number of packed blocks for current term's docids or frequencies. In particular, PackedDocBlockNum = floor(DocFreq/PackedBlockSize) TrimmedDocFreq = DocFreq % PackedBlockSize == 0 ? DocFreq - 1 : DocFreq. We use this trick since the definition of skip entry is a little different from base interface. In , skip data is assumed to be saved for skipIntervalth, 2*skipIntervalth ... posting in the list. However, in Lucene41PostingsFormat, the skip data is saved for skipInterval+1th, 2*skipInterval+1th ... posting (skipInterval==PackedBlockSize in this case). When DocFreq is multiple of PackedBlockSize, MultiLevelSkipListWriter will expect one more skip data than Lucene41SkipWriter. SkipDatum is the metadata of one skip entry. For the first block (no matter packed or VInt), it is omitted. DocSkip records the document number of every PackedBlockSizeth document number in the postings (i.e. last document number in each packed block). On disk it is stored as the difference from previous value in the sequence. DocFPSkip records the file offsets of each block (excluding )posting at PackedBlockSize+1th, 2*PackedBlockSize+1th ... , in DocFile. The file offsets are relative to the start of current term's TermFreqs. On disk it is also stored as the difference from previous SkipDatum in the sequence. Since positions and payloads are also block encoded, the skip should skip to related block first, then fetch the values according to in-block offset. PosFPSkip and PayFPSkip record the file offsets of related block in .pos and .pay, respectively. While PosBlockOffset indicates which value to fetch inside the related block (PayBlockOffset is unnecessary since it is always equal to PosBlockOffset). Same as DocFPSkip, the file offsets are relative to the start of current term's TermFreqs, and stored as a difference sequence. PayByteUpto indicates the start offset of the current payload. It is equivalent to the sum of the payload lengths in the current block up to PosBlockOffset
-
Positions
The .pos file contains the lists of positions that each term occurs at within documents. It also sometimes stores part of payloads and offsets for speedup. PosFile(.pos) --> Header, <TermPositions> TermCount, Footer Header --> CodecHeader ( ) TermPositions --> <PackedPosDeltaBlock> PackedPosBlockNum, VIntBlock? VIntBlock --> <PositionDelta[, PayloadLength?], PayloadData?, OffsetDelta?, OffsetLength?>PosVIntCount PackedPosDeltaBlock --> PackedInts ( ) PositionDelta, OffsetDelta, OffsetLength --> VInt ( ) PayloadData --> byte ( )PayLength Footer --> CodecFooter ( )
Notes: TermPositions are order by term (terms are implicit, from the term dictionary), and position values for each term document pair are incremental, and ordered by document number. PackedPosBlockNum is the number of packed blocks for current term's positions, payloads or offsets. In particular, PackedPosBlockNum = floor(totalTermFreq/PackedBlockSize) PosVIntCount is the number of positions encoded as VInt format. In particular, PosVIntCount = totalTermFreq - PackedPosBlockNum*PackedBlockSize The procedure how PackedPosDeltaBlock is generated is the same as PackedDocDeltaBlock in chapter Frequencies and Skip Data. PositionDelta is, if payloads are disabled for the term's field, the difference between the position of the current occurrence in the document and the previous occurrence (or zero, if this is the first occurrence in this document). If payloads are enabled for the term's field, then PositionDelta/2 is the difference between the current and the previous position. If payloads are enabled and PositionDelta is odd, then PayloadLength is stored, indicating the length of the payload at the current term position. For example, the TermPositions for a term which occurs as the fourth term in one document, and as the fifth and ninth term in a subsequent document, would be the following sequence of VInts (payloads disabled): 4, 5, 4 PayloadData is metadata associated with the current term position. If PayloadLength is stored at the current position, then it indicates the length of this payload. If PayloadLength is not stored, then this payload has the same length as the payload at the previous position. OffsetDelta/2 is the difference between this position's startOffset from the previous occurrence (or zero, if this is the first occurrence in this document). If OffsetDelta is odd, then the length (endOffset-startOffset) differs from the previous occurrence and an OffsetLength follows. Offset data is only written for .
-
Payloads and Offsets
The .pay file will store payloads and offsets associated with certain term-document positions. Some payloads and offsets will be separated out into .pos file, for performance reasons. PayFile(.pay): --> Header, <TermPayloads, TermOffsets?> TermCount, Footer Header --> CodecHeader ( ) TermPayloads --> <PackedPayLengthBlock, SumPayLength, PayData> PackedPayBlockNum TermOffsets --> <PackedOffsetStartDeltaBlock, PackedOffsetLengthBlock> PackedPayBlockNum PackedPayLengthBlock, PackedOffsetStartDeltaBlock, PackedOffsetLengthBlock --> PackedInts ( ) SumPayLength --> VInt ( ) PayData --> byte ( ) SumPayLength Footer --> CodecFooter ( )
Notes: The order of TermPayloads/TermOffsets will be the same as TermPositions, note that part of payload/offsets are stored in .pos. The procedure how PackedPayLengthBlock and PackedOffsetLengthBlock are generated is the same as PackedFreqBlock in chapter Frequencies and Skip Data. While PackedStartDeltaBlock follows a same procedure as PackedDocDeltaBlock. PackedPayBlockNum is always equal to PackedPosBlockNum, for the same term. It is also synonym for PackedOffsetBlockNum. SumPayLength is the total length of payloads written within one block, should be the sum of PayLengths in one packed block. PayLength in PackedPayLengthBlock is the length of each payload associated with the current position.
PackedInts.COMPACT A fields data file (extension .fdt ). this file stores a compact representation of documents in compressed blocks of 16KB or more. When writing a segment, documents are appended to an in-memorybyte[] buffer. When its size reaches 16KB or more, some metadata about the documents is flushed to disk, immediately followed by a compressed representation of the buffer using the LZ4 compression format.Here is a more detailed description of the field data file format: FieldData (.fdt) --> <Header>, PackedIntsVersion, <Chunk>ChunkCount Header --> CodecHeader ( ) PackedIntsVersion --> as a VInt ( ) ChunkCount is not known in advance and is the number of chunks necessary to store all document of the segment Chunk --> DocBase, ChunkDocs, DocFieldCounts, DocLengths, <CompressedDocs> DocBase --> the ID of the first document of the chunk as a VInt ( ) ChunkDocs --> the number of documents in the chunk as a VInt ( ) DocFieldCounts --> the number of stored fields of every document in the chunk, encoded as followed: if chunkDocs=1, the unique value is encoded as a VInt ( ) else read a VInt ( ) (let's call it bitsRequired )if bitsRequired is0 then all values are equal, and the common value is the following VInt () else bitsRequired is the number of bits required to store any value, and values are stored in a packed () array where every value is stored on exactly bitsRequired bits
DocLengths --> the lengths of all documents in the chunk, encoded with the same method as DocFieldCounts CompressedDocs --> a compressed representation of <Docs> using the LZ4 compression format Docs --> <Doc>ChunkDocs Doc --> <FieldNumAndType, Value>DocFieldCount FieldNumAndType --> a VLong ( ), whose 3 last bits are Type and other bits are FieldNum Type --> 0: Value is String 1: Value is BinaryValue 2: Value is Int 3: Value is Float 4: Value is Long 5: Value is Double 6, 7: unused
FieldNum --> an ID of the field Value --> String ( ) | BinaryValue | Int | Float | Long | Double depending on Type BinaryValue --> ValueLength <Byte>ValueLength
Notes If documents are larger than 16KB then chunks will likely contain only one document. However, documents can never spread across several chunks (all fields of a single document are in the same chunk). When at least one document in a chunk is large enough so that the chunk is larger than 32KB, the chunk will actually be compressed in several LZ4 blocks of 16KB. this allows s which are only interested in the first fields of a document to not have to decompress 10MB of data if the document is 10MB, but only 16KB. Given that the original lengths are written in the metadata of the chunk, the decompressor can leverage this information to stop decoding as soon as enough data has been decompressed. In case documents are incompressible, CompressedDocs will be less than 0.5% larger than Docs.
A fields index file (extension .fdx ).FieldsIndex (.fdx) --> <Header>, <ChunkIndex> Header --> CodecHeader ( ) ChunkIndex: See
Delta-compressed Numerics: per-document integers written in blocks of 4096. For each block the minimum value is encoded, and each entry is a delta from that minimum value. Table-compressed Numerics: when the number of unique values is very small, a lookup table is written instead. Each per-document entry is instead the ordinal to this table. Uncompressed Numerics: when all values would fit into a single byte, and the acceptableOverheadRatio would pack values into 8 bits per value anyway, they are written as absolute values (with no indirection or packing) for performance.GCD-compressed Numerics: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics. Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed by maxDoc*length .Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written in blocks of 4096, with the current absolute start for the block, and the average (expected) delta per entry. For each document the deviation from the delta (actual - expected) is written. Sorted: an FST mapping deduplicated terms to ordinals is written, along with the per-document ordinals written using one of the numeric strategies above. SortedSet: an FST mapping deduplicated terms to ordinals is written, along with the per-document ordinal list written using one of the binary strategies above.
.dvd : DocValues data.dvm : DocValues metadata
The DocValues metadata or .dvm file. For DocValues field, this stores metadata, such as the offset into the DocValues data (.dvd) DocValues metadata (.dvm) --> Header,<FieldNumber,EntryType,Entry>NumFields,Footer Entry --> NumericEntry | BinaryEntry | SortedEntry NumericEntry --> DataOffset,CompressionType,PackedVersion BinaryEntry --> DataOffset,DataLength,MinLength,MaxLength,PackedVersion?,BlockSize? SortedEntry --> DataOffset,ValueCount FieldNumber,PackedVersion,MinLength,MaxLength,BlockSize,ValueCount --> VInt ( ) DataOffset,DataLength --> Int64 ( ) EntryType,CompressionType --> Byte ( ) Header --> CodecHeader ( ) Footer --> CodecFooter ( )
Sorted fields have two entries: a SortedEntry with the FST metadata, and an ordinary NumericEntry for the document-to-ord metadata. SortedSet fields have two entries: a SortedEntry with the FST metadata, and an ordinary BinaryEntry for the document-to-ord-list metadata. FieldNumber of -1 indicates the end of metadata. EntryType is a 0 (NumericEntry), 1 (BinaryEntry, or 2 (SortedEntry) DataOffset is the pointer to the start of the data in the DocValues data (.dvd) CompressionType indicates how Numeric values will be compressed: 0 --> delta-compressed. For each block of 4096 integers, every integer is delta-encoded from the minimum value within the block. 1 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document. 2 --> uncompressed. When the acceptableOverheadRatio parameter would upgrade the number of bits required to 8, and all values fit in a byte, these are written as absolute binary values for performance.3 --> gcd-compressed. When all integers share a common divisor, only quotients are stored using blocks of delta-encoded ints.
MinLength and MaxLength represent the min and max byte[] value lengths for Binary values. If they are equal, then all values are of a fixed size, and can be addressed as DataOffset + (docID * length) . Otherwise, the binary values are of variable size, and packed integer metadata (PackedVersion,BlockSize) is written for the addresses.The DocValues data or .dvd file. For DocValues field, this stores the actual per-document data (the heavy-lifting) DocValues data (.dvd) --> Header,<NumericData | BinaryData | SortedData>NumFields,Footer NumericData --> DeltaCompressedNumerics | TableCompressedNumerics | UncompressedNumerics | GCDCompressedNumerics BinaryData --> Byte ( ) DataLength,Addresses SortedData --> FST<Int64> ( ) DeltaCompressedNumerics --> BlockPackedInts(blockSize=4096) ( ) TableCompressedNumerics --> TableSize, Int64 ( ) TableSize, PackedInts ( ) UncompressedNumerics --> Byte ( ) maxdoc Addresses --> MonotonicBlockPackedInts(blockSize=4096) ( ) Footer --> CodecFooter (
SortedSet entries store the list of ordinals in their BinaryData as a sequences of increasing vLongs ( ), delta-encoded.
Binary doc values can be at most in length.
Header --> CodecHeader FieldsCount --> VInt FieldName --> String FieldBits, DocValuesBits --> Byte FieldNumber --> VInt Attributes --> IDictionary<String,String>
FieldsCount: the number of fields in this file. FieldName: name of the field as a UTF-8 String. FieldNumber: the field's number. Note that unlike previous versions of Lucene, the fields are not numbered implicitly by their order in the file, instead explicitly. FieldBits: a byte containing field options. The low-order bit is one for indexed fields, and zero for non-indexed fields. The second lowest-order bit is one for fields that have term vectors stored, and zero for fields without term vectors. If the third lowest order-bit is set (0x4), offsets are stored into the postings list in addition to positions. Fourth bit is unused. If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field. If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field. If the seventh lowest-order bit is set (0x40), term frequencies and positions omitted for the indexed field. If the eighth lowest-order bit is set (0x80), positions are omitted for the indexed field.
DocValuesBits: a byte containing per-document value types. The type recorded as two four-bit integers, with the high-order bits representing norms options, and the low-order bits representingoptions. Each four-bit integer can be decoded as such: 0: no DocValues for this field. 1: NumericDocValues. ( ) 2: BinaryDocValues. ( ) 3: SortedDocValues. ( )
Attributes: a key-value map of codec-private attributes.
.nvd : DocValues data.nvm : DocValues metadata
textual data is compressed using the very light, LZ4 compression algorithm, binary data is written using fixed-size blocks of packed s ( ).
a data file where terms, frequencies, positions, offsets and payloads are stored, an index file, loaded into memory, used to locate specific documents in the data file.
A vector data file (extension .tvd ). this file stores terms, frequencies, positions, offsets and payloads for every document. Upon writing a new segment, it accumulates data into memory until the buffer used to store terms and payloads grows beyond 4KB. Then it flushes all metadata, terms and positions to disk using LZ4 compression for terms and payloads and blocks of packeds ( ) for positions. Here is a more detailed description of the field data file format: VectorData (.tvd) --> <Header>, PackedIntsVersion, ChunkSize, <Chunk>ChunkCount, Footer Header --> CodecHeader ( ) PackedIntsVersion --> as a VInt ( ) ChunkSize is the number of bytes of terms to accumulate before flushing, as a VInt ( ) ChunkCount is not known in advance and is the number of chunks necessary to store all document of the segment Chunk --> DocBase, ChunkDocs, < NumFields >, < FieldNums >, < FieldNumOffs >, < Flags >, < NumTerms >, < TermLengths >, < TermFreqs >, < Positions >, < StartOffsets >, < Lengths >, < PayloadLengths >, < TermAndPayloads > DocBase is the ID of the first doc of the chunk as a VInt ( ) ChunkDocs is the number of documents in the chunk NumFields --> DocNumFieldsChunkDocs DocNumFields is the number of fields for each doc, written as a VInt ( ) if ChunkDocs==1 and as a array otherwise FieldNums --> FieldNumDeltaTotalDistincFields, a delta-encoded list of the sorted unique field numbers present in the chunk FieldNumOffs --> FieldNumOffTotalFields, as a array FieldNumOff is the offset of the field number in FieldNums TotalFields is the total number of fields (sum of the values of NumFields) Flags --> Bit < FieldFlags > Bit is a single bit which when true means that fields have the same options for every document in the chunk FieldFlags --> if Bit==1: FlagTotalDistinctFields else FlagTotalFields Flag: a 3-bits int where: the first bit means that the field has positions the second bit means that the field has offsets the third bit means that the field has payloads
NumTerms --> FieldNumTermsTotalFields FieldNumTerms: the number of terms for each field, using blocks of 64 packed s ( ) TermLengths --> PrefixLengthTotalTerms SuffixLengthTotalTerms TotalTerms: total number of terms (sum of NumTerms) PrefixLength: 0 for the first term of a field, the common prefix with the previous term otherwise using blocks of 64 packed s ( ) SuffixLength: length of the term minus PrefixLength for every term using blocks of 64 packed s ( ) TermFreqs --> TermFreqMinus1TotalTerms TermFreqMinus1: (frequency - 1) for each term using blocks of 64 packed s ( ) Positions --> PositionDeltaTotalPositions TotalPositions is the sum of frequencies of terms of all fields that have positions PositionDelta: the absolute position for the first position of a term, and the difference with the previous positions for following positions using blocks of 64 packed s ( ) StartOffsets --> (AvgCharsPerTermTotalDistinctFields) StartOffsetDeltaTotalOffsets TotalOffsets is the sum of frequencies of terms of all fields that have offsets AvgCharsPerTerm: average number of chars per term, encoded as a float on 4 bytes. They are not present if no field has both positions and offsets enabled. StartOffsetDelta: (startOffset - previousStartOffset - AvgCharsPerTerm * PositionDelta). previousStartOffset is 0 for the first offset and AvgCharsPerTerm is 0 if the field has no positions using blocks of 64 packed s ( ) Lengths --> LengthMinusTermLengthTotalOffsets LengthMinusTermLength: (endOffset - startOffset - termLength) using blocks of 64 packed s ( ) PayloadLengths --> PayloadLengthTotalPayloads TotalPayloads is the sum of frequencies of terms of all fields that have payloads PayloadLength is the payload length encoded using blocks of 64 packed s ( ) TermAndPayloads --> LZ4-compressed representation of < FieldTermsAndPayLoads >TotalFields FieldTermsAndPayLoads --> Terms (Payloads) Terms: term bytes Payloads: payload bytes (if the field has payloads) Footer --> CodecFooter ( )
An index file (extension .tvx ).VectorIndex (.tvx) --> <Header>, <ChunkIndex>, Footer Header --> CodecHeader ( ) ChunkIndex: See Footer --> CodecFooter ( )
Delta-compressed: per-document integers written in blocks of 16k. For each block the minimum value in that block is encoded, and each entry is a delta from that minimum value. Each block of deltas is compressed with bitpacking. For more information, see . Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as ), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking ( ). GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
Fixed-width Binary: one large concatenated is written, along with the fixed length. Each document's value can be addressed directly with multiplication ( docID * length ).Variable-width Binary: one large concatenated is written, along with end addresses for each document. The addresses are written in blocks of 16k, with the current absolute start for the block, and the average (expected) delta per entry. For each document the deviation from the delta (actual - expected) is written. Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. Chunk addresses are written in blocks of 16k, with the current absolute start for the block, and the average (expected) delta per entry. For each chunk the deviation from the delta (actual - expected) is written.
Sorted: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, along with the per-document ordinals written using one of the numeric strategies above.
SortedSet: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.
.dvd : DocValues data.dvm : DocValues metadata
The DocValues metadata or .dvm file. For DocValues field, this stores metadata, such as the offset into the DocValues data (.dvd) DocValues metadata (.dvm) --> Header,<Entry>NumFields,Footer Entry --> NumericEntry | BinaryEntry | SortedEntry | SortedSetEntry NumericEntry --> GCDNumericEntry | TableNumericEntry | DeltaNumericEntry GCDNumericEntry --> NumericHeader,MinValue,GCD TableNumericEntry --> NumericHeader,TableSize,Int64 ( ) TableSize DeltaNumericEntry --> NumericHeader NumericHeader --> FieldNumber,EntryType,NumericType,MissingOffset,PackedVersion,DataOffset,Count,BlockSize BinaryEntry --> FixedBinaryEntry | VariableBinaryEntry | PrefixBinaryEntry FixedBinaryEntry --> BinaryHeader VariableBinaryEntry --> BinaryHeader,AddressOffset,PackedVersion,BlockSize PrefixBinaryEntry --> BinaryHeader,AddressInterval,AddressOffset,PackedVersion,BlockSize BinaryHeader --> FieldNumber,EntryType,BinaryType,MissingOffset,MinLength,MaxLength,DataOffset SortedEntry --> FieldNumber,EntryType,BinaryEntry,NumericEntry SortedSetEntry --> EntryType,BinaryEntry,NumericEntry,NumericEntry FieldNumber,PackedVersion,MinLength,MaxLength,BlockSize,ValueCount --> VInt ( EntryType,CompressionType --> Byte ( Header --> CodecHeader ( ) MinValue,GCD,MissingOffset,AddressOffset,DataOffset --> Int64 ( ) TableSize --> vInt ( ) Footer --> CodecFooter ( )
Sorted fields have two entries: a with the value metadata, and an ordinary for the document-to-ord metadata. SortedSet fields have three entries: a with the value metadata, and two s for the document-to-ord-index and ordinal list metadata. FieldNumber of -1 indicates the end of metadata. EntryType is a 0 ( ) or 1 ( ) DataOffset is the pointer to the start of the data in the DocValues data (.dvd) NumericType indicates how Numeric values will be compressed: 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block. 1 --> gcd-compressed. When all integers share a common divisor, only quotients are stored using blocks of delta-encoded ints. 2 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.
BinaryType indicates how Binary values will be stored: 0 --> fixed-width. All values have the same length, addressing by multiplication. 1 --> variable-width. An address for each value is stored. 2 --> prefix-compressed. An address to the start of every interval'th value is stored.
MinLength and MaxLength represent the min and max byte[] value lengths for Binary values. If they are equal, then all values are of a fixed size, and can be addressed as DataOffset + (docID * length). Otherwise, the binary values are of variable size, and packed integer metadata (PackedVersion,BlockSize) is written for the addresses. MissingOffset points to a containing a bitset of all documents that had a value for the field. If its -1, then there are no missing values. Checksum contains the CRC32 checksum of all bytes in the .dvm file up until the checksum. this is used to verify integrity of the file on opening the index. The DocValues data or .dvd file. For DocValues field, this stores the actual per-document data (the heavy-lifting) DocValues data (.dvd) --> Header,<NumericData | BinaryData | SortedData>NumFields,Footer NumericData --> DeltaCompressedNumerics | TableCompressedNumerics | GCDCompressedNumerics BinaryData --> Byte ( ) DataLength,Addresses SortedData --> FST<Int64> ( ) DeltaCompressedNumerics --> BlockPackedInts(blockSize=16k) ( ) TableCompressedNumerics --> PackedInts ( ) GCDCompressedNumerics --> BlockPackedInts(blockSize=16k) ( ) Addresses --> MonotonicBlockPackedInts(blockSize=16k) ( ) Footer --> CodecFooter ( )
SortedSet entries store the list of ordinals in their BinaryData as a sequences of increasing vLongs ( ), delta-encoded.
Header --> CodecHeader ( ) FieldsCount --> VInt ( ) FieldName --> String ( ) FieldBits, DocValuesBits --> Byte ( ) FieldNumber --> VInt ( ) Attributes --> IDictionary<String,String> ( ) DocValuesGen --> Int64 ( ) Footer --> CodecFooter ( )
FieldsCount: the number of fields in this file. FieldName: name of the field as a UTF-8 string. FieldNumber: the field's number. Note that unlike previous versions of Lucene, the fields are not numbered implicitly by their order in the file, instead explicitly. FieldBits: a containing field options. The low-order bit is one for indexed fields, and zero for non-indexed fields. The second lowest-order bit is one for fields that have term vectors stored, and zero for fields without term vectors. If the third lowest order-bit is set (0x4), offsets are stored into the postings list in addition to positions. Fourth bit is unused. If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field. If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field. If the seventh lowest-order bit is set (0x40), term frequencies and positions omitted for the indexed field. If the eighth lowest-order bit is set (0x80), positions are omitted for the indexed field.
DocValuesBits: a containing per-document value types. The type recorded as two four-bit integers, with the high-order bits representing norms options, and the low-order bits representingoptions. Each four-bit integer can be decoded as such: 0: no DocValues for this field. 1: . ( ) 2: . ( ) 3: . ( )
DocValuesGen is the generation count of the field's . If this is -1, there are no updates to that field. Anything above zero means there are updates stored by . Attributes: a key-value map of codec-private attributes.
.si : Header, SegVersion, SegSize, IsCompoundFile, Diagnostics, Files, Footer
Header --> CodecHeader ( ) SegSize --> Int32 ( ) SegVersion --> String ( ) Files --> ISet<String> ( ) Diagnostics --> IDictionary<String,String> ( ) IsCompoundFile --> Int8 ( ) Footer --> CodecFooter ( )
SegVersion is the code version that created the segment. SegSize is the number of documents contained in the segment index. IsCompoundFile records whether the segment is written as a compound file or not. If this is -1, the segment is not a compound file. If it is 1, the segment is a compound file. The Diagnostics Map is privately written by , as a debugging aid, for each segment it creates. It includes metadata like the current Lucene version, OS, .NET/Java version, why the segment was created (merge, flush, addIndexes), etc. Files is a list of files referred to by this segment.
Example for skipInterval = 3:
c (skip level 2)
c c c (skip level 1)
x x x x x x x x x x (skip level 0)
d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d (posting list)
3 6 9 12 15 18 21 24 27 30 (df)
d - document
x - skip data
c - skip data with child pointer
Skip level i contains every skipInterval-th entry from skip level i-1.
Therefore the number of entries on level i is: floor(df / ((skipInterval ^ (i + 1))).
Each skip entry on a level i>0 contains a pointer to the corresponding skip entry in list i-1.
this guarantees a logarithmic amount of skips to find the target document.
While this class takes care of writing the different skip levels,
subclasses must define the actual format of the skip data.
PostingsConsumer is returned for each term by . is called for each document where the term occurs, specifying id and term frequency for that document. If positions are enabled for the field, then will be called for each occurrence in the document. is called when the producer is done adding positions to the document.
Subclass this class. Subclass , override , and add the line base.ScanForPostingsFormats(typeof(YourPostingsFormat).Assembly) . If you have any format classes in your assembly that are not meant for reading, you can add theto them so they are ignored by the scan. Set the new by calling at application startup.
For every document, is called, informing the Codec how many fields will be written. is called for each field in the document. After all documents have been written, is called for verification/sanity-checks. Finally the writer is disposed ( )
TermsConsumer is returned for each field by . TermsConsumer returns a for each term in . When the producer (e.g. IndexWriter) is done adding documents for the term, it calls , passing in the accumulated term statistics. Producer calls with the accumulated collection statistics when it is finished adding terms to the field.
For every document, is called, informing the how many fields will be written. is called for each field in the document, informing the codec how many terms will be written for that field, and whether or not positions, offsets, or payloads are enabled. Within each field, is called for each term. If offsets and/or positions are enabled, then will be called for each term occurrence. After all documents have been written, is called for verification/sanity-checks. Finally the writer is disposed ( )
Register Additional Codecs
// Register the factory at application start up.
Codec.SetCodecFactory(new DefaultCodecFactory {
CustomCodecTypes = new Type[] { typeof(MyCodec), typeof(AnotherCodec) }
});
Only Use Explicitly Defined Codecs
public class ExplicitCodecFactory : DefaultCodecFactory
{
protected override void Initialize()
{
// Load specific codecs in a specific order.
PutCodecType(typeof(MyCodec));
PutCodecType(typeof(AnotherCodec));
}
protected override Codec NewCodec(Type type)
{
// Special case: AnotherCodec has a required dependency
if (typeof(AnotherCodec).Equals(type))
return new AnotherCodec(new SomeDependency());
return base.NewCodec(type);
}
}
// Register the factory at application start up.
Codec.SetCodecFactory(new ExplicitCodecFactory());
See the Use Reflection to Scan an Assembly for Codecs
public class ScanningCodecFactory : DefaultCodecFactory
{
protected override void Initialize()
{
// Load all default codecs
base.Initialize();
// Load all of the codecs inside of the same assembly that MyCodec is defined in
ScanForCodecs(typeof(MyCodec).Assembly);
}
}
// Register the factory at application start up.
Codec.SetCodecFactory(new ScanningCodecFactory());
Codecs in the target assemblie(s) can be excluded from the scan by decorating them with
the Register Additional DocValuesFormats
// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new DefaultDocValuesFormatFactory {
CustomDocValuesFormatTypes = new Type[] { typeof(MyDocValuesFormat), typeof(AnotherDocValuesFormat) }
});
Only Use Explicitly Defined DocValuesFormats
public class ExplicitDocValuesFormatFactory : DefaultDocValuesFormatFactory
{
protected override void Initialize()
{
// Load specific codecs in a specific order.
PutDocValuesFormatType(typeof(MyDocValuesFormat));
PutDocValuesFormatType(typeof(AnotherDocValuesFormat));
}
protected override DocValuesFormat NewDocValuesFormat(Type type)
{
// Special case: AnotherDocValuesFormat has a required dependency
if (typeof(AnotherDocValuesFormat).Equals(type))
return new AnotherDocValuesFormat(new SomeDependency());
return base.NewDocValuesFormat(type);
}
}
// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new ExplicitDocValuesFormatFactory());
See the Use Reflection to Scan an Assembly for DocValuesFormats
public class ScanningDocValuesFormatFactory : DefaultDocValuesFormatFactory
{
protected override void Initialize()
{
// Load all default codecs
base.Initialize();
// Load all of the codecs inside of the same assembly that MyDocValuesFormat is defined in
ScanForDocValuesFormats(typeof(MyDocValuesFormat).Assembly);
}
}
// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new ScanningDocValuesFormatFactory());
Doc values formats in the target assembly can be excluded from the scan by decorating them with
the Register Additional PostingsFormats
// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new DefaultPostingsFormatFactory {
CustomPostingsFormatTypes = new Type[] { typeof(MyPostingsFormat), typeof(AnotherPostingsFormat) }
});
Only Use Explicitly Defined PostingsFormats
public class ExplicitPostingsFormatFactory : DefaultPostingsFormatFactory
{
protected override void Initialize()
{
// Load specific codecs in a specific order.
PutPostingsFormatType(typeof(MyPostingsFormat));
PutPostingsFormatType(typeof(AnotherPostingsFormat));
}
protected override PostingsFormat NewPostingsFormat(Type type)
{
// Special case: AnotherPostingsFormat has a required dependency
if (typeof(AnotherPostingsFormat).Equals(type))
return new AnotherPostingsFormat(new SomeDependency());
return base.NewPostingsFormat(type);
}
}
// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new ExplicitPostingsFormatFactory());
See the Use Reflection to Scan an Assembly for PostingsFormats
public class ScanningPostingsFormatFactory : DefaultPostingsFormatFactory
{
protected override void Initialize()
{
// Load all default codecs
base.Initialize();
// Load all of the codecs inside of the same assembly that MyPostingsFormat is defined in
ScanForPostingsFormats(typeof(MyPostingsFormat).Assembly);
}
}
// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new ScanningPostingsFormatFactory());
Postings formats in the target assembly can be excluded from the scan by decorating them with
the
document.Add(new BinaryDocValuesField(name, new BytesRef("hello")));
document.Add(new ByteDocValuesField(name, (byte) 22));
document.Add(new DerefBytesDocValuesField(name, new BytesRef("hello")));
document.Add(new DoubleField(name, 6.0, Field.Store.NO));
For optimal performance, re-use the
DoubleField field = new DoubleField(name, 0.0, Field.Store.NO);
Document document = new Document();
document.Add(field);
for (all documents)
{
...
field.SetDoubleValue(value)
writer.AddDocument(document);
...
}
See also the field is neither stored nor indexed the field is not indexed but termVector is
document.Add(new SingleField(name, 6.0F, Field.Store.NO));
For optimal performance, re-use the
FloatField field = new SingleField(name, 0.0F, Field.Store.NO);
Document document = new Document();
document.Add(field);
for (all documents)
{
...
field.SetSingleValue(value)
writer.AddDocument(document);
...
}
See also
document.Add(new Int32DocValuesField(name, 22));
If you also need to store the value, you should add a
separate
document.Add(new Int32Field(name, 6, Field.Store.NO));
For optimal performance, re-use the
Int32Field field = new Int32Field(name, 6, Field.Store.NO);
Document document = new Document();
document.Add(field);
for (all documents)
{
...
field.SetInt32Value(value)
writer.AddDocument(document);
...
}
See also
document.Add(new Int64DocValuesField(name, 22L));
document.Add(new Int64Field(name, 6L, Field.Store.NO));
For optimal performance, re-use the
Int64Field field = new Int64Field(name, 0L, Field.Store.NO);
Document document = new Document();
document.Add(field);
for (all documents) {
...
field.SetInt64Value(value)
writer.AddDocument(document);
...
}
See also
document.Add(new NumericDocValuesField(name, 22L));
If you also need to store the value, you should add a
separate
document.Add(new PackedInt64DocValuesField(name, 22L));
document.Add(new Int16DocValuesField(name, (short) 22));
document.Add(new SortedBytesDocValuesField(name, new BytesRef("hello")));
document.Add(new SortedDocValuesField(name, new BytesRef("hello")));
document.Add(new SortedSetDocValuesField(name, new BytesRef("hello")));
document.Add(new SortedSetDocValuesField(name, new BytesRef("world")));
document.Add(new StraightBytesDocValuesField(name, new BytesRef("hello")));
As long as matches are successful, keep reading sequentially. When a match fails, skip to the next string in lexicographic order that does not enter a reject state.
if (IsVerbose)
{
Message("your message");
}
Each document points to a list of term numbers that are contained in that document. Term numbers are in sorted order, and are encoded as variable-length deltas from the previous term number. Real term numbers start at 2 since 0 and 1 are reserved. A term number of 0 signals the end of the termNumber list. There is a single int[maxDoc()] which either contains a pointer into a byte[] for the termNumber lists, or directly contains the termNumber list if it fits in the 4 bytes of an integer. If the first byte in the integer is 1, the next 3 bytes are a pointer into a byte[] where the termNumber list starts. There are actually 256 byte arrays, to compensate for the fact that the pointers into the byte arrays are only 3 bytes long. The correct byte array for a document is a function of it's id. To save space and speed up faceting, any term that matches enough documents will not be un-inverted... it will be skipped while building the un-inverted field structure, and will use a set intersection method during faceting. To further save memory, the terms (the actual string values) are not all stored in memory, but a TermIndex is used to convert term numbers to term values only for the terms needed after faceting has completed. Only every 128th term value is stored, along with it's corresponding term number, and this is used as an index to find the closest term and iterate until the desired number is hit (very much like Lucene's own internal term index).
consumes a document and finishes its processing updates its private either by calling or (if the document has a delTerm) applies all deletes in the slice to its private and resets it increments its internal document id
true iff the given item is identical to the item
hold by the slices tail, otherwise false.
- applies pending delete operations based on the global number of buffered delete terms iff is enabled - flushes either on the number of documents per ( ) or on the global active memory consumption in the current indexing session iff or is enabled respectively - calls and in order
RAM consumption - configured via Number of RAM resident documents - configured via
ext is not empty.
: These indexes do not consist of several sub-readers, they are atomic. They support retrieval of stored fields, doc values, terms, and postings. : Instances (like ) of this reader can only be used to get stored fields from the underlying s, but it is not possible to directly retrieve postings. To do that, get the sub-readers via . Alternatively, you can mimic an (with a serious slowdown), by wrapping composite readers with .
try
{
writer.Dispose();
}
finally
{
if (IndexWriter.IsLocked(directory))
{
IndexWriter.Unlock(directory);
}
}
after which, you must be certain not to use the writer
instance anymore.
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
{
Codec = Lucene46Codec(),
OpenMode = OpenMode.CREATE
};
However, if you prefer to match the syntax of Lucene using chained setter methods,
there are extension methods in the Lucene.Net.Index.Extensions namespace. Example usage:
using Lucene.Net.Index.Extensions;
..
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
.SetCodec(new Lucene46Codec())
.SetOpenMode(OpenMode.CREATE);
@since 3.1
public class MyLucene45Codec : Lucene45Codec
{
//customize Lucene41PostingsFormat, passing minBlockSize=50, maxBlockSize=100
private readonly PostingsFormat tweakedPostings = new Lucene41PostingsFormat(50, 100);
public override PostingsFormat GetPostingsFormatForField(string field)
{
if (field.Equals("fieldWithTonsOfTerms", StringComparison.Ordinal))
return tweakedPostings;
else
return base.GetPostingsFormatForField(field);
}
}
...
iwc.Codec = new MyLucene45Codec();
Note that other implementations may have their own parameters, or no parameters at all.
false.
segments.gen : GenHeader, Generation, Generation, Footersegments_N : Header, Version, NameCounter, SegCount, <SegName, SegCodec, DelGen, DeletionCount, FieldInfosGen, UpdatesFiles>SegCount, CommitUserData, Footer
Header --> GenHeader, NameCounter, SegCount, DeletionCount --> Generation, Version, DelGen, Checksum, FieldInfosGen --> SegName, SegCodec --> CommitUserData --> UpdatesFiles --> Footer -->
Version counts how often the index has been changed by adding or deleting documents. NameCounter is used to generate names for new segment files. SegName is the name of the segment, and is used as the file name prefix for all of the files that compose the segment's index. DelGen is the generation count of the deletes file. If this is -1, there are no deletes. Anything above zero means there are deletes stored by . DeletionCount records the number of deleted documents in this segment. SegCodec is the of the that encoded this segment. CommitUserData stores an optional user-supplied opaque that was passed to . FieldInfosGen is the generation count of the fieldInfos file. If this is -1, there are no updates to the fieldInfos in that segment. Anything above zero means there are updates to fieldInfos stored by . UpdatesFiles stores the list of files that were updated in that segment.
IndexWriterConfig iwc = new IndexWriterConfig(LuceneVersion.LUCENE_XX, new KeywordAnalyzer());
iwc.MergePolicy = new UpgradeIndexMergePolicy(iwc.MergePolicy);
using (IndexWriter w = new IndexWriter(dir, iwc))
{
w.ForceMerge(1);
}
-
Less than zero precedes y in the sort order. -
Zero is equal to . -
Greater than zero follows in the sort order.
if (Verbose) {
Message("your message");
}
var booleanQuery = new BooleanQuery() {
{ new WildcardQuery(new Term("field2", "foobar")), Occur.SHOULD },
{ new MultiPhraseQuery() {
new Term("field", "microsoft"),
new Term("field", "office")
}, Occur.SHOULD }
};
// or
var booleanQuery = new BooleanQuery() {
new BooleanClause(new WildcardQuery(new Term("field2", "foobar")), Occur.SHOULD),
new BooleanClause(new MultiPhraseQuery() {
new Term("field", "microsoft"),
new Term("field", "office")
}, Occur.SHOULD)
};
foreach (BooleanClause clause in booleanQuery) {}
is an abstract base class that assumes you will retrieve the top N docs, according to some criteria, after collection is done. is a concrete subclass and sorts according to score + docID. This is used internally by the search methods that do not take an explicit . It is likely the most frequently used collector. subclasses and sorts according to a specified object (sort by field). This is used internally by the search methods that take an explicit . , which wraps any other Collector and aborts the search if it's taken too much time. wraps any other and prevents collection of hits whose score is <= 0.0
private class MySearchCollector : ICollector
{
private readonly OpenBitSet bits;
private int docBase;
public MySearchCollector(OpenBitSet bits)
{
if (bits is null) throw new ArgumentNullException("bits");
this.bits = bits;
}
// ignore scorer
public void SetScorer(Scorer scorer)
{
}
// accept docs out of order (for a BitSet it doesn't matter)
public bool AcceptDocsOutOfOrder
{
get { return true; }
}
public void Collect(int doc)
{
bits.Set(doc + docBase);
}
public void SetNextReader(AtomicReaderContext context)
{
this.docBase = context.DocBase;
}
}
IndexSearcher searcher = new IndexSearcher(indexReader);
OpenBitSet bits = new OpenBitSet(indexReader.MaxDoc);
searcher.Search(query, new MySearchCollector(bits));
IndexSearcher searcher = new IndexSearcher(indexReader);
OpenBitSet bits = new OpenBitSet(indexReader.MaxDoc);
int docBase;
searcher.Search(query,
Collector.NewAnonymous(setScorer: (scorer) =>
{
// ignore scorer
}, collect: (doc) =>
{
bits.Set(doc + docBase);
}, setNextReader: (context) =>
{
docBase = context.DocBase;
}, acceptsDocsOutOfOrder: () =>
{
return true;
})
);
var disjunctionMaxQuery = new DisjunctionMaxQuery(0.1f) {
new TermQuery(new Term("field1", "albino")),
new TermQuery(new Term("field2", "elephant"))
};
var docIdSet = DocIdSet.NewAnonymous(getIterator: () =>
{
OpenBitSet bitset = new OpenBitSet(5);
bitset.Set(0, 5);
return new DocIdBitSet(bitset);
});
var docIdSet = DocIdSet.NewAnonymous(getIterator: () =>
{
OpenBitSet bitset = new OpenBitSet(5);
bitset.Set(0, 5);
return new DocIdBitSet(bitset);
}, bits: () =>
{
return bits;
});
var docIdSet = DocIdSet.NewAnonymous(getIterator: () =>
{
OpenBitSet bitset = new OpenBitSet(5);
bitset.Set(0, 5);
return new DocIdBitSet(bitset);
}, isCacheable: () =>
{
return true;
});
var docIdSet = DocIdSet.NewAnonymous(getIterator: () =>
{
OpenBitSet bitset = new OpenBitSet(5);
bitset.Set(0, 5);
return new DocIdBitSet(bitset);
}, bits: () =>
{
return bits;
}, isCacheable: () =>
{
return true;
});
-1 or if or were not called yet. if the iterator has exhausted. Otherwise it should return the doc ID it is currently on.
int Advance(int target)
{
int doc;
while ((doc = NextDoc()) < target)
{
}
return doc;
}
Some implementations are considerably more efficient than that.
Compare a hit at 'slot a' with hit 'slot b'. This method is called by to notify the of the current weakest ("bottom") slot. Note that this slot may not hold the weakest value according to your comparer, in cases where your comparer is not the primary one (ie, is only used to break ties from the comparers before it). Compare a new hit (docID) against the "weakest" (bottom) entry in the queue. This method is called by to notify the of the top most value, which is used by future calls to . Compare a new hit (docID) against the top value previously set by a call to . Installs a new hit into the priority queue. The calls this method when a new hit is competitive. Invoked when the search is switching to the next segment. You may need to update internal state of the comparer, for example retrieving new values from the . Return the sort value stored in the specified slot. This is only called at the end of the search, in order to populate when returning the top results.
var filter = Filter.NewAnonymous(getDocIdSet: (context, acceptDocs) =>
{
if (acceptDocs is null) acceptDocs = new Bits.MatchAllBits(5);
OpenBitSet bitset = new OpenBitSet(5);
if (acceptDocs.Get(1)) bitset.Set(1);
if (acceptDocs.Get(3)) bitset.Set(3);
return new DocIdBitSet(bitset);
});
true. Otherwise this strategy falls back to a "zig-zag join" (
PriorityQueue<ScoreDoc> pq = new HitQueue(10, true); // pre-populate.
ScoreDoc top = pq.Top;
// Add/Update one element.
top.Score = 1.0f;
top.Soc = 0;
top = (ScoreDoc) pq.UpdateTop();
int totalHits = 1;
// Now pop only the elements that were *truly* inserted.
// First, pop all the sentinel elements (there are pq.Count - totalHits).
for (int i = pq.Count - totalHits; i > 0; i--) pq.Pop();
// Now pop the truly added elements.
ScoreDoc[] results = new ScoreDoc[totalHits];
for (int i = totalHits - 1; i >= 0; i--)
{
results[i] = (ScoreDoc)pq.Pop();
}
NOTE:
.IndexReader.Document(docID) .IndexReader.Document(docID, fieldVisitor) .IndexReader.Document(docID, fieldsToLoad) n
hits for Filters out the null collectors, so they are not used during search time.If the input contains 1 real collector (i.e. non- null ), it is returned.Otherwise the method returns a which wraps the non- nullones.
var multiPhraseQuery = new MultiPhraseQuery() {
new Term("field", "microsoft"),
new Term("field", "office")
};
Note that as long as you specify all of the parameters, you can use either
GetTermsEnum(terms, new AttributeSource())
var phraseQuery = new NGramPhraseQuery(2) {
new Term("field", "ABCD"),
new Term("field", "EFGH")
};
Note that as long as you specify all of the parameters, you can use either
Filter f = NumericRangeFilter.NewFloatRange("weight", 0.03f, 0.10f, true, true);
Accepts all documents whose float valued "weight" field
ranges from 0.03 to 0.10, inclusive.
See
Query q = NumericRangeQuery.NewFloatRange("weight", 0.03f, 0.10f, true, true);
matches all documents whose How it works
Schindler, U, Diepenbroek, M, 2008. Generic XML-based Framework for Metadata Portals. Computers & Geosciences 34 (12), 1947-1955. doi:10.1016/j.cageo.2008.02.023
7*255*2 + 255 = 3825 distinct terms (when there is a term for every distinct value of an
8-byte-number in the index and the range covers almost all of them; a maximum of 255 distinct values is used
because it would always be possible to reduce the full 256 values to one term with degraded precision).
In practice, we have seen up to 300 terms in most cases (index with 500,000 metadata records
and a uniform value distribution).Precision Step
![\mathrm{maxQueryTerms} = \left[ \left( \mathrm{indexedTermsPerValue} - 1 \right) \cdot \left(2^\mathrm{precisionStep} - 1 \right) \cdot 2 \right] + \left( 2^\mathrm{precisionStep} - 1 \right)](doc-files/nrq-formula-2.png)
The default for all data types is 4, which is used, when no precisionStepis given.Ideal value in most cases for 64 bit data types (long, double) is 6 or 8. Ideal value in most cases for 32 bit data types (int, float) is 4. For low cardinality fields larger precision steps are good. If the cardinality is < 100, it is fair to use (see below). Steps >=64 for long/double and >=32 for int/float produces one token per value in the index and querying is as slow as a conventional . But it can be used to produce fields, that are solely used for sorting (in this case simply use as ). Using , , or for sorting is ideal, because building the field cache is much faster than with text-only numbers. These fields have one term per value and therefore also work with term enumeration for building distinct lists (e.g. facets / preselected values to search for). Sorting is also possible with range query optimized fields using one of the above s.
var phraseQuery = new PhraseQuery() {
new Term("field", "microsoft"),
new Term("field", "office")
};
Note that as long as you specify all of the parameters, you can use either
Comparisons are fast The term dictionary is enumerated in an intelligent way, to avoid comparisons. See for more details.
reqScorer != null, exclScorer != null, reqScorer was advanced once via Next() or SkipTo() and reqScorer.Doc may still be excluded.
public IndexSearcher NewSearcher(IndexReader r)
{
return new IndexSearcher(r);
}
You can pass your own factory instead if you want custom behavior, such as:
Setting a custom scoring model: Parallel per-segment search: Return custom subclasses of (for example that implement distributed scoring) Run queries to warm your before it is used. Note: when using near-realtime search you may want to also set to warm newly merged segments in the background, outside of the reopen path.
SearcherLifetimeManager mgr = new SearcherLifetimeManager();
Per search-request, if it's a "new" search request, then
obtain the latest searcher you have (for example, by
using
// Record the current searcher, and save the returend
// token into user's search results (eg as a hidden
// HTML form field):
long token = mgr.Record(searcher);
When a follow-up search arrives, for example the user
clicks next page, drills down/up, etc., take the token
that you saved from the previous search and:
// If possible, obtain the same searcher as the last
// search:
IndexSearcher searcher = mgr.Acquire(token);
if (searcher != null)
{
// Searcher is still here
try
{
// do searching...
}
finally
{
mgr.Release(searcher);
// Do not use searcher after this!
searcher = null;
}
}
else
{
// Searcher was pruned -- notify user session timed
// out, or, pull fresh searcher again
}
Finally, in a separate thread, ideally the same thread
that's periodically reopening your searchers, you should
periodically prune old searchers:
mgr.Prune(new PruneByAge(600.0));
IndexSearcher s = manager.Acquire();
try
{
// Do searching, doc retrieval, etc. with s
}
finally
{
manager.Release(s);
// Do not use s after this!
s = null;
}
k1 = 1.2 ,b = 0.75 .
Idf(docFreq, searcher.MaxDoc);
Note that -
: Basic model of information content: : Limiting form of Bose-Einstein : Geometric approximation of Bose-Einstein : Poisson approximation of the Binomial : Divergence approximation of the Binomial : Inverse document frequency : Inverse expected document frequency [mixture of Poisson and IDF] : Inverse term frequency [approximation of I(ne)]
-
: First normalization of information gain: : Laplace's law of succession : Ratio of two Bernoulli processes : no first normalization
-
: Second (length) normalization: : Uniform distribution of term frequency : term frequency density inversely related to length : term frequency normalization provided by Dirichlet prior : term frequency normalization provided by a Zipfian relation : no second normalization
xqw is the query boost; Xw is a random variable that counts the occurrences of word w; tdw is the normalized term frequency; λw is a parameter.
-
: Probabilistic distribution used to model term occurrence : Log-logistic : Smoothed power-law
-
: λw parameter of the probability distribution : Nw/N or average number of documents where w occurs: Fw/N or average number of occurrences of w in the collection
-
: Term frequency normalization Any supported DFR normalization (listed in )
, which defines a new statistic, the probability that the collection language model generates the current term; , which is a strategy interface for object that compute the collection language model p(w|C) ;, an implementation of the former, that computes the term probability as the number of occurrences of the term in the collection, divided by the total number of tokens.
The method is called a single time, allowing the implementation to compute any statistics (such as IDF, average document length, etc) across the entire collection. The and passed in already contain all of the raw statistics involved, so a can freely use any combination of statistics without causing any additional I/O. Lucene makes no assumption about what is stored in the returned object. The query normalization process occurs a single time: is called for each query leaf node, is called for the top-level query, and finally passes down the normalization value and any top-level boosts (e.g. from enclosing s). For each segment in the index, the creates a The GetScore() method is called for each matching document.
state.Length is small.
-
-
cosine-similarity(q,d) = V(q) · V(d) ––––––––– |V(q)| |V(d)|
-
-
VSM Score
Normalizing V(d) to the unit vector is known to be problematic in that it removes all document length information. For some documents removing this info is probably ok, e.g. a document made by duplicating a certain paragraph 10 times, especially if that paragraph is made of distinct terms. But for a document which contains no duplicated paragraphs, this might be wrong. To avoid this problem, a different document length normalization factor is used, which normalizes to a vector equal to or larger than the unit vector: doc-len-norm(d). At indexing, users can specify that certain documents are more important than others, by assigning a document boost. For this, the score of each document is also multiplied by its boost value doc-boost(d). Lucene is field based, hence each query term applies to a single field, document length normalization is by the length of the certain field, and in addition to document boost there are also document fields boosts. The same field can be added to a document during indexing several times, and so the boost of that field is the multiplication of the boosts of the separate additions (or parts) of that field within the document. At search time users can specify boosts to each query, sub-query, and each query term, hence the contribution of a query term to the score of a document is multiplied by the boost of that query term query-boost(q). A document may match a multi term query without containing all the terms of that query (this is correct for some of the queries), and users can further reward documents matching more query terms through a coordination factor, which is usually larger when more terms are matched: coord-factor(q,d).
-
-
score(q,d) = coord-factor(q,d) · query-boost(q) · V(q) · V(d) ––––––––– |V(q)|
· doc-len-norm(d) · doc-boost(d)
-
-
Lucene Conceptual Scoring Formula
Query-boost for the query (actually for each query term) is known when search starts. Query Euclidean norm |V(q)| can be computed when search starts, as it is independent of the document being scored. From search optimization perspective, it is a valid question why bother to normalize the query at all, because all scored documents will be multiplied by the same |V(q)|, and hence documents ranks (their order by score) will not be affected by this normalization. There are two good reasons to keep this normalization: Recall that Cosine Similarity can be used find how similar two documents are. One can use Lucene for e.g. clustering, and use a document as a query to compute its similarity to other documents. In this use case it is important that the score of document d3 for query d1 is comparable to the score of document d3 for query d2. In other words, scores of a document for two distinct queries should be comparable. There are other applications that may require this. And this is exactly what normalizing the query vector V(q) provides: comparability (to a certain extent) of two or more queries. Applying query normalization on the scores helps to keep the scores around the unit vector, hence preventing loss of score data because of floating point precision limitations.
Document length norm doc-len-norm(d) and document boost doc-boost(d) are known at indexing time. They are computed in advance and their multiplication is saved as a single value in the index: norm(d). (In the equations below, norm(t in d) means norm(field(t) in doc d) where field(t) is the field associated with term t.)
-
-
score(q,d) = coord(q,d) · queryNorm(q) · ∑ ( tf(t in d) · idf(t)2 · t.Boost · norm(t,d) ) -
t in q
-
-
Lucene Practical Scoring Function
tf(t in d) correlates to the term's frequency, defined as the number of times term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score. Note that tf(t in q) is assumed to be 1 and therefore it does not appear in this equation, However if a query contains twice the same term, there will be two term-queries with that same term and hence the computation would still be correct (although not very efficient). The default computation for tf(t in d) in DefaultSimilarity ( ) is: -
tf(t in d) = frequency½
-
idf(t) stands for Inverse Document Frequency. this value correlates to the inverse of DocFreq (the number of documents in which the term t appears). this means rarer terms give higher contribution to the total score. idf(t) appears for t in both the query and the document, hence it is squared in the equation. The default computation for idf(t) in DefaultSimilarity ( ) is: -
idf(t) = 1 + log ( NumDocs ––––––––– DocFreq+1
)
-
coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. this is a search time factor computed in coord(q,d) ( ) by the Similarity in effect at search time. queryNorm(q) is a normalizing factor used to make scores between queries comparable. this factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. this is a search time factor computed by the Similarity in effect at search time. The default computation in DefaultSimilarity ( ) produces a Euclidean norm: -
queryNorm(q) = queryNorm(sumOfSquaredWeights) = 1 –––––––––––––– sumOfSquaredWeights½
The sum of squared weights (of the query terms) is computed by the query object. For example, a computes this value as: -
sumOfSquaredWeights = q.Boost 2 · ∑ ( idf(t) · t.Boost ) 2 -
t in q
and q.Boost is -
t.Boost is a search time boost of term t in the query q as specified in the query text (see query syntax), or as set by application calls to . Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi objects, and so the boost of a term in the query is accessible by calling the sub-query . norm(t,d) encapsulates a few (indexing time) boost and length factors: Field boost - set before adding the field to a document. lengthNorm - computed when the document is added to the index in accordance with the number of tokens of this field in the document, so that shorter fields contribute more to the score. LengthNorm is computed by the class in effect at indexing.
method is responsible for combining all of these factors into a single . When a document is added to the index, all the above factors are multiplied. If the document has multiple fields with the same name, all their boosts are multiplied together: -
norm(t,d) = lengthNorm · ∏ -
field f in d named as t
for search.
Idf(docFreq, searcher.MaxDoc);
Note that Check if there are repetitions If there are, find groups of repetitions.
no repetitions: "ho my"~2 >repetitions: "ho my my"~2 repetitions: "my ho my"~2
document.Add(new Field("byNumber", x.ToString(CultureInfo.InvariantCulture), Field.Store.NO, Field.Index.NOT_ANALYZED));
Valid Types of Values
Object Reuse
Memory Usage
4 * IndexReader.MaxDoc * (# of different fields actually used to sort)
teacherid: 1
studentfirstname: james
studentsurname: jones
teacherid: 2
studenfirstname: james
studentsurname: smith
studentfirstname: sally
studentsurname: jones
SpanQuery q1 = new SpanTermQuery(new Term("studentfirstname", "james"));
SpanQuery q2 = new SpanTermQuery(new Term("studentsurname", "jones"));
SpanQuery q2m = new FieldMaskingSpanQuery(q2, "studentfirstname");
Query q = new SpanNearQuery(new SpanQuery[] { q1, q2m }, -1, false);
to search for 'studentfirstname:james studentsurname:jones' and find
teacherid 1 without matching teacherid 2 (which has a 'james' in position 0
and 'jones' in position 1).
bool SkipTo(int target)
{
do
{
if (!Next())
return false;
} while (target > Doc);
return true;
}
Most implementations are considerably more efficient than that.
WildcardQuery wildcard = new WildcardQuery(new Term("field", "bro?n"));
SpanQuery spanWildcard = new SpanMultiTermQueryWrapper<WildcardQuery>(wildcard);
// do something with spanWildcard, such as use it in a SpanFirstQuery
o is equal to this.
bool SkipTo(int target)
{
do
{
if (!Next())
return false;
} while (target > Doc);
return true;
}
Most implementations are considerably more efficient than that.
// Counter is in the Lucene.Net.Util namespace
Counter clock = Counter.NewCounter(true);
long baseline = clock.Get();
// ... prepare search
TimeLimitingCollector collector = new TimeLimitingCollector(c, clock, numTicks);
collector.SetBaseline(baseline);
indexSearcher.Search(query, collector);
A is constructed by a top-level query, given a ( ). The method is called on the to compute the query normalization factor of the query clauses contained in the query. The query normalization factor is passed to . At this point the weighting is complete. A is constructed by .
var searcherManager = new SearcherManager(indexWriter, true, null);
using (var context = searcherManager.GetContext())
{
IndexSearcher searcher = context.Reference;
// use searcher...
}
protected override void Dispose(bool disposing)
{
if (!CompareAndSetIsOpen(expect: true, update: false)) return;
// Dispose unmanaged resources
if (disposing)
{
// Dispose managed resources
}
}
.cfs : An optional "virtual" file consisting of all the other index files for systems that frequently run out of file handles..cfe : The "virtual" compound file's entry table holding all entries in the corresponding .cfs file.
Compound (.cfs) --> Header, FileData FileCount Compound Entry Table (.cfe) --> Header, FileCount, <FileName, DataOffset, DataLength> FileCount, Footer Header --> FileCount --> DataOffset,DataLength --> FileName --> FileData --> raw file data Footer -->
FileCount indicates how many files are contained in this compound file. The entry table that follows has that many entries. Each directory entry contains a long pointer to the start of this file's data section, the files length, and a with that file's name.
-
0 00000000 -
1 00000001 -
2 00000010 -
... -
127 01111111 -
128 10000000 00000001 -
129 10000001 00000001 -
130 10000010 00000001 -
... -
16,383 11111111 01111111 -
16,384 10000000 10000000 00000001 -
16,385 10000001 10000000 00000001 -
...
implementation of RAM-based indices; implementation indices stored in a database; implementation of an index as a single file;
Throws if the file does not exist. Returns a value >=0 if the file exists, which specifies its length.
NOTE: Clients may call this method for same files over and over again, so some impls might optimize for that. For other impls the operation can be a noop, for various reasons.
Directory to; // the directory to copy to
foreach (string file in dir.ListAll()) {
dir.Copy(to, file, newFile, IOContext.DEFAULT); // newFile can be either file, or a new name
}
is a straightforward implementation using , which is ideal for writing without using much RAM. However, it has poor concurrent performance (multiple threads will bottleneck) as it synchronizes when multiple threads read from the same file. uses 's positional seeking, which makes it slightly less efficient than using during reading, with similar write performance. uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit runtime, or you are running on a 32 bit runtime but your index sizes are small enough to fit into the virtual memory space.
var result = Lock.With.NewAnonymous<string>(
@lock: directory.MakeLock("my.lock"),
lockWaitTimeout: Lock.LOCK_OBTAIN_WAIT_FOREVER,
doBody: () =>
{
//... code to execute while locked ...
return "the result";
}).Run();
var result = Lock.With.NewAnonymous<string>(
@lock: directory.MakeLock("my.lock"),
lockWaitTimeout: Lock.LOCK_OBTAIN_WAIT_FOREVER,
doBody: () =>
{
//... code to execute while locked ...
return "the result";
}).Run();
using Directory fsDir = FSDirectory.Open(new DirectoryInfo("/path/to/index"));
using NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0);
IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_48, analyzer);
using IndexWriter writer = new IndexWriter(cachedFSDir, conf);
default as the second parameter.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the virtual machine.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
The file denoted by the returned abstract pathname did not exist before this method was invoked, and Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the application.
[WritableArray, SuppressMessage("Microsoft.Performance", "CA1819", Justification = "Lucene's design requires some writable array properties")]
If the type is , the comparer returned uses the to make the comparison to ensure that the current culture doesn't affect the results. This is the default string comparison used in Java, and what Lucene's design depends on. If the type implements , the comparer uses for the comparison. This allows the use of types with custom comparison schemes. If neither of the above conditions are true, will default to .
if =true: "AttributeClass.Key=value,AttributeClass.Key=value" if =false: "key=value,key=value"
public void ReflectWith(IAttributeReflector reflector)
{
reflector.Reflect(typeof(ICharTermAttribute), "term", GetTerm());
reflector.Reflect(typeof(IPositionIncrementAttribute), "positionIncrement", GetPositionIncrement());
}
public String ToString()
{
return "start=" + startOffset + ",end=" + endOffset;
}
This method may be overridden by subclasses.
I from the prefix.
if =true: "AttributeClass.Key=value,AttributeClass.Key=value" if =false: "key=value,key=value"
An automaton is either represented explicitly (with and objects) or with a singleton string (see and ) in case the automaton is known to accept exactly one string. (Implicitly, all states and transitions of an automaton are reachable from its initial state.) Automata are always reduced (see ) and have no transitions to dead states (see ). If an automaton is nondeterministic, then returns false (but the converse is not required).Automata provided as input to operations are generally assumed to be disjoint.
They have an identical number of outgoing transitions, labeled with the same labels. Corresponding outgoing transitions lead to the same states (to states with an identical right-language).
They are deterministic (DFA). There are no transitions to dead states. They are not minimal (some transitions could be combined).
The number of states: The set of final states: The transition function: Minimal boundary function:
-
regexp ::= unionexp -
| -
unionexp ::= interexp | unionexp (union) -
| interexp -
interexp ::= concatexp & interexp (intersection) [OPTIONAL] -
| concatexp -
concatexp ::= repeatexp concatexp (concatenation) -
| repeatexp -
repeatexp ::= repeatexp ? (zero or one occurrence) -
| repeatexp * (zero or more occurrences) -
| repeatexp + (one or more occurrences) -
| repeatexp {n} (n occurrences) -
| repeatexp {n,} (n or more occurrences) -
| repeatexp {n,m} (n to m occurrences, including both) -
| complexp -
complexp ::= ~ complexp (complement) [OPTIONAL] -
| charclassexp -
charclassexp ::= [ charclasses ] (character class) -
| [^ charclasses ] (negated character class) -
| simpleexp -
charclasses ::= charclass charclasses -
| charclass -
charclass ::= charexp - charexp (character range, including end-points) -
| charexp -
simpleexp ::= charexp -
| . (any single character) -
| # (the empty language) [OPTIONAL] -
| @ (any string) [OPTIONAL] -
| " <Unicode string without double-quotes> " (a string) -
| ( ) (the empty string) -
| ( unionexp ) (precedence override) -
| < <identifier> > (named automaton) [OPTIONAL] -
| <n-m> (numerical interval) [OPTIONAL] -
charexp ::= <Unicode character> (a single non-reserved character) -
| \ <Unicode character> (a single character)
def bits2int(val):
arr=0
for shift in range(8,0,-1):
if val & 0x80:
arr = (arr << 4) | shift
val = val << 1
return arr
def int_table():
tbl = [ hex(bits2int(val)).strip('L') for val in range(256) ]
return ','.join(tbl)
(i >>> (4 * n)) & 0x0F is the offset of the n-th set bit of
the given byte plus one, or 0 if there are n or less bits set in the given
byte. For example bitList(12) returns 0x43:
0x43 & 0x0Fis 3, meaning the the first bit set is at offset 3-1 = 2,(0x43 >>> 4) & 0x0Fis 4, meaning there is a second bit set at offset 4-1=3,(0x43 >>> 8) & 0x0Fis 0, meaning there is no more bit set in this byte.
algorithm 1: , count of set bits in a algorithm 2: , selection of a set bit in a , bytewise signed smaller <8 operator: . shortwise signed smaller <16 operator: . some of the Lk and Hk constants that are used by the above: L8 , H8 , L9 , L16 and H16 .
var iter = BytesRefEnumerator.EMPTY;
var iter = BytesRefIterator.EMPTY;
using (TextWriter sw = new StreamWriter("out.dot"))
{
Util.ToDot(fst, sw, true, true);
}
and then, from command line:
dot -Tpng -o out.png out.dot
IDisposable resource1 = null, resource2 = null, resource3 = null;
ExpectedException priorE = null;
try
{
resource1 = ...; resource2 = ...; resource3 = ...; // Acquisition may throw ExpectedException
..do..stuff.. // May throw ExpectedException
}
catch (ExpectedException e)
{
priorE = e;
}
finally
{
IOUtils.CloseWhileHandlingException(priorE, resource1, resource2, resource3);
}
IDisposable resource1 = null, resource2 = null, resource3 = null;
ExpectedException priorE = null;
try
{
resource1 = ...; resource2 = ...; resource3 = ...; // Acquisition may throw ExpectedException
..do..stuff.. // May throw ExpectedException
}
catch (ExpectedException e)
{
priorE = e;
}
finally
{
IOUtils.DisposeWhileHandlingException(priorE, resource1, resource2, resource3);
}
If the argument is NaN, then the result is NaN. If the argument is zero, then the result is a zero with the same sign as the argument. If the argument is infinite, then the result is infinity with the same sign as the argument.
If the argument is NaN, then the result is NaN. If the argument is +1, then the result is a zero. If the argument is positive infinity, then the result is positive infinity. If the argument is less than 1, then the result is NaN.
If the argument is NaN, then the result is NaN. If the argument is zero, then the result is a zero with the same sign as the argument. If the argument is +1, then the result is positive infinity. If the argument is -1, then the result is negative infinity. If the argument's absolute value is greater than 1, then the result is NaN.
The behavior is undefined if the iterators are not actually sorted. Null elements are unsupported. If is set to true and if a single iterator contains duplicates then they will not be deduplicated.When elements are deduplicated it is not defined which one is returned. If is set to false then the order in which duplicates are returned isn't defined.
The behavior is undefined if the iterators are not actually sorted. Null elements are unsupported. If is set to true and if a single iterator contains duplicates then they will not be deduplicated.When elements are deduplicated it is not defined which one is returned. If is set to false then the order in which duplicates are returned isn't defined.
(two bytes) length of the following byte array, exactly the above count of bytes for the sequence to be sorted.
Performance Results
Test system: Pentium 4, Sun Java 1.5_06 -server -Xbatch -Xmx64M-
50% full 3.36 3.96 1.44 1.46 1.99 1.58 -
1% full 3.31 3.90 1.04 0.99
-
50% full 2.50 3.50 1.00 1.03 1.12 1.25 -
1% full 2.51 3.49 1.00 1.02
<BLock>BlockCount BlockCount: ⌈ ValueCount / BlockSize ⌉ Block: <Header, (Ints)> Header: <Token, (MinValue)> Token: a byte ( ), first 7 bits are the number of bits per value ( bitsPerValue ). If the 8th bit is 1, then MinValue (see next) is0 , otherwise MinValue and needs to be decodedMinValue: a zigzag-encoded variable-length ( ) whose value should be added to every int from the block to restore the original values Ints: If the number of bits per value is 0 , then there is nothing to decode and all ints are equal to MinValue. Otherwise: BlockSize packed ints () encoded on exactly bitsPerValue bits per value. They are the subtraction of the original values and MinValue
is negative, or is non negative and is negative, or the low bits do not fit in a long[] :(L * numValues / 64) > System.Int32.MaxValue , orthe high bits do not fit in a long[] :(2 * numValues / 64) > System.Int32.MaxValue , orindexInterval < 2 ,the index bits do not fit in a long[] :(numValues / indexInterval * ceil(2log(3 * numValues)) / 64) > System.Int32.MaxValue .
is smaller than an earlier encoded value, or is larger than .
<BLock>BlockCount BlockCount: ⌈ ValueCount / BlockSize ⌉ Block: <Header, (Ints)> Header: <B, A, BitsPerValue> B: the B from f: x → A × x + B using a variable-length ( ) A: the A from f: x → A × x + B encoded using on 4 bytes ( ) BitsPerValue: a variable-length ( ) Ints: if BitsPerValue is 0 , then there is nothing to read and all values perfectly match the result of the function. Otherwise, these are the zigzag-encoded packed () deltas from the expected value (computed from the function) using exaclty BitsPerValue bits per value
(using ), , , .
// Implements ISentinelFactory<T>.Create(PriorityQueue<T>)
var sentinelFactory = new MySentinelFactory<MyObject>();
PriorityQueue<MyObject> pq = new MyQueue<MyObject>(sentinelFactory);
// save the 'top' element, which is guaranteed to not be default .
MyObject pqTop = pq.Top;
<...>
// now in order to add a new element, which is 'better' than top (after
// you've verified it is better), it is as simple as:
pqTop.Change();
pqTop = pq.UpdateTop();
NOTE:
pq.Top.Change();
pq.UpdateTop();
instead of
o = pq.Pop();
o.Change();
pq.Push(o);
pq.Top.Change();
pq.UpdateTop();
instead of
o = pq.Pop();
o.Change();
pq.Push(o);
QueryBuilder builder = new QueryBuilder(analyzer);
Query a = builder.CreateBooleanQuery("body", "just a test");
Query b = builder.CreatePhraseQuery("body", "another test");
Query c = builder.CreateMinShouldMatchQuery("body", "another test", 0.5f);
SentinelIntSet set = ...
foreach (int v in set.keys)
{
if (v == set.EmptyVal)
continue;
//use v...
}
If the argument is or an infinity, then the result is .
If the argument is or its absolute value is greater than 1, then the result is .
The extra amount of memory to perform merges is configurable. This allows small merges to be very fast while large merges will be performed in-place (slightly slower). You can make sure that the fast merge routine will always be used by having maxTempSlots equal to half of the length of the slice of data to sort.Only the fast merge routine can gallop (the one that doesn't run in-place) and it only gallops on the longest slice.
internal static readonly VirtualMethod newMethod =
new VirtualMethod(typeof(BaseClass), "newName", parameters...);
internal static readonly VirtualMethod oldMethod =
new VirtualMethod(typeof(BaseClass), "oldName", parameters...);
bool isDeprecatedMethodOverridden =
oldMethod.GetImplementationDistance(this.GetType()) > newMethod.GetImplementationDistance(this.GetType());
// alternatively (more readable):
bool isDeprecatedMethodOverridden =
VirtualMethod.CompareImplementationDistance(this.GetType(), oldMethod, newMethod) > 0
> 1, iff is overridden/implemented in a subclass of the class overriding/declaring < 1, iff is overridden in a subclass of the class overriding/declaring 0, iff both methods are overridden in the same class (or are not overridden at all)
1 byte 0-n bytes 0-n bytes 0-n bytes
Token encodes whether clean means full of zeros or ones in the first bit, the number of clean words minus 2 on the next 3 bits and the number of dirty words on the last 4 bits. The higher-order bit is a continuation bit, meaning that the number is incomplete and needs additional bytes to be read. Clean length+: If clean length has its higher-order bit set, you need to read a vint ( ), shift it by 3 bits on the left side and add it to the 3 bits which have been read in the token. Dirty length+ works the same way as Clean length+ but on 4 bits and for the length of dirty words. Dirty words are the dirty words, there are Dirty length of them.
catch (Exception ex) when (ex.IsAssertionError())
catch (Exception ex) when (ex.IsError())
catch (Exception ex) when (ex.IsNoClassDefFoundError())
catch (Exception ex) when (ex.IsOutOfMemoryError())
The format of a provider-configuration file violates the specification; An occurs while reading a provider-configuration file; A concrete provider class named in a provider-configuration file cannot be found; A concrete provider class is not a subclass of the service class; A concrete provider class cannot be instantiated; or Some other kind of error occurs.
catch (Exception ex) when (ex.IsServiceConfigurationError())
catch (Exception ex) when (ex.IsStackOverflowError())
catch (Lucene.Net.QueryParsers.Surround.Parser.ParseException e)
The forName method in classClass .The findSystemClass method in classClassLoader .The loadClass method in classClassLoader .
catch (Exception ex) when (ex.IsClassNotFoundException())
catch (Exception ex) when (ex.IsAlreadyClosedException())
Lucene made a custom
catch (Exception ex) when (ex.IsEOFException())
catch (Exception ex) when (ex.IsNoSuchMethodException())
catch (Exception ex) when (ex.IsArrayIndexOutOfBoundsException())
catch (Exception ex) when (ex.IsIllegalArgumentException())
catch (Exception ex) when (ex.IsIllegalStateException())
catch (Exception ex) when (ex.IsIndexOutOfBoundsException())
Calling the instance method of a null object.Accessing or modifying the field of a null object.Taking the length of null as if it were an array.Accessing or modifying the slots of null as if it were an array.Throwing null as if it were a Throwable value.
catch (Exception ex) when (ex.IsNullPointerException())
The static
Integer someInt = new Integer(43);
int primitiveInt = someInt; // Implicit cast by the Java compiler
If
int? someInt = 43;
int primitiveInt = someInt; // Compile error
So, to get the same behavior as in Java (provided the nullable cannot be factored away), the
appropriate translation would be:
int? someInt = 43;
int primitiveInt;
if (someInt.HasValue)
primitiveInt = someInt.Value;
else
throw new NullReferenceException();
However, do note in most cases it would be better to try to refactor so the nullable
(and therefore the exception) isn't required.
catch (Exception ex) when (ex.IsNumberFormatException())
catch (Exception ex) when (ex.IsRuntimeException())
catch (Exception ex) when (ex.IsStringIndexOutOfBoundsException())
catch (Exception ex) when (ex.IsUnsupportedOperationException())