++ **For/PFor/PForDelta** + - **Novel** **"TurboPFor"** (Patched Frame-of-Reference,PFor/PForDelta) scheme with **direct access** or bulk decoding. + Outstanding compression and speed. More efficient than **ANY** other fast "integer compression" scheme. + - Compress 70 times faster and decompress up to 4 times faster than OptPFD + - :new: **(2017) TurboPFor AVX2, now 50%! more faster!!!!** + - :new: **(2017) TurboPFor Hybrid, better compression and more faster** +
++ **Bit Packing** + - :sparkles: Fastest and most efficient **"SIMD Bit Packing"** + - :new: **(2017) TurboPack AVX2, now more faster. Decoding 10 Billions intergers/seconds** + - Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios + - Bit Packing with **Direct/Random Access** without decompressing entire blocks + - Access any single bit packed entry with **zero decompression** + - :sparkles: **Direct Update** of individual bit packed entries + - Reducing **Cache Pollution** +
++ **Variable byte** + - :sparkles: Scalar **"Variable Byte"** faster and more efficient than **ANY** other (incl. SIMD MaskedVByte) implementation + - :new: **(2017) new scheme w. better compression and 30% more faster** +
++ **Simple family** + - :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b + or other "simple family" implementation +
++ **Elias fano** + - :sparkles: Fastest **"Elias Fano"** implementation w/ or w/o SIMD +
++ **Transform** + - :sparkles: Scalar & SIMD Transform: Delta, Zigzag, Transpose/Shuffle, Floating point<->Integer +
++ **Inverted Index ...do less, go fast!**
+ - Direct Access to compressed *frequency* and *position* data in inverted index with zero decompression
+ - :sparkles: **Novel** **"Intersection w/ skip intervals"**, decompress the minimum necessary blocks (~10-15%)!.
+ - **Novel** Implicit skips with zero extra overhead
+ - **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression".
+ - more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core
+ - :sparkles: Revolutionary Parallel Query Processing on Multicores w/ more than **7000!!! queries/sec** on a quad core PC.
+ **...forget** ~~Map Reduce, Hadoop, multi-node clusters,~~ ...
+
+### Integer Compression Benchmark:
+- Realistic and practical "integer compression" benchmark with **large** integer arrays.
+- No **PURE** cache benchmark
+
+##### - Synthetic data (2017):
+ - Generate and test (zipfian) skewed distribution (100.000.000 integers, Block size=128/256)
+ Note: Unlike general purpose compression, a small fixed size (ex. 128 integers) is in general used in "integer compression".
+ Large blocks involved, while processing queries (inverted index, search engines, databases, graphs, in memory computing,...) need to be entirely decoded
+
+ ./icbench -a1.5 -m0 -M255 -n100M ZIPF
+
+CPU: Skylake i7-6700 w/ only 3.7GHz gcc 6.2 single thread
+
+
+|C Size|ratio%|Bits/Integer|C MI/s|D MI/s|Name 128/256 |
+|--------:|-----:|--------:|----------:|----------:|--------------|
+|62939886| 15.7| 5.04|**397**|**2311**|**TurboPFor256**|
+|63392759| 15.8| 5.07|330|1608|**TurboPFor**|
+|63392801| 15.8| 5.07|326|231|**TurboPForDA**|
+|65060504| 16.3| 5.20|15|687|[FP.SIMDOptPFor](#FastPFor)|
+|65359916|16.3| 5.23| 8| 609|PC.OptPFD|
+|73477088|18.4| 5.88|102| 621|PC.Simple16|
+|73481096| 18.4| 5.88|156|2187|[FP.SimdFastPFor](#FastPFor) 64k *|
+|76345136| 19.1| 6.11|245|653|**VSimple**|
+|95915096|24.0| 7.67| 212|958|Simple-8b|
+|91956582| 25.5| 8.15|65|2141|[QMX](#QMX) 64k *|
+|99910930| 25.0| 7.99|**3290**|**2968.35**|**TurboPackV**|
+|99910930| 25.0| 7.99|2122|2347|**TurboPack**|
+|99910930| 25.0| 7.99|2105|2219|**TurboFor**|
+|100332929| 25.1| 8.03|**3580**|**2998.17**|**TurboPack256V**|
+|101015650| 25.3| 8.08|2380|2371|**TurboVByte**|
+|102074663| 25.5| 8.17|1428|1979|[MaskedVByte](#MaskedVByte)|
+|102074663| 25.5| 8.17|565|1052|[PC.Vbyte](#PolyCom)|
+|102083036| 25.5| 8.17|1300|1067|[FP.VByte](#FastPFor)|
+|112500000| 28.1| 9.00|382|**3035**|[VarintG8IU](#VarintG8IU)|
+|125000000| 31.2|10.00|1111|2948|[StreamVbyte](#StreamVByte)|
+|400000000| 100.00| 32.00| 2240|2237|Copy|
+| | | | N/A | N/A |EliasFano|
+
+(*) inefficent for small block size
+
+MI/s: 1.000.000 integers/second. **1000 MI/s = 4 GB/s**
+**#BOLD** = pareto frontier.
+FP=FastPFor SC:simdcomp PC:Polycom
+TurboPForDA,TurboForDA: Direct Access is normally used when accessing few individual values.
+
+------------------------------------------------------------------------
+##### - Data files:
+ - gov2.sorted from [DocId data set](#DocId data set) Block size=128 (lz4+blosc+VSimple w/ 64Ki)
+
+
+ ./icbench -fS -r gov2.sorted
+
+|Size |Ratio %|Bits/Integer|C Time MI/s|D Time MI/s|Function |
+|-----------:|------:|-----:|-------:|-------:|---------------------|
+| 3.321.663.893| 13.9| 4.44|**328**|**1452**|**TurboPFor**|
+| 3.337.758.854| 14.0| 4.47| 5| 513|PC.OptPFD|
+| 3.350.717.959| 14.0| 4.48|**365**|**1744**|**TurboPFor256**|
+| 3.501.671.314| 14.6| 4.68| 322| 827|**VSimple**|
+| 3.766.174.764| 15.8| 5.04|**618**| 712|**EliasFano**|
+| 3.820.190.182| 16.0| 5.11| 119| 650|PC.Simple16|
+| 4.521.326.518| 18.9| 6.05| 209| 824|Simple-8b|
+| 4.649.671.427| 19.4| 6.22|**771**|962|**TurboVbyte**|
+| 4.953.768.342| 20.7| 6.63|1419|1513|**TurboPack**|
+| 4.955.740.045| 20.7| 6.63|**1766**|**2567**|**TurboPackV**|
+| 5.203.353.057| 21.8| 6.96|1560|1807|FP.SIMDPackD1|
+| 5.393.769.503| 22.5| 7.21|**2260.63**|**2715.30**|**TurboPackV256**|
+| 6.221.886.390| 26.0| 8.32|1667|1738|**TurboFor**|
+| 6.221.886.390| 26.0| 8.32|1661| 565|**TurboForDA**|
+| 6.699.519.000| 28.0| 8.96| 472| 495|FP.Vbyte|
+| 6.700.989.563| 28.0| 8.96| 729| 992|MaskedVByte|
+| 7.622.896.878| 31.9|10.20| 209|1198|VarintG8IU|
+| 8.594.342.216| 35.9|11.50|1307|1594|libfor|
+|23.918.861.764|100.0|32.00|1456|1481|Copy|
+
+
+|Size |Ratio %|Bits/Integer|C Time MI/s|D Time MI/s|Function |
+|----------:|-----:|----:|------:|------:|---------------------|
+| 3.214.763.689| 13.44| 4.30| 339.90| 837.69|**VSimple 64Ki**|
+| 3.958.888.197| 16.55| 5.30| 279.19| 618.60|[lz4](#lz4)+DT 64Ki|
+| 6.074.995.117| 25.40| 8.13| 494.70| 729.97|[blosc_lz4](#blosc) 64Ki|
+| 8.773.150.644| 36.68|11.74| 637.83|1301.05|blosc_lz 64Ki|
+
+
+Ki=1024 Integers. 64Ki = 256k bytes
+"lz4+DT 64Ki" = Delta+Transpose from TurboPFor + lz4
+"blosc_lz4" tested w/ lz4 compressor+vectorized shuffle
+
+##### - Compressed Inverted Index Intersections with GOV2
+ GOV2: 426GB, 25 Millions documents, average doc. size=18k.
+
+ + Aol query log: 18.000 queries
+ **~1300** queries per second (single core)
+ **~5000** queries per second (quad core)
+ Ratio = 14.37% Decoded/Total Integers.
+
+ + TREC Million Query Track (1MQT):
+ **~1100** queries per second (Single core)
+ **~4500** queries per second (Quad core CPU)
+ Ratio = 11.59% Decoded/Total Integers.
+
+- Benchmarking intersections (Single core, AOL query log)
+
+| max.docid/q|Time s| q/s | ms/q | % docid found|
+|-----------------:|---:|----:|-----:|-------:|
+|1.000|7.88|2283.1|0.438|81|
+|10.000|10.54|1708.5|0.585|84|
+| ALL |13.96|1289.0|0.776|100|
+q/s: queries/second, ms/q:milliseconds/query
+
+- Benchmarking Parallel Query Processing (Quad core, AOL query log)
+
+| max.docid/q|Time s| q/s | ms/q | % docids found|
+|-----------------:|----:|----:|-----:|-------:|
+|1.000|2.66|6772.6|0.148|81|
+|10.000|3.39|5307.5|0.188|84|
+|ALL|3.57|5036.5|0.199|100|
+
+###### Notes:
+- Search engines are spending 90% of the time in intersections when processing queries.
+- Most search engines are using pruning strategies, caching popular queries,... to reduce the time for intersections and query processing.
+- As indication, google is processing [40.000 Queries per seconds](http://www.internetlivestats.com/google-search-statistics/),
+using [900.000 multicore servers](https://www.cloudyn.com/blog/10-facts-didnt-know-server-farms/) for searching [8 billions web pages](http://searchenginewatch.com/sew/study/2063479/coincidentally-googles-index-size-jumps) (320 X size of GOV2).
+- Recent "integer compression" GOV2 experiments (best paper at ECIR 2014) [On Inverted Index Compression for Search Engine Efficiency](http://www.dcs.gla.ac.uk/~craigm/publications/catena14compression.pdf) using 8-core Xeon PC are reporting 1.2 seconds per query (for 1.000 Top-k docids).
+
+### Compile:
+
+ git clone --recursive git://github.com/powturbo/TurboPFor.git
+ cd TurboPFor
+ make
+
+ or
+
+ make AVX2=1
+
+### Testing:
+##### - Synthetic data (use ZIPF parameter):
+ + benchmark groups of "integer compression" functions
+
+
+ ./icbench -eBENCH -a1.2 -m0 -M255 -n100M ZIPF
+ ./icbench -eBITPACK/VBYTE -a1.2 -m0 -M255 -n100M ZIPF
+
+ >*Type "icbench -l1" for a list*
+
+
+ >*-zipfian distribution alpha = 1.2 (Ex. -a1.0=uniform -a1.5=skewed distribution)
+ -number of integers = 100.000.000
+ -integer range from 0 to 255
*
+
+ + individual function test (ex. Copy TurboPack TurboPFor)
+
+
+ ./icbench -a1.5 -m0 -M255 -ecopy/turbopack/turbopfor/turbopack256v ZIPF
+
+
+##### - Data files:
+ - Raw 32 bits binary data file (see option "-f" for other formats)
+
+
+ ./icbench file
+
+
+
+ - Multiblock sorted (unique) data file like gov2 file from [DocId data set](#DocId data set)
+
+
+ ./icbench -fS -r gov2.sorted
+
+
+##### - Intersections:
+ 1 - Download Gov2 (or ClueWeb09) + query files (Ex. "1mq.txt") from [DocId data set](#DocId data set)
+ 8GB RAM required (16GB recommended for benchmarking "clueweb09" files).
+
+ 2 - Create index file
+
+
+ ./idxcr gov2.sorted .
+
+
+ >*create inverted index file "gov2.sorted.i" in the current directory*
+
+ 3 - Test intersections
+
+
+ ./idxqry gov2.sorted.i 1mq.txt
+
+
+ >*run queries in file "1mq.txt" over the index of gov2 file*
+
+##### - Parallel Query Processing:
+ 1 - Create partitions
+
+
+ ./idxseg gov2.sorted . -26m -s8
+
+
+ >*create 8 (CPU hardware threads) partitions for a total of ~26 millions document ids*
+
+ 2 - Create index file for each partition
+
+
+ ./idxcr gov2.sorted.s*
+
+
+ >*create inverted index file for all partitions "gov2.sorted.s00 - gov2.sorted.s07" in the current directory*
+
+ 3 - Intersections:
+
+ delete "idxqry.o" file and then type "make para" to compile "idxqry" w. multithreading
+
+
+ ./idxqry gov2.sorted.s*.i 1mq.txt
+
+ >*run queries in file "1mq.txt" over the index of all gov2 partitions "gov2.sorted.s00.i - gov2.sorted.s07.i".*
+
+### Function usage:
+See benchmark "icbench" program for "integer compression" usage examples.
+In general encoding/decoding functions are of the form:
+
+
+ >**char *endptr = encode( unsigned *in, unsigned n, char *out, [unsigned start], [int b])**
+ endptr : set by encode to the next character in "out" after the encoded buffer
+ in : input integer array
+ n : number of elements
+ out : pointer to output buffer
+ b : number of bits. Only for bit packing functions
+ start : previous value. Only for integrated delta encoding functions
+
+
+
+ >**char *endptr = decode( char *in, unsigned n, unsigned *out, [unsigned start], [int b])**
+ endptr : set by decode to the next character in "in" after the decoded buffer
+ in : pointer to input buffer
+ n : number of elements
+ out : output integer array
+ b : number of bits. Only for bit unpacking functions
+ start : previous value. Only for integrated delta decoding functions
+
+header files to use with documentation:
+
+| header file|Integer Compression functions|
+|------------|-----------------------------|
+|vint.h|variable byte|
+|vsimple.h|variable simple|
+|vp4c.h, vp4d.h|TurboPFor|
+|bitpack.h bitunpack.h|Bit Packing, For, +Direct Access|
+|eliasfano.h|Elias Fano|
+
+### Environment:
+###### OS/Compiler (64 bits):
+- Linux: GNU GCC (>=4.6)
+- clang (>=3.2)
+- Windows: MinGW-w64 (no parallel query processing)
+
+###### Multithreading:
+- All TurboPFor integer compression functions are thread safe
+
+### References:
+
+ + [FastPFor](https://github.com/lemire/FastPFor) + [Simdcomp](https://github.com/lemire/simdcomp): SIMDPack FPF, Vbyte FPF, VarintG8IU, StreamVbyte
+ + [Optimized Pfor-delta compression code](http://jinruhe.com): OptPFD/OptP4, Simple16 (limited to 28 bits integers)
+ + [MaskedVByte](http://maskedvbyte.org/). See also: [Vectorized VByte Decoding](http://engineering.indeed.com/blog/2015/03/vectorized-vbyte-decoding-high-performance-vector-instructions/)
+ + [Streamvbyte](https://github.com/lemire/streamvbyte).
+ + [Index Compression Using 64-Bit Words](http://people.eng.unimelb.edu.au/ammoffat/abstracts/am10spe.html): Simple-8b (speed optimized version tested)
+ + [libfor](https://github.com/cruppstahl/for)
+ + [Compression, SIMD, and Postings Lists](http://www.cs.otago.ac.nz/homepages/andrew/papers/) QMX integer compression from the "simple family"
+ + [lz4](https://github.com/Cyan4973/lz4). included w. block size 64K as indication. Tested after preprocessing w. delta+transpose
+ + [blosc](https://github.com/Blosc/c-blosc). blosc is like transpose/shuffle+lz77. Tested blosc+lz4 and blosclz incl. vectorizeed shuffle.
+ + [Document identifier data set](http://lemire.me/data/integercompression2014.html)
+
+ + **Integer compression publications:**
+ - [In Vacuo and In Situ Evaluation of SIMD Codecs (TurboPackV,TurboPFor/QMX)](http://dl.acm.org/citation.cfm?id=3015023) + [paper](http://www.cs.otago.ac.nz/homepages/andrew/papers/)
+ - [SIMD Compression and the Intersection of Sorted Integers](http://arxiv.org/abs/1401.6399)
+ - [Partitioned Elias-Fano Indexes](http://www.di.unipi.it/~ottavian/files/elias_fano_sigir14.pdf)
+ - [On Inverted Index Compression for Search Engine Efficiency](http://www.dcs.gla.ac.uk/~craigm/publications/catena14compression.pdf)
+ - [Google's Group Varint Encoding](http://static.googleusercontent.com/media/research.google.com/de//people/jeff/WSDM09-keynote.pdf)
+
+ + **Applications:**
+ - [Graph500](https://github.com/julianromera/graph500)
+ - [Small Polygon Compression](http://abhinavjauhri.com/publications/dcc_poster_2016.pdf) + [code](https://github.com/ajauhri/bignum_compression)
+ - [Parallel Graph Analysis (Lecture 18)](http://www.cs.rpi.edu/~slotag/classes/FA16/) + [code](http://www.cs.rpi.edu/~slotag/classes/FA16/handson/lec18-comp2.cpp)
+
+Last update: 03 JAN 2017