hcq/TurboPFor-Integer-Compression
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

.

Browse Source
This commit is contained in:
x
2018-01-14 09:12:51 +01:00
parent d26400f28f
commit 6727850dda
1 changed files with 9 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
README.md
View File

@ -12,23 +12,23 @@ TurboPFor: Fastest Integer Compression [![Build Status](https://travis-ci.org/po
* Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme. * Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme.
* Compress 70 times faster and decompress up to 4 times faster than OptPFD * Compress 70 times faster and decompress up to 4 times faster than OptPFD
* **Bit Packing** * **Bit Packing**
* :sparkles: Fastest and most efficient **"SIMD Bit Packing"** **10 Billions integers/sec (40Gb/s)** * Fastest and most efficient **"SIMD Bit Packing"** **10 Billions integers/sec (40Gb/s!)**
* Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios * Scalar **"Bit Packing"** decoding nearly as fast as SIMD-Packing in realistic (No "pure cache") scenarios
* **Direct/Random Access** : Access any single bit packed entry with **zero decompression** * **Direct/Random Access** : Access any single bit packed entry with **zero decompression**
* **Variable byte** * **Variable byte**
* :sparkles: Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation * Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation
* **Simple family** * **Simple family**
* :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b * **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b
* **Elias fano** * **Elias fano**
* :sparkles: Fastest **"Elias Fano"** implementation w/ or w/o SIMD/AVX2 * Fastest **"Elias Fano"** implementation w/ or w/o SIMD/AVX2
+ **Transform** + **Transform**
* :sparkles: Scalar & SIMD Transform: Delta, Zigzag, Transpose/Shuffle * Scalar & SIMD Transform: Delta, Zigzag, Transpose/Shuffle
* **Floating Point Compression** * **Floating Point Compression**
* :sparkles: (Differential) Finite Context Method FCM/DFCM floating point compression * (Differential) Finite Context Method FCM/DFCM floating point compression
* Using **TurboPFor**, more than 2 GB/s throughput * Using **TurboPFor**, more than 2 GB/s throughput
* **Inverted Index ...do less, go fast!** * **Inverted Index ...do less, go fast!**
* Direct Access to compressed *frequency* and *position* data w/ zero decompression * Direct Access to compressed *frequency* and *position* data w/ zero decompression
* :sparkles: **Novel** **"Intersection w/ skip intervals"**, decompress the minimum necessary blocks (**~10-15%)!**. * **Novel** **"Intersection w/ skip intervals"**, decompress the minimum necessary blocks (**~10-15%)!**.
* **Novel** Implicit skips with zero extra overhead * **Novel** Implicit skips with zero extra overhead
* **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression". * **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression".
* more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core * more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core
@ -385,5 +385,5 @@ header files to use with documentation:<br />
* [Small Polygon Compression](https://arxiv.org/abs/1509.05505) + [Poster](http://abhinavjauhri.me/publications/dcc_poster_2016.pdf) + [code](https://github.com/ajauhri/bignum_compression) * [Small Polygon Compression](https://arxiv.org/abs/1509.05505) + [Poster](http://abhinavjauhri.me/publications/dcc_poster_2016.pdf) + [code](https://github.com/ajauhri/bignum_compression)
* [Parallel Graph Analysis (Lecture 18)](http://www.cs.rpi.edu/~slotag/classes/FA16/) + [code](http://www.cs.rpi.edu/~slotag/classes/FA16/handson/lec18-comp2.cpp) * [Parallel Graph Analysis (Lecture 18)](http://www.cs.rpi.edu/~slotag/classes/FA16/) + [code](http://www.cs.rpi.edu/~slotag/classes/FA16/handson/lec18-comp2.cpp)
Last update: 13 Jan 2018 Last update: 14 Jan 2018