daiduo/TurboPFor-Integer-Compression
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Readme

Browse Source
This commit is contained in:
x
2017-03-19 13:30:06 +01:00
parent 88b34d96a9
commit d13b91c839
1 changed files with 35 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
README.md
View File

@ -1,49 +1,43 @@
TurboPFor: Fastest Integer Compression [![Build Status](https://travis-ci.org/powturbo/TurboPFor.svg?branch=master)](https://travis-ci.org/powturbo/TurboPFor) TurboPFor: Fastest Integer Compression [![Build Status](https://travis-ci.org/powturbo/TurboPFor.svg?branch=master)](https://travis-ci.org/powturbo/TurboPFor)
====================================== ======================================
+ **TurboPFor: The new synonym for "integer compression"** * **TurboPFor: The new synonym for "integer compression"**
- 100% C (C++ headers), as simple as memcpy * 100% C (C++ headers), as simple as memcpy
- :+1: **Java** Critical Natives/JNI. Access TurboPFor **incl. SIMD/AVX2!** from Java as fast as calling from C * :+1: **Java** Critical Natives/JNI. Access TurboPFor **incl. SIMD/AVX2!** from Java as fast as calling from C
- :sparkles: **FULL** range 8/16/32/64 bits lists * :sparkles: **FULL** range 8/16/32/64 bits lists
- No other "Integer Compression" compress/decompress faster * No other "Integer Compression" compress/decompress faster
- :sparkles: Direct Access, **integrated** (SIMD/AVX2) FOR/delta/Zigzag for sorted/unsorted arrays * :sparkles: Direct Access, **integrated** (SIMD/AVX2) FOR/delta/Zigzag for sorted/unsorted arrays
<p> * **For/PFor/PForDelta**
+ **For/PFor/PForDelta** * **Novel** **"TurboPFor"** (PFor/PForDelta) scheme w./ **direct access**.
- **Novel** **"TurboPFor"** (PFor/PForDelta) scheme w./ **direct access**. * Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme.
- Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme. * Compress 70 times faster and decompress up to 4 times faster than OptPFD
- Compress 70 times faster and decompress up to 4 times faster than OptPFD * :new: **TurboPFor AVX2, now 50%! more faster!!!!**
- :new: **TurboPFor AVX2, now 50%! more faster!!!!** * :new: **TurboPFor Hybrid, better compression and more faster**
- :new: **TurboPFor Hybrid, better compression and more faster** * **Bit Packing**
<p> * :sparkles: Fastest and most efficient **"SIMD Bit Packing"**
+ **Bit Packing** * :new: TurboPack AVX2 more faster. **Decoding 10 Billions intergers/seconds (40Gb/s)**
- :sparkles: Fastest and most efficient **"SIMD Bit Packing"** * :new: more faster. Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios
- :new: TurboPack AVX2 more faster. **Decoding 10 Billions intergers/seconds (40Gb/s)** * **Direct/Random Access** : Access any single bit packed entry with **zero decompression**
- :new: more faster. Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios * **Variable byte**
- **Direct/Random Access** : Access any single bit packed entry with **zero decompression** * :sparkles: Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation
<p> * :new: **new scheme : better compression and 30% faster**
+ **Variable byte** * **Simple family**
- :sparkles: Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation * :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b
- :new: **new scheme : better compression and 30% faster** * **Elias fano**
<p> * :sparkles: Fastest **"Elias Fano"** implementation w/ or w/o SIMD/AVX2
+ **Simple family**
- :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b
<p>
+ **Elias fano**
- :sparkles: Fastest **"Elias Fano"** implementation w/ or w/o SIMD/AVX2
<p>
+ **Transform** + **Transform**
- :sparkles: Scalar & SIMD Transform: Delta, Zigzag, Transpose/Shuffle * :sparkles: Scalar & SIMD Transform: Delta, Zigzag, Transpose/Shuffle
<p> <p>
+ **Floating Point Compression** * **Floating Point Compression**
- :new: (Differential) Finite Context Method FCM/DFCM floating point compression * :new: (Differential) Finite Context Method FCM/DFCM floating point compression
- Using **TurboPFor**, more than 2 GB/s throughput * Using **TurboPFor**, more than 2 GB/s throughput
<p> <p>
+ **Inverted Index ...do less, go fast!** * **Inverted Index ...do less, go fast!**
- Direct Access to compressed *frequency* and *position* data w/ zero decompression * Direct Access to compressed *frequency* and *position* data w/ zero decompression
- :sparkles: **Novel** **"Intersection w/ skip intervals"**, decompress the minimum necessary blocks (**~10-15%)!**. * :sparkles: **Novel** **"Intersection w/ skip intervals"**, decompress the minimum necessary blocks (**~10-15%)!**.
- **Novel** Implicit skips with zero extra overhead * **Novel** Implicit skips with zero extra overhead
- **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression". * **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression".
- more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core * more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core
- :sparkles: Revolutionary Parallel Query Processing on Multicores w/ more than **7000!!! queries/sec** on a simple quad core PC.<br> * :sparkles: Revolutionary Parallel Query Processing on Multicores w/ more than **7000!!! queries/sec** on a simple quad core PC.<br>
**...forget** ~~Map Reduce, Hadoop, multi-node clusters,~~ ... **...forget** ~~Map Reduce, Hadoop, multi-node clusters,~~ ...
### Integer Compression Benchmark: ### Integer Compression Benchmark: