hcq/TurboPFor-Integer-Compression
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

.

Browse Source
This commit is contained in:
powturbo
2017-06-13 09:15:24 +02:00
parent a30c39fb1e
commit ec82948657
1 changed files with 13 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@ -7,19 +7,16 @@ TurboPFor: Fastest Integer Compression [![Build Status](https://travis-ci.org/po
* No other "Integer Compression" compress/decompress faster * No other "Integer Compression" compress/decompress faster
* :sparkles: Direct Access, **integrated** (SIMD/AVX2) FOR/delta/Zigzag for sorted/unsorted arrays * :sparkles: Direct Access, **integrated** (SIMD/AVX2) FOR/delta/Zigzag for sorted/unsorted arrays
* **For/PFor/PForDelta** * **For/PFor/PForDelta**
* **Novel** **"TurboPFor"** (PFor/PForDelta) scheme w./ **direct access**. * **Novel TurboPFor** (PFor/PForDelta) scheme w./ **direct access** + **SIMD/AVX2**.
* Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme. * Outstanding compression/speed. More efficient than **ANY** other fast "integer compression" scheme.
* Compress 70 times faster and decompress up to 4 times faster than OptPFD * Compress 70 times faster and decompress up to 4 times faster than OptPFD
* :new: **TurboPFor AVX2, now 50%! more faster!!!!**
* :new: **TurboPFor Hybrid, better compression and more faster** * :new: **TurboPFor Hybrid, better compression and more faster**
* **Bit Packing** * **Bit Packing**
* :sparkles: Fastest and most efficient **"SIMD Bit Packing"** * :sparkles: Fastest and most efficient **"SIMD Bit Packing"** **10 Billions integers/sec (40Gb/s)**
* :new: TurboPack AVX2 more faster. **Decoding 10 Billions intergers/seconds (40Gb/s)** * Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios
* :new: more faster. Scalar **"Bit Packing"** decoding as fast as SIMD-Packing in realistic (No "pure cache") scenarios
* **Direct/Random Access** : Access any single bit packed entry with **zero decompression** * **Direct/Random Access** : Access any single bit packed entry with **zero decompression**
* **Variable byte** * **Variable byte**
* :sparkles: Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation * :sparkles: Scalar **"Variable Byte"** faster than **ANY** other (incl. SIMD) implementation
* :new: **new scheme : better compression and 30% faster**
* **Simple family** * **Simple family**
* :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b * :sparkles: **Novel** **"Variable Simple"** (incl. **RLE**) faster and more efficient than simple16, simple-8b
* **Elias fano** * **Elias fano**
@ -35,7 +32,7 @@ TurboPFor: Fastest Integer Compression [![Build Status](https://travis-ci.org/po
* **Novel** Implicit skips with zero extra overhead * **Novel** Implicit skips with zero extra overhead
* **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression". * **Novel** Efficient **Bidirectional** Inverted Index Architecture (forward/backwards traversal) incl. "integer compression".
* more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core * more than **2000! queries per second** on GOV2 dataset (25 millions documents) on a **SINGLE** core
* :sparkles: Revolutionary Parallel Query Processing on Multicores w/ more than **7000!!! queries/sec** on a simple quad core PC.<br> * :sparkles: Revolutionary Parallel Query Processing on Multicores **> 7000!!! queries/sec** on a simple quad core PC.<br>
**...forget** ~~Map Reduce, Hadoop, multi-node clusters,~~ ... **...forget** ~~Map Reduce, Hadoop, multi-node clusters,~~ ...
### Integer Compression Benchmark: ### Integer Compression Benchmark:
@ -180,6 +177,8 @@ using [900.000 multicore servers](https://www.cloudyn.com/blog/10-facts-didnt-kn
git clone --recursive git://github.com/powturbo/TurboPFor.git git clone --recursive git://github.com/powturbo/TurboPFor.git
cd TurboPFor cd TurboPFor
###### Linux, Windows (MingW), Clang,...
make make
or or
@ -189,6 +188,11 @@ using [900.000 multicore servers](https://www.cloudyn.com/blog/10-facts-didnt-kn
Minimum build w/ TurboPFor scalar functions Minimum build w/ TurboPFor scalar functions
make NSIMD=1 make NSIMD=1
###### Windows visual c++
nmake /f makefile.vs
### Testing: ### Testing:
##### - Synthetic data (use ZIPF parameter): ##### - Synthetic data (use ZIPF parameter):
+ benchmark groups of "integer compression" functions <br /> + benchmark groups of "integer compression" functions <br />
@ -346,7 +350,7 @@ header files to use with documentation:<br />
###### OS/Compiler (64 bits): ###### OS/Compiler (64 bits):
- Linux: GNU GCC (>=4.6) - Linux: GNU GCC (>=4.6)
- clang (>=3.2) - clang (>=3.2)
- Windows: MinGW-w64 (no parallel query processing) - Windows: MinGW-w64 + Visual c++ (no parallel query processing app)
###### Multithreading: ###### Multithreading:
- All TurboPFor integer compression functions are thread safe - All TurboPFor integer compression functions are thread safe
@ -377,5 +381,5 @@ header files to use with documentation:<br />
* [Small Polygon Compression](https://arxiv.org/abs/1509.05505) + [Poster](http://abhinavjauhri.me/publications/dcc_poster_2016.pdf) + [code](https://github.com/ajauhri/bignum_compression) * [Small Polygon Compression](https://arxiv.org/abs/1509.05505) + [Poster](http://abhinavjauhri.me/publications/dcc_poster_2016.pdf) + [code](https://github.com/ajauhri/bignum_compression)
* [Parallel Graph Analysis (Lecture 18)](http://www.cs.rpi.edu/~slotag/classes/FA16/) + [code](http://www.cs.rpi.edu/~slotag/classes/FA16/handson/lec18-comp2.cpp) * [Parallel Graph Analysis (Lecture 18)](http://www.cs.rpi.edu/~slotag/classes/FA16/) + [code](http://www.cs.rpi.edu/~slotag/classes/FA16/handson/lec18-comp2.cpp)
Last update: 19 MAR 2017 Last update: 13 JUN 2017