daiduo/TurboPFor-Integer-Compression
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

🆕 Java+64 bits lists for BitPacking, VSimple, VByte, Elias Fano,…

Browse Source
This commit is contained in:
powturbo
2015-06-21 19:06:14 +02:00
parent 62863bc6fe
commit a73275a7cc
1 changed files with 5 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@ -71,7 +71,7 @@ MI/s: 1.000.000 integers/second. 1000 MI/s = 4 GB/s<br>
TurboPForDA,TurboForDA: Direct Access is normally used when accessing individual values.
##### - Data files:
- gov2.sorted from [DocId data set](#DocId data set) Block size=128 (lz4+VSimple 64k)
- gov2.sorted from [DocId data set](#DocId data set) Block size=128 (lz4+blosc+VSimple w/ 64k)
./icbench -c1 gov2.sorted
@ -90,7 +90,7 @@ TurboPForDA,TurboForDA: Direct Access is normally used when accessing individual
| 4.953.768.342| 20.71| 6.63|**1766.05**|**1943.87**|**TurboPackV**|
| 4.953.768.342| 20.71| 6.63|1419.35|1512.86|**TurboPack**|
| 5.203.353.057| 21.75| 6.96|1560.34|1806.60|SIMDPackD1 FPF|
| 6.074.995.117| 25.40| 8.13| 494.70| 729.97| blosc lz4 64K|
| 6.074.995.117| 25.40| 8.13| 494.70| 729.97|[blosc](#blosc) lz4 64K|
| 6.221.886.390| 26.01| 8.32|1666.76|1737.72|**TurboFor**|
| 6.221.886.390| 26.01| 8.32|1660.52| 565.25|**TurboForDA**|
| 6.699.519.000| 28.01| 8.96| 472.01| 495.12|Vbyte FPF|
@ -99,7 +99,8 @@ TurboPForDA,TurboForDA: Direct Access is normally used when accessing individual
| 8.594.342.216| 35.93|11.50|1307.22|1593.07|libfor|
|23.918.861.764|100.00|32.00|1456.17|1480.78|Copy|
lz4 w/ delta+transpose similar to delta + [blosc](#blosc)
lz4 after preprocessing delta+transpose similar to delta + [blosc](#blosc)<br>
blosc tested w/ lz4 compressor
##### - Compressed Inverted Index Intersections with GOV2<br />
GOV2: 426GB, 25 Millions documents, average doc. size=18k.
@ -262,8 +263,8 @@ header files to use with documentation:<br />
+ <a name="Simple-8b"></a>[Index Compression Using 64-Bit Words](http://people.eng.unimelb.edu.au/ammoffat/abstracts/am10spe.html): Simple-8b (speed optimized version tested)
+ <a name="libfor"></a>[libfor](https://github.com/cruppstahl/for)
+ <a name="lz4"></a>[lz4](https://github.com/Cyan4973/lz4). included w. block size 64K as indication. Tested after preprocessing w. delta+transpose
+ <a name="blosc"></a>[blosc](https://github.com/Blosc/c-blosc). blosc is like transpose/shuffle+lz77. Tested blosc+lz4
+ <a name="DocId data set"></a>[Document identifier data set](http://lemire.me/data/integercompression2014.html)
+ <a name="blosc"></a>[blosc](https://github.com/Blosc/c-blosc). blosc is like transpose/shuffle+lz77
+ **Publications:**
- [SIMD Compression and the Intersection of Sorted Integers](http://arxiv.org/abs/1401.6399)
- [Partitioned Elias-Fano Indexes](http://www.di.unipi.it/~ottavian/files/elias_fano_sigir14.pdf)