Commit Graph

29 Commits

Author SHA1 Message Date
9b5028785d [fix](prepare) fix datetimev2 return err when binary_row_format (#34662)
fix datetimev2 return err when binary_row_format. before pr, Backend return datetimev2 alwary by to_string.
fix datatimev2 return metadata loss scale.
2024-05-18 18:37:41 +08:00
0ae1b9c70a [chore](remove code) Remove dragonbox related (#34528)
* Revert "[refactor](mysql result format) use new serde framework to tuple convert (#25006)"

This reverts commit e5ef0aa6d439c3f9b1f1fe5bc89c9ea6a71d4019.

* run buildall

* MORE

* FIX
2024-05-13 22:16:57 +08:00
193600ad9d [Performance](sink) opt mysql result writer (#31816) 2024-04-10 11:34:30 +08:00
f8d3b20911 [improve](fmt) use format_to and FMT_COMPILE to speed up write data #29682 2024-01-12 11:53:57 +08:00
c1d64a7128 [Feature](datatype) Add IPv4/v6 data type for doris (#24965) 2023-10-26 17:33:28 +08:00
Pxl
642c149e6a remove datetime_value and move vecdatetime_value to doris namespace (#25695)
remove datetime_value and move vecdatetime_value to doris namespace
2023-10-20 22:08:17 +08:00
e5ef0aa6d4 [refactor](mysql result format) use new serde framework to tuple convert (#25006) 2023-10-14 19:46:42 +08:00
8973610543 [feature](datetime) "timediff" supports calculating microseconds (#21371) 2023-07-10 19:21:32 +08:00
08fff8923f [improvement](serde) Optimizing the performance of mysql result writter (#20928)
When converting query results into MySQL format, it involves transforming columnar data storage into row-based storage. This process raises the question of choosing between sequential reading and sequential writing. In reality, sequential writing is the better choice for performance optimization.

Test with 9M rows contains more than 20 columns, this patch can reduce the conversion time from 20s to 11s.
2023-06-19 12:29:01 +08:00
03a679e33a [improvement](sink) reuse rows buffer in msyql_result_writer (#20482)
Creating a rows buffer for each block can impact non-negligible performance.
So it is necessary to reuse the rows buffer.

Test with a total of 1.7M rows, the AppendBatchTime reduced from 500ms to 280ms.
2023-06-07 10:09:32 +08:00
ee34b6de2d [Refact] (serde) refact mysql serde with data type (#19543)
refact mysql output (de)serialize with data type serde , avoid accoriding switch case Primitive type writed in mysqlWriter
2023-05-26 14:11:17 +08:00
e76b3a316f [Bug](mysql proto) fix binary proto with dynamic mode (#19055)
Dynamic mode used in array type when serialize it to mysql row buffer using dynamic mode, when combine binary row format with dynamic mode,something goes wrong, and lead to invalid binary row format.
2023-04-27 11:18:01 +08:00
e412dd12e8 [chore](build) Use include-what-you-use to optimize includes (PART II) (#18761)
Currently, there are some useless includes in the codebase. We can use a tool named include-what-you-use to optimize these includes. By using a strict include-what-you-use policy, we can get lots of benefits from it.
2023-04-19 23:11:48 +08:00
bd8e3e6405 [refactor](date) unify DateTimeValue and VecDateTimeValue (#17670) 2023-03-20 16:27:08 +08:00
Pxl
16fc3a0e22 [Chore](compile) remove some unused static on inline function to reduce compile time (#17603)
remove some unused static on inline function to reduce compile time
2023-03-13 11:11:59 +08:00
116e17428b [Enhancement](point query optimize) improve performace of point query on primary keys (#15491)
1. support row format using codec of jsonb
2. short path optimize for point query
3. support prepared statement for point query
4. support mysql binary format
2023-01-20 13:33:01 +08:00
4ccdd65bf6 [Fix](array) fix mysql_row_buffer may use after free when reserve() delete original address in dynamic_mode (#11395)
```
if (!_dynamic_mode) {
	int8store(_len_pos, _pos - _len_pos - 8);
	_len_pos = nullptr;
}
```

_len_pos may be pointed to the pos which already deleted in reserve, int8store will asign value to the freed address,
and lead to use after free when build in ASAN.So I changed _len_pos to the offset of _buf
2022-08-01 22:52:19 +08:00
50864aca7d [refactor] fix warings when compile with clang (#8069) 2022-02-19 11:29:02 +08:00
d8ba6e3eb6 1. Fix an error when fetch string type field may cause malform packet error. (#7262)
This is beacuse of an const MAX_PHYSICAL_PACKET_LENGTH  in fe should be 2^24 -1,
   but it is set as 2^24 -2 by mistake.
2. Fix bitmap_to_string may failed when the result is large than 2G
2021-12-01 10:02:34 +08:00
6c6380969b [refactor] replace boost smart ptr with stl (#6856)
1. replace all boost::shared_ptr to std::shared_ptr
2. replace all boost::scopted_ptr to std::unique_ptr
3. replace all boost::scoped_array to std::unique<T[]>
4. replace all boost:thread to std::thread
2021-11-17 10:18:35 +08:00
7e30b28f3a [Optimize] Speed up converting the data of other types to string in mysql_result_writer (#6384)
Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-24 22:30:58 +08:00
ed3ff470ce [ARRAY] Support array type load and select not include access by index (#5980)
This is part of the array type support and has not been fully completed. 
The following functions are implemented
1. fe array type support and implementation of array function, support array syntax analysis and planning
2. Support import array type data through insert into
3. Support select array type data
4. Only the array type is supported on the value lie of the duplicate table

this pr merge some code from #4655 #4650 #4644 #4643 #4623 #2979
2021-07-13 14:02:39 +08:00
2b5e4dc951 Add fmt library to speed up mysql text result serialization (#5554)
* Add fmt library to speed up mysql text result serialization

* use BUILD_SYSTEM instead of make

Co-authored-by: gaodayue <gaodayue@bytedance.com>
2021-04-08 09:16:58 +08:00
6fedf5881b [CodeFormat] Clang-format cpp sources (#4965)
Clang-format all c++ source files.
2020-11-28 18:36:49 +08:00
09f97f8a05 [Refactor] Fixes some be typo part 2 (#4747) 2020-10-20 09:28:57 +08:00
37b4cafe87 Change variable and namespace name in BE (#268)
Change 'palo' to 'doris'
2018-11-02 10:22:32 +08:00
2868793b6b Change license to Apache License 2.0 (#262) 2018-11-01 09:06:01 +08:00
bfe1bc7cf3 remove mysql_dtoa 2018-06-13 08:33:48 +08:00
e2311f656e baidu palo 2017-08-11 17:51:21 +08:00