Commit Graph

13073 Commits

Author SHA1 Message Date
05c05cfc83 Add apache arrow IPC module for Spark-Doris-Connector usage (#1958)
Add FLATBUFFERS to TP archive
2019-10-18 14:27:19 +08:00
3f325e001a Change the priority of different type in function (#2003)
This commit fix the issue [ISSUE-2002].
It changes the priority of coalesce, ifnull, nullif function etc.
The priority of decimal is higher then varchar in the IS_SUPERTYPE_OF compare mode.

Example:
select coalesce(decimal_column, 1) from table;
    the return type of coalesce should be decimal instead of varchar.

Add supertype about datetime and date
The supertype of datetime is bigint, largeint etc.
In IS_SUPERTYPE_OF compare mode, the function(bigint, bigint, bigint) is a supertype of function(datetime, bigint, int).

Example:
select coalesce(now(), 1)) from web_returns;
    the return type of coalesce should be bigint instead of varchar.
2019-10-18 09:35:49 +08:00
c3b5046940 Fix bug of invalid stream load task rollback (#1999)
If stream load be committed with result PUBLISH_TIMEOUT, it should not rollback
this transaction, but only return this message to user.
2019-10-17 21:08:29 +08:00
d9bb494d7f Fix bug that insert stmt with label return label already exist. (#2006) 2019-10-17 20:00:12 +08:00
4f7cc7e033 add predicate filter(#1652) (#1775) 2019-10-17 19:20:00 +08:00
3bca253fb3 Fix beta rowset read slow (#1994)
[Bug][BetaRowset] fix beta rowset read slowly with limit

beta rowset do not update raw_rows_read in statistics and will read all
data in tablet when query with limit, which lead to long query time.
2019-10-17 19:19:46 +08:00
3c12af4dcc Limit the memory consumption of broker scan node (#1996)
If memory exceed limit, no more row batch will be pushed to batch queue
2019-10-17 14:40:16 +08:00
ac16318c9b [Bug-fix][Broker-load] Fix the bug of the label already exists when the txn has been finished (#1992)
If FE is restarted between txn committed and visible, the load job will be rescheduled and failed with label already exists.
The reason is that there are inconsistency between transaction of load job and meta of load job.
So, the replay of the txn attachment need to be done in function replayOnCommitted.
The load job state and progress is correct after that.
2019-10-16 16:35:18 +08:00
d2bc47d2cc Add introduction of label_keep_max_second (#1993)
[Docs]
2019-10-16 16:05:13 +08:00
41e55cfca9 Modify fixed partition feature (#1989)
1. Not support MAVALUE in multi partition column.
2. Fix the incorrect show create table stmt.
2019-10-16 16:03:46 +08:00
2fcb79e3ef Fix wrong group by result bug (#1987) 2019-10-16 07:19:53 +08:00
63fa260d3f Support prepare/close in UDF (#1985)
The prepare/close step of scalar function is already supported in execution framework, We only need to do is that support it in syntax and meta in frontend.

In addition, 'Hive' binary type of scalar function NOT supports prepare/close step, we need to make it supports.
2019-10-16 07:19:20 +08:00
ee5b79ac2b Fix bug that memtable should be destroyed before finishing the load process (#1983)
The parent mem tracker may be release before visiting it in child mem tracker,
which cause segfault.
2019-10-15 22:46:19 +08:00
ec7c8a2c6f Support adding fixed range partition
eg: ALTER TABLE test_table ADD PARTITION p0125 VALUES [("20190125"), ("20190126"));
2019-10-15 09:50:30 +08:00
62acf5d098 Limit the memory usage of Loading process (#1954) 2019-10-15 09:26:20 +08:00
f130bd3e7b Use Env function to operate directory (#1980)
Now Env has unify all environment operation, such as file operation.
However some of our old functions don't leverage it. This change unify
FileUtils::scan_dir to use Env's function.
2019-10-15 09:25:12 +08:00
4391152168 Make variable argument UDAF work (#1982) 2019-10-15 09:24:53 +08:00
c3a3212ae5 Bump netty-all from 4.1.25.Final to 4.1.42.Final in /fe (#1959)
Bumps [netty-all](https://github.com/netty/netty) from 4.1.25.Final to 4.1.42.Final.
- [Release notes](https://github.com/netty/netty/releases)
- [Commits](https://github.com/netty/netty/compare/netty-4.1.25.Final...netty-4.1.42.Final)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-14 23:05:00 +08:00
9fb9dbefca Get rid of compaction on rowset when making snapshot (#1977)
When making snapshot for incremental clone, missed singleton versions may have been compacted.
So get_rowset_by_version() will not acquire any rowset.
This rowset should be acquired from _inc_rs_version_map.
2019-10-14 22:07:31 +08:00
b84ef013eb Fix the mistake for HLL in mini load (#1981)
[Docs] Fix mistakes for HLL column in mini load
2019-10-14 19:46:23 +08:00
ccc236484b Fix bug that failed to add KEY column to DUPLICATE KEY table (#1973) 2019-10-14 16:40:34 +08:00
01e71def63 Update engine_clone_task.cpp (#1979) 2019-10-14 16:23:04 +08:00
a6a9b0021f Check tablet state before update it (#1974)
After alter BE try to set tablet state to running but the tablet maybe dropped
2019-10-14 16:04:47 +08:00
a323a190a2 Update monitor-alert.md (#1975) 2019-10-14 12:22:51 +08:00
fb7e63038b Fix compile fail (#1971) 2019-10-14 10:24:13 +08:00
e3cc0ee93e Fix empty string bug in dict encoding (#1970) 2019-10-14 10:05:00 +08:00
d68b1b287c Support segment-level zone map (#1931) 2019-10-13 22:06:09 +08:00
7eece1e9e2 Support variable arguments for UDAF (#1968) 2019-10-13 22:04:23 +08:00
80e9b21fb0 Make Segment v2 use string's real length(#1943) (#1944) 2019-10-13 13:23:43 +08:00
8232261df1 Lost rowset during tablet revise tablet meta (#1967) 2019-10-12 23:30:11 +08:00
463b462b8d Add create_time to information_schema.tables 2019-10-12 21:45:14 +08:00
de4e45cd08 Make Schema Change work on segment V2 (#1965) 2019-10-12 21:32:53 +08:00
ce236bfcd4 add alter table modify limit: Cannot change DATETIME to DATE (#1963) 2019-10-12 19:11:17 +08:00
7370b44ab2 Tablet report does not set version miss (#1961) 2019-10-12 14:36:49 +08:00
4678ec8dd9 Delete unused log (#1957) 2019-10-11 23:12:38 +08:00
bbb3fdef8c Fix bug that OlapTableSink use invalid column as distribution column for RANDOM distribution table. (#1956)
RANDOM distribution is deprecated long time ago, this is just for compatibility and bug fix.
2019-10-11 20:07:25 +08:00
71731b25f4 Ignore some compaction errors to reduce logs (#1955) 2019-10-11 19:58:38 +08:00
4a17152f40 Add tdigest compression param for pencentile_approx function (#1939) 2019-10-11 18:56:59 +08:00
948f497dd4 Fix some warning when compile type is debug using -Werror flag(because of use deprecated funcrtions) (#1953) 2019-10-11 17:21:41 +08:00
0e4b3755a2 Refactor txn manager methods (#1950) 2019-10-11 17:16:13 +08:00
d5493fb20a Replace std::regex with RE2 (#1930)
In Storage Engine GC, TabletManger use std::regex to extract tablet id
and schema hash from path. But it will construct regex pattern for
every path to check, this is a huge waste. This change list make this
pattern a global static pattern, and replace it with RE2, which has
better performance.
2019-10-11 15:57:53 +08:00
e267d031bb Enhance the speed of avg function (#1889)
This commit enable the avg operator in fe instead of converting the avg function into sum/count.
Also, this commit fix the bug of deciamlv2 avg which cause the core in be.
The int128 could not be assinged directly.

The speed of avg function is similar to sum function after enhancement.
2019-10-10 22:43:46 +08:00
e4f3e8fda7 Remove redundant method in rowset meta manager (#1949) 2019-10-10 19:29:59 +08:00
b72a4a4bc6 Add tablet meta checkpoint mechanism (#1936) 2019-10-10 09:39:02 +08:00
d46fc59cc3 Add send_clear_alter_tasks operation
ALTER TABLE tbl SET ("send_clear_alter_tasks" = "true");
2019-10-09 22:32:48 +08:00
024348d74b Enable auto convert when check in (#1926)
Leverage gitattributes to enable auto convert end-of-line to LF when
checking in. Convert already exist CRLF to LF by removing all files and
checking out with new .gitattributes file. Except .gitattributes, all
files are only modified at the end of line.
2019-10-09 22:31:27 +08:00
a6b843c04d Fix OrdinalPageIndex header size (#1938) 2019-10-09 22:07:27 +08:00
1c99e88fc0 Invalid hash value of DateLiteral (#1933) 2019-10-08 11:07:02 +08:00
0d729b1191 Filter empty strings of properties in file fe.conf (#1932) 2019-10-07 23:21:05 +08:00
0072712c80 Add address reuesd option for http server (#1915)
Avoid test failure accidentally because of the BindException (Address already in use)
2019-09-30 20:34:10 +08:00