Support to get tables in materialized view when collecting table in plan
table scehma as fllowing:
create materialized view mv1
BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ('replication_num' = '1')
as
select
t1.c1,
t3.c2
from
table1 t1
inner join table3 t3 on t1.c1 = t3.c2
if get table from the plan as follwoing, we can get [table1, table3, table2], the mv1 is expanded to get base tables;
SELECT
mv1.*,
uuid()
FROM
mv1 LEFT SEMI
JOIN table2 ON mv1.c1 = table2.c1
WHERE
mv1.c1 IN (
SELECT
c1
FROM
table2
)
OR mv1.c1 < 10
Fix Reach limit of connections error
in fe.conf , arrow_flight_token_cache_size is mandatory less than qe_max_connection/2. arrow flight sql is a stateless protocol, connection is usually not actively disconnected, bearer token is evict from the cache will unregister ConnectContext.
Fix ConnectContext.command not be reset to COM_SLEEP in time, this will result in frequent kill connection after query timeout.
Fix bearer token evict log and exception.
TODO: use arrow flight session: https://mail.google.com/mail/u/0/#inbox/FMfcgzGxRdxBLQLTcvvtRpqsvmhrHpdH
Currently, when reading a hive on cosn table, doris return empty result, but the table has data.
iceberg on cosn is ok.
The reason is misuse of cosn's file sytem. according to cosn's doc, its fs.cosn.impl should be org.apache.hadoop.fs.CosFileSystem
MySQL [test]> show views;
+----------------+
| Tables_in_test |
+----------------+
| t1_view |
| t2_view |
+----------------+
2 rows in set (0.00 sec)
MySQL [test]> show views like '%t1%';
+----------------+
| Tables_in_test |
+----------------+
| t1_view |
+----------------+
1 row in set (0.01 sec)
MySQL [test]> show views where create_time > '2024-03-18';
+----------------+
| Tables_in_test |
+----------------+
| t2_view |
+----------------+
1 row in set (0.02 sec)
fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about #31585
doc pr: apache/doris-website#488
Previously, local tvf can only query data on one BE node.
But if the storage is shared(eg, NAS), it can be executed on multi nodes.
This PR mainly changes:
1. Add a new property `"shared_stoage" = "false/true"`
Default is false, if set to true, "backend_id" is optional. If "backend_id" is set,
it still be executed on that BE, if not set, "shared_stoage" must be "true"
and it will be executed on multi nodes.
Doc: https://github.com/apache/doris-website/pull/494
1. Check the return value of avro reader's init_fetch_table_schema_reader()
2. Also fix a bug but the parse exception of Nereids may suppress the real exception from old planner
It will result unable to see the real error msg.
In previously, when enabling FQDN, Doris will call dns resolver to get IP from hostname
each time when 1) FE gets BE's grpc client. 2) BE gets other BE's brpc client.
So when in high concurrency case, the dns resolver be overloaded and failed to resolve hostname.
This PR mainly changes:
1. Add DNSCache for both FE and BE.
The DNSCache will run on every FE and BE node. It has a cache, key is hostname and value is IP.
Caller can get IP by hostname from this cache, and if hostname does not exist, it will try to resolve it
and update the cache.
In addition, DNSCache has a daemon thread to refresh the cache every 1 min, in case that the IP may
be changed at anytime.
There are other implements of this dns cache:
1. 36fed13997
This is for BE side, but it does not handle the IP change case.
3. https://github.com/apache/doris/pull/28479
This is for FE side, but it can only work with Master FE. Other FE node will not be aware of the IP change.
And there are a bunch of BackendServiceProxy, this PR only handle cache in one of them.
In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.
In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.
Some of mysql connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection.
eg, `select @@autocommit` should with column type `BIGINT`, not `BIT`, otherwise it will throw error like:
```
System.FormatException: The input string 'True' was not in a correct format.
at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
at System.Convert.ToInt32(String value)
at MySql.Data.MySqlClient.Driver.LoadCharacterSetsAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.Driver.ConfigureAsync(MySqlConnection connection, Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Boolean execAsync, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlConnection.Open()
```
In this PR, I add a new field of `VarAttr`: `convertBoolToLongMethod`, if set, it will convert boolean to long.
And set it for `autocommit`
