At present, there are defects in the chapter on debugging FE in doc. My colleagues and I stepped on the pit when
building the debugging environment, so I want to improve this chapter in combination with my own stepping on the pit
experience.
The following is my explanation of the changes:
1. mkdir -p ./thirdparty/installed/bin
explain: When I downloaded versions 0.14 and 0.15, there were no files under thirdparty, so I didn't know whether to
create it myself or what to do. Finally, I decided to create it myself. I think it's necessary to add instructions here.
2. Add installation thrift@0.13.0 Failed handling method.
explain: My colleagues and I failed to find the installation package when executing the installation command, and finally
found a solution on GitHub. Therefore, I added the handling method of the problem to avoid other Mac users from
getting stuck in this place.
3. Fixed an error in the generated code description.
explain: Before I finished building the code, I debugged FE, and I failed all the time. Idea hints that no files can be found.
Later, after consulting with morningman in wechat group, it was understood that `mvn install -DskipTests` does not
need to execute `mvn generate-sources` after execution. This is inconsistent with the description in the document and
needs to be corrected.
Close related #7334
1. Fix bug describe in [Bug] show frontends cause FE oom #7334
2. Fix error of CurrentConnected fields in show frontends result.
3. Add more FAQ
Add `sink.batch.size` `sink.max-retries` options in `Doris Spark-connector`.
Be consistent with `link-connector` options .
eg:
```scala
df.write
.format("doris")
// specify maximum number of lines in a single flushing
.option("sink.batch.size",2048)
// specify number of retries after writing failed
.option("sink.max-retries",3)
.save()
```
Transfer RowBatch in Protobuf Request to Controller Attachment,
when the maximum length of the RowBatch in the Protobuf Request is exceeded.
This can avoid reaching the upper limit of the Protobuf Request length (2G),
and it is expected that performance can be improved.
1. The clang format action will be triggered when a PR is submitted.
2. Skywalking eyes actions will be triggered when a PR is submitted and after merging to master branch.
We found that many commit messages submitted at present have ambiguous information.
Clear commit messages can help developers submit pull requests more readable,
committers merge easily and Release Manager easy to release.
Therefore, we have sorted out a version of the commit format specification.
We hope that subsequent contributors can sort out the commit messages according to
the specification when submitting Pull Request.
Now minidump file will be created when BE crashes.
And user can manually trigger a minidump by sending SIGUSR1 to BE process.
More details can be found in minidump.md documents
Add a new field `Lag` in result of `show routine load` stmt.
`Lag: {"0":10, "1":0}` means kafka partition 0 has 10 msg behind and partition 1 is update-to-date.
Users can directly query the data in the hive table in Doris, and can use join to perform complex queries without laboriously importing data from hive.
Main changes list below:
FE:
Extend HiveScanNode from BrokerScanNode
HiveMetaStoreClientHelper communicate with HIVE and HDFS.
BE:
Treate HiveScanNode as BrokerScanNode, treate HiveTable as BrokerTable.
broker_scanner.cpp: suppot read column from HDFS path.
orc_scanner.cpp: support read hdfs file.
POM:
Add hive.version=2.3.7, hive-metastore and hive-exec
Add hadoop.version=2.8.0, hadoop-hdfs
Upgrade commons-lang to fix incompatiblity of Java 9 and later.
Thrift:
Add THiveTable
Add read_by_column_def in TBrokerRangeDesc
The new session variable 'close_join_reorder' is used to turn off all automatic join reorder algorithms.
If close_join_reorder is true, the Doris will execute query by the order in the original query.
1. Migrate some of the best practice articles to the Blog
2. Changed the names of performance tests and best practices to performance tests and examples
Mainly changes:
1. Fix [Bug] Colocate group can not redistributed after dropping a backend #7019
2. Add detail msg about why a colocate group is unstable.
3. Add more suggestion when upgrading Doris cluster.
Add the sharing blog function to the document site, including the blog list and detail page. At the same time, a guide on how to share blogs has been added to the developer guide.
