Support sql mode (#2083)

At present, we do not support SQL MODE which is similar to MySQL. In MySQL, SQL MODE is stored in global session and session with a 64 bit address,and every bit 0 or 1 on this address represents a mode state. Besides, MySQL supports combine mode which is composed of several modes.

We should support SQL MODE to deal with sql dialect problems. We can heuristically use the MySQL way to store SQL MODE in session and parse it into string when we need to return it back to client.

This commit suggests a solution to support SQL MODE. But it's just a sample, and the mode types in SqlModeHelper.java are not really meaningful from now on.
This commit is contained in:
xy720
2019-11-01 23:21:00 +08:00
committed by Mingyu Chen
parent e1a8f9d30f
commit ac5dd0c9f2
13 changed files with 400 additions and 8 deletions

View File

@ -0,0 +1,42 @@
# SQL MODE
Doris新支持的sql mode参照了 Mysql 的sql mode管理机制,每个客户端都能设置自己的sql mode,拥有Admin权限的数据库管理员可以设置全局sql mode。
##sql mode 介绍
sql mode使用户能在不同风格的sql语法和数据校验严格度间做切换,使Doris对其他数据库有更好的兼容性。例如在一些数据库里,'||'符号是一个字符串连接符,但在Doris里却是与'or'等价的,这时用户只需要使用sql mode切换到自己想要的风格。每个客户端都能设置sql mode,并在当前对话中有效,只有拥有Admin权限的用户可以设置全局sql mode。
##原理
sql mode用一个64位的Long型存储在SessionVariables中,这个地址的每一位都代表一个mode的开启/禁用(1表示开启,0表示禁用)状态,只要知道每一种mode具体是在哪一位,我们就可以通过位运算方便快速的对sql mode进行校验和操作。
每一次对sql mode的查询,都会对此Long型进行一次解析,变成用户可读的字符串形式,同理,用户发送给服务器的sql mode字符串,会被解析成能够存储在SessionVariables中的Long型。
已被设置好的全局sql mode会被持久化,因此对全局sql mode的操作总是只需一次,即使程序重启后仍可以恢复上一次的全局sql mode。
##操作方式
1、设置sql mode
```
set global sql_mode = "DEFAULT"
set session sql_mode = "DEFAULT"
```
>目前Doris的默认sql mode是DEFAULT(但马上会在后续修改中会改变)。
>设置global sql mode需要Admin权限,并会影响所有在此后连接的客户端。
>设置session sql mode只会影响当前对话客户端,默认为session方式。
2、查询sql mode
```
select @@global.sql_mode
select @@session.sql_mode
```
>除了这种方式,你还可以通过下面方式返回所有session variables来查看当前sql mode
```
show global variables
show session variables
```
##已支持mode
(后续补充)
##复合mode
(后续补充)

View File

@ -0,0 +1,42 @@
#SQL MODE
The SQL MODE supported by Doris refers to the sql mode management mechanism of MySQL. Each client can set its own sql mode, and the database administrator with admin permission can set the global sql mode.
##Sql mode introduction
SQL MODE enables users to switch between different styles of SQL syntax and data verification strictness, making Doris more compatible with other databases. For example, in some databases, the '||' symbol is a string connector, but in Doris it is equivalent to 'or'. At this time, users only need to use SQL mode to switch to the style they want. Each client can set sql mode, which is valid in the current conversation. Only users with admin permission can set global SQL mode.
##Theory
SQL MODE is stored in session variables with a 64 bit long type. Each bit of this address represents the on / off (1 for on, 0 for off) state of a mode. As long as we know the specific bit of each mode, we can easily and quickly verify and operate SQL mode through bit operation.
Every time you query sql mode, the long type will be parsed into a user-readable string. Similarly, the sql mode string sent by the user to the server will be parsed into a long type that can be stored in session variables.
The set global sql mode will be persisted, so the operation on the global sql mode is always only once, even after the program is restarted, the last global sql mode can be recovered.
##Operation
1、set sql mode
```
set global sql_mode = "DEFAULT"
set session sql_mode = "DEFAULT"
```
>At present, Doris's default sql mode is DEFAULT (but it will be changed in the future modification).
>Setting global sql mode requires admin permission and affects all clients that connect later.
>Setting session sql mode will only affect the current conversation client. The default setting way is session.
2、select sql mode
```
select @@global.sql_mode
select @@session.sql_mode
```
>In addition to this method, you can also view the current sql mode by returning all session variables as follows
```
show global variables
show session variables
```
##supported mode
(Work in progress)
##combine mode
(Work in progress)

View File

@ -19,6 +19,7 @@ package org.apache.doris.analysis;
import com.google.common.base.Strings;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
@ -27,6 +28,7 @@ import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.mysql.privilege.UserResource;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.SessionVariable;
import org.apache.doris.qe.SqlModeHelper;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@ -111,8 +113,22 @@ public class SetVar {
if (!(literalExpr instanceof LiteralExpr)) {
throw new AnalysisException("Set statement does't support computing expr:" + literalExpr.toSql());
}
result = (LiteralExpr)literalExpr;
if (variable.equalsIgnoreCase(SessionVariable.SQL_MODE)) {
// For the case like "set sql_mode = PIPES_AS_CONCAT"
if (result instanceof StringLiteral) {
String sqlMode = result.getStringValue();
result = new StringLiteral(SqlModeHelper.encode(sqlMode).toString());
}
// For the case like "set sql_mode = 3"
else if (result instanceof IntLiteral) {
String sqlMode = SqlModeHelper.decode(result.getLongValue());
result = new IntLiteral(SqlModeHelper.encode(sqlMode).toString(), Type.BIGINT);
}
}
// Need to check if group is valid
if (variable.equalsIgnoreCase(SessionVariable.RESOURCE_VARIABLE)) {
if (result != null && !UserResource.isValidGroup(result.getStringValue())) {

View File

@ -17,7 +17,11 @@
package org.apache.doris.analysis;
import com.google.common.base.Strings;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.qe.SessionVariable;
import org.apache.doris.qe.SqlModeHelper;
import org.apache.doris.qe.VariableMgr;
import org.apache.doris.thrift.TBoolLiteral;
import org.apache.doris.thrift.TExprNode;
@ -63,6 +67,10 @@ public class SysVariableDesc extends Expr {
@Override
public void analyzeImpl(Analyzer analyzer) throws AnalysisException {
VariableMgr.fillValue(analyzer.getContext().getSessionVariable(), this);
if (!Strings.isNullOrEmpty(name) && name.equalsIgnoreCase(SessionVariable.SQL_MODE)) {
setType(Type.VARCHAR);
setStringValue(SqlModeHelper.decode(intValue));
}
}
public String getName() {

View File

@ -38,5 +38,5 @@ public class FeConstants {
// general model
// Current meta data version. Use this version to write journals and image
public static int meta_version = FeMetaVersion.VERSION_64;
public static int meta_version = FeMetaVersion.VERSION_65;
}

View File

@ -138,4 +138,6 @@ public final class FeMetaVersion {
public static final int VERSION_63 = 63;
// for table create time
public static final int VERSION_64 = 64;
// support sql mode, change sql_mode from string to long
public static final int VERSION_65 = 65;
}

View File

@ -91,8 +91,9 @@ public class SessionVariable implements Serializable, Writable {
@VariableMgr.VarAttr(name = IS_REPORT_SUCCESS)
private boolean isReportSucc = false;
// Set sqlMode to empty string
@VariableMgr.VarAttr(name = SQL_MODE)
private String sqlMode = "";
private long sqlMode = 0L;
@VariableMgr.VarAttr(name = RESOURCE_VARIABLE)
private String resourceGroup = "normal";
@ -213,11 +214,11 @@ public class SessionVariable implements Serializable, Writable {
return waitTimeout;
}
public String getSqlMode() {
public long getSqlMode() {
return sqlMode;
}
public void setSqlMode(String sqlMode) {
public void setSqlMode(long sqlMode) {
this.sqlMode = sqlMode;
}
@ -505,7 +506,7 @@ public class SessionVariable implements Serializable, Writable {
Text.writeString(out, txIsolation);
out.writeBoolean(autoCommit);
Text.writeString(out, resourceGroup);
Text.writeString(out, sqlMode);
out.writeLong(sqlMode);
out.writeBoolean(isReportSucc);
out.writeInt(queryTimeoutS);
out.writeLong(maxExecMemByte);
@ -540,7 +541,11 @@ public class SessionVariable implements Serializable, Writable {
txIsolation = Text.readString(in);
autoCommit = in.readBoolean();
resourceGroup = Text.readString(in);
sqlMode = Text.readString(in);
if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_65) {
sqlMode = in.readLong();
} else {
sqlMode = 0L;
}
isReportSucc = in.readBoolean();
queryTimeoutS = in.readInt();
maxExecMemByte = in.readLong();

View File

@ -0,0 +1,185 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
package org.apache.doris.qe;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
import com.google.common.collect.Maps;
import org.apache.doris.common.AnalysisException;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
public class SqlModeHelper {
private static final Logger LOG = LogManager.getLogger(SqlModeHelper.class);
// TODO(xuyang): these mode types are copy from MYSQL mode types, which are example
// of how they works and to be compatible with MySQL, so for now they are not
// really meaningful.
/* Bits for different SQL MODE modes, you can add custom SQL MODE here */
public static final long MODE_REAL_AS_FLOAT = 1L;
public static final long MODE_PIPES_AS_CONCAT = 2L;
public static final long MODE_ANSI_QUOTES = 4L;
public static final long MODE_IGNORE_SPACE = 8L;
public static final long MODE_NOT_USED = 16L;
public static final long MODE_ONLY_FULL_GROUP_BY = 32L;
public static final long MODE_NO_UNSIGNED_SUBTRACTION = 64L;
public static final long MODE_NO_DIR_IN_CREATE = 128L;
public static final long MODE_NO_AUTO_VALUE_ON_ZERO = 1L << 19;
public static final long MODE_NO_BACKSLASH_ESCAPES = 1L << 20;
public static final long MODE_STRICT_TRANS_TABLES = 1L << 21;
public static final long MODE_STRICT_ALL_TABLES = 1L << 22;
// NO_ZERO_IN_DATE and NO_ZERO_DATE are removed in mysql 5.7 and merged into STRICT MODE.
// However, for backward compatibility during upgrade, these modes are kept.
@Deprecated
public static final long MODE_NO_ZERO_IN_DATE = 1L << 23;
@Deprecated
public static final long MODE_NO_ZERO_DATE = 1L << 24;
public static final long MODE_INVALID_DATES = 1L << 25;
public static final long MODE_ERROR_FOR_DIVISION_BY_ZERO = 1L << 26;
public static final long MODE_HIGH_NOT_PRECEDENCE = 1L <<29;
public static final long MODE_NO_ENGINE_SUBSTITUTION = 1L << 30;
public static final long MODE_PAD_CHAR_TO_FULL_LENGTH = 1L << 31;
public static final long MODE_TIME_TRUNCATE_FRACTIONAL = 1L << 32;
/* Bits for different COMBINE MODE modes, you can add custom COMBINE MODE here */
public static final long MODE_ANSI = 1L << 18;
public static final long MODE_TRADITIONAL = 1L << 27;
public final static long MODE_LAST = 1L << 33;
public final static long MODE_ALLOWED_MASK =
(MODE_REAL_AS_FLOAT | MODE_PIPES_AS_CONCAT | MODE_ANSI_QUOTES |
MODE_IGNORE_SPACE | MODE_NOT_USED | MODE_ONLY_FULL_GROUP_BY |
MODE_NO_UNSIGNED_SUBTRACTION | MODE_NO_DIR_IN_CREATE |
MODE_NO_AUTO_VALUE_ON_ZERO | MODE_NO_BACKSLASH_ESCAPES |
MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES | MODE_NO_ZERO_IN_DATE |
MODE_NO_ZERO_DATE | MODE_INVALID_DATES | MODE_ERROR_FOR_DIVISION_BY_ZERO |
MODE_HIGH_NOT_PRECEDENCE | MODE_NO_ENGINE_SUBSTITUTION |
MODE_PAD_CHAR_TO_FULL_LENGTH | MODE_TRADITIONAL | MODE_ANSI |
MODE_TIME_TRUNCATE_FRACTIONAL);
public final static long MODE_COMBINE_MASK = (MODE_ANSI | MODE_TRADITIONAL);
private final static Map<String, Long> sqlModeSet = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
private final static Map<String, Long> combineModeSet = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
static {
sqlModeSet.put("REAL_AS_FLOAT", MODE_REAL_AS_FLOAT);
sqlModeSet.put("PIPES_AS_CONCAT", MODE_PIPES_AS_CONCAT);
sqlModeSet.put("ANSI_QUOTES", MODE_ANSI_QUOTES);
sqlModeSet.put("IGNORE_SPACE", MODE_IGNORE_SPACE);
sqlModeSet.put("NOT_USED", MODE_NOT_USED);
sqlModeSet.put("ONLY_FULL_GROUP_BY", MODE_ONLY_FULL_GROUP_BY);
sqlModeSet.put("NO_UNSIGNED_SUBTRACTION", MODE_NO_UNSIGNED_SUBTRACTION);
sqlModeSet.put("NO_DIR_IN_CREATE", MODE_NO_DIR_IN_CREATE);
sqlModeSet.put("ANSI", MODE_ANSI);
sqlModeSet.put("NO_AUTO_VALUE_ON_ZERO", MODE_NO_AUTO_VALUE_ON_ZERO);
sqlModeSet.put("NO_BACKSLASH_ESCAPES", MODE_NO_BACKSLASH_ESCAPES);
sqlModeSet.put("STRICT_TRANS_TABLES", MODE_STRICT_TRANS_TABLES);
sqlModeSet.put("STRICT_ALL_TABLES", MODE_STRICT_ALL_TABLES);
sqlModeSet.put("NO_ZERO_IN_DATE", MODE_NO_ZERO_IN_DATE);
sqlModeSet.put("NO_ZERO_DATE", MODE_NO_ZERO_DATE);
sqlModeSet.put("INVALID_DATES", MODE_INVALID_DATES);
sqlModeSet.put("ERROR_FOR_DIVISION_BY_ZERO", MODE_ERROR_FOR_DIVISION_BY_ZERO);
sqlModeSet.put("TRADITIONAL", MODE_TRADITIONAL);
sqlModeSet.put("HIGH_NOT_PRECEDENCE", MODE_HIGH_NOT_PRECEDENCE);
sqlModeSet.put("NO_ENGINE_SUBSTITUTION", MODE_NO_ENGINE_SUBSTITUTION);
sqlModeSet.put("PAD_CHAR_TO_FULL_LENGTH", MODE_PAD_CHAR_TO_FULL_LENGTH);
sqlModeSet.put("TIME_TRUNCATE_FRACTIONAL", MODE_TIME_TRUNCATE_FRACTIONAL);
combineModeSet.put("ANSI", (MODE_REAL_AS_FLOAT | MODE_PIPES_AS_CONCAT |
MODE_ANSI_QUOTES | MODE_IGNORE_SPACE | MODE_ONLY_FULL_GROUP_BY));
combineModeSet.put("TRADITIONAL", (MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES |
MODE_NO_ZERO_IN_DATE | MODE_NO_ZERO_DATE | MODE_ERROR_FOR_DIVISION_BY_ZERO |
MODE_NO_ENGINE_SUBSTITUTION));
}
// convert long type SQL MODE to string type that user can read
public static String decode(Long sqlMode) throws AnalysisException {
// 0 parse to empty string
if (sqlMode == 0) {
return "";
}
if ((sqlMode & ~MODE_ALLOWED_MASK) != 0) {
throw new AnalysisException("Variable 'sql_mode' can't be set to the value of: " + sqlMode);
}
List<String> names = new ArrayList<String>();
for (Map.Entry<String, Long> mode : getSupportedSqlMode().entrySet()) {
if ((sqlMode & mode.getValue()) != 0) {
names.add(mode.getKey());
}
}
return Joiner.on(',').join(names);
}
// convert string type SQL MODE to long type that session can store
public static Long encode(String sqlMode) throws AnalysisException {
List<String> names =
Splitter.on(',').trimResults().omitEmptyStrings().splitToList(sqlMode);
// empty string parse to 0
long value = 0L;
for (String key : names) {
// the SQL MODE must be supported, set sql mode repeatedly is not allowed
if (!isSupportedSqlMode(key) || (value & getSupportedSqlMode().get(key)) != 0) {
throw new AnalysisException("Variable 'sql_mode' can't be set to the value of: " + key);
}
if (isCombineMode(key)) {
// set multiple combine mode is not allowed
if ((value & MODE_COMBINE_MASK) != 0) {
throw new AnalysisException("Variable 'sql_mode' can't be set to the value of: " + key);
}
value |= getCombineMode().get(key);
}
value |= getSupportedSqlMode().get(key);
}
return value;
}
// check if this SQL MODE is supported
public static boolean isSupportedSqlMode(String sqlMode) {
// empty string is valid and equals to 0L
if (sqlMode == null || !getSupportedSqlMode().containsKey(sqlMode)) {
return false;
}
return true;
}
// check if this SQL MODE is combine mode
public static boolean isCombineMode(String key) {
return combineModeSet.containsKey(key);
}
public static Map<String, Long> getSupportedSqlMode() {
return sqlModeSet;
}
public static Map<String, Long> getCombineMode() {
return combineModeSet;
}
}

View File

@ -420,6 +420,15 @@ public class VariableMgr {
row.add(getValue(ctx.getObj(), ctx.getField()));
}
if (row.size() > 1 && row.get(0).equalsIgnoreCase(SessionVariable.SQL_MODE)) {
try {
row.set(1, SqlModeHelper.decode(Long.valueOf(row.get(1))));
} catch (AnalysisException e) {
row.set(1, "");
LOG.warn("Decode sql mode failed");
}
}
rows.add(row);
}
} finally {

View File

@ -23,6 +23,7 @@ import org.apache.doris.mysql.privilege.MockedAuth;
import org.apache.doris.mysql.privilege.PaloAuth;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.SqlModeHelper;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
@ -73,4 +74,18 @@ public class SetVarTest {
var.analyze(analyzer);
Assert.fail("No exception throws.");
}
@Test(expected = AnalysisException.class)
public void testInvalidSqlModeValue() throws UserException, AnalysisException {
SetVar var = new SetVar(SetType.SESSION, "sql_mode", new IntLiteral(SqlModeHelper.MODE_LAST));
var.analyze(analyzer);
Assert.fail("No exception throws");
}
@Test(expected = AnalysisException.class)
public void testInvalidSqlMode() throws UserException, AnalysisException {
SetVar var = new SetVar(SetType.SESSION, "sql_mode", new StringLiteral("WRONG_MODE"));
var.analyze(analyzer);
Assert.fail("No exception throws");
}
}

View File

@ -49,4 +49,5 @@ public class SysVariableDescTest {
desc.analyze(AccessTestUtil.fetchAdminAnalyzer(false));
Assert.fail("No exception throws.");
}
}

View File

@ -0,0 +1,61 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
package org.apache.doris.qe;
import org.apache.doris.common.AnalysisException;
import org.junit.Assert;
import org.junit.Test;
public class SqlModeHelperTest {
@Test
public void testNormal() throws AnalysisException {
String sqlMode = "PIPES_AS_CONCAT";
Assert.assertEquals(new Long(2L), SqlModeHelper.encode(sqlMode));
sqlMode = "";
Assert.assertEquals(new Long(0L), SqlModeHelper.encode(sqlMode));
long sqlModeValue = 2L;
Assert.assertEquals("PIPES_AS_CONCAT", SqlModeHelper.decode(sqlModeValue));
sqlModeValue = 0L;
Assert.assertEquals("", SqlModeHelper.decode(sqlModeValue));
}
@Test(expected = AnalysisException.class)
public void testInvalidSqlMode() throws AnalysisException {
String sqlMode = "PIPES_AS_CONCAT, WRONG_MODE";
SqlModeHelper.encode(sqlMode);
Assert.fail("No exception throws");
}
@Test(expected = AnalysisException.class)
public void testMultiSqlMode() throws AnalysisException {
String sqlMode = "ANSI, TRADITIONAL";
SqlModeHelper.encode(sqlMode);
Assert.fail("No exception throws");
}
@Test(expected = AnalysisException.class)
public void testInvalidDecode() throws AnalysisException {
long sqlMode = SqlModeHelper.MODE_LAST;
SqlModeHelper.decode(sqlMode);
Assert.fail("No exception throws");
}
}

View File

@ -71,6 +71,7 @@ public class VariableMgrTest {
Assert.assertEquals(2147483648L, var.getMaxExecMemByte());
Assert.assertEquals(300, var.getQueryTimeoutS());
Assert.assertEquals(false, var.isReportSucc());
Assert.assertEquals(0L, var.getSqlMode());
List<List<String>> rows = VariableMgr.dump(SetType.SESSION, var, null);
Assert.assertTrue(rows.size() > 5);
@ -117,6 +118,12 @@ public class VariableMgrTest {
// Get from name
SysVariableDesc desc = new SysVariableDesc("exec_mem_limit");
Assert.assertEquals(var.getMaxExecMemByte() + "", VariableMgr.getValue(var, desc));
SetVar setVar4 = new SetVar(SetType.SESSION, "sql_mode", new StringLiteral(
SqlModeHelper.encode("PIPES_AS_CONCAT").toString()));
VariableMgr.setVar(var, setVar4);
Assert.assertEquals(2L, var.getSqlMode());
}
@Test(expected = DdlException.class)
@ -161,7 +168,6 @@ public class VariableMgrTest {
Assert.fail("No exception throws.");
}
@Test(expected = DdlException.class)
public void testReadOnly() throws AnalysisException, DdlException {
SysVariableDesc desc = new SysVariableDesc("version_comment");