Commit Graph

3079 Commits

Author SHA1 Message Date
539474a28d Disable RocksDB tests in 2.1
The RocksDB based cache storage is not built by default so the tests
should be disabled.
2017-06-05 13:20:15 +03:00
4c78e2c99a Fix avrorouter memory leaks
The loading of the JSON schemas leaked memory as functions that increment
the reference count were used.
2017-06-05 13:20:14 +03:00
d5a72d1b75 Fix avrorouter client counts
The avro client numbers were decremented twice for each client.
2017-06-05 13:19:48 +03:00
285598e0e5 Fix avrorouter type and name handling
The type and name parsing functions could move outside of allocated memory
as they didn't check for the terminating null character. Also fixed the
printf format string used when the list of used tables is being created.

Fixed CDC testing connector to abort on error and added some extra output
to the cdc_datatypes test.
2017-06-05 13:19:47 +03:00
a37a665bd2 MXS-1271: Remove redundant JSON parsing
As the cdc.py script outputs newline delimited JSON, there's no need to
parse the JSON in the cdc_kafka_producer.py script.
2017-06-05 13:17:12 +03:00
9124de86df Add default value to MaskingFilter "fill" setting
MXS-1261. If "fill" is not defined, an "X" is used as default.
Setting an empty string as "fill" or "value" is disallowed.
2017-06-05 11:30:06 +03:00
b5d1504e6a Merge branch '2.0' into 2.1 2017-05-26 15:22:43 +03:00
ad109408b5 MXS-1216: Add real_type and length to cdc_schema.py
The schema generator program needs to add the real_type and length fields
if the data types define them.

Also fixed a bug where the real_type and length fields were checked for
generated fields.
2017-05-16 13:27:46 +03:00
0c9409ae7f Merge branch '2.0' into 2.1-merge-2.0 2017-05-15 10:35:44 +03:00
a12d19591e MXS-1216: Store field real type and length in Avro schema
The avro schema allows custom properties to be defined for the schema
fields. The avrorouter stored extra information about the table into the
schema for later use.

Currently, this information is only generated by the avrorouter
itself. Further improvements to the schema generator scripts need to be
done.
2017-05-15 10:25:12 +03:00
6b6a7fa4a1 Do checkpoint processing at end of binlog
When the binlog has been read, it needs to be treated as if the
transaction or row limit has been hit. This will cause all tables to be
flushed to disk before the files are indexed.
2017-05-15 10:25:11 +03:00
8a288110a9 Rename conflicting Avro fields
When a user defined field conflicts with an internal MaxScale field, the
field is suffixed with an underscore.
2017-05-15 10:25:11 +03:00
926930e241 MXS-1216: Correct CHAR(n) handling
The field length was wrongly compared to less than 255 for two byte field
lengths. In addition to that, the metadata was interpreted in the wrong
way.
2017-05-15 10:25:09 +03:00
c988735a03 MXS-1216: Fix crash on MariaDB 10.0 DATETIME(n)
When a MariaDB 10.0 DATETIME field with a custom length was defined, the
field offsets weren't calculated properly.

As there is no metadata for pre-10.1 DATETIME types with decimal
precision, the metadata (i.e. decimal count) needs to be gathered from the
CREATE TABLE statement. This information is then used to calculate the
correct field length when the value is decoded.

This change does not fix the incorrect interpretation of the old DATETIME
value. The converted values are still garbled due to the fact that the
value needs to be shifted out of the decimal format before it can be
properly converted.
2017-05-15 10:25:06 +03:00
ed653ac729 Merge branch '2.0' into 2.1-merge-2.0 2017-05-14 10:25:12 +03:00
c464bb4ee1 MXS-1216: Store field real type and length in Avro schema
The avro schema allows custom properties to be defined for the schema
fields. The avrorouter stored extra information about the table into the
schema for later use.

Currently, this information is only generated by the avrorouter
itself. Further improvements to the schema generator scripts need to be
done.
2017-05-12 16:47:20 +03:00
85b49520a2 Do checkpoint processing at end of binlog
When the binlog has been read, it needs to be treated as if the
transaction or row limit has been hit. This will cause all tables to be
flushed to disk before the files are indexed.
2017-05-12 11:30:19 +03:00
9ed972e1ac Rename conflicting Avro fields
When a user defined field conflicts with an internal MaxScale field, the
field is suffixed with an underscore.
2017-05-12 11:28:07 +03:00
da2f7b1efe MXS-1216: Correct CHAR(n) handling
The field length was wrongly compared to less than 255 for two byte field
lengths. In addition to that, the metadata was interpreted in the wrong
way.
2017-05-12 11:27:02 +03:00
30bd869f1c MXS-1216: Fix crash on MariaDB 10.0 DATETIME(n)
When a MariaDB 10.0 DATETIME field with a custom length was defined, the
field offsets weren't calculated properly.

As there is no metadata for pre-10.1 DATETIME types with decimal
precision, the metadata (i.e. decimal count) needs to be gathered from the
CREATE TABLE statement. This information is then used to calculate the
correct field length when the value is decoded.

This change does not fix the incorrect interpretation of the old DATETIME
value. The converted values are still garbled due to the fact that the
value needs to be shifted out of the decimal format before it can be
properly converted.
2017-05-12 11:22:04 +03:00
8e2c7fd952 Merge branch '2.0' into 2.1 2017-05-10 09:12:42 +03:00
898bc3444e MXS-1216: Fix processing of long fixed strings
The fixed length string processing assumed that the string lengths were
contained in the first byte. This is not true for large fixed length
strings that take more than 255 bytes to store. This consists of
multi-byte character strings that can take up to 1024 bytes to store.
2017-05-09 14:46:39 +03:00
09349aaa22 Add missing string versions of MySQL commands
The STRPACKETTYPE macro was missing a number of the commands from the enum.

Added executed command byte value to readwritesplit info level log output.
2017-05-09 10:40:34 +03:00
e8b14dd8b0 Remove debug output 2017-05-08 14:54:23 +03:00
656cb3c517 Added depth and SERVER_SLAVE_OF_EXTERNAL_MASTER for build_mysql51_replication_tree
Now build_mysql51_replication_tree assigns depth and handles
SERVER_SLAVE_OF_EXTERNAL_MASTER status
2017-04-26 13:40:28 +03:00
57700215ac MXS-1244: MySQL monitor "detect_replication_lag=true" doesn't work with "mysql51_replication=true"
Added the missing handle->master assignment
2017-04-26 13:39:55 +03:00
e6b34ea85c Added depth and SERVER_SLAVE_OF_EXTERNAL_MASTER for build_mysql51_replication_tree
Now build_mysql51_replication_tree assigns depth and handles
SERVER_SLAVE_OF_EXTERNAL_MASTER status
2017-04-21 17:09:28 +02:00
6b9a2cd0b5 MXS-1244: MySQL monitor "detect_replication_lag=true" doesn't work with "mysql51_replication=true"
Added the missing handle->master assignment
2017-04-21 16:11:30 +02:00
33b809eec1 Do not build storage_rocksdb by default 2017-04-20 13:12:46 +03:00
d03b059107 Cache length of buffered data
If a complete response is delivered in many buffers, then calling
gwbuf_length() whenever you need the complete size starts to hurt.
By caching the length of the data received sofar and by updating
the length in clientReply(), gwbuf_length() will be called exactly
once for each buffer(chain) delivered to routeQuery().
2017-04-19 18:15:56 +03:00
a418387d0a MXS-1218 Poll statistics changed to 64bit to avoid looparound
Statistics calculation, printing and MaxInfo are modified.
n_fds remains 32bit.
2017-04-18 13:14:47 +03:00
f4a5134a83 MXS-1212: use res.length for buffer length instead of computing the size of the whole buffer chain
This reduces CPU usage and decreases the response time for result sets
with many rows.
2017-04-07 12:16:50 +02:00
41a9663611 MXS-1191: Fix crash on column drop
When a column is dropped from a table with a schema that has more columns
than the row event has, avrorouter will crash.
2017-04-04 17:08:07 +03:00
c67894b011 Add Avro schema generator Python script
The script is the Python version of the cdc_schema.go application.
2017-04-04 16:42:39 +03:00
e0a98f6539 Fix calls of pcre2_substitute
If the output buffer given to pcre2_substitute is too small, an error
value is written to the last parameter (output length). That value
should not be used for calculations. This patch gives a copy as
parameter instead.

Coincidentally, this commit fixes the crashes of query classifier tests.

Also, increase buffer growth rate in utils.c.
2017-04-04 16:19:21 +03:00
872f69b681 Also check IPv6 mapped IPv4 addresses as IPv4
If a client connects from an IPv4 address, but the listener listens on an
IPv6 address, the client IP will be a IPv6 mapped IPv4 address
e.g. ::ffff:127.0.0.1. A grant for an IPv4 address should still match an
IPv6 mapped IPv4 address.
2017-04-04 15:28:39 +03:00
9f14f3659f Only check user permissions on startup
When users were loaded, the permissions for the service user were
checked. The conditional that makes sure the check is executed only at
startup was checking the listener's users instead of the SQLite handle
which caused all reloads of users to check the permissions.
2017-04-04 15:28:39 +03:00
860b14526c Fix mxs_mysql_is_result_set
The function assumed that the packet payload was always at least two
bytes.
2017-04-04 15:28:39 +03:00
1ff83150f0 Revert "Route statements in larger batches"
This reverts commit f3c83770903151a0a3b53593c3e05fa0af94cd5f. The
functionality was used implicitly by modules that declare the
RCAP_TYPE_CONTIGUOUS_OUTPUT capability.
2017-04-04 09:56:51 +03:00
0c3f9ffa2e Maxrows: documentation update and use of MXS_MODULE_PARAM_SIZE for max_resultset_size parameter
max_resultset_size parameter now uses MXS_MODULE_PARAM_SIZE and
config_get_size

Documentation change follows the change.
2017-04-04 08:43:55 +02:00
815780aa4a In cache testrules.c ensure there seems to be at least one thread 2017-04-03 14:20:54 +03:00
0f2f9d48b7 Improve maxadmin help messages
The help messages are now more descriptive and have usage information in
them. This should help users use the commands without relying on the
online documentation.
2017-04-03 12:09:59 +03:00
f3c8377090 Route statements in larger batches
The RCAP_TYPE_STMT_OUTPUT is not used in its previous form. It can be
altered to route only complete packets back to the client. This allows
routers to do safer parsing on the results.
2017-04-03 12:07:45 +03:00
e6d2c96f5b MXS-1211: maxrows should be configurable to return error when limit has been exceeded
New parameter added to maxsrows filter:
max_resultset_return=empty|error|ok
Default, 'empty' is to return an empty set, as the current
implementation.
'err' will return an ERR reply with the input SQL statement
'ok' will return an OK packet
2017-03-31 14:26:44 +02:00
cbc1e864d9 Use RFC 3986 compliant addresses in log messages
When log messages are written with both address and port information, IPv6
addresses can cause confusion if the normal address:port formatting is
used. The RFC 3986 suggests that all IPv6 addresses are expressed as a
bracket enclosed address optionally followed by the port that is separate
from the address by a colon.

In practice, the "all interfaces" address and port number 3306 can be
written in IPv4 numbers-and-dots notation as 0.0.0.0:3306 and in IPv6
notation as [::]:3306. Using the latter format in log messages keeps the
output consistent with all types of addresses.

The details of the standard can be found at the following addresses:

     https://www.ietf.org/rfc/rfc3986.txt

     https://www.rfc-editor.org/std/std66.txt
2017-03-31 14:12:58 +03:00
726610b67d Use 64-bit integers for statistics in rwsplit
Wraparound is not likely anymore.
2017-03-30 12:44:27 +03:00
b458b63756 Use IPv6 for created listeners
When listeners are created, use the default values of [::]:3306.
2017-03-29 17:14:39 +03:00
4b2d94d0f8 When a transaction has been started, no other checks are needed
If we do check then we will end up turning off the read only flag
that just was set on.
2017-03-28 16:17:13 +03:00
f2734f895e Revert "Update dbfwfilter build configuration"
This reverts commit 09ef292283311483354e6cd628318a5cafd96b2c. The old
syntax is still required on SLES 11.
2017-03-27 10:01:54 +03:00
d0a9571da0 Introduce new qc_parse() prototype
It is now possible to specify what information the caller is interested
in. With this the cost for collecting information during the query parsing
that nobody is interested in can be avoided.
2017-03-22 11:09:36 +02:00