Merge branch '2.1' into develop

Documentation/About/Limitations.md

@@ -231,5 +231,7 @@ executed in separate parts.
 query will be routed to the first available server. This possibly returns an
 error about database rights instead of a missing database.
 
-* As text protocol prepared statements are relatively rare, prepared statements
+* The preparation of a prepared statement is routed to all servers. The
-are not supported in schemarouter.
+execution of a prepared statement is routed to the first available server or to
+the server pointed by a routing hint attached to the query. In practice this
+means that prepared statements aren't supported by the schemarouter.

Documentation/Changelog.md

@@ -1,5 +1,22 @@
 #Changelog
 
+## MariaDB MaxScale 2.1
+* Hierarchical configuration files are now supported.
+* Logging is now performed in a way compatible with logrotate(8).
+* Persistent connections are reset upon resuse.
+* Galera monitor now consistently chooses the same node as master.
+* Galera Monitor can set the preferred donor nodes list.
+* The configuration can now be altered dynamically and the changes are persisted.
+* There is now a monitor for Amazon Aurora clusters.
+* MySQL Monitor now has a multi-master mode.
+* MySQL Monitor now has a failover mode.
+* Named Server Filter now supports wildcards for source option.
+* Binlog Server can now be configured to encrypt binlog files.
+* New filters, _cache, _ccrfilter_, _masking_, and _maxrows_ are introduced.
+
+For more details, please refer to:
+* [MariaDB MaxScale 2.1.0 Release Notes](Release-Notes/MaxScale-2.1.0-Release-Notes.md)
+
 ## MariaDB MaxScale 2.0
 * MariaDB MaxScale is licensed under MariaDB BSL.
 * SSL can be used in the communication between MariaDB MaxScale and the backend servers.

Documentation/Getting-Started/Configuration-Guide.md

@@ -589,42 +589,6 @@ filters=counter | QLA
 The requests pass through the filters from left to right in the order defined in
 the configuration parameter.
 
-#### `monitor`
-
-The monitor parameter tells the service which cluster of servers to use. The
-value of the option is the name of a monitor object. This removes the redundant
-definition of servers in both services and monitors.
-
-If both the `monitor` and `servers` parameters are defined, only the `monitor`
-parameter will be used.
-
-Only one cluster can be defined for a service. Multi-cluster setups should use
-the `servers` parameter to list the servers that the service should use.
-
-Here is an excerpt from an example configuration with a service that defines the
-cluster parameter.
-
-```
-[db-cluster-1-monitor]
-type=monitor
-module=mysqlmon
-servers=server1,server2,server3
-user=maxuser
-passwd=maxpwd
-monitor_interval=1000
-
-[my-readwrite-service]
-type=service
-router=readwritesplit
-monitor=db-cluster-1-monitor
-user=maxuser
-passwd=maxpwd
-```
-
-The _db-cluster-1-monitor_ cluster consists of three servers monitored by a
-_mysqlmon_ monitor module. The _my-readwrite-service_ uses this monitor as the
-source for servers.
-
 #### `servers`
 
 The servers parameter in a service definition provides a comma separated list of

Documentation/Release-Notes/MaxScale-2.0.4-Release-Notes.md

@@ -1,4 +1,4 @@
-# MariaDB MaxScale 2.0.4 Release Notes
+# MariaDB MaxScale 2.0.4 Release Notes -- 2017-02-01
 
 Release 2.0.4 is a GA release.
 

Documentation/Release-Notes/MaxScale-2.1.0-Release-Notes.md

@@ -3,7 +3,7 @@
 Release 2.1.0 is a Beta release.
 
 This document describes the changes in release 2.1.0, when compared to
-release 2.0.X.
+release 2.0.4.
 
 For any problems you encounter, please consider submitting a bug
 report at [Jira](https://jira.mariadb.org).
@@ -71,8 +71,8 @@ MaxScale to rotate the the log file.
 This behaviour is now compatible with logrotate(8).
 
 Further, if MaxScale is configured to use shared memory for the log file,
-the file is created into the directory "/dev/shm/maxscale". Earlier the
+the file is created into the directory `/dev/shm/maxscale`. Earlier the
-log file was created into the directory "/dev/shm/maxscale.PID", where PID
+log file was created into the directory `/dev/shm/maxscale.PID`, where PID
 was the pid of the MaxScale process.
 
 In addition, there is now a mechanism that prevents the flooding of the log, in
@@ -132,12 +132,11 @@ For more details, read the [Galeramon documentation](../Monitors/Galera-Monitor.
 MaxAdmin now defaults to Emacs editing mode instead of VIM. To activate
 with VIM-mode start MaxAdmin with option -i.
 
-### Support for text wildcards in hostnames
+### Named Server Filter
+The source option can now handle wildcards such as:
+192.168.%.%
 
-Wildcards in text-form hostnames in the mysql.user-table of the backend
+For more details, read the [Named Server Filter documentation](../Filters/Named-Server-Filter.md).
-database are now supported. For matching, the client IP-address must be
-converted to text-form, which requires a reverse DNS lookup. Please see
-[Limitations](../About/Limitations.md) for more information.
 
 ## New Features
 
@@ -189,15 +188,6 @@ allowing runtime tuning of parameters.
 - `destroy monitor`: Destroy a created monitor
 - `alter monitor`: Alter monitor parameters
 
-### Monitors as server sources for services
-
-Services now accept a `monitor` parameter which can be used to point a service
-to a cluster of servers that are monitored by a monitor. The value of the
-`monitor` parameter should be the name of a monitor in the configuration name.
-
-For more details, refer to the [Configuration
-Guide](../Getting-Started/Configuration-Guide.md#cluster) section on clusters.
-
 ### Module commands
 
 Introduced in MaxScale 2.1, the module commands are special, module-specific
@@ -281,10 +271,46 @@ to large sets of data with a single query.
 For more information, refer to the [Maxrows](../Filters/Maxrows.md)
 documentation.
 
+### Galeramon Monitor new option
+The `set_donor_nodes` option allows the setting of _global variable_ _wsrep_sst_donor_  with a list the preferred donor nodes (among slave ones).
+
+For more details, read the [Galeramon documentation](../Monitors/Galera-Monitor.md).
+
+### Binlog Server encrypted binlogs
+The binlog server can optionally encrypt the events received from the master server: the setup requires MariaDB 10.1 master (with Encryption active) and the `mariadb10-compatibility=1` option set.
+
+For more details, read the [Binlogrouter documentation](../Routers/Binlogrouter.md).
+
 ## Bug fixes
 
-[Here is a list of bugs fixed since the release of MaxScale 2.0.X.](https://jira.mariadb.org/browse/MXS-739?jql=project%20%3D%20MXS%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20in%20(Fixed%2C%20Done)%20AND%20fixVersion%20%3D%202.0.0)
+[Here is a list of bugs fixed since the release of MaxScale 2.0.4.](https://jira.mariadb.org/browse/MXS-951?jql=project%20%3D%20MXS%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20in%20(Fixed%2C%20Done)%20AND%20fixVersion%20%3D%202.1.0%20AND%20fixVersion%20!%3D%202.0.1%20AND%20fixVersion%20!%3D%202.0.2%20AND%20fixVersion%20!%3D%202.0.3%20AND%20fixVersion%20!%3D%202.0.4)
 
+* [MXS-1025](https://jira.mariadb.org/browse/MXS-1025) qc_sqlite always reports " Statement was parsed, but not classified"
+* [MXS-977](https://jira.mariadb.org/browse/MXS-977) MaxAdmin show monitor output missing formatting
+* [MXS-951](https://jira.mariadb.org/browse/MXS-951) Using utf8mb4 on galera hosts stops maxscale connections
+* [MXS-889](https://jira.mariadb.org/browse/MXS-889) "Ungrade Test" Jenkins job fails with CeentOS/RHEL 5 and SLES11
+* [MXS-887](https://jira.mariadb.org/browse/MXS-887) create_env Jenkin job fails
+* [MXS-873](https://jira.mariadb.org/browse/MXS-873) Changing server status via maxadmin is not atomic
+* [MXS-832](https://jira.mariadb.org/browse/MXS-832) Problem with Regex filter as readconnroute doesn't wait for complete packets
+* [MXS-831](https://jira.mariadb.org/browse/MXS-831) new_master event not triggered by galeramon
+* [MXS-828](https://jira.mariadb.org/browse/MXS-828) Remove "Syslog logging is disabled." to stdout when starting without syslog
+* [MXS-825](https://jira.mariadb.org/browse/MXS-825) --execdir option does not work
+* [MXS-805](https://jira.mariadb.org/browse/MXS-805) Server weights don't work with LEAST_BEHIND_MASTER
+* [MXS-804](https://jira.mariadb.org/browse/MXS-804) Grants for user@IP/Netmask doesn't work
+* [MXS-799](https://jira.mariadb.org/browse/MXS-799) fatal signal 11 when socket could not be opened
+* [MXS-769](https://jira.mariadb.org/browse/MXS-769) Malloc return value must be checked.
+* [MXS-711](https://jira.mariadb.org/browse/MXS-711) All service ports use the same user data
+* [MXS-650](https://jira.mariadb.org/browse/MXS-650) Connection attempt w/o SSL to SSL service gives confusing error
+* [MXS-626](https://jira.mariadb.org/browse/MXS-626) Don't log anything to maxlog until it is known whether that is wanted.
+* [MXS-590](https://jira.mariadb.org/browse/MXS-590) MaxScale doesn't log an error when .secrets file is not owned by current user
+* [MXS-586](https://jira.mariadb.org/browse/MXS-586) Tee filter hangs when using range
+* [MXS-576](https://jira.mariadb.org/browse/MXS-576) Maxscale does not generate warning/error if incorrect values is set for persistpoolmax
+* [MXS-397](https://jira.mariadb.org/browse/MXS-397) Unsafe handling of dcb_readqueue
+* [MXS-390](https://jira.mariadb.org/browse/MXS-390) Lack of checks of dynamic memory allocation
+* [MXS-350](https://jira.mariadb.org/browse/MXS-350) Return value of realloc must not be assigned to provided pointer.
+* [MXS-348](https://jira.mariadb.org/browse/MXS-348) Incorrect use of strncat
+* [MXS-253](https://jira.mariadb.org/browse/MXS-253) Use of strncpy is dangerous
+* [MXS-126](https://jira.mariadb.org/browse/MXS-126) debug assert in TEE filter test
 
 ## Known Issues and Limitations
 

include/maxscale/service.h

@@ -90,7 +90,7 @@ typedef struct server_ref_t
 #define SERVICE_MAX_RETRY_INTERVAL 3600 /*< The maximum interval between service start retries */
 
 /** Value of service timeout if timeout checks are disabled */
-#define SERVICE_NO_SESSION_TIMEOUT LONG_MAX
+#define SERVICE_NO_SESSION_TIMEOUT 0
 
 /**
  * Parameters that are automatically detected but can also be configured by the
@@ -145,7 +145,7 @@ typedef struct service
     SERVICE_REFRESH_RATE rate_limit;   /**< The refresh rate limit for users table */
     MXS_FILTER_DEF **filters;          /**< Ordered list of filters */
     int n_filters;                     /**< Number of filters */
-    long conn_idle_timeout;            /**< Session timeout in seconds */
+    uint64_t conn_idle_timeout;            /**< Session timeout in seconds */
     char *weightby;                    /**< Service weighting parameter name */
     struct service *next;              /**< The next service in the linked list */
     bool retry_start;                  /**< If starting of the service should be retried later */

server/core/dcb.c

@@ -1915,14 +1915,11 @@ dprintOneDCB(DCB *pdcb, DCB *dcb)
     }
 }
 
-/*
+static bool dprint_all_dcbs_cb(DCB *dcb, void *data)
- * @brief Print DCB list statistics
- *
- * @param       pdcb    DCB to print results to
- */
-void
-dprintDCBList(DCB *pdcb)
 {
+    DCB *pdcb = (DCB*)data;
+    dprintOneDCB(pdcb, dcb);
+    return true;
 }
 
 /**
@@ -1930,23 +1927,19 @@ dprintDCBList(DCB *pdcb)
  *
  * @param       pdcb    DCB to print results to
  */
-void
+void dprintAllDCBs(DCB *pdcb)
-dprintAllDCBs(DCB *pdcb)
 {
+    dcb_foreach(dprint_all_dcbs_cb, pdcb);
+}
 
-    int nthr = config_threadcount();
+static bool dlist_dcbs_cb(DCB *dcb, void *data)
-
+{
-    for (int i = 0; i < nthr; i++)
+    DCB *pdcb = (DCB*)data;
-    {
+    dcb_printf(pdcb, " %-16p | %-26s | %-18s | %s\n",
-        spinlock_acquire(&all_dcbs_lock[i]);
+               dcb, gw_dcb_state2string(dcb->state),
-
+               ((dcb->session && dcb->session->service) ? dcb->session->service->name : ""),
-        for (DCB *dcb = all_dcbs[i]; dcb; dcb = dcb->thread.next)
+               (dcb->remote ? dcb->remote : ""));
-        {
+    return true;
-            dprintOneDCB(pdcb, dcb);
-        }
-
-        spinlock_release(&all_dcbs_lock[i]);
-    }
 }
 
 /**
@@ -1962,24 +1955,24 @@ dListDCBs(DCB *pdcb)
     dcb_printf(pdcb, " %-16s | %-26s | %-18s | %s\n",
                "DCB", "State", "Service", "Remote");
     dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n");
+    dcb_foreach(dlist_dcbs_cb, pdcb);
+    dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n\n");
+}
 
-    int nthr = config_threadcount();
+static bool dlist_clients_cb(DCB *dcb, void *data)
+{
+    DCB *pdcb = (DCB*)data;
 
-    for (int i = 0; i < nthr; i++)
+    if (dcb->dcb_role == DCB_ROLE_CLIENT_HANDLER)
     {
-        spinlock_acquire(&all_dcbs_lock[i]);
+        dcb_printf(pdcb, " %-15s | %16p | %-20s | %10p\n",
-        for (DCB *dcb = all_dcbs[i]; dcb; dcb = dcb->thread.next)
+                   (dcb->remote ? dcb->remote : ""),
-        {
+                   dcb, (dcb->session->service ?
-            dcb_printf(pdcb, " %-16p | %-26s | %-18s | %s\n",
+                         dcb->session->service->name : ""),
-                       dcb, gw_dcb_state2string(dcb->state),
+                   dcb->session);
-                       ((dcb->session && dcb->session->service) ? dcb->session->service->name : ""),
-                       (dcb->remote ? dcb->remote : ""));
-        }
-
-        spinlock_release(&all_dcbs_lock[i]);
     }
 
-    dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n\n");
+    return true;
 }
 
 /**
@@ -1995,27 +1988,7 @@ dListClients(DCB *pdcb)
     dcb_printf(pdcb, " %-15s | %-16s | %-20s | %s\n",
                "Client", "DCB", "Service", "Session");
     dcb_printf(pdcb, "-----------------+------------------+----------------------+------------\n");
-
+    dcb_foreach(dlist_clients_cb, pdcb);
-    int nthr = config_threadcount();
-
-    for (int i = 0; i < nthr; i++)
-    {
-        spinlock_acquire(&all_dcbs_lock[i]);
-        for (DCB *dcb = all_dcbs[i]; dcb; dcb = dcb->thread.next)
-        {
-            if (dcb->dcb_role == DCB_ROLE_CLIENT_HANDLER)
-            {
-                dcb_printf(pdcb, " %-15s | %16p | %-20s | %10p\n",
-                           (dcb->remote ? dcb->remote : ""),
-                           dcb, (dcb->session->service ?
-                                 dcb->session->service->name : ""),
-                           dcb->session);
-            }
-        }
-
-        spinlock_release(&all_dcbs_lock[i]);
-    }
-
     dcb_printf(pdcb, "-----------------+------------------+----------------------+------------\n\n");
 }
 
@@ -3512,7 +3485,9 @@ void dcb_process_idle_sessions(int thr)
             {
                 MXS_SESSION *session = dcb->session;
 
-                if (session->service && session->client_dcb && session->client_dcb->state == DCB_STATE_POLLING &&
+                if (session->service && session->client_dcb &&
+                    session->client_dcb->state == DCB_STATE_POLLING &&
+                    session->service->conn_idle_timeout &&
                     hkheartbeat - session->client_dcb->last_read > session->service->conn_idle_timeout * 10)
                 {
                     poll_fake_hangup_event(dcb);

server/core/poll.c

@@ -1522,17 +1522,6 @@ void poll_fake_hangup_event(DCB *dcb)
     poll_add_event_to_dcb(dcb, NULL, ev);
 }
 
-/**
- * Print the event queue contents
- *
- * @param pdcb          The DCB to print the event queue to
- */
-void
-dShowEventQ(DCB *pdcb)
-{
-}
-
-
 /**
  * Print the event queue statistics
  *

server/core/server.c

@@ -597,7 +597,7 @@ spin_reporter(void *dcb, char *desc, int value)
 }
 
 /**
- * Diagnostic to print all DCBs in persistent pool for a server
+ * Diagnostic to print number of DCBs in persistent pool for a server
  *
  * @param       pdcb    DCB to print results to
  * @param       server  SERVER for which DCBs are to be printed
@@ -605,22 +605,7 @@ spin_reporter(void *dcb, char *desc, int value)
 void
 dprintPersistentDCBs(DCB *pdcb, const SERVER *server)
 {
-    DCB *dcb;
+    dcb_printf(pdcb, "Number of persistent DCBs: %d\n", server->stats.n_persistent);
-    int nthr = config_threadcount();
-
-    for (int i = 0; i < nthr; i++)
-    {
-#if SPINLOCK_PROFILE
-        dcb_printf(pdcb, "DCB List Spinlock Statistics:\n");
-        spinlock_stats(&server->persistlock, spin_reporter, pdcb);
-#endif
-        dcb = server->persistent[i];
-        while (dcb)
-        {
-            dprintOneDCB(pdcb, dcb);
-            dcb = dcb->nextpersistent;
-        }
-    }
 }
 
 /**

server/core/session.c

@@ -460,16 +460,6 @@ printAllSessions()
     dcb_foreach(printAllSessions_cb, NULL);
 }
 
-/*
- * @brief Print session list statistics
- *
- * @param       pdcb    DCB to print results to
- */
-void
-dprintSessionList(DCB *pdcb)
-{
-}
-
 /** Callback for dprintAllSessions */
 bool dprintAllSessions_cb(DCB *dcb, void *data)
 {

server/modules/filter/cache/cachefiltersession.cc

@@ -404,9 +404,8 @@ int CacheFilterSession::handle_expecting_response()
         switch ((int)MYSQL_GET_COMMAND(header))
         {
         case MYSQL_REPLY_OK:
-        case MYSQL_REPLY_ERR:
             store_result();
-
+        case MYSQL_REPLY_ERR:
             rv = send_upstream();
             m_state = CACHE_IGNORING_RESPONSE;
             break;

server/modules/routing/avrorouter/avro.c

@@ -1074,7 +1074,7 @@ void converter_func(void* data)
     /** We reached end of file, flush unwritten records to disk */
     if (router->task_delay == 1)
     {
-        avro_flush_all_tables(router, true);
+        avro_flush_all_tables(router, AVROROUTER_FLUSH);
         avro_save_conversion_state(router);
     }
 

server/modules/routing/avrorouter/avro_file.c

@@ -68,7 +68,12 @@ bool avro_open_binlog(const char *binlogdir, const char *file, int *dest)
 
     if ((fd = open(path, O_RDONLY)) == -1)
     {
-        MXS_ERROR("Failed to open binlog file %s.", path);
+        if (errno != ENOENT)
+        {
+            char err[MXS_STRERROR_BUFLEN];
+            MXS_ERROR("Failed to open binlog file %s: %d, %s", path, errno,
+                      strerror_r(errno, err, sizeof(err)));
+        }
         return false;
     }
 
@@ -745,7 +750,7 @@ avro_binlog_end_t avro_read_all_events(AVRO_INSTANCE *router)
                 router->trx_count >= router->trx_target)
             {
                 update_used_tables(router);
-                avro_flush_all_tables(router, false);
+                avro_flush_all_tables(router, AVROROUTER_SYNC);
                 avro_save_conversion_state(router);
                 notify_all_clients(router);
                 total_rows += router->row_count;

server/modules/routing/avrorouter/avro_rbr.c

@@ -272,7 +272,6 @@ bool handle_row_event(AVRO_INSTANCE *router, REP_HEADER *hdr, uint8_t *ptr)
     /** There should always be a table map event prior to a row event.
      * TODO: Make the active_maps dynamic */
     TABLE_MAP *map = router->active_maps[table_id % sizeof(router->active_maps)];
-    ss_dassert(map);
 
     if (map)
     {

server/modules/routing/avrorouter/avro_schema.c

@@ -479,7 +479,15 @@ static const char *extract_field_name(const char* ptr, char* dest, size_t size)
     if (ptr > start)
     {
         /** Valid identifier */
-        snprintf(dest, size, "%.*s", (int)(ptr - start), start);
+        size_t bytes = ptr - start;
+
+        if (bt)
+        {
+            bytes--;
+        }
+
+        memcpy(dest, start, bytes);
+        dest[bytes] = '\0';
 
         make_valid_avro_identifier(dest);
         ptr = next_field_definition(ptr);

server/modules/routing/debugcli/debugcmd.c

@@ -134,12 +134,6 @@ struct subcommand showoptions[] =
         {0, 0, 0}
     },
 #endif
-    {
-        "dcblist", 0, 0, dprintDCBList,
-        "Show DCB statistics",
-        "Show statistics for the list of all DCBs(descriptor control blocks)",
-        {0}
-    },
     {
         "dcbs", 0, 0, dprintAllDCBs,
         "Show all DCBs",
@@ -166,12 +160,6 @@ struct subcommand showoptions[] =
         "Show the epoll polling system statistics",
         {0, 0, 0}
     },
-    {
-        "eventq", 0, 0, dShowEventQ,
-        "Show event queue",
-        "Show the queue of events waiting to be processed",
-        {0, 0, 0}
-    },
     {
         "eventstats", 0, 0, dShowEventStats,
         "Show event queue statistics",
@@ -262,12 +250,6 @@ struct subcommand showoptions[] =
         "Show a single session in MaxScale, e.g. show session 5",
         {ARG_TYPE_SESSION, 0, 0}
     },
-    {
-        "sessionlist", 0, 0, dprintSessionList,
-        "Show session list statistics",
-        "Show statistics for the list of all sessions",
-        {0, 0, 0}
-    },
     {
         "sessions", 0, 0, dprintAllSessions,
         "Show all sessions",