StatsControl¶
-
class
borneo.StatsControl(config, logger, is_rate_limiting_enabled)[source]¶ Bases:
objectStatsControl allows user to control the collection of driver statistics at runtime.
The statistics data is collected for an interval of time. At the end of the interval, the stats data is logged in a specified JSON format that can be filtered and parsed. After the logging, the counters are cleared and collection of data resumes.
Collection intervals are aligned to the top of the hour. This means first interval logs may contain stats for a shorter interval.
Collection of stats are controlled by the following environment variables:
NOSQL_STATS_PROFILE=[none|regular|more|all]
Specifies the stats profile:
- none - disabled,
- regular - per request: counters, errors, latencies, delays, retries. This incurs minimum overhead.
- more - stats above with 95th and 99th percentile latencies. This may add 0.5% overhead compared to none stats profile.
- all - stats above with per query information. This may add 1% overhead compared to none stats profile.
NOSQL_STATS_INTERVAL=600 Interval in seconds to log the stats, by default is 10 minutes.
NOSQL_STATS_PRETTY_PRINT=true Option to enable pretty printing of the JSON data, default value is false.
Collection of stats can also be used by using the API:
NoSQLHandleConfig.set_stats_profile()orStatsControl.set_profile(). At runtime stats collection can be enabled selectively by usingStatsControl.start()ondStatsControl.stop(). The following example shows how to use a stats handler and how to control the stas at runtime:def stats_handler(stats): # type: (Dict) -> None print("Stats : " + str(stats)) ... config = NoSQLHandleConfig( endpoint ) config.set_stats_profile(StatsProfile.REGULAR) config.set_stats_interval(600) config.set_stats_pretty_print(False) config.set_stats_handler(stats_handler) handle = NoSQLHandle(config) handle = get_handle(tenant_id) stats_control = handle.get_stats_control() #... application code without stats # enable observations stats_control.start(); #... application code with REGULAR stats # For particular parts of code profile can be changed to collect more stats. stats_control.set_stats_profile(StatsProfile.ALL) #... more sensitive code with ALL stats stats_control.set_stats_profile(StatsProfile.REGULAR) #... application code with REGULAR stats # disable observations stats_control.stop() #... application code without stats handle.close()
The following is an example of stats log entry using the ALL profile:
A one time entry containing stats id and options:
INFO: Client stats|{ // INFO log entry "sdkName" : "Oracle NoSQL SDK for Python", // SDK name "sdkVersion" : "5.2.4", // SDK version "clientId" : "f595b333", // NoSQLHandle id "profile" : "ALL", // stats profile "intervalSec" : 600, // interval length in seconds "prettyPrint" : true, // JSON pretty print "rateLimitingEnabled" : false} // if rate limiting is enabled
An entry at the end of each interval containing the stats values:
``INFO: Client stats|{ "clientId" : "b7bc7734", // id of NoSQLHandle object "startTime" : "2021-09-20T20:11:42Z", // UTC start interval time "endTime" : "2021-09-20T20:11:47Z", // UTC end interval time "requests" : [{ // array of types of requests "name" : "Get", // stats for GET request type "httpRequestCount" : 2, // count of http requests "errors" : 0, // number of errors in interval "httpRequestLatencyMs" : { // response time of http requests "min" : 4, // minimum value in interval "avg" : 4.5, // average value in interval "max" : 5, // maximum value in interval "95th" : 5, // 95th percentile value "99th" : 5 // 99th percentile value }, "requestSize" : { // http request size in bytes "min" : 42, // minimum value in interval "avg" : 42.5, // average value in interval "max" : 43 // maximum value in interval }, "resultSize" : { // http result size in bytes "min" : 193, // minimum value in interval "avg" : 206.5, // average value in interval "max" : 220 // maximum value in interval }, "rateLimitDelayMs" : 0, // delay in milliseconds introduced by the rate limiter "retry" : { // retries "delayMs" : 0, // delay in milliseconds introduced by retries "authCount" : 0, // no of auth retries "throttleCount" : 0, // no of throttle retries "count" : 0 // total number of retries } }, { "name" : "Query", // stats for all QUERY type requests "httpRequestCount" : 14, "errors" : 0, "httpRequestLatencyMs" : { "min" : 3, "avg" : 13.0, "max" : 32, "95th" : 32, "99th" : 32 }, "resultSize" : { "min" : 146, "avg" : 7379.71, "max" : 10989 }, "requestSize" : { "min" : 65, "avg" : 709.85, "max" : 799 }, "rateLimitDelayMs" : 0, "retry" : { "delayMs" : 0, "authCount" : 0, "throttleCount" : 0, "count" : 0 } }, { "name" : "Put", // stats for PUT type requests "httpRequestCount" : 1002, "errors" : 0, "httpRequestLatencyMs" : { "min" : 1, "avg" : 4.41, "max" : 80, "95th" : 8, "99th" : 20 }, "requestSize" : { "min" : 90, "avg" : 90.16, "max" : 187 }, "resultSize" : { "min" : 58, "avg" : 58.0, "max" : 58 }, "rateLimitDelayMs" : 0, "retry" : { "delayMs" : 0, "authCount" : 0, "throttleCount" : 0, "count" : 0 } }], "queries" : [{ // query stats aggregated by query statement // query statement "query" : "SELECT * FROM audienceData ORDER BY cookie_id", // query plan description "plan" : "SFW([6]) [ FROM: RECV([3]) [ DistributionKind : ALL_PARTITIONS, Sort Fields : sort_gen, ] as $from-0 SELECT: FIELD_STEP([6]) [ VAR_REF($from-0)([3]), audienceData ] ]", "doesWrites" : false, "httpRequestCount" : 12, // number of http calls to the server "unprepared" : 1, // number of query requests without prepare "simple" : false, // type of query "count" : 20, // number of handle.query() API calls "errors" : 0, // number of calls trowing exception "httpRequestLatencyMs" : {// response time of http requests in milliseconds "min" : 8, // minimum value in interval "avg" : 14.58, // average value in interval "max" : 32, // maximum value in interval "95th" : 32, // 95th percentile value in interval "99th" : 32 // 99th percentile value in interval }, "requestSize" : { // http request size in bytes "min" : 65, // minimum value in interval "avg" : 732.5, // average value in interval "max" : 799 // maximum value in interval }, "resultSize" : { // http result size in bytes "min" : 914, // minimum value in interval "avg" : 8585.33, // average value in interval "max" : 10989 // maximum value in interval }, "rateLimitDelayMs" : 0, // total delay introduced by rate limiter in milliseconds "retry" : { // automatic retries "delayMs" : 0, // delay introduced by retries "authCount" : 0, // count of auth related retries "throttleCount" : 0, // count of throttle related retries "count" : 0 // total count of retries } }] }``
The log entries go to the logger configured in NoSQLHandlerConfig. By default, if no logger is configured the statistics entries, if enabled, will be logged to file logs/driver.log in the local directory.
Stats collection is not dependent of logging configuration, even if logging is disabled, collection of stats will still happen if stats profile other than none is used. In this case, the stats are available by using the stats handler.
Depending on the type of query, if client processing is required, for example in the case of ordered or aggregate queries, indicated by the false simple field of the query entry, the count and httpRequestsCount numbers will differ. count represents the number of
handle.query()API calls and httpRequestCount represents the number of internal http requests from server. For these type of queries, the driver executes several simpler queries, per shard or partition, and than combines the results locally.Note: connection statistics are not available for NoSQL Python driver.Attributes Summary
LOG_PREFIXMethods Summary
get_id()Returns a pseudo unique string to identify the NoSQLHandle object. get_interval()Returns the current collection interval. get_logger()Returns the current logger. get_pretty_print()Returns the current JSON pretty print flag. get_profile()Returns the stats collection profile. get_stats_handler()Returns the registered handler. is_started()Returns true if collection of stats is enabled, otherwise returns false. observe(request, req_size, res_size, …)Internal method only. observe_error(request)Internal method only. observe_query(query_request)Internal method only. set_pretty_print(pretty_print)Enable JSON pretty print for easier human reading. set_profile(profile)Set the stats collection stats_profile. set_stats_handler(stats_handler)Registers a user defined stats handler. shutdown()Logs the stats collected and stops the timer. start()Collection of stats is enabled only between start and stop or from the beginning if environment property NOSQL_STATS_PROFILE is not “none”. stop()Stops collection of stats. Attributes Documentation
-
LOG_PREFIX= 'Client stats|'¶
Methods Documentation
-
get_interval()[source]¶ Returns the current collection interval. Default interval is 600 seconds, i.e. 10 min.
-
set_pretty_print(pretty_print)[source]¶ Enable JSON pretty print for easier human reading. Default is disabled.
-
set_profile(profile)[source]¶ Set the stats collection stats_profile. Default stats stats_profile is NONE.
-
set_stats_handler(stats_handler)[source]¶ Registers a user defined stats handler. The handler is called at the end of the interval with a structure containing the logged stat values.