This section shows the "focus" of the release (via radar/spider chart) in terms of ticket distribution across 5 focus areas:
- Customer Request
- Speed of Deployment
- General Enhancement
SIGNIFICANT CHANGES FROM v5.2.3_ESR3 to v6.0.0
- New Components/RPM package dependencies:
- New search/indexing component - replaces sphinx
- Listens on localhost:9200 by default
- Java application that runs with heap allocation -Xmx2g
- New static content provider/tomcat proxy - replaces httpd
- Listens on ports 80 (http) and 443 (https) by default
- New log rotation utility - replaces rotatelogs
- The 6.0.0 RPMs are now el6/el7 independent i.e. same set can be installed on both el6 and el7 platforms
- Minimum JDK version supported: v1.8.0_121 (up from v1.8.0_20)
- apache-tomcat changes:
- Minimum version supported: v8.0.36
- moogpoller servlet now offers websocket implementation, rather than long polling
- No longer listens on ports 8009, 8080, or 8081
- No longer running with SSL enabled connectors (SSL is terminated at nginx)
- Runs with new increased heap allocation (-Xmx2g) in service script (up from -Xmx512m)
- The moogpoller/hastatus endpoint has been removed (due to switch to websockets)
- Removal of Twitter data from users and Twitter module from the Notifier moolet
- Removal of the Journaller moolet (as the Journal thread no longer exists)
- Situation column changes
- Removal of 'External Severity' field (and all associated graze/moogdb functions/endpoints)
- Removal of 'Primary Process' and 'Primary Service' (and all associated graze/moogdb functions/endpoints)
- The moogtoolrunner user is created out-of-the-box during RPM installation and added to the 'moogsoft' group
- Moog services (moogfarmd and LAMs) no longer start on server boot by default. This prevents port clashes on reboot etc
- The migration utility for upgrade to v6.0.0 will remove any installed Integrations/SolutionPaks and any associated UI Cookbooks and recipes. These will need to be re-added post-upgrade
- Dashboards no longer support embedded Alert or Situation views
[BESSIE-242] - Improve the Incident.MOOG User Experience
Moogsoft AIOps V6.0.0 brings with it a new user interface to simplify and enhance the user experience.
The workflow for Situations is central to the UI and visibility of metrics such as MTTA (Mean Time To Acknowledge) and MTTR (Mean Time To Resolve) are visible on the Summary screen:
Use of WebSockets:
- Behind the scenes, the UI now uses WebSockets rather than long polling to keep user sessions up to date
- This mechanism maintains one or more constant tcp connection(s) between a user's browser client and the server through which two-way communication occurs
- The WebSockets mechanism offers significant efficiency improvements over the request-response handling associated with long polling
- In side-by-side tests with multiple concurrent users, WebSockets has been seen to offer significant Tomcat CPU usage reductions compared to long polling when serving those users in busy systems
- Note that, although still named moogpoller, this servlet is now re-purposed for the handling of WebSockets
The proxy_read_timeout property for the moogpoller proxy in nginx (in /etc/nginx/conf.d/moog-ssl.conf) has been set to the maximum allowable (30d) to prevent periodic disconnection of the websocket (which would cause a users UI session to refresh by itself).
If, in deployments, the AIOps system needs to be fronted by another nginx (or similar application e.g HAProxy) then ensure this (or similar) property is set.
The UI will not function if AIOps is installed on an Amazon Web Services (AWS) instance that is fronted by their CloudFront technology. This technology does not support WebSockets.
For a full list of all the changes in the UI please see What's New in Moogsoft AIOps.
Some hotkeys available in Incident.MOOG v5.2.3 (and previous releases) have been deprecated in v6.0.0 and will be disabled as part of an upgrade
[BESSIE-24] - Mobile Experience (Beta)
Moogsoft AIOps V6.0.0 now has a new mobile interface:
|Login Screen||Open Situations||Situation Details|
This interface is enabled using System Administration → Labs → Configure.
Please note: This feature is BETA and is disabled by default
[MOOG-5766] - Replace Sphinx with Elasticsearch
Moogsoft AIOps now uses Elasticsearch to index its data and provide search functionality.
Elasticsearch is installed by the moogsoft-search package as a dependency. Elasticsearch is available from the new elastic yum repository (see installation guide).
Elasticsearch is installed (by default) in /usr/share/elasticsearch/
Configuration files are stored (by default) in /etc/elasticsearch/ and logs are written to /var/log/elasticsearch/
Elasticsearch runs via a service script: /etc/init.d/elasticsearch and starts its own java process.
The new Moogsoft AIOps indexing utility is here: $MOOGSOFT_HOME/bin/utils/moog_indexer and provides the following options:
As for Sphinx, periodic indexing is carried out by the indexer once-a-minute via a cron job. The default indexing behaviour out of the box (via crontab) is the following:
This means that the indexer will use five threads and a batch size of 1000. Any changes in alerts or Situations since the last run of the moog_indexer will get sent to Elastic.
The default crontab call to moog_indexer DOES NOT provide the -p flag which is only required for 'Private Teams' functionality (when the 'all_data' permission removed from a role). If this flag is not provided, then all alerts will be searchable by any team member
To perform a complete re-index of the data in Moogsoft AIOps, the following moog_indexer command must be used:
That command will: Delete old index data, run a full index (with batch size of one thousand), and use five threads.
To allow AIOps to communicate with Elasticsearch, $MOOGSOFT_HOME/config/system.conf has been updated (relevant code block below):
- Adding or removing a column via the UI 'Column Management' interface does not require/trigger a full re-index unlike previous releases.
Elasticsearch runs as a Java process and by default uses 2GB RAM (defined as -Xmx2g in /etc/elasticsearch/jvm.options).
By default, Elasticsearch listens only on localhost:9200 as it is typically co-located with the UI components. To make it listen on all interfaces (e.g. for a distributed install where it is separate from the UI components) the following line should be added to the /etc/elasticsearch/elasticsearch.yml file and the Elasticsearch service restarted:
[BESSIE-451] - Replace httpd with Nginx
AIOps now uses Nginx instead of httpd to provide static UI content and as a proxy for Tomcat.
Nginx is installed by the moogsoft-ui package as a dependency. Nginx is available from the epel yum repository.
Nginx stores its configuration files under /etc/nginx/ and logs are written to /var/log/nginx/
The AIOps configuration file for Nginx is here: /etc/nginx/conf.d/moog-ssl.conf
The two default ssl certificate files used by Nginx are stored here:
Nginx uses the logrotate package to handle log rotation of the moogfarmd and LAM log files. logrotate is run via a cronjob which is deployed by either the moog_init_server.sh or moog_init_lams.sh init scripts.
The /etc/logrotate.d/moog-logrotate file contains configuration for logrotate and is deployed during installation of the moogsoft-server or moogsoft-lams packages.
[MOOG-6139] - Syslog LAM
A new syslog_lam is now available, which can support UDP, TCP and file monitored messages. The LAM uses regular expressions to reject or accept and extract events from syslog messages.
Syslog is a way for network devices to send event messages to a logging server – usually known as a Syslog server. The Syslog protocol is supported by a wide range of devices and can be used to log different types of events.
The lam binary is $MOOGSOFT_HOME/bin/syslog_lam and the configuration file is : $MOOGSOFT_HOME/config/syslog_lam.conf
[MOOG-6072] - Feedback Sigaliser (Beta)
Improvements have been made to the Feedback Sigaliser:
- The brain name now reflects the Situation where it was generated from, as well the brain category (that can be defined by users)
- The Feedback Sigaliser UI allows a user to edit existing brains (name, category, and description), and select a training setting on a per-brain basis (i.e. recommended or strict). Users can also choose to delete brains that are no longer needed
- An initial version of Import/Export functionality for brains has been added. In the current release, it means that brains are persisted between restarts of moogfarmd
The Feedback Sigaliser is also now accessible via the System Administration dialog:
The following section of $MOOGSOFT_HOME/config/moog_farmd.conf (under the 'Feedback' moolet) has been updated for this release:
- Similar to the UI Cookbook, the UI Feedback Sigaliser controls only work in environments with a single moogfarmd instance
- There are remaining known issues and performance improvements which will be addressed in an upcoming release
Improvements included in this Release
[MOOG-4774 / MOOG-6141 / MOOG-6044] - Improved Entropy Calculation
AIOps v6 now ships with an improved entropy algorithm (EntropyV2) which is enabled by default.
The events_analyser utility/AlertBuilder moolet now calculates the entropy of an alert by taking into consideration the significance of tokens in the context of the alert rather than the entire dataset. Tokens within an alert which change frequently now contribute negatively to alert entropy (the alert may not be as significant). This is in contrast to 'EntropyClassic' where the entropy of each alert takes into consideration the significance of tokens only in the context of the entire dataset.
Similar to previous functionality, if the AlertBuilder moolet receives an event signature it has seen before (via a previous run of the events_analyser utility), it will set the event (and hence alert) entropy to match the value saved in the moog_reference.alert_properties table.
However, EntropyV2 improves the quality of entropies for event signatures not seen before by calculating the value in real-time based on any tokens seen before. After a certain number of events with the same signature have been received by the AlertBuilder moolet, the calculated entropy value for that signature is saved in the moog_reference.alert_properties table.
Tests on some sample datasets has shown that the spread of alert entropies is less evenly distributed in EntropyV2 compared to EntropyClassic (on a scale of 0-1). It was observed that it is now more likely for alerts to be assigned a lower entropy under EntropyV2 (the highest concentration of alert entropies might be for example between 0.15 and 0.5).
It is important that the events_analyser (incremental at least) is run frequently in environments with a high number of alerts with a low event count to ensure that moogfarmd does not need to keep in memory a large number of event signatures it has not seen before.
The setting is configured in $MOOGSOFT_HOME/config/events_analyser.conf:
The out-of-the-box partitioning fields have also changed to only tokenize the 'description' field of events but this is customizable:
If 'EntropyV2' mode is enabled:
- The 'token_variation_threshold' setting in $MOOGSOFT_HOME/config/events_analyser.conf is not used
- The 'properties_from_db' and 'update_properties_in_db' AlertBuilder moolet settings in $MOOGSOFT_HOME/config/moog_farmd.conf are not used (the behaviour in 'EntropyV2' mode is 'true'/'true' for those settings)
For backwards compatibility, the previous entropy algorithm (EntropyClassic) is still available, but must be configured using the 'entropy_calc' field and 'EntropyClassic' value in $MOOGSOFT_HOME/config/events_analyser.conf
Summary of new behaviour:
- If an event signature has been seen before, and the value has been saved into the moog_reference.alert_properties table (via a previous run of the events_analyser), then that entropy value will be used.
- If an event signature has not been seen before, the entropy will be calculated in real-time until a certain number of events with that signature have been received. At that point, the entropy will be saved into the moog_reference.alert_properties table for future retrieval.
- moogfarmd will store token entropies for new signatures until either the events_analyser has been run or a period of time has elapsed. If this happens, the entropy will be calculated properly the next time the events_analyser is run.
- If entropy is not used at all in an AIOps deployment (or the events_analyser is not run at all), it is recommended to set 'entropy_calc' to 'EntropyClassic' and set the 'properties_from_db' setting to 'false' for all running AlertBuilder moolets
- Entropy data for 'EntropyClassic' and 'EntropyV2' are not compatible. If a user wishes to switch between the two modes, the events_analyser full (non-incremental) run must be executed after the setting has been changed to ensure all the entropy data matches the right configuration type
- The new 'EntropyV2' algorithm calculates signature entropies in real-time (in the AlertBuilder moolet) and this requires that moogfarmd stores data (in memory) for this calculation. Systems with a higher number of alerts with a low event count and systems where the events_analyser is not run often, should be aware that moogfarmd will use more memory than systems that run events_analyser often
- An event signature will be stored in moogfarmd memory until either the events_analyser utility has been run, or no new instances of that signature have been seen for a while
- An event signature will be stored in moogfarmd memory until either the events_analyser utility has been run, or no new instances of that signature have been seen for a while
- Due to the new alert entropy distribution under EntropyV2, the previous values used by the alert field 'Significance' are no longer as evenly distributed
- It is worth being aware that any Sigalisers using an 'entropy_threshold' value of more than 0 might need adjusting due to the new alert entropy distribution under EntropyV2
[MOOG-VARIOUS] - Transaction Packaging Improvements
A number of functions have now been updated to take advantage of the mysql deadlock retry settings in system.conf (maxRetries and retryWait) and code changes have been made to other functions to improve their reliability:
- [MOOG-5277] - Wrap mergeSigs in a retryable transaction
- [MOOG-5364] - Wrap loadSystemConfigs and getThreadEntries in retryable transactions
- [MOOG-5521] - [add_alerts_to_sig] Design/efficiency improvements
- [MOOG-5966] - Wrap createUser in a retryable transaction
- [MOOG-6026] - Wrap HA Status in a retryable transaction
- [BESSIE-1090]- Wrap Housekeeper in a retryable transaction
[BESSIE-1130] - Wrap CLoadAlertTimeline in a retryable transaction
[BESSIE-1139] - Wrap CInviteUser in a retryable transaction
[BESSIE-1140] - Wrap CLoadSituationTimeline in a retryable transaction
- [BESSIE-1228] - Wrap CSetSigResolveState in a retryable transaction
[BESSIE-1107] - "Generic" Integration renamed to "Webhook"
The 'Generic' REST LAM integration has been renamed to Webhook.
[BESSIE-303] - As an operator, I would like a new version to be applied without clearing my browser cache
Moogsoft AIOps UI files are now versioned against each release, which means that from this point onwards, the browser cache does not need to be cleared between upgrades of AIOps.
[MOOG-3825] - UI Servlets now addressed as single HA Instance
The three UI servlets (moogsvr, moogpoller and toolrunner) are no longer classed as distinct HA "instances" and are now rolled-up together as a single "servlets" instance. This is because it only makes sense to carry out failovers of servlets within a Tomcat instance altogether, rather than separately.
Thus, in $MOOGSOFT_HOME/config/servlets.conf, ha configuration on an individual servlet basis is no longer supported and a single "ha" block is defined to specify the HA settings for that Tomcat server.
For example in a 2 server setup with an active (SURBITON) and passive (KINGSTON) UI stack the server1 servlets.conf might look like:
...and the server2 servlets.conf might look like:
...and ha_cntl will report this as:
...and the UI - Self Monitoring page will represent this as:
[MOOG-5485] - Increase allowed Mooms message sizes for messages on Events and Alerts topics
To facilitate future improvements, messages on the events and alerts topics are now permitted up to 128k (up from 64k).
Improved logging has also been added to indicate oversized messages and on which topic they occurred.
Despite this new allowance, it is generally not recommended to allow messages to exceed 64k (e.g. via moobot code).
[MOOG-6049] - Split moog_sigdb.sql into more manageable files
The moogdb and moog_reference schemas have been split into multiple files to simplify upgrades and make the schemas more maintainable.
There are two new folders under $MOOGSOFT_HOME/etc/moog/:
Both of these folders contain:
- An SQL file to insert default data into the schema
- A script to initialize that database
- A folder containing stored procedure SQL files
- A folder containging table definition SQL files
- A folder containing trigger definition SQL files
- A folder containing view definition SQL files
There is also a new script to initialize both databases from scratch ( $MOOGSOFT_HOME/etc/moog/load_db.sh ). This is called by the $MOOGSOFT_HOME/bin/utils/moog_init_db.sh script when initializing the database:
The above command can also be used to reset the schemas to their default states (clean).
[MOOG-6028] - Improvements to better support busy environments out of the box
Several configuration settings have been changed to better support busy systems out of the box:
- mysql → maxRetries: The default is now 10. It used to be 5
- mysql → retryWait: The default is now 50. It used to be 10
- failover → margin: The default is now 10. It used to be 3
- moogsvr → priority_db_connections: The default is now 25. It used to be 10
- The -Xmx setting has been increased from 512M to 2G
[MOOG-4831 / MOOG-6063 / STHREE-90] - Redesigned Integration/SolutionPak framework (Service scripts in RHEL7 need to be runnable by non-root user to support toolrunner functionality)
Due to changes in the way that RHEL7/CentOS7 service scripts work, Integrations have been re-written to run as processes rather than services. Therefore, LAMs are no longer run using service scripts when run as part of an Integration or Solution Pak.
Starting or stopping an Integration/SolutionPak is now controlled via a new utility: $MOOGSOFT_HOME/bin/utils/process_cntl (This is not a utility that the end user needs to run manually).
A new service script has been added to support this: /etc/init.d/moogstartupd and this is a run-on-startup script responsible for restarting any integrations which were running before a system was rebooted/shutdown.
As part of this ticket, the moogtoolrunner user is now created out-of-the-box and put into the moogsoft group.
Also changed, is the 'System Administration → Self Monitoring' process control behaviour. In Incident.MOOG v5.2.3 and previous, this functionality used service scripts to start/stop/restart moogfarmd and LAM instances. This now uses the process_cntl utility.
If the default moogtoolrunner user is not used to start and stop integrations, then the /etc/init.d/moogstartupd script needs to be updated accordingly (swap out the old toolrunner username with the one being used)
Integrations and Solution Paks are now extracted using a utility ($MOOGSOFT_HOME/bin/utils/integration_installer) which is run on a clean install moog_init operation ($MOOGSOFT_HOME/bin/utils/moog_init.sh -Iu root) and an upgrade operation($MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -x).
[MOOG-6164] - Add SSL/TLS support for mailer
The 'mailer' moobot module (used out-of-the-box by the $MOOGSOFT_HOME/bots/moobots/Notifier.js moobot) has been updated to support SSL/TLS via two new flags in the initTransport method:
- If using port 587, set start_tls to true and use_tls to false
- If using port 465, set start_tls to true and use_tls to true
- If using port 25, set start_tls to false and use_tls to false (or comment both flags out)
[MOOG-6154] - Remove statistic values that are no longer required
The following values in the 'situation_summary' object are no longer available (via graze or moobot):
The following values in the 'system_summary' object are no longer available (via graze or moobot):
[MOOG-6162] - Introduce additional indexes to improve statistic performance
Indexes have been added to various database tables which will improve the performance of statistics calculations.
[ MOOG-6019 / MOOG-6090] - Improve create_alert performance
The performance of the alert creation process has been improved by up to 30% under certain conditions.
[MOOG-6096 / MOOG-6099 / MOOG-6100] - Add framework to add deltas to messages on the bus
Work has been done to allow future iterations of AIOps to know more about changes occurring in the system in order to improve performance.
[MOOG-6155] - Limit "open ended" statistics to the last week's sigs
The graze endpoint getSystemSummary will now only return statistics for the last week (alert_count and total_events). The v6.0.0 UI Summary page metrics are also limited to the last week:
- System Overview
- Team Overview
- Services Overview
[MOOG-6149] - Archiver: Ensure large export or removal operations are resilient to network/connection issues
The moog_archiver has been updated to improve the reliability of the removal/export process by batching up the remove/export operations. To reduce the impact on the database, the archiver will action operations on a days worth of data at a time. This is not configurable. The archiver output log is more verbose than the previous release which allows for more detailed debug logging.
New command line options have been added to the $MOOGSOFT_HOME/bin/utils/moog_archiver to support the new behaviour (see below). If not provided, defaults values will be used for each argument.
Bugs Fixed in this Release
[MOOG-5852] - Feedback sigaliser: fixing brain naming and descriptions
An issue which meant that all brains in use by the Feedback sigaliser had the same name has been fixed. Now, the brain name is configurable by the user and consists of the situation it was generated from and the category, which is a customizable text field.
[MOOG-5873] - Feedback sigaliser: unlearn is not working
An issue where 'unlearn actions' were not carried out by the Feedback Sigaliser has been fixed.
[MOOG-5949] - Feedback sigaliser: simhash sometimes produces wrong result
An issue where alerts that were expected to match a brain were not correctly clustered (and vice versa) has been fixed.
[MOOG-6023] - Logfile LAM load_at_start does not read file from start until file is updated
An issue which prevented the logfile_lam from reading the 'target' file from the beginning when the 'load_at_start' flag was set to true has been fixed.
[MOOG-6043] - Tomcat dbpool deadlocks when processing concurrent logins
An issue which prevented tomcat from serving user requests when under heavy load has been fixed.
[MOOG-6050] - moog_init scripts do not process -d mysql flag arguments
An issue which prevented the init scripts from configuring the database correctly has been fixed.
[MOOG-6151] - Lambots and Moobots: Large JS object size can cause compilation errors.
An issue which prevented a LAM from processing large Lambot files (>64k) has been fixed. Similarly large moobots now require the moobot_optimization parameter to be set to -1 in $MOOGSOFT_HOME/config/moog_farmd.conf.
[MOOG-6150] - Sort the sig_event_codes table
An issue where the user would see an entry in the Journal thread for the wrong type of situation/alert/user event has been fixed.
[MOOG-6118] - Cookbook filter object null behaviour
An issue which meant that Cookbook exclusion/inclusion filters containing 'is null' did not work as expected has been fixed.
[MOOG-6114] - Feedback Sigaliser: same alert not recognized by the brain
An issue which meant that in rare circumstances one of the alerts chosen to create a brain will not be included in future Feedback Situations. This has been fixed.
[MOOG-6086] - moog_init_functions.sh does not correctly replace database names
An issue which meant that the moog_init scripts did not correctly replace the database name in $MOOGSOFT_HOME/config/system.conf has been fixed.
[MOOG-6012] - [moog_farmd]: Change "'agent_time' > 1200 seconds before latest event" log entry from WARN to INFO
The following moogfarmd.log message has been decreased from WARN level to INFO level:
[MOOG-5991] - Only one instance of events_analyser should run at a time
To prevent multiple runs of the events_analyser from overlapping, the events_analyser script has been updated to only allow one instance of the script to run at a time.
[MOOG-5919] - MySQLSyntaxErrorException: Failure getTokenEntropies
An issue caused when an event containing no tokens was sent into moogfarmd has been fixed.
[MOOG-5576] - security.conf should not contain plaintext password
$MOOGSOFT_HOME/config/security.conf supports a new key type - encryptedUserDnLookupPassword
This key can be used instead of the userDnLookupPassword if the password cannot be in plain text
The $MOOGSOFT_HOME/bin/utils/moog_encryptor utility can be used to generate a secure password.
Below is an example of how the new key can be used in security.conf:
[MOOG-4833] - moog_install_validator.sh script does not correctly handle paths with spaces in
The $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh script no longer reports errors if a file or folder containing spaces has been added somewhere under $MOOGSOFT_HOME.
[MOOG-4551] - Moog services should not run on startup by default
The following AIOps services no longer run on system startup by default:
The services can be set to run on startup using the following commands (run as root) if needed:
[MOOG-6104] - Rehup messages to farmd to start an already running moolet report an error (and vice versa)
An issue whereby moogfarmd would report an error if asked to start a moolet which was already running has been fixed.
[MOOG-5942] - correct usage text in servicenow_authentication script
An issue which meant that the command line utility $MOOGSOFT_HOME/bin/utils/servicenow_authentication would return incorrect usage information when given incorrect arguments has been fixed.
[MOOG-5070] - Service Now Integration Installation Issues / Improvements
The ServiceNow integration has been updated as part of MOOG-4831 / MOOG-6063 / STHREE-90 and multiple minor issues have been resolved.
[MOOG-6182] - PRC is enabled checking affects the performance on high number of concurrently users
An issue which meant that the status of the 'Probable Root Cause' feature was checked against the database more often than necessary has been fixed.
Upgrade instructions are available:
Known issues with functionality in this release:
- [MOOG-6679] - Using createAlert(event, true) on alert with no custom_info and event which has custom_info causes exception
- [MOOG-6681] - Propagate Situation Actions not working
- [MOOG-6673] - Variable substitution doesn't work for Alert Server Tools
- [MOOG-4616] - [SERVER] After rotation of the target logfile, the logfile_lam stops reading the target
- [MOOG-6671] - Team room attachments not working correctly
- [MOOG-5552] - [UI] Self monitoring tree-view occasional render issue
- [MOOG-5665] - [UI] Self Monitoring -> Processing Metrics: Cannot distinguish between multiple components of same process type
- [MOOG-5672] - [SERVER] With Hazelcast enabled, moobot constants cannot use objects as values.
- [MOOG-5741] - [SERVER] Depending on mooms config, moog component "Control" queues spread across Rabbit Cluster on startup - leads to problems if rabbit node (or server) lost.
- [MOOG-5796] - [HA] Not possible to change moogfarmd log level at runtime on passive instance[MOOG-5891] - [UI] In MSEdge, Situation and Alert view context menu option to Copy Cell(s) and Row(s) does not work
- Workaround: Use the normal keyboard shortcuts for copying text
- [MOOG-5979] - [Graze] Inconsistency with sitn_id return parameters (for successful requests)
- [MOOG-4549] - [Graze] Error messages should refer to the correct field name (for unsuccessful requests)
- [MOOG-6027] - [HA] The asynchronous Persistence Store operations should guarantee that the passed Object will not be concurrently modified