How can I setup health monitoring for my BOC product?

Do I need health monitoring?

If you operate a BOC product in your own infrastructure (on premise) you may want to setup monitoring for your product services. Health monitoring can help to identify the building of bottlenecks (e.g. because of scenario growth), stability issues or outages early on, so that counter measures can be taken before real problems happen.

This article lists several monitoring possibilities. You do not have to implement all of them. Just select the ones that fit your organization and your needs.

Note: This article refers to the operation of ADONIS, ADOIT or ADOGRC in your own infrastructure (on premise).

If you are using a BOC product in the BOC cloud (SaaS), BOC will take care of health monitoring.

How do I set up monitoring?

Monitor service availability

Calls to the auth.view of your service URL with a short timeout (recommendation: 10 seconds).

If the call returns status code HTTP 200 as response, the full service stack (web server, application server, database) is available and response times are reasonable.

Any other return code indicates a problem. In this case follow the guidelines in "How to fix an outage (on premise)?".

Example: https://adoxx.[mycompany].com/auth.view?forceConnectorType=STANDARD

Monitor process health

Monitor if all processes are running that are needed for your BOC product to function.

Setup alerts if any of these processes are stopped or become unresponsive:

• Web server service (Tomcat.exe).
• Application server service of your BOC product (aserver.exe).
• Query engine service of your BOC product (aqe_server.exe)
• FTS service of your BOC product (afts_server.exe)
• Database instance.

System monitoring

Make sure, system monitoring of the following servers is in place:

• Web server
• Application server
• Database server

Typical indicators to monitor include:

• CPU usage
• RAM usage
• Free disk space

If any of these resources becomes a bottleneck, make sure that you are alerted early and resolve the problem quickly.

Database monitoring

Contact your database team to monitor resource consumption, workload and number of used database connections on the database instance of your BOC product.

Over time resource consumption may rise because of scenario growth, growth of the user base and/or individual workload spikes. Make sure the database instance is not overloaded and allocate more resouces as it becomes necessary.

Log monitoring

Scan the following log-files of your BOC product for entries that indicate health problems:

• aworker-log: "\<application server log directory>\<timestamp>_aworker.log"
• aserver-log: "\<application server log directory>\<timestamp>_aserver.log"
• webserver-log: "\<web server log directory>\<BOC product and version>.log" (e.g. ADONIS15.0.log)
• database-errorlog: "\<application server log directory>\<timestamp>_adoxx_db.log"

aserver-log
Keywords	"System memory status"
Example entry	"System memory status (total / available): 16383 MB / 543 MB"
Measure	If there is little memory available (less than 2 GB or less than 10% of total memory), increase RAM on the application server machine.

aworker-log
Keywords	"[WARN] Performance issue: m_pDataSource"
Example entry	2023-09-07 21:55:57.182 [4040][1612009049] [52341][WARN] Performance issue: m_pDataSource->ExistsRepoInstInDB(pID->GetUUID(), bRet) took 16.421 seconds
Measure	If encountered more frequently than 10 times / day it may indicate upcoming performance problems that are most likely related to database or network performance.

webserver-log
Keywords	"Worker lost"
Example entry	2023-02-06 15:19:04,171 ERROR [pool-3-thread-14][S] WORKER LOST AS1:41403 - com.boc.axw.core.applicationserver.PingAWorkerThread.onAWorkerLost(PingAWorkerThread.java:131)
Measure	If encountered more frequently than 3 times / day, it indicates problem. Contact BOC technical support and provide a Support Information Package.

database-errorlog
The database-errorlog should always be empty.	The log contains client-side error messages that have been returned from the database server to the client.	If the log contains any entries, follow-up on the reported database error message.