How to diagnose “Server Error 500” on Oracle AVDF?

Hello,

In this blog post, I will write about diagnosing “Server Error 500” on Oracle AVDF web console. When you login via AVDF Web console, connect is performed to a repository database. However, if this connection is not established by some reasons, a user cannot logon via Web Console and “Server Error 500” is occured. So, the direct reason of Server Error 500 is login failure to a repository database. Also, such login failures are caused by various reasons.
– User account or Password problem
– A repository database is not available
– The service is not started correctly
– Connection is prevented by some limitations of database
– Timeout
– The iNodes has no ifree on /var/lib/oracle
– Others
User can clarify by following steps the reason of Server Error 500 and can find out the workaround of this.
1. User account or Password problem
In this case, user may encounter following problems.

1) password is incorrect
2) account is locked
3) passphrase is not allowed
Need to check whether login from sqlplus can be success or not.

su – oracle
sqlplus /

If connect is failed, please try to change the password and unlock account.

To change password in 12.1.2 and unlock account by sysdba.

su – oracle
sqlplus “/as sysdba”
SQL> alter user identified by [password] account unlock.

2. a repository database is not available
If a repository database is terminated by some reasons, user cannot login to a database and encounter Server Error 500.
In this case, there are specific errors in /var/log/messages around login periods.

Failed to connect to database:ORA-01034: ORACLE not available

Almost of such cases, a repository database is terminated by some problems of generic database issues which is not only AVDF depends issues.
(For example, PMON is terminated by internal errors ORA-7445 or ORA-600)
For further analysys, please check alert.log and incident files of a repository database which are included in an AVS downloaded diagnostic file.

The recovery action in this case, please reboot AVS server. A repository database will be startup and user can login via Web console without Server Error 500.

3. The service is not started correctly
This is simiar with above “2. A repository database is not available”.

A repository database is configured as RAC Restart env and using +ASM instance.
So the trouble of GI resources, +ASM instance, TNS listener and so on, will prevent the connection to a repository database.
For example, a repository database cannot sync with +ASM instance since +ASM instance encounters hang, I/O or performance problem. Also, if TNS Listener is down, this Server Error 500 is caused. Upto 12.2.0.4, there is an issue with ASM audit files by which inodes gets exhausted due to which the ASM instance will be in Hung state, in such cases also server error 500 can be seen. By the above reasons, the restarting AVS will become workaround of this in this case. A diagnostic file in the current version of AVDF includes the log files of GI resources, so please check GI logs for further analysys.

4. Connection is prevented by some limitations of database
In this case, ORA-20 or ORA-18 may be the reason of this. If such errors are logged into /var/log/messages or alert.log intermittently or oftenly, please reboot AVS server for workaround Server Error 500. Also the reason of increasing sessions are variously. If need to investigate the reason of such increasing sessions, please contact Oracle Support with AVDF diagnostic files.

5.Timeout
In 12.2, loading a dashboad pageServer takes long periods and will cause Server Error 500. In previous releases, Proxy Error 502 will be caused in similar situation.

Bug 25074036 AV WEBCONSOLE PROXY 502 WHEN LOGGING IN AS AVAUDITOR USER (BUT NOT AVADMIN)
Base Bug 24383894 122BP3: AV SERVER LOGIN PAGE NOT LOADING. GETTING ERROR ‘SERVER ERROR 500’

This is a kind of performance issue by some SQLs on a repository database for loading information on Web page. For example, if there are huge number of alerts, this will be caused. For invesigating the direct reason of this, please get AWR reports or AWR SQL reports around this was caused and please check what sql took long periods for execution. In 12.2 BP#5 and later, the default timeout setting is expand from 2min to 5min. So upgrading 12.2 BP#5 and later may help this case.