Contents

Introduction

Troubleshoot Cisco VOS Based Finesse Server With CLI

Generic Overview

Clean Up Disk Space

Audit CLI Commands Entered

Review Logs Online

Find a Relevant Log File

Review the File Online

Browse Directory in VOS

Check Platform History

Download Logs

Introduction

This document describes multiple troubleshooting scenarios for Cisco Finesse Server or any other Cisco Voice Operating System (VOS) based product with the use of java based command line interface (CLI).

Note: The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Troubleshoot Cisco VOS Based Finesse Server With CLI

Generic Overview

Cisco VOS CLI provides a set of commands that allows to accomplish certain troubleshoting tasks in a very effecient way.

Clean Up Disk Space

A very common task for any VOS based server is to clean up space in the common partition. This command can be used for that.

First, list the log files and specify a path and optional keys.

admin:file list activelog /desktop/logs/webservices date detail

11 Aug,2016 20:26:23 1,927,946 Desktop-webservices.2016-08-11T15-53-39.646.startup.log.zip

11 Aug,2016 21:44:42 1,105,322 Error-Desktop-webservices.2016-08-11T15-53-40.166.startup.log.zip

12 Aug,2016 01:48:58 1,741,638 Desktop-webservices.2016-08-11T20-26-23.164.log.zip

12 Aug,2016 03:35:30 1,126,078 Error-Desktop-webservices.2016-08-11T21-44-42.664.log.zip

12 Aug,2016 07:11:33 1,753,064 Desktop-webservices.2016-08-12T01-48-58.164.log.zip

12 Aug,2016 09:26:17 1,125,984 Error-Desktop-webservices.2016-08-12T03-35-30.164.log.zip

12 Aug,2016 12:34:08 1,741,476 Desktop-webservices.2016-08-12T07-11-33.664.log.zip

12 Aug,2016 15:17:05 1,125,790 Error-Desktop-webservices.2016-08-12T09-26-17.664.log.zip

Then remove the old log files. Optionally you can specify noconfirm key to avoid and not press yes as a confirmation.

admin:file delete activelog /desktop/logs/webservices/*webservices.2016-08-11* noconfirm

files: found = 4, deleted = 4

Audit CLI Commands Entered

This command lists all VOS CLI commands entered. Even the commands executed unsuccessfully.

From the output, you can see that the system was restarted, tomcat-trust certificate was imported and Tomcat service was restarted after that.

admin:file search activelog platform/log/cli* "running command" ignorecase recurs

Searching path: /var/log/active/platform/log/cli*

2016-11-03 09:59:12,558 INFO [main] sdMain.main - running command -> [utils system restart ]

2016-11-03 09:38:52,623 INFO [main] sdMain.main - running command -> [set cert import trust tomcat]

2016-11-03 09:39:30,978 INFO [main] sdMain.main - running command -> [utils service restart Cisco Tomcat]

2016-11-03 09:41:16,664 INFO [main] sdMain.main - running command -> [show cert own tomcat]

Search completed

Note: On some VOS based servers ignorecase key is not available.

Review Logs Online

Find a Relevant Log File

In case of a specific error message displayed in Finesse client, a search option case be used to determine the exact timestamp and the log file.

admin:file search activelog /desktop/logs/ "100011" recurs ignorecase

Searching path: /var/log/active//desktop/logs/

/var/log/active//desktop/logs/realm/realm.log:1289383032: Nov 18 2016 08:14:15.076 +0100:

[http-bio-8445-exec-10] INFO CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 100011

Search completed

The output shows the exact log file, timestamp with msec precision and the error message generated "Unable to authenticate user: 100011".

Review the File Online

Next step after you find relevant log file is to review it. And use file dump option for big log files is not very beneficial. Instead file view can be used, so you can navigate straight to the end of the file.

admin:file view activelog /desktop/logs/realm/realm.log

0: Aug 11 2016 15:52:14.423 +0200: [localhost-startStop-1] INFO

Loaded realm log config file at: /opt/cisco/desktop/conf/realm/log4j.xml

55577: Aug 11 2016 15:53:10.000 +0200: [Infrastructure Initialization, completed] INFO

Loaded realm log config file at: /opt/cisco/desktop/conf/realm/log4j.xml

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 374) :

956921239: Nov 14 2016 11:53:13.283 +0100: [http-bio-8445-exec-12] INFO

CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 1012

1206572479: Nov 17 2016 09:14:04.523 +0100: [http-bio-8082-exec-6] INFO

CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: MEDIASENSE_API

1217996207: Nov 17 2016 12:24:28.251 +0100: [http-bio-8082-exec-10] INFO

CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: MEDIASENSE_API

1289292089: Nov 18 2016 08:12:44.133 +0100: [http-bio-8445-exec-22] INFO

CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 5678

1289383032: Nov 18 2016 08:14:15.076 +0100: [http-bio-8445-exec-10] INFO

CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 100011

end of the file reached

options: q=quit, n=next, p=prev, b=begin, e=end (lines 361 - 374 of 374) :

Browse Directory in VOS

Can be used for multiple reasons. Amongst them are checks that the log file is generated to prove the process runs, check logs size for disk clean up or self-study.

This command will give a list of log files filtered by the name provided. Option detail displays date and size. An additional key date sorts the output based on the time files were generated.

admin:file list activelog /desktop/logs/webservices/*2016-08-12* det date

12 Aug,2016 07:11:33 1,753,064 Desktop-webservices.2016-08-12T01-48-58.164.log.zip

12 Aug,2016 09:26:17 1,125,984 Error-Desktop-webservices.2016-08-12T03-35-30.164.log.zip

12 Aug,2016 12:34:08 1,741,476 Desktop-webservices.2016-08-12T07-11-33.664.log.zip

12 Aug,2016 15:17:05 1,125,790 Error-Desktop-webservices.2016-08-12T09-26-17.664.log.zip

12 Aug,2016 17:56:43 1,741,988 Desktop-webservices.2016-08-12T12-34-08.664.log.zip

12 Aug,2016 21:07:52 1,125,050 Error-Desktop-webservices.2016-08-12T15-17-05.164.log.zip

12 Aug,2016 23:19:18 1,753,089 Desktop-webservices.2016-08-12T17-56-43.663.log.zip

13 Aug,2016 02:58:40 1,125,382 Error-Desktop-webservices.2016-08-12T21-07-52.664.log.zip

13 Aug,2016 04:41:53 1,742,441 Desktop-webservices.2016-08-12T23-19-18.664.log.zip

dir count = 0, file count = 9

Check Platform History

Basic diagnostic information about the server upgrades, Cisco Options Package (COP) file installations and reboots is very helpful. Especially during the troubleshoot process.

admin:file dump install system-history.log

=======================================

Product Name - Cisco Finesse

Product Version - 11.0.1.10000-24

Kernel Image - 2.6.32-431.20.3.el6.x86_64

=======================================

08/11/2016 12:46:19 | root: Install 11.0.1.10000-24 Start

08/11/2016 06:02:15 | root: Boot 11.0.1.10000-24 Start

08/11/2016 15:45:52 | root: Install 11.0.1.10000-24 Success

08/11/2016 15:45:53 | root: Boot 11.0.1.10000-24 Start

08/26/2016 12:50:14 | root: Cisco Option Install finesse-cce.1101.ES05.10000.cop Start

08/26/2016 13:25:51 | root: Cisco Option Install finesse-cce.1101.ES05.10000.cop Success

08/26/2016 13:51:14 | root: Restart 11.0.1.10000-24 Start

08/26/2016 13:51:40 | root: Boot 11.0.1.10000-24 Start

10/19/2016 17:16:48 | root: Shutdown 11.0.1.10000-24 Start

10/20/2016 12:17:10 | root: Boot 11.0.1.10000-24 Start

Download Logs

In case there is a need to pull the logs from the system for further analysis, then this command can be used. It is irreplaceble when Cisco Tomcat web server is down and Real-Time Monitoring Tool (RTMT) can not be used.

You can pull the logs based on time filter. In this example for the last two hours.

admin:file get activelog /desktop/logs/webservices/ reltime hours 2 compress

This command can take significantly long time,

and can also affect the system wide IOWAIT on your system.

Would you like to proceed [y/n]?y

Please wait while the system is gathering files info ...done.

Sub-directories were not traversed.

Number of files affected: 1

Total size in Bytes: 3460015

Total size in Kbytes: 3378.921

Would you like to proceed [y/n]? y

SFTP server IP:

Also, logs can be pulled based on the name filter. In this example, only files with log extension are pulled from install directory. The files are not compressed.

admin:file get install /*.log

Please wait while the system is gathering files info ...done.

Sub-directories were not traversed.

Number of files affected: 10

Total size in Bytes: 2640585

Total size in Kbytes: 2578.6963

Would you like to proceed [y/n]? n

Files transfer cancelled.

Another useful case is to collect the logs for a root cause analysis for a specific time range.

admin:file get activelog /desktop/logs/ abstime 09:00:11/13/16 09:00:11/14/16 recurs

Please wait while the system is gathering files info ...done.

Sub-directories were traversed.

Number of files affected: 2

Total size in Bytes: 7446444

Total size in Kbytes: 7271.918

Would you like to proceed [y/n]?

Note: Ensure to verify total size value and if it is too big either make lesser time range or put more specific directory.