Contents

Introduction

HyperFlex Installation

HyperFlex Upgrades

HyperFlex Bootstrapping

HX Connect

HX & Intersight

Network Logs

Data Replication

Stretch Cluster

HX Plugin

Audit Logs

Core REST APIs / AAA

ASUP

Data at Rest Encryption

Introduction

This document will be an inclusive list of which log files should be referenced for various processes and troubleshooting of various issues.

Tip: f you aren't sure what log file to look at, you can do a long listing of all files the log directory and sort by time to see the last logs written to!  Use the command ls -lah. For live troubleshooting, you may want to run ls -lahtr /var/log/springpath to see which log file was edited most recently at the bottom of the list.

root@SpringpathControllerG9ES5WGWDG:/var/log/springpath# ls -lah

total 433M

-rw-r-----   1 root    adm     123M Sep 28 16:15 stat-storfs.log

-rw-r-----   1 root    adm      63M Sep 28 16:14 debug-storfs.log

-rw-rw-rw-   1 root    root     50M Sep 28 16:14 stNodeMgr.log

-rw-r-----   1 root    adm      47M Sep 28 16:15 debug-cipmonitor.log

-rw-r-----   1 root    adm      41M Sep 28 16:15 stat-scvmclient.log

-rw-rw-rw-   1 root    root     22M Sep 28 16:14 stMgr.log

HyperFlex Installation

Topic

Log File

Notes

Troubleshooting GUI

/var/log/springpath/hxinstaller.log

The log contains all of the API's fired from client to server and their payload/responses

Troubleshooting Backend

/var/log/springpath/hxinstaller.log

GO Lang Server

/var/log/springpath/ucs_logs/ucs_automation

UCSM Failure

Host Related Issues

C:\ProgramData\Cisco\HyperFlex\Install\Log

Hyper-V

/var/log/hostd.log

ESXi

/var/run/log/vmkernel.log

ESXi

/var/run/log/scvmclient.log

ESXi Datastore mounts

Deploy and Create/Expand Failures

/var/log/tomact7/catalina.out

Tomcat logs.  Tomcat versions may differ across versions

/var/log/springpath/stDeploy.out

Thrift Failures

/var/log/springpath/stBootstrapGuiBackend.log

REST Failures

/var/log/springpath/springpath_validation.log

Validation Services

ESXi Agent Manager

C:\Program FIles\VMware\vCenter Server\vpxd\scripts

ESXi Agent Manager user fails to log in to VC

HyperFlex Upgrades

Log File

Notes

/var/log/springpath/stMgr.log

Check on all controller VM's when cluster upgrade fails

/var/log/springpath/stUpgradeSvc.log

All backend logs

/var/log/tomcat7/catalina.out

REST Logs

/var/log/springpath/springpath_validation.log

Validations

HyperFlex Bootstrapping

Log File

Notes

/var/log/springpath/stMgr.log

ESXi Cluster

/var/log/springpath/stUpgradeSvc.log

Hyper-V Cluster

HX Connect

Log File

Notes

/var/log/springpath/hxmanager.log

GUI Logs

All URL requests are logged.  To also enable response logging, edit:

/opt/springpath/storfs-mgmt-ui/config/config.json

LogHttpRequest and/or LogHttpResponse to "true"

restart hxmanager

**remember to turn off, else logging is too verbose

HX & Intersight

Log FIle

Notes

/opt/partner/cisco-hxdc-run/hxdc_current/bin/hxdp

Intersight Binary

/opt/partner/cisco-hxdc-run/hxdc_current/version

Current Version

/nfs/system/hx_device_connector/db/connector.db

Cloud Connection Information

/var/log/springpath/hx_device_connector.log

grep for "HX_PLAT"

Network Logs

Log File

Notes

/var/log/springpath/ucs_logs

Rack Unit Log

Data Replication

Log File

Notes

/var/log/springpath/stDataSvcMgr.log

/var/log/springpathdebug-replsvc.log

/var/log/springpathdebug-repl-cipmonitor.log

/var/log/springpathreplicationNetworkConfig.log

/var/log/springpath/debug-storfs.log

Stretch Cluster

Log File

Notes

/opt/springpath/config/stretch.tunes

Install & Deploy

HX Plugin

Log File

Notes

/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log

vCenter Log

/var/log/spingpath/stMgr.log

STCTL

Audit Logs

Log File

Notes

/var/log/springpath/audit-rest.log

REST

/var/log/vmware-vsphere-client/logs/vsphere_client_virg.log

Plugin

/var/log/shell.log

Stcli

Core REST APIs / AAA

Log File

Notes

/var/log/nginx/ssl-access.log

Nginx

/var/log/tomcat8/catalina.out

Tomcat.  Tomcat versions may change between HX versions

/var/log/springpath/stMgr.log

StMgr

/var/log/springpath/hxSvcMgr.log

HxSvcMgr

/var/log/springpath/hxHyperVSvcMgr.log

HxHyperVSvcMgr

/var/log/springpath/audit-rest.log

/var/log/springpath/aaa.log

Debug log of AAA subsystem

ASUP

Log File

Notes

/var/log/springpath/asupcl.log

/etc/cron.d/callhome

ASUP  cron job

Data at Rest Encryption

Topic

Log File

Notes

/var/log/springpath/hxmanager.log

Logs the calls that are made from the browser to the GUI proxy server that is forwarded to the REST server

/var/log/tomcat7/catalina.out

Logs the REST server calls.  Tomcat versions may change between HX versions

REST API's

all calls are proxied through NGINX & hxmanager (go-server) to actual web-server (ex: tomcat)

/var/log/springpath/hxmanager.log

HX Manager (go-server)

/varlog/nginx/*.log

Nginx Logs

/etc/nginx/conf.d/springpath.conf

Nginx conf file, look for routings

/var/log/tomcat7/catalina.out

Tomcat

/var/log/tomcat7/*

Tomcat

grep SED /var/log/springpath/stMgr.log | cut -c 1-40

REST API Calls stMgr service APIs

SED Server/PPP

/var/log/springpath/stevents.log

Events that start with Encryption

/var/log/springpath/encryption.log

grep for enable-security, modify-security, disable-security, secure-drives, rescan-inventory, secure-erase

SED Client

/usr/share/springpath/stevents.logs

Reports current SED state (Capable, Enabled, Locked)

/var/log/syslog

Debugging UCSM  Logs

DME

-Verify that lstoragelocal or lstorageRemote sent as part of OobStorageConfig

-Search for "drive security" or "KMIP" to find errors logged in DME code

-If using remote policy and OOB CIMC Mgmt IP is configured, make sure ConfigMgmtIpRules stage ran

BladeAG

-Search logs after receiving "OobStorageConfigFSM" or "AdminOobStorageConfigFSM" stimulus

-Check if route addition failure is reported as part of ConfigMgmtIpRules

-Check for XML starting with "lock-operation" in case of Enable/Modify/Disable Unlock Operation

-Check for XML starting with "clear-secure-drive", "enable-security-on-jbod", "secure-drive-group", "clear-secure-foreign, config-drive"

NOTE:  Last 2 XML checks for UCSM specific issues, all are for UCSM and certificate management issues

CIMC

/obfl/obfl-log*

/var/log/messages*

grepping for "storage" extracts the lines written by storaged and by mcserver involving storaged

/var/log/storaged_startup_log.txt

-contains the first 5000 lines written by storaged and by mcserver involving storaged

-search for same keywords as with bladeAG

-during certificate create, search for "cert" to see if there is a further log indicating the reason for failure