Contents

Introduction

Supported HX Systems

When to Use

How to Use

Analyze Tool Output - Next Steps

CLI Commands

Introduction

This document describes the process to run Hypercheck Health Check & Pre-Upgrade tool on Hyperflex HyperV cluster. This tool is a utility to perform pro-active self checks on hyperflex systems to ensure its stability and resiliency. It helps automate a list of health and pre-upgrade checks on hyperflex systems to save time during hyperflex upgrade and maintenance operations.

NOTE: Always download latest version of the tool before you use it. Since the tool is enhanced frequently, using older version may result in missing important checks.

Supported HX Systems

- Hyperflex Versions - 3.0, 3.5, 4.0

- Hyperflex Standard Cluster

- Only supported on Hyperflex cluster in Microsoft Hyper-V

NOTE : How to run Hypercheck on hyperflex ESXi cluster, Please visit-

https://www.cisco.com/c/en/us/support/docs/hyperconverged-infrastructure/hyperflex-hx-data-platform/214101-hypercheck-hyperflex-health-pre-upgr.html

When to Use

- Before Hyperflex upgrades.

- Hyperflex Health Check before and after Maintenance Windows

- To identify failed drives/disks.

- When working with Cisco TAC

- Pro-active Health Check anytime.

How to Use

Step 1. Download Hyperflex-Hypercheck.zip from Cisco github devnet account here .Please get the latest copy which will have the latest improvements and updates.

NOTE: Use only the script downloaded from the Cisco github devnet account.

Step 2. Upload It to the Storage Controller VM (SCVM) with the Cluster Management IP (CMIP).

Use your preferred method - scp/sftp/ftp/tftp - to copy the Hypercheck-HyperV-master.zip to the /tmp directory

For MAC:

Perform SCP from CLI (please confirm that the Hyperflex-Hypercheck.zip is in the same folder from where you are running scp)

# scp Hypercheck-HyperV-master.zip root@<scvm-eth0:mgmtip>:/tmp/

Use the following to identify the Cluster Management IP in your HX environment- Hyperflex playbook

For Windows:

We can use WINSCP to transfer the files as shown below-

Step 3. Extract the Contents of Hypercheck-HyperV-master.zip

Type cd /tmp to change to the /tmp directory

root@hyper11-stc:~# cd /tmp/

Type unzip Hypercheck-HyperV-master.zip to extract the files

root@hyper11-stc:/tmp# unzip Hypercheck-HyperV-master.zip

Archive: Hypercheck-HyperV-master.zip

67f0d63639785f5c18b8ed35ff789f748157e944

creating: Hypercheck-HyperV-master/

inflating: Hypercheck-HyperV-master/README.md

inflating: Hypercheck-HyperV-master/ReadMe.txt

inflating: Hypercheck-HyperV-master/TestInfo.txt

inflating: Hypercheck-HyperV-master/VHXTool.py

inflating: Hypercheck-HyperV-master/prettytable.py

inflating: Hypercheck-HyperV-master/progressbar.py

root@hyper11-stc:/tmp#

Step 4. Execute the VHXTool Python script

Type cd Hypercheck-HyperV-master  to go to the Hypercheck-HyperV-master directory

root@hyper11-stc:/tmp# cd Hypercheck-HyperV-master

Type python VHXTool.py to execute the script

root@hyper11-stc:/tmp/Hypercheck-HyperV-master# python VHXTool.py

Step 5. Enter the HyperV admin username ,Password and Cluster Root Password When Prompted

Please enter below info of Hyper-V Cluster:

Enter the Hyper-V Username(Ex: Domain\Username):

Enter the Hyper-V Password:

Enter the HX-Cluster Root Password:

NOTE: To stop the script execution use the key [CTRL+Z] and it will stop immediately

Step 6. Hyperflex-Hypercheck  tool will prompt you to enquire if Active Directory is installed on a bare metal in your environment.

NOTE: Script will continue to run irrespective of the input provided( Yes/No) here. Below message will be displayed as a word of caution.

“Please be aware that all Active Directory Servers/ DNS Servers should not be nested in Hyperflex datastore virtual machines. There should always be physical (bare metal) ADs in your environment”

Step 7. Hyperflex-Hypercheck tool starts its checks. It will take about 5-10 mins for the execution to complete depending on the number of converged nodes in the cluster

Step 8. Understanding Outputs / Checks Performed

The following checks are performed by Hyperflex-Hyperchecktool

Hyperflex Checks: (Below checks are performed on all the storage controller VMs)

Cluster services check - Verifies the status of storfs, stMgr and stNodeMgr services.

Enospc state check - Checks if the cluster space usage is above the warning threshold or no.

Zookeeper check - Checks whether the Zookeeper is running or no.

Exhibitor check - Verfies the status of the Exhibitor service which manages the ZK.

System Disks Usage - Checks if /sda1, var/stv and /var/zookeeper is less than 80%.

HDD health check - Reports if you have any blacklisted disk in your cluster.

DNS check - Checks whether DNS is configured and reachable.

Timestamp check - Checks if all the controller VMs have the exact same time.

NTP sync check - Checks whether NTP is reachable from the storage controller VMs and synced.

Check package & versions - Checks for packages and versions on Storage Controller VMs.

Check Iptables count - Checks for Iptables count on and ensure it is same on all Storage Controller VMs.

Cache Disks check     - Checks the number of Cache Disks.

Extra pnodes check - Looks for any extra/duplicate pnode entries in the cluster.

Memory usage check - Checks for available memory more than 2GB.

Incidence of OOM in the log file - Checks for any previous incidence of Out Of Memory Condition.

Check permissions for /tmp -  Checks if the /tmp permissions are set correctly.

Check Cluster Access Policy - Checks the Configured Cluster Access Policy

CMIP Hostname – Check if the clustermanagementip has hostname defined

Check domain join health – checking domain join health of the node.

HYPER-V Checks: (Below checks are performed on each HyperV node)

Hostname – check and print the hostname of the node.

Cluster Failover – check and ensure cluster failover is enabled.

Hyper-V Role – Check and ensure HyperV role is enabled.

Node State - Checks the state of the node.

Network Interfaces State - Checks the Network Interfaces State of the node.

Remote Management Enabled - Check if the Remote Management is enabled on the node.

MTU for Storage Data Network - Check MTU for the Storage Data Network of the node.

Domain and forest details -  Check the Domain and forest details of the cluster.

Host file entries- Check if the host file have correct entries.

Check Adapter details - Check Adapter details of the node.

Drivers test - Check the status of minifilter drivers.

SMB Test - Checking SMB Path reachability from the node.

VMMS – Checking the VMMS Service status

Step 9. Get report of the script outputs. You can get it as shown below

Hypercheck Report tar file is saved under /var/log/springpath and /tmp/Hypercheck-HyperV-master. So, you can download the tar bundle from under /var/log/springpath or/tmp/Hypercheck-HyperV-master. Alternatively, you can just generate and upload a storfs-support bundle which will also contain the hypercheck report tar.

Hypercheck Report tar file is saved under /var/log/springpath.

Report tar file example - VHX_Report_2020_08_29_08_40_20.tar is copied to path: /var/log/springpath

Type ls -l | grep VHX_Report to review the files created by Hyperflex-Hypercheck tool

Under /var/log/springpath,

root@hyper11-stc:/var/log/springpath# ls -l | grep VHX_Report

-rw-r--r--  1 root    root    370K Aug 29 08:47 VHX_Report_2020_08_29_08_40_20.tar

root@hyper10-stc:/var/log/springpath#

Under /tmp/Hypercheck-HyperV-master

root@hyper11-stc:/tmp/Hypercheck-HyperV-master# ls -ltrh

total 576K

-rwxrwxrwx 1 root root  53K Apr 10  2018 prettytable.py

-rwxrwxrwx 1 root root 1.4K Jan 22  2019 progressbar.py

-rwxrwxrwx 1 root root 1.6K Aug 28 00:27 ReadMe.txt

-rwxrwxrwx 1 root root  75K Aug 28 06:32 VHXTool.py

-rwxrwxrwx 1 root root 3.5K Aug 28 06:46 TestInfo.txt

-rw-r--r-- 1 root root  49K Aug 29 08:40 prettytable.pyc

-rw-r--r-- 1 root root 1.8K Aug 29 08:40 progressbar.pyc

drwxr-xr-x 2 root root 4.0K Aug 29 08:47 VHX_Report_2020_08_29_08_40_20

-rw-r--r-- 1 root root 370K Aug 29 08:47 VHX_Report_2020_08_29_08_40_20.tar

Files and logs in the Hypercheck log bundle-

root@hyper11-stc: tmp/Hypercheck-HyperV-master # cd VHX_Report_2020_08_29_08_40_20

root@hyper11-stc: tmp/Hypercheck-HyperV-master # ls -ltrh

-rw-r--r-- 1 root root  27K Aug 29 08:44 VHX_Report_10.8.16.65.txt

-rw-r--r-- 1 root root  27K Aug 29 08:45 VHX_Report_10.8.16.66.txt

-rw-r--r-- 1 root root  27K Aug 29 08:47 VHX_Report_10.8.16.67.txt

-rw-r--r-- 1 root root 101K Aug 29 08:47 VHX_Tool_Main_Report_2020-08-29_08-47-43_HYPER11-SMB.txt

-rw-r--r-- 1 root root 186K Aug 29 08:47 VHXTool_2020-08-29_08-40-20.log

Step 10: Export HX_YYYY_MM_DD_HH_MM_SS.tar and share it with TAC.

Use your preferred method to export the Hypercheck logs using scp/sftp/ftp/tftp from the SCVM or you can simply download the storfs support bundle which will contain VHX_Report tar bundle.

Step 11. Sample VHXTool Output from a 3-node Cluster

Please enter below info of Hyper-V Cluster:

Enter the Hyper-V Username(Ex: Domain\Username): hx.local\hxadmin

Enter the Hyper-V Password:

Enter the HX-Cluster Root Password:

Is the Active Directory installed on Physical (bare metal) in your Environment (Enter Yes/No): yes

Note: Please be aware that all Active Directory Servers/ DNS Servers should not be nested in Hyperflex datastore virtual machines. There should always be physical (bare metal) ADs in your environment.

SMB Name: HYPER10-SMB

SSH connection established to HX Node: 10.8.16.65

SSH connection established to HX Node: 10.8.16.66

SSH connection established to HX Node: 10.8.16.67

HX Cluster Nodes:

+-------+-----------------+-------------+-----------------+----------+--------------------------+-----------------+------------+

| Nodes | Eth0 IP Address | HostName    | Eth1 IP Address | Eth1 MTU | Cluster Mgmt IP          | Cluster Data IP | CRM Master |

+-------+-----------------+-------------+-----------------+----------+--------------------------+-----------------+------------+

| 1     | 10.8.16.65      | hyper10-stc | 10.8.18.65      | 9000     | HYPER10-MGMT-IP.hx.local | 10.8.18.69      | 10.8.18.67 |

+-------+-----------------+-------------+-----------------+----------+--------------------------+-----------------+------------+

| 2     | 10.8.16.66      | hyper11-stc | 10.8.18.66      | 9000     | HYPER10-MGMT-IP.hx.local | 10.8.18.69      | 10.8.18.67 |

+-------+-----------------+-------------+-----------------+----------+--------------------------+-----------------+------------+

| 3     | 10.8.16.67      | hyper12-stc | 10.8.18.67      | 9000     | HYPER10-MGMT-IP.hx.local | 10.8.18.69      | 10.8.18.67 |

+-------+-----------------+-------------+-----------------+----------+--------------------------+-----------------+------------+

HX Controller: 10.8.16.65

Cluster services check      [########################] COMPLETE

ZooKeeper & Exhibitor check [########################] COMPLETE

HDD health check            [########################] COMPLETE

Pre-Upgrade Check           [########################] COMPLETE

Hyper-V check               [########################] COMPLETE

HX Controller: 10.8.16.66

Cluster services check      [########################] COMPLETE

ZooKeeper & Exhibitor check [########################] COMPLETE

HDD health check            [########################] COMPLETE

Pre-Upgrade Check           [########################] COMPLETE

Hyper-V check               [########################] COMPLETE

HX Controller: 10.8.16.67

Cluster services check      [########################] COMPLETE

ZooKeeper & Exhibitor check [########################] COMPLETE

HDD health check            [########################] COMPLETE

Pre-Upgrade Check           [########################] COMPLETE

Hyper-V check               [########################] COMPLETE

HX Controller: 10.8.16.65

Test Summary:

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Name                             | Result                                                        | Comments                                                       |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cluster services check           | PASS                                                          | Checks storfs, stMgr, sstNodeMgr service running on each node. |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Enospc state check               | PASS                                                          | Checks if the cluster storage utilization is above threshold.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Zookeeper check                  | PASS                                                          | Checks if Zookeeper service is running.                        |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Exhibitor check                  | PASS                                                          | Checks if Exhibitor in running.                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| System Disks Usage               | PASS                                                          | Checks if /sda1, var/stv and /var/zookeeper is less than 80%.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| HDD health check                 | PASS                                                          | Checks if any drive is in blacklisted state.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| DNS check                        | PASS                                                          | Checks if configured DNS is reachable.                         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Timestamp check                  | PASS                                                          | Checks if the timestamp is same across all Nodes.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| NTP sync check                   | PASS                                                          | Checks if the NTP is synced with NTP server.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check package & versions         | PASS                                                          | Checks for count and version of HX packages on each node.      |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Iptables count             | PASS                                                          | Checks if the IP Table count matches on all nodes.             |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cache Disks check                | PASS                                                          | Checks the number of Cache Disks.                              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Extra pnodes check               | PASS                                                          | Checks for any stale Node entry.                               |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Memory usage check               | PASS                                                          | Checks for available memory more than 2GB.                     |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Incidence of OOM in the log file | PASS                                                          | Checks for any previous incidence of Out Of Memory Condition.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check permissions for /tmp       | PASS                                                          | Checks if the /tmp permissions are set correctly.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Cluster Access Policy      | Lenient                                                       | Checks the Configured Cluster Access Policy                    |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check CMIP Hostname              | PASS                                                          | Check if the clustermanagementip has hostname defined.         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Domain join health               | Name = hyper10-stc                                            | Checking domain join health of the Node.                       |

|                                  | Domain = HX.LOCAL                                             |                                                                |

|                                  | Distinguished Name = CN=HYPER10-STC,OU=HYPER10,DC=hx,DC=local |                                                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

HX Controller: 10.8.16.66

Test Summary:

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Name                             | Result                                                        | Comments                                                       |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cluster services check           | PASS                                                          | Checks storfs, stMgr, sstNodeMgr service running on each node. |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Enospc state check               | PASS                                                          | Checks if the cluster storage utilization is above threshold.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Zookeeper check                  | PASS                                                          | Checks if Zookeeper service is running.                        |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Exhibitor check                  | PASS                                                          | Checks if Exhibitor in running.                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| System Disks Usage               | PASS                                                          | Checks if /sda1, var/stv and /var/zookeeper is less than 80%.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| HDD health check                 | PASS                                                          | Checks if any drive is in blacklisted state.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| DNS check                        | PASS                                                          | Checks if configured DNS is reachable.                         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Timestamp check                  | PASS                                                          | Checks if the timestamp is same across all Nodes.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| NTP sync check                   | PASS                                                          | Checks if the NTP is synced with NTP server.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check package & versions         | PASS                                                          | Checks for count and version of HX packages on each node.      |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Iptables count             | PASS                                                          | Checks if the IP Table count matches on all nodes.             |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cache Disks check                | PASS                                                          | Checks the number of Cache Disks.                              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Extra pnodes check               | PASS                                                          | Checks for any stale Node entry.                               |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Memory usage check               | PASS                                                          | Checks for available memory more than 2GB.                     |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Incidence of OOM in the log file | PASS                                                          | Checks for any previous incidence of Out Of Memory Condition.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check permissions for /tmp       | PASS                                                          | Checks if the /tmp permissions are set correctly.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Cluster Access Policy      | Lenient                                                       | Checks the Configured Cluster Access Policy                    |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check CMIP Hostname              | PASS                                                          | Check if the clustermanagementip has hostname defined.         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Domain join health               | Name = hyper11-stc                                           | Checking domain join health of the Node.                       |

|                                  | Domain = HX.LOCAL                                             |                                                                |

|                                  | Distinguished Name = CN=HYPER11-STC,OU=HYPER10,DC=hx,DC=local |                                                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

HX Controller: 10.8.16.67

Test Summary:

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Name                             | Result                                                        | Comments                                                       |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cluster services check           | PASS                                                          | Checks storfs, stMgr, sstNodeMgr service running on each node. |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Enospc state check               | PASS                                                          | Checks if the cluster storage utilization is above threshold.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Zookeeper check                  | PASS                                                          | Checks if Zookeeper service is running.                        |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Exhibitor check                  | PASS                                                          | Checks if Exhibitor in running.                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| System Disks Usage               | PASS                                                          | Checks if /sda1, var/stv and /var/zookeeper is less than 80%.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| HDD health check                 | PASS                                                          | Checks if any drive is in blacklisted state.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| DNS check                        | PASS                                                          | Checks if configured DNS is reachable.                         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Timestamp check                  | PASS                                                          | Checks if the timestamp is same across all Nodes.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| NTP sync check                   | PASS                                                          | Checks if the NTP is synced with NTP server.                   |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check package & versions         | PASS                                                          | Checks for count and version of HX packages on each node.      |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Iptables count             | PASS                                                          | Checks if the IP Table count matches on all nodes.             |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Cache Disks check                | PASS                                                          | Checks the number of Cache Disks.                              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Extra pnodes check               | PASS                                                          | Checks for any stale Node entry.                               |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Memory usage check               | PASS                                                          | Checks for available memory more than 2GB.                     |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Incidence of OOM in the log file | PASS                                                          | Checks for any previous incidence of Out Of Memory Condition.  |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check permissions for /tmp       | PASS                                                          | Checks if the /tmp permissions are set correctly.              |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check Cluster Access Policy      | Lenient                                                       | Checks the Configured Cluster Access Policy                    |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Check CMIP Hostname              | PASS                                                          | Check if the clustermanagementip has hostname defined.         |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

| Domain join health               | Name = hyper12-stc                                           | Checking domain join health of the Node.                       |

|                                  | Domain = HX.LOCAL                                             |                                                                |

|                                  | Distinguished Name = CN=HYPER12-STC,OU=HYPER10,DC=hx,DC=local  |                                                                |

+----------------------------------+---------------------------------------------------------------+----------------------------------------------------------------+

################################################################################

Hyper-V check:

################################################################################

Hyper-V Clusters: 10.8.16.61, 10.8.16.62, 10.8.16.63

Hyper-V Host: 10.8.16.61

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Name                                     | Status                                                                              | Comments                                                    |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| HostName                                 | HYPER10                                                                             | Check if the hostname is defined.                           |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Cluster Failover                         | Installed                                                                           | Check if the Failover Cluster Manager feature is installed. |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Hyper-V Role                             | Installed                                                                           | Check if the Hyper-V Manager feature is installed.          |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Node State                               | PASS                                                                                | Check the Node State.                                       |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Network Interfaces State                 | PASS                                                                                | Check the Network Interfaces State.                         |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Remote Management Enabled                | PASS                                                                                | Check if the Remote Management is enabled on the node.      |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| MTU for Storage Data Network             | 9000                                                                                | Check MTU for the Storage Data Network.                     |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check the Domain and forest details      | Description             : HXDC-DOMAIN                                               | Check the Domain and forest details of the cluster.         |

|                                          | DnsForestName           : hx.local                                                  |                                                             |

|                                          | DomainControllerAddress : \\10.8.12.254                                             |                                                             |

|                                          | DomainControllerName    : \\HXDC                                                    |                                                             |

|                                          | DomainName              : HXDC-DOMAIN                                               |                                                             |

|                                          | Status                  : OK                                                        |                                                             |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check host file entries                  |                                                                                     | Check if the host file have correct entries.                |

|                                          | 10.8.18.65     HYPER10-SMB.hx.local                                                 |                                                             |

|                                          | 10.8.18.69     HYPER10-SMB.hx.local                                                 |                                                             |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check Adapter details                    | InterfaceAlias                 InterfaceDescription                IPv4Address      | Check Adapter details of the node.                          |

|                                          | --------------                 --------------------                -----------      |                                                             |

|                                          | vswitch-hx-livemigration       Hyper-V Virtual Ethernet Adapter #4 {10.8.17.61}     |                                                             |

|                                          | vswitch-hx-vm-network          Hyper-V Virtual Ethernet Adapter #3 {10.8.19.61}     |                                                             |

|                                          | vswitch-hx-storage-data        Hyper-V Virtual Ethernet Adapter #2 {10.8.18.61}     |                                                             |

|                                          | vswitch-hx-inband-mgmt         Hyper-V Virtual Ethernet Adapter    {10.8.16.61}     |                                                             |

|                                          | vEthernet (New Virtual Switch) Hyper-V Virtual Ethernet Adapter #5 {169.254.84.129} |                                                             |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Drivers test                             | Filter Name                     Num Instances    Altitude    Frame                  | Check the status of minifilter drivers.                     |

|                                          | ------------------------------  -------------  ------------  -----                  |                                                             |

|                                          | CsvNSFlt                                1       404900         0                    |                                                             |

|                                          | CsvFlt                                  0       404800         0                    |                                                             |

|                                          | CCFFilter                               1       261160         0                    |                                                             |

|                                          | storqosflt                              1       244000         0                    |                                                             |

|                                          | ResumeKeyFilter                         0       202000         0                    |                                                             |

|                                          | wcifs                                   0       189900         0                    |                                                             |

|                                          | CldFlt                                  0       180451         0                    |                                                             |

|                                          | FileCrypt                               0       141100         0                    |                                                             |

|                                          | svhdxflt                                0       135100         0                    |                                                             |

|                                          | luafv                                   1       135000         0                    |                                                             |

|                                          | npsvctrig                               1        46000         0                    |                                                             |

|                                          | Wof                                     1        40700         0                    |                                                             |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Virtual Machine Management service check | PASS                                                                                | Checking if VMMS service is Up and Running.                 |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

| SMB Test                                 | PASS                                                                                | Checking SMB reachability of node.                          |

+------------------------------------------+-------------------------------------------------------------------------------------+-------------------------------------------------------------+

Hyper-V Host: 10.8.16.62

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Name                                     | Status                                                                                      | Comments                                                    |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| HostName                                 | HYPER11                                                                                     | Check if the hostname is defined.                           |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Cluster Failover                         | Installed                                                                                   | Check if the Failover Cluster Manager feature is installed. |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Hyper-V Role                             | Installed                                                                                   | Check if the Hyper-V Manager feature is installed.          |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Node State                               | PASS                                                                                        | Check the Node State.                                       |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Network Interfaces State                 | PASS                                                                                        | Check the Network Interfaces State.                         |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Remote Management Enabled                | PASS                                                                                        | Check if the Remote Management is enabled on the node.      |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| MTU for Storage Data Network             | 9000                                                                                        | Check MTU for the Storage Data Network.                     |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check the Domain and forest details      | Description             : HXDC-DOMAIN                                                       | Check the Domain and forest details of the cluster.         |

|                                          | DnsForestName           : hx.local                                                          |                                                             |

|                                          | DomainControllerAddress : \\10.8.12.254                                                     |                                                             |

|                                          | DomainControllerName    : \\HXDC                                                            |                                                             |

|                                          | DomainName              : HXDC-DOMAIN                                                       |                                                             |

|                                          | Status                  : OK                                                                |                                                             |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check host file entries                  |                                                                                             | Check if the host file have correct entries.                |

|                                          | 10.8.18.66     HYPER10-SMB.hx.local                                                         |                                                             |

|                                          | 10.8.18.69     HYPER10-SMB.hx.local                                                         |                                                             |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check Adapter details                    | InterfaceAlias                 InterfaceDescription                IPv4Address              | Check Adapter details of the node.                          |

|                                          | --------------                 --------------------                -----------              |                                                             |

|                                          | vswitch-hx-inband-mgmt         Hyper-V Virtual Ethernet Adapter    {10.8.16.70, 10.8.16.62} |                                                             |

|                                          | vswitch-hx-livemigration       Hyper-V Virtual Ethernet Adapter #4 {10.8.17.62}             |                                                             |

|                                          | vswitch-hx-storage-data        Hyper-V Virtual Ethernet Adapter #2 {10.8.18.62}             |                                                             |

|                                          | vswitch-hx-vm-network          Hyper-V Virtual Ethernet Adapter #3 {10.8.19.62}             |                                                             |

|                                          | vEthernet (New Virtual Switch) Hyper-V Virtual Ethernet Adapter #5 {169.254.247.198}        |                                                             |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Drivers test                             | Filter Name                     Num Instances    Altitude    Frame                          | Check the status of minifilter drivers.                     |

|                                          | ------------------------------  -------------  ------------  -----                          |                                                             |

|                                          | CsvNSFlt                                1       404900         0                            |                                                             |

|                                          | CsvFlt                                  0       404800         0                            |                                                             |

|                                          | CCFFilter                               1       261160         0                            |                                                             |

|                                          | storqosflt                              2       244000         0                            |                                                             |

|                                          | ResumeKeyFilter                         0       202000         0                            |                                                             |

|                                          | wcifs                                   0       189900         0                            |                                                             |

|                                          | CldFlt                                  0       180451         0                            |                                                             |

|                                          | FileCrypt                               0       141100         0                            |                                                             |

|                                          | svhdxflt                                0       135100         0                            |                                                             |

|                                          | luafv                                   1       135000         0                            |                                                             |

|                                          | npsvctrig                               1        46000         0                            |                                                             |

|                                          | Wof                                     1        40700         0                            |                                                             |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Virtual Machine Management service check | PASS                                                                                        | Checking if VMMS service is Up and Running.                 |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

| SMB Test                                 | PASS                                                                                        | Checking SMB reachability of node.                          |

+------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------+

Hyper-V Host: 10.8.16.63

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Name                                     | Status                                                                             | Comments                                                    |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| HostName                                 | HYPER12                                                                            | Check if the hostname is defined.                           |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Cluster Failover                         | Installed                                                                          | Check if the Failover Cluster Manager feature is installed. |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Hyper-V Role                             | Installed                                                                          | Check if the Hyper-V Manager feature is installed.          |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Node State                               | PASS                                                                               | Check the Node State.                                       |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Network Interfaces State                 | PASS                                                                               | Check the Network Interfaces State.                         |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Remote Management Enabled                | PASS                                                                               | Check if the Remote Management is enabled on the node.      |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| MTU for Storage Data Network             | 9000                                                                               | Check MTU for the Storage Data Network.                     |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check the Domain and forest details      | Description             : HXDC-DOMAIN                                              | Check the Domain and forest details of the cluster.         |

|                                          | DnsForestName           : hx.local                                                 |                                                             |

|                                          | DomainControllerAddress : \\10.8.12.254                                            |                                                             |

|                                          | DomainControllerName    : \\HXDC                                                   |                                                             |

|                                          | DomainName              : HXDC-DOMAIN                                              |                                                             |

|                                          | Status                  : OK                                                       |                                                             |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check host file entries                  |                                                                                    | Check if the host file have correct entries.                |

|                                          | 10.8.18.67     HYPER10-SMB.hx.local                                                |                                                             |

|                                          | 10.8.18.69     HYPER10-SMB.hx.local                                                |                                                             |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Check Adapter details                    | InterfaceAlias                 InterfaceDescription                IPv4Address     | Check Adapter details of the node.                          |

|                                          | --------------                 --------------------                -----------     |                                                             |

|                                          | vswitch-hx-inband-mgmt         Hyper-V Virtual Ethernet Adapter    {10.8.16.63}    |                                                             |

|                                          | vswitch-hx-storage-data        Hyper-V Virtual Ethernet Adapter #2 {10.8.18.63}    |                                                             |

|                                          | vswitch-hx-vm-network          Hyper-V Virtual Ethernet Adapter #3 {10.8.19.63}    |                                                             |

|                                          | vswitch-hx-livemigration       Hyper-V Virtual Ethernet Adapter #4 {10.8.17.63}    |                                                             |

|                                          | vEthernet (New Virtual Switch) Hyper-V Virtual Ethernet Adapter #5 {169.254.18.96} |                                                             |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Drivers test                             | Filter Name                     Num Instances    Altitude    Frame                 | Check the status of minifilter drivers.                     |

|                                          | ------------------------------  -------------  ------------  -----                 |                                                             |

|                                          | CsvNSFlt                                1       404900         0                   |                                                             |

|                                          | CsvFlt                                  0       404800         0                   |                                                             |

|                                          | CCFFilter                               1       261160         0                   |                                                             |

|                                          | storqosflt                              2       244000         0                   |                                                             |

|                                          | ResumeKeyFilter                         0       202000         0                   |                                                             |

|                                          | wcifs                                   0       189900         0                   |                                                             |

|                                          | CldFlt                                  0       180451         0                   |                                                             |

|                                          | FileCrypt                               0       141100         0                   |                                                             |

|                                          | svhdxflt                                0       135100         0                   |                                                             |

|                                          | luafv                                   1       135000         0                   |                                                             |

|                                          | npsvctrig                               1        46000         0                   |                                                             |

|                                          | Wof                                     1        40700         0                   |                                                             |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| Virtual Machine Management service check | PASS                                                                               | Checking if VMMS service is Up and Running.                 |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

| SMB Test                                 | PASS                                                                               | Checking SMB reachability of node.                          |

+------------------------------------------+------------------------------------------------------------------------------------+-------------------------------------------------------------+

Main Report File: VHX_Tool_Main_Report_2020-09-13_09-49-38_HYPER10-SMB.txt

Report tar file: VHX_Report_2020_09_13_21_42_18.tar

Report file copied to path: /var/log/springpath

Release Notes:

https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html

Upgrade Guides:

https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

Note:

1) Hypercheck doesnot perform FAILOVER TEST, so please ensure that the upstream is configured for network connectivity for JUMBO or NORMAL MTU size as needed.

Analyze Tool Output - Next Steps

- The tool automates the process of running manual commands on Hyperflex Systems.

- If the tool runs OK and gives PASS/COMPLETED on all tests. The HX system is good for all the checks which the script has performed.

- In situations where, the tool FAILS on some checks or doesn't run successfully, you can use the CLI commands (listed below) to perform the same checks on Hyperflex System as done by the script Manually.

- The tool DOES NOT check for any old/new/open/resolved caveats and hence it is highly recommended to review Hyperflex Release Notes and Upgrade Guides before any upgrade or maintenance activity.

NOTE: DO NOT open a TAC case because the script failed to run. Please run the commands manually, identify the issue and open a SR for the problem identified.

CLI Commands

On Hyperflex SCVM-

ssh to All Hyperflex SCVMs-

# service_status.sh

# sysmtool --ns cluster --cmd enospcinfo

# echo srvr | nc 0 2181

# pidof exhibitor

# sysmtool --ns disk --cmd list | grep -i blacklisted | wc -l

# stcli services dns show (and ping the IPs listed)

# date ; compare the time on all SCVMs. They should ideally be identical

# stcli services ntp show

# sysmtool --ns cleaner --cmd status

# ntpq -p -4

# dpkg -l | grep -i springpath

# iptables -L -n | wc -l

# hxcli cluster info

# df -h ; check that /var/stv should not be more than 80% used

# zgrep -i “out of memory” /var/log/springpath/debug-storfs.*

# domainjoin-cli query

On HyperV Node -

Open the Windows Powershell

# Get-ClusterNode

# Get-ClusterNetwork

# Get-Service WinRM, Make sure windows remote management service is running

# Get-Content $env:SystemRoot\System32\Drivers\etc\hosts , Make sure you have correct host entried

# test-path\\<smb-share name fqdn>\<datastorename> This should return true ,if not datastore is not reachable via smb.

# Get-NetIPConfiguration

# Get-NetIPInterface -AddressFamily IPv4 -InterfaceAlias vswitch-hx-storage-data | select NlMtu*