Contents

Introduction

Problem Description

Observations and Actions Performed

Procedure to Delete PM Files

Introduction

This document describes how to troubleshoot and clear the "START_BACKING_THREAD:BIND: no SPACE LEFT" error in NCS1K.

Problem Description

Observing start_backing_thread:bind: No space left on the device unable to monitor alarms in the device (rootfs is filled 100%). PM logs collection in rootfs directory and filling it up. Version: XR 7.3.2

Observations and Actions Performed

- Command  <show alarms b s a>  output is not displayed and observed error "start_backing_thread:bind: No space left on device" and rootfs is fully occupied.

- Rootfs is filled 100%.

Show Media Output

- Pm log files are filled in rootfs directory.
PM logs stored in rootfs

Procedure to Delete PM Files

This command displays all the files in the root directory.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#run

[xr-vm_node0_RP0_CPU0:~]$cd /root

[xr-vm_node0_RP0_CPU0:/root]$ls -ltrah

This command deletes the file from the directory.

rm “filename” ----à

Verify the size of rootfs using  show media command.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#show media

- It is important to know why auto deletion did not occur in the system.

- It is identified there is a Python script enabled for automatic log collection every 1 hour in rootfs.

- The pm log generation is not done by the device by default, It is configured manually. Hence, this auto-deletion also needs to be done manually.

- Need to verify the source from where the pm log files are being stored in rootfs to kill this process.

- It is found that some Python scripts are running in the background to collect pm logs in rootfs in 2 nodes out of 4 ncs1k4 nodes in their network.

- These were added during deployment time.

Hard Disk Files

To kill the Python process, use the  reload  command in XR.

Resolution:

To terminate these Python processes, you need to  XR RELOAD  on the nodes.

XR RELOAD  - Command reloads the VM.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#reload  (Reloads the VM.)

Reloading does not impact service but you must lose management visibility for around 5-10 mins. You can also clean up the pm-related files using  rm  command:

<rm -rf "/misc/disk1/pm_data">

<rm -rf "/root/*pm.txt">

<rm -rf "/root/PM-data*">

<rm -rf *2022.tgz (deletes all the files ending with 2022.tgz)>

Similarly, you can delete files that are not needed.