Article

Table of Contents
Theme:
Was this article helpful?
Try Vultr Today with

$50 Free on Us!

Want to contribute?

You could earn up to $300 by adding new articles!

Troubleshoot your VPS with Bootable ISOs

Last Updated: Mon, May 18, 2020
Linux Guides System Admin Troubleshooting Guides

Introduction

Like physical servers, virtual servers sometimes need rescue. System problems can occur for many reasons. The system may run out of space, Misbehaving applications can corrupt the disk, or failed updates may leave the system unbootable. If the system isn't accessible, use a bootable rescue ISO to resolve the issue.

Make a Backup

Before attempting any rescue, back up your VPS. Vultr offers manual snapshots and automated backups for VPS instances. If you do not have automatic backups enabled, make a manual snapshot. Attempting to rescue a server without a backup is reckless. There is no substitute for a sound backup strategy. You can create a server snapshot in a few clicks in the customer portal.

You are responsible to back up all your mission-critical data, following your local laws and organizational policies. Vultr does not maintain additional backups outside your customer portal.

Mount a Bootable ISO

There are several bootable ISOs in the Vultr ISO library suitable for system rescue work.

  • Finnix
  • SystemRescueCD
  • Trinity Rescue Kit

You may choose any ISO in the Vultr ISO library, or upload one. To attach an ISO to your server, go to the Server Information page in the customer portal. Select Settings > Custom ISO. Select your ISO, then click Attach ISO and Reboot.

Attach ISO

Your server will reboot from the ISO. Connect to your server with the web console to perform the rescue steps.

Note: The Finnix ISO will time out and revert to the VPS operating system if you wait more than 60 seconds. If this happens, click the Send CtrlAltDel button at the top-right of the console to reboot the server to the ISO boot screen again. The SystemRescueCD and Trinity Rescue Kit ISOs do not revert back to the VPS operating system after booting.

Connect via SSH

The Vultr web console is limited by the capabilities of noVNC, which is HTML-based. For easier access to the server, enable SSH in your rescue environment.

Finnix is lightweight and useful for low-memory environments, but many of its tools are outdated. For more about Finnix, see the Vultr Doc, "Using Finnix Rescue CD to Rescue, Repair, or Backup Your Linux System".

If possible, we recommend using SystemRescueCD or Trinity Rescue Kit for most tasks. The majority of examples in this guide use SystemRescueCD.

To enable SSH in SystemRescueCD:

  1. Boot the SystemRescueCD ISO.
  2. Select Boot SystemRescueCD using default options
  3. Set the root password with passwd.

    # passwd
    Enter new UNIX password:
    Retype new UNIX password:
    passwd: password updated successfully
    

Until the next reboot, SystemRescueCD uses the new password. Setting the root password for SystemRescueCD does not permanently change the root password of your server.

  1. SystemRescueCD automatically starts the SSH server. You can stop it with systemctl sshd stop or restart it with systemctl sshd restart.

Connect to the server as root with an SSH client, or use SFTP to access mounted disks. Locate the server IP address on the server information page. If you have previously connected to the server over SSH, be aware that the server's fingerprint will be different. You may need to remove the old entry from ~/.ssh/known_hosts before SSH connects.

Common Linux Rescue Scenarios

Locate the name of the VPS disk device

Use lsblk to locate your VPS disk and partition device names.

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 836.9M  1 loop /run/archiso/sfs/airootfs
sr0     11:0    1   888M  0 rom  /run/archiso/bootmnt
vda    254:0    0    55G  0 disk
└─vda1 254:1    0    55G  0 part

This example shows vda1 is a 55GB partition. Inspect further with blkid.

# blkid
/dev/vda1: UUID="10c173ba-bc37-4b35-a771-976098b870bc" TYPE="ext4" PARTUUID="bd17e170-01"
/dev/sr0: UUID="2019-02-09-12-53-31-00" LABEL="SYSRCD601" TYPE="iso9660" PTUUID="1e4ed97b" PTTYPE="dos"
/dev/loop0: TYPE="squashfs"

/dev/vda1 is the only ext4 partition on the server, confirming that /dev/vda1 is the VPS data device. Your server configuration may be different, make sure you clearly understand your data devices and names before proceeding.

Check the filesystem

Use fsck to check and correct filesystem corruption.

  1. Use df -h to see the mounted disks.

    # df -h
    Filesystem      Size  Used Avail Use% Mounted on
    tmpfs           739M 1016K  738M   1% /media/ramdisk
    /dev/sdh        160M  160M     0 100% /media/sdh
    /dev/loop0      146M  146M     0 100% /media/compressed_root
    unionfs         739M 1016K  738M   1% /
    devtmpfs         10M     0   10M   0% /dev
    
  2. Verify the VPS disk (vda in this example) is not in the list of mounted devices. Never run fsck on a mounted disk. Do not continue until you are sure the disk is unmounted.

  3. Check the filesystem with fsck.

    # fsck -f /dev/vda1
    
  4. If no problems are detected:

    # fsck -f /dev/vda1
    fsck from util-linux 2.33.1
    e2fsck 1.44.5 (15-Dec-2018)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/vda1: 114923/3520000 files (0.1% non-contiguous), 907391/14417659 blocks
    
  5. If errors are found, fsck will request permission to fix the issue.

    Superblock has an invalid journal (inode 4).
    Clear<y>?
    Press enter to automatically attempt to fix the problems.
    
  6. Remove the ISO and reboot your server to verify the problem is resolved once fsck completes.

Mount VPS filesystem

Mount /dev/vda1 to a folder if you'd like to access the files directly. This mounts the vda1 partition to /mnt.

# mount /dev/vda1 /mnt

Browse /mnt to see the entire root filesystem. Use SCP or SFTP to transfer files from the system.

Change Root

Changing root (chroot) changes the apparent disk root directory, which is useful for system maintenance, such as reinstalling the bootloader or resetting passwords. This topic is complex. We recommend reviewing "What's the proper way to prepare chroot to recover a broken Linux installation?" to set up chroot properly. Once chroot is enabled, you can change user passwords, remove or install packages, and other tasks.

Check VPS system logs

Check the log files for clues about server problems. For example:

  1. Verify the logfiles exist and have current dates.

    # ls -l /mnt/var/log
    
  2. View the last 500 lines of syslog.

    # tail -n 500 /mnt/var/log/syslog | less
    

Check VPS disk space / filesystem inodes

If the VPS disk runs out of either blocks or inodes, your server will not function properly.

  1. Check free blocks on device /dev/vda1 with df -h.

    # df -h /dev/vda1
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1        55G  2.6G   49G   5% /mnt
    
  2. Check the inodes on device /dev/vda1 with the -i switch.

    # df -i /dev/vda1
    Filesystem      Inodes  IUsed   IFree IUse% Mounted on
    /dev/vda1      3520000 114923 3405077    4% /mnt
    

If the server is out of either blocks or inodes, delete some files or move them to another device.

Repair the Linux Bootloader

Bootloader problems vary by distribution, and specific instructions are beyond this short troubleshooting guide. This summary of GRUB and GRUB2 repair is useful if you are using SystemRescueCD. Here are some guides for popular distributions.

If your distribution isn't listed, we recommend a web search for "how to repair boot loader [distro name]".

Windows Password Reset

The Vultr Doc "Reset Windows Server Administrator Password" explains how to reset a lost Windows administrator password.

Want to contribute?

You could earn up to $300 by adding new articles