Author: David FinsterLast Updated: Mon, May 18, 2020
Like physical servers, virtual servers sometimes need rescue. System problems can occur for many reasons. The system may run out of space, Misbehaving applications can corrupt the disk, or failed updates may leave the system unbootable. If the system isn't accessible, use a bootable rescue ISO to resolve the issue.
Before attempting any rescue, back up your VPS. Vultr offers manual snapshots and automated backups for VPS instances. If you do not have automatic backups enabled, make a manual snapshot. Attempting to rescue a server without a backup is reckless. There is no substitute for a sound backup strategy. You can create a server snapshot in a few clicks in the customer portal.
You are responsible to back up all your mission-critical data, following your local laws and organizational policies. Vultr does not maintain additional backups outside your customer portal.
There are several bootable ISOs in the Vultr ISO library suitable for system rescue work.
You may choose any ISO in the Vultr ISO library, or upload one. To attach an ISO to your server, go to the Server Information page in the customer portal. Select Settings > Custom ISO. Select your ISO, then click Attach ISO and Reboot.
Your server will reboot from the ISO. Connect to your server with the web console to perform the rescue steps.
Note: The Finnix ISO will time out and revert to the VPS operating system if you wait more than 60 seconds. If this happens, click the Send CtrlAltDel button at the top-right of the console to reboot the server to the ISO boot screen again. The SystemRescueCD and Trinity Rescue Kit ISOs do not revert back to the VPS operating system after booting.
The Vultr web console is limited by the capabilities of noVNC, which is HTML-based. For easier access to the server, enable SSH in your rescue environment.
Finnix is lightweight and useful for low-memory environments, but many of its tools are outdated. For more about Finnix, see the Vultr Doc, "Using Finnix Rescue CD to Rescue, Repair, or Backup Your Linux System".
If possible, we recommend using SystemRescueCD or Trinity Rescue Kit for most tasks. The majority of examples in this guide use SystemRescueCD.
To enable SSH in SystemRescueCD:
Set the root password with passwd.
# passwd Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully
Until the next reboot, SystemRescueCD uses the new password. Setting the root password for SystemRescueCD does not permanently change the root password of your server.
systemctl sshd stopor restart it with
systemctl sshd restart.
Connect to the server as root with an SSH client, or use SFTP to access mounted disks. Locate the server IP address on the server information page. If you have previously connected to the server over SSH, be aware that the server's fingerprint will be different. You may need to remove the old entry from ~/.ssh/known_hosts before SSH connects.
lsblk to locate your VPS disk and partition device names.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 836.9M 1 loop /run/archiso/sfs/airootfs sr0 11:0 1 888M 0 rom /run/archiso/bootmnt vda 254:0 0 55G 0 disk └─vda1 254:1 0 55G 0 part
This example shows vda1 is a 55GB partition. Inspect further with
# blkid /dev/vda1: UUID="10c173ba-bc37-4b35-a771-976098b870bc" TYPE="ext4" PARTUUID="bd17e170-01" /dev/sr0: UUID="2019-02-09-12-53-31-00" LABEL="SYSRCD601" TYPE="iso9660" PTUUID="1e4ed97b" PTTYPE="dos" /dev/loop0: TYPE="squashfs"
/dev/vda1 is the only ext4 partition on the server, confirming that /dev/vda1 is the VPS data device. Your server configuration may be different, make sure you clearly understand your data devices and names before proceeding.
Use fsck to check and correct filesystem corruption.
Use df -h to see the mounted disks.
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 739M 1016K 738M 1% /media/ramdisk /dev/sdh 160M 160M 0 100% /media/sdh /dev/loop0 146M 146M 0 100% /media/compressed_root unionfs 739M 1016K 738M 1% / devtmpfs 10M 0 10M 0% /dev
Verify the VPS disk (vda in this example) is not in the list of mounted devices. Never run fsck on a mounted disk. Do not continue until you are sure the disk is unmounted.
Check the filesystem with fsck.
# fsck -f /dev/vda1
If no problems are detected:
# fsck -f /dev/vda1 fsck from util-linux 2.33.1 e2fsck 1.44.5 (15-Dec-2018) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/vda1: 114923/3520000 files (0.1% non-contiguous), 907391/14417659 blocks
If errors are found, fsck will request permission to fix the issue.
Superblock has an invalid journal (inode 4). Clear<y>? Press enter to automatically attempt to fix the problems.
Remove the ISO and reboot your server to verify the problem is resolved once fsck completes.
Mount /dev/vda1 to a folder if you'd like to access the files directly. This mounts the vda1 partition to /mnt.
# mount /dev/vda1 /mnt
Browse /mnt to see the entire root filesystem. Use SCP or SFTP to transfer files from the system.
Changing root (chroot) changes the apparent disk root directory, which is useful for system maintenance, such as reinstalling the bootloader or resetting passwords. This topic is complex. We recommend reviewing "What's the proper way to prepare chroot to recover a broken Linux installation?" to set up chroot properly. Once chroot is enabled, you can change user passwords, remove or install packages, and other tasks.
Check the log files for clues about server problems. For example:
Verify the logfiles exist and have current dates.
# ls -l /mnt/var/log
View the last 500 lines of syslog.
# tail -n 500 /mnt/var/log/syslog | less
If the VPS disk runs out of either blocks or inodes, your server will not function properly.
Check free blocks on device /dev/vda1 with
# df -h /dev/vda1 Filesystem Size Used Avail Use% Mounted on /dev/vda1 55G 2.6G 49G 5% /mnt
Check the inodes on device /dev/vda1 with the
# df -i /dev/vda1 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda1 3520000 114923 3405077 4% /mnt
If the server is out of either blocks or inodes, delete some files or move them to another device.
Bootloader problems vary by distribution, and specific instructions are beyond this short troubleshooting guide. This summary of GRUB and GRUB2 repair is useful if you are using SystemRescueCD. Here are some guides for popular distributions.
If your distribution isn't listed, we recommend a web search for "how to repair boot loader [distro name]".
The Vultr Doc "Reset Windows Server Administrator Password" explains how to reset a lost Windows administrator password.