Unable to start AWS Linux instance after instance type change – “a start job is running for…”

We’ve got a handful of “old” AWS EC2 instances, running small t2.micro, t2.nano, and t2.small instances. From time to time we have to change the instance type to a larger and more modern one. In such a case, it is common that Linux is not able to mount the secondary volume (not the root one – the root one works just fine). It is due to the change in the volume device descriptors.

The symptoms

  • In the AWS console, the instance looks like working fine, but you are not able to log in to it
  • When you take a look at the “instance screenshot” during boot time, you may notice that the system is waiting on the mounting step with the message “a start job is running for …” followed by the device descriptor
  • When you take a look at the “instance screenshot” once booted, you will see the prompt of the emergency mode

The solution

The solution is to adjust entries in the /etc/fstab file – instead of the device descriptor, you will have to use the device UUID.

First, connect to the instance using the Serial Console (AWS Console -> Select instance -> Actions -> Monitor and Troubleshoot -> EC2 Serial Console)

Once there, you can list your drives and their UUIDs:

In my case, I was trying to mount the /dev/nvme1n1p1, so I copied the UUID and I edited the /etc/fstab file:

I simply commented out the second line (starting with /dev/xvdb1), copied it to the third line, and replaced the /dev/xvdb1 with “UUID=” followed by the UUID of my secondary drive.

This way I gave the system the proper ID of the volume to mount and it worked fine on the next reboot.

What if I can not use the serial console?

If your instance is not able to work with a serial console, you will have to do this the other way:

  1. Turn off the instance
  2. Disconnect the root volume of this instance and mount it to the separate Linux instance as a secondary or tertiary drive
  3. Using this separate instance, edit /etc/fstab file to comment out the problematic drives
  4. Disconnect this volume from the helper instance and connect back as a root device to your source instance

This way your system will start – without a secondary drive, but it will work. You will be able to log in to the system and adjust /etc/fstab properly