How to create Raid1 mirroring with 3rd & 4th Drives in OVH servers

OVH panel offers easy Raid configuration for the partitions in up to 2 drives. Recently I was working on a server that has 2 SSDs and 2 HDDs. The server is intended to serve as a Shared Hosting server. My idea was to install the OS on the SSDs and use the HDDs for hosting website data. It was not much of a hassle to install the OS on SSds on Raid1. I can easily add the two HDDs as additional drives and mount them both to be used for storage. But there’s a drawback in such configuration. There will be no redundancy for the storage drives. If any of those HDDs crash – I will lose all data on it. So, my idea is to create another Raid1 with the two HDDs. Unfortunately, it is not possible from the OVH panel and OVH denied any support on this issue. Therefore I had to do it myself.

Since many of the server providers are offering servers with 4 drives – I believe some of you might find this useful. The idea is to create two Raid arrays of 4 drives. In this case – the two SSDs were already in Raid 1 – configured by the OVH panel. So what I needed to do is to create another Raid 1 with the other two HDDs.

Here is how the initial configuration looked like:

#lsblk

ovh default settings with 4 drives

There already is a Raid1 between the second partitions of the two SSDs (nvme0n1p2 & nvme1n1p2). The two HDDs are “sda” & “sdb” which are unformatted and no partitions are created.

The basic steps are as follows:

  1. Create a partition in sda with full volume
  2. Create a partition in sdb with full volume
  3. Create a Raid1 with the two partitions from sda & sdb
  4.  Set the file system (ext4) for the newly created Raid1
  5. Mount the newly created Raid1 to be used as the new volume in the server.

IMPORTANT NOTES:

To check the current Raid configuration:
#cat /proc/mdstat

To create a new partition in a drive:
# fdisk /dev/sd[x]

fdisk can create partitions with max 2TB. Here in my case the two HDDs were of 4TB each. Therefore I had to use GPT parted to create the partition. If you want to learn more about GPT click here.

1. Create a partition in sda with full volume:

# parted /dev/sda

Output:

GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted)
Creates a new GPT disklabel i.e. partition table:
#(parted) mklabel gpt
Sample outputs:
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted)

Next, set the default unit to TB, enter:
#(parted) unit TB

To create a 3.7TB partition size, enter:
#(parted) mkpart primary 0 0
OR
#(parted) mkpart primary 0.00TB 3.70TB

To print the current partitions, enter:
#(parted) print

Sample outputs:
Model: ATA ST33000651AS (scsi)
Disk /dev/sda: 3.70TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 0.00TB 4.00TB 4.00TB ext4 primary

Quit and save the changes, enter:
#(parted) quit
Sample outputs:
Information: You may need to update /etc/fstab. (Since we will be creating Raid with this partition – not need to add this in /etc/fstab in this case)

Use the mkfs.ext3 or mkfs.ext4 (ext4 preferred) command to format the file system, enter:
# mkfs.ext3 /dev/sda1

OR
# mkfs.ext4 /dev/sda1

2. Create a partition in sdb with full volume

# parted /dev/sdb

#(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted)

Next, set the default unit to TB, enter:
#(parted) unit TB

To create a 3.7TB partition size, enter:
#(parted) mkpart primary 0 0
OR
#(parted) mkpart primary 0.00TB 3.70TB

To print the current partitions, enter:
#(parted) print
Sample outputs:

Model: SATA ST33000651AS (scsi)
Disk /dev/sdb: 3.70TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 0.00TB 4.00TB 4.00TB ext4 primary

Quit and save the changes, enter:

#(parted) quit

Sample outputs:
Information: You may need to update /etc/fstab.

Use the mkfs.ext3 or mkfs.ext4 (ext4 preferred) command to format the file system, enter:
# mkfs.ext3 /dev/sdb1
OR
# mkfs.ext4 /dev/sdb1

IMPORTANT NOTE:
BEFORE PROCEEDING TO NEXT STEP CHECK THE FILE SYSTEMS OF THE NEWLY CREATED PARTITIONS OF THE TWO DRIVES:

#fdisk -l

If you see something like the following:
partition file type microsoft basic

You need to do some extra steps to convert these partitions to ext4. Please check the (#13 reply) THIS LINK for details.
=============================================================
Solution: Install gdisk (yum install gdisk) (Check: https://bbs.archlinux.org/viewtopic.php?id=191163 (check #13 reply))
NEED TO CHANGE THE FILE SYSTEMS (reported by fdisk -l) from Microsoft Basic to “Linux File System” with the above-mentioned solution)

# yum install gdisk
# gdisk /dev/sda 🙁
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

#Command (? for help): t
#Partition number (1-1): 1
Current type is ‘Microsoft basic data’
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to ‘Linux filesystem’

#Command (? for help): p
#Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.

DO THE SAME FOR SDB
=============================================================

3. Create a Raid1 with the two partitions from sda & sdb

We already had a Raid1 of the two partitions from SSD named “md2”. We can find it:

# cat /proc/mdstat

In steps 1 & 2 – we created two new partitions in “sda” & “sdb” named “sda1” & “sdb1” respectively.
partitions 3rd 4th drive no raid yet

Check the Raid status of sda1 & sdb1:
# mdadm –examine /dev/sd[a-b]1
– it will show no raid

NOW run the following to create the Raid1 between sda1 and sdb1:
# mdadm –create /dev/md3 –level=mirror –raid-devices=2 /dev/sd[a-b]1

The name of the new Raid1 array is “md3”.

Now check the “lsblk” again and it will show “md3” 3.7TB
md3 raid sda1 sdb1

Now if you check Raid status ( # cat /proc/mdstat ) you’ll see the Raid partitions sda1 & sdb1 are syncing.
md3 sda1 sdb1 sync

The 4TB disks took almost 5 hours to complete the syncing.

4. Set the file system (ext4) for the newly created Raid1

Execute the following command to set the file system (ext4) of the newly created md3 Raid drive:

# mkfs.ext4 /dev/md3

5. Mount the newly created Raid1

We want to mount the newly created Raid1 (md3) to the directory “/home2”. We need to do the following:

# mkdir /home2
# mount /dev/md3 /home2

md3 mounted home2

Now we need to edit /etc/fstab and add the new Raid volume – so that the storage volume is not lost after the server reboots.
edit etc fstab to save the storage volume home2

# nano /etc/fstab

add the following line: /dev/md3 /home2 ext4 defaults 0 0

and save the file. That’s it. The new storage volume will remain after server reboot and we can use this to store our data. If we now check the storage details – we will see the following:

final storage details after adding second raid1 storage

The process will be the same for servers from any provider and not only OVH. Feel free to add your suggestion if you think anything can be improved here or if you encounter any problems by following this article. Thank you.

How to clean /tmp directory automatically in Linux/cPanel using tmpwatch

The “tmpwatch ” command in linux is to removes files which haven’t been accessed for a period of time. The tmpwatch recursively removes files which haven’t been accessed for a given time. Normally, it’s used to clean up directories which are used for temporary holding space such as /tmp.

If you are noticing “/tmp” getting overloaded with files and not sure which files/folders to delete – you better use “tmpwatch” to cleanup or delete files/folders from the “/tmp” directory.

You’ll need SSH root access to install tmpwatch and add it in the cron. If your server is inaccessible due to “/tmp” getting full – you may restart the server and that should free up some space after reboot.

  1. Login to the server as root using SSH
  2. Run the following command:

    #yum install tmpwatch -y

  3. To delete temporary files (for example after every 12 hours) run the following command:

    #/usr/sbin/tmpwatch -am 12 /tmp

  4. The next step is to configure tmpwatch to run automatically through a cron. To do that type the following command:

    #crontab -e

  5. The above command will open the cron job list for the user root. Now go to the bottom and add the following line and save the file:

    0 4 * * * /usr/sbin/tmpwatch -am 12 /tmp

    If you are unable to add the above line, you may navigate to “/var/spool/cron” and open the cron file “root” with a text editor (such as, vi, nano). Add the line at the bottom and save the file:

    0 4 * * * /usr/sbin/tmpwatch -am 12 /tmp

Check the usage of “/tmp” and it should be clean by now.

Thank you.

Whitelist IP or IP range in/out using iptables

#Flush existing rules
iptables -F

# Set up default DROP rule for eth0 (Assuming eth0 is the Ethernet Port)
iptables -P INPUT DROP

# Allow existing connections to continue
iptables -A INPUT -i eth0 -m state –state ESTABLISHED,RELATED -j ACCEPT

# Accept everything from the 192.168.0.x network
iptables -A INPUT -i eth0 -s 192.168.0.0/24 -j ACCEPT

# Allow connections from this host to 192.168.1.10
iptables -A OUTPUT -o eth0 -d 192.168.1.10 -j ACCEPT

How to change default SSH port 22 to something else

Internet connected Linux servers are always vulnerable to exploitation by the hackers/intruders. One of their common attack is through SSH. By default SSH servers are configured to listen to port 22 for SSH connection. Changing the port to something else will definitely add an additional measure in securing the server.

How to Change Default SSH port 22 to something else:

  1. Connect to your server via SSH (your ssh port is still 22) as the root user.
  2. Run this command: # vi /etc/ssh/sshd_config
  3. Find out this line: # port 22
  4. Delete the hash “#” sign from the beginning of the line and change the port to something else. Note down this new port very carefully. You’ll need to use this port for the SSH connection from the next login.
  5. Save the file and exit.
  6. Restart the SSH service by executing this command: # service sshd restart 

Now disconnect and try reconnecting with the new SSH port. Make sure you memorize the new port number or note it down somewhere.

How to Clear bash / SSH / Terminal command history in Linux

You might need to wipe off the command history that you’ve executed in your linux system for various reason. Normally when logged into the bash / SSH terminal – if the Up Arrow key is pressed – previously executed commands are shown. To completely delete the history type the following command:

# history -c
# history -w

 

How to find the IP addresses of a Linux system

The following command in a linux based system will show all the IPs assigned to that system:
# ip addr show

Or alternatively, if you’d like to know the IP address(es) of a certain port (for example eth0) type the following:
# ip addr show eth0

Sample output:
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:30:67:b4:b7:4d brd ff:ff:ff:ff:ff:ff
inet 72.144.122.10/29 brd 72.144.122.15 scope global eth0
inet6 ef80::120:67ef:deb4:f74e/64 scope link
valid_lft forever preferred_lft forever

The IPv4 assigned to the system are: 72.144.122.10/29  (72.144.122.10 – 72.144.122.15 = 6 IPs)
The IPv6 assigned to the system are: ef80::120:67ef:deb4:f74e/64

Determine if the storage is SSD or HDD – using linux command

You’ve purchased a Cloud server or a Virtual server. The server provider says it comes with an SSD (Solid State Drive). Now you definitely would like to verify whether the storage is indeed SSD or is it simple HDD (Hard Disk Drive). If you’re server is running on any version of Linux OS (kernel version 2.6.29 onwards) – just run the following command:

cat /sys/block/sda/queue/rotational

Linux command to identify if a storage is ssd or hdd

The result will be Either 1 or 0 . If it is “0” – Congratulations – your storage is indeed an SSD one and if the result is “1” – your storage device is an HDD.

I ran the above command in one of my VPSs. The result is “1” – that is the storage device used here is an HDD.

GHOST vulnerability in Linux distributions and its Solution

A GNU C Library (glibc) vulnerability – termed as “GHOST Vulnerability” was announced to general public on 27  January 2015 – after a Cloud research company Qualys discovered this major security vulnerability. This vulnerability enables hackers to remotely take control of systems without even knowing any system IDs or passwords.

Ghost Vulnerability - A serious security hole in Linux systems

How to Quickly Identify if your system is vulnerable:

The GHOST vulnerability can be exploited on Linux systems that use versions of the GNU C Library prior to glibc-2.18. That is, systems that use glibc-2.2 to glibc-2.17 are at risk. Many Linux distributions like CentOS (6,7), Debian 7, RHEL (6,7), Ubuntu and Distributions with end of life are vulnerable and should be patched immediately.

You may check the version of glibc by looking up the version of ldd (which uses glibc) with the following command:

#ldd –version

The first line of the output will tell you the glib version, the output could be like this:

# ldd (GNU libc) 2.12

As mentioned earlier – if it is older than 2.18 – your system is vulnerable.

How to solve (patch the system) this problem:

Ubuntu / Debian:

Update all of your packages to the latest version available using this command:

# sudo apt-get update && sudo apt-get dist-upgrade

Respond to the confirmation prompt with: “y” and finally reboot using this command: # sudo reboot

 

CentOS / RHEL:

Update glibc to the latest version available via “yum”:

# sudo yum update glibc

Respond to the confirmation prompt with: “y” and finally reboot using this command: # sudo reboot

 

IF UPDATE FAILS:

To mitigate the problem you before the patch change the value of UseDNS to “no” in “etc/ssh/sshd_config”. This disables the Reverse DNS checks in public facing services.

Although you can be relived for the time being – make sure you patch your distribution whenever it is available using the steps mentioned above.

How to block IP address in Linux – Using IPTables Rule

Using IPTables rules we can block a Single IP address or a block of IP Addresses.

The following command (via ssh) will drop any packet coming from the IP address 1.2.3.4 :

# iptables -I INPUT -s 1.2.3.4 -j DROP

or you can use append

# iptables -A INPUT -s 1.2.3.4 -j DROP

How To Block Subnet (ip.Add.re.ss/subnet):

If your Machines public interface card name is eth1 and if you’d like to block the subnet 10.0.0.0/8 -Use the following syntax:

# iptables -i eth1 -A INPUT -s 10.0.0.0/8 -j DROP

How to View Blocked IP Address(es):

Simply use the following command:

# iptables -L -v

How to Save Blocked IP Address(es) in IPTables:

# service iptables save