How to
contribute a limited amount
of storage as a slave to the cluster?

Integrating Hadoop with LVM

Hi friends, I am back with another blog in this blog we have gone discuss Hadoop, LVM, and elasticity storage like Increasing, decreasing, and merging storage devices and the same LVM i.e, integration with LVM.

What is Hadoop?

Apache Hadoop is the software it is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.

What is LVM?

In Linux, Logical Volume Manager is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux dCristributions are LVM-aware to the point of being able to have their root file systems on a logical volume. In this, we can extend two storage devices, increase, decrease sizes this is the main benefit.

What is the Elasticity storage of hadoop?

The storage will be flexible we can increase, decreas,e and merge storages to one data node. This allows us easy to maintain flexible/elasticity storage.

Create HDFS Hadoop Cluster:

Click on this Image to setup Hadoop Cluster

Hadoop Cluster

I have set up a Hadoop cluster you guys can go through by clicking on this

Create an LVM

adding two storage devices to my VM

Before creating LVM we can see new devices not mounted.

>> df -Th

Creating PV, Volume Group, and Logical Volume

Created two Physical volume

>> pvcreate /dev/sdh
>> pvcreate /dev/sdi
vgcreate hadoopLVM /dev/sdh /dev/sdi
lvcreate --size 40G --name hadoopLV hadoopLVM
>> mkfs.ext4 /dev/hadoopLVM/hadoopLV
>> mount /dev/hadoopLVM/hadoopLV  /datanode/

Increase Datanode size

Here we can see increased storage from 35GiB to 50Gib

>> lvextend --size +<Size_to_increase>G /dev/<group_name>/<LV_name>
>> resize2fs /dev/<group_name>/<LV_name>

Decrease A Logical Volume Size

Before decreasing make sure your storage is unmounted

>> umount <mounted_path>
>> e2fsck  -f /dev/<group_name>/<LV_name>
>> resize2fs /dev/<group_name>/<LV_name> <sizeToFormat>G
>> lvreduce -L <sizeToReduce>G /dev/<group_name>/<LV_name>

Thanks for Reading !!



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store