| dc.description.abstract | Both private and public sector organizations are constantly looking for new ways to keep their information safe and accessible at all times. Over the past few decades, replication has always been a reliable way to make sure data is constantly available, even though it has been proven to induce higher costs due to the additional required storage. Since the early 2000s, erasure codes have been developed as a means to drastically reduce the overhead, while enormously increasing efficiency and providing significant error-correcting capabilities. One of the most well-known erasure coding policies is Reed-Solomon (RS), a highly consistent, reliable, and efficient technique to store and recover data, currently used at Facebook's data centers. Other frequently mentioned policies are Pyramid codes, a variant of Locally Repairable Codes (LRCs) that make use of a pyramid-based scheme to generate additional parity groups for each level, and has been used at Microsoft's Windows Live servers. Apache Hadoop is an open-source distributed framework used for scalable processing that has recently introduced erasure coding policies to their storage capabilities. NorNet Core (or NorNet Core Testbed {website: www.nntb.no.}), a distributed academic network, will be used as the main scenario to measure, compare, and analyze these different erasure coding policies and their efficiency. Based on simulations of physically distributed storage, this thesis will show how minimal alterations in commonly known codes (such as RS codes) can converge in a Pyramid-based code that could severely enhance fault-tolerance and performance. Additionally, in a side-to-side comparison, it will be detailed how bigger codes (of higher dimension and length), more often than not, provide a more beneficial trade-off. | en_US |
