19 Storage over IP (HyperSCSI)

19.1 Introduction

The HyperSCSI protocol (see [HSC]) is a network protocol whose goal is transporting SCSI commands and data over a network. It was developed in year 2000 at the Data Storage Institute (DSI) as an alternative to the iSCSI protocol (see [SSC03]), which again as an alternative to the Fiber Channel did not work well on Ethernet networks. It evolved from a research of possible SCSI encapsulation into Ethernet frames.

Similarly to iSCSI, the HyperSCSI (hereinafter "HSCSI") mainly tries to offer a cheaper variant for building Storage Area Networks (SANs) compared with the Fiber Channel (hereinafter "FC"). Compared with the iSCSI, HSCSI tries to deliver higher performance, which in both cases is worse than that using the FC technology.

Goals of the project were

verifying the HSCSI protocol usability in the Linux operating system (both as a client and as a server) and Microsoft Windows OS (client only)
testing the data throughput using the HSCSI protocol
testing the HSCSI protocol usability with transport security mechanisms
attempt to directly compare the HSCSI and iSCSI protocols.

Experience gained during testing within the project was described in the Technical report 23/2003.

19.2 HSCSI features

The protocol itself is designed for two variants of network transport. The first one actually implemented and described here is called HS/Eth and is based on data transport directly over the Ethernet. The authors claim that this variant works also on 802.11b networks. The second variant is meant to transport data over the IP (called HS/IP). Unfortunately the HS/IP variant is not yet available and it is not known when it will be.

Only a software implementation for operating systems Linux and MS Windows 2000 is currently available. The client-server approach is used. The project is Open Source but only for the Linux platform. Both source codes and binary packages of server and client are available for Linux. For the MS Windows platform, only a binary trial version of the client is available, without any source codes.

The HSCSI protocol also deals with data transport security, using the following mechanisms:

Unauthorized access to data protection: The client has to authenticate itself to the server using the correct login/password combination. Unique combination can be assigned for each exported volume.
Securing the data integrity: Only the SHA1 HMAC algorithm is currently implemented but many others can be used in future (when implemented). The architecture is modular, so other mechanisms can be easily added.
Encrypting transported data: Only the AES (Rijndael) algorithm is currently implemented. The 128-bit variant is used. Just as with data integrity, plans for using other algorithms exist, benefitting from the modular architecture.

19.3 Comparison of the HSCSI and iSCSI protocols

As the only available implementation of the HSCSI protocol is the Ethernet variant, the main disadvantage is the impossibility of routing HSCSI and the resulting impossibility of wider deployment. Another important disadvantage of HSCSI is the fact that this protocol is no official standard (unlike, e.g., iSCSI) and so it is unsupported in any way by manufacturers developing hardware data storage solutions. No manufacturer even plans to support and/or implement this protocol currently.

The main advantage of HSCSI compared to iSCSI is especially a lower network load as well as an end system (server and client) load. It results from both the protocol design and implementation and from the fact that for data transport, iSCSI uses the TCP/IP protocol with a bigger overhead. Another advantage when compared to iSCSI is a fully functional software implementation of both client and server (unlike iSCSI with server implementation problems); as a result, HSCSI can be used for a solution built on commonly available hardware, no expensive and specialized hardware is needed. Therefore, HSCSI can be used for building small and cheap SANs.

19.4 Practical experience

Testing general functionality of client and server: We have tested both single- and multi-client (server) variants. Tested versions using the Intel/Linux platform were functional but only with kernel versions 2.4.18 and higher. Older versions of HSCSI support even older kernel versions. We tested only the Ethernet variant because of the unavailability of the IP variant implementation. The client was connected directly to the server (and over a gigabit Ethernet for the performance test) or through a switch (for multi-client tests). Both the multi-client/single server approach as well as a setup where a single client connected to devices on multiple servers were functional.
Functionality tests over the 802.11b wireless networks: Though the tested versions allowed the client to access the remote device over the wireless network, all attempts to transport a contiguous data block failed, producing errors that terminated not only the data transport prematurely but also disallowed the client to connect to the server any more.
Testing the security mechanisms (integrity and encryption): We examined the functionality of mechanisms for data integrity control as well as for data encryption during the transport using HSCSI. The implementation includes only one algorithm for integrity control (SHA1 MAC) as well as for encryption (128-bit AES). Though the authors warned that encrypting transported data may cause data corruption, we could not reproduce this. Unfortunately, data encryption increased the system load up to three times. Versions prior to 20030903 could not provide the data integrity control, probably because of incorrect implementation, although the authors claimed this feature to be fully functional. The data integrity control worked flawlessly in the last tested version (20030930) although no change was mentioned in the change log.
Functionality of access to exported SCSI hard drives: The HSCSI protocol worked flawlessly with SCSI hard drives.
Functionality of access to exported IDE drives (both hard and optical): IDE disk drives worked natively. The IDE optical drives had to be used as SCSI drives using the IDE-SCSI emulation layer on both client and server sides. This applied also to the read-only CD-ROM drives. Writing on CD-RW drives worked as well. This implementation included a bug that caused the server to send the contents of previous medium to the client after a media change.
Testing the MS Windows 2000 Professional client implementation: We tested the usability of the first client available for a non-Linux operating system. Compared to the Linux client, features of the MS client are reduced considerably. No data integrity control and no data encryption is implemented, sessions close incorrectly and only a single-server variant is available. The problems with incorrect session closing affected only the hard disk drives, not the CD-ROM drives. Generally it is clear that this was the first version which is not as advanced as the Linux one.
Testing the client functionality on a handheld device: Unfortunately we could not compile a kernel for a handheld device with HSCSI support and so we could not test it. Another problem was that the connection should have been using a wireless network which had turned out as inappropriate.

19.5 Performance tests

For measuring the disk operation performance, a benchmarking program IOzone was used. We used the function close() and the results were written to a binary MS Excel file.

IOzone parameters:

iozone -Rb hscsi.wks -n 4g -g 4g -z -c -a -i 0 -i 1

The graphs show that while the data transport integrity control has no impact on performance (graph and graph), the data transport encryption has a considerable negative impact (graph and graph).

The fact that the RedHat optimised kernels deliver better performance than standard (vanilla) kernels was confirmed (graph and graph).

Figure 19.1: Results of the HSCSI benchmark

Figure 19.2: Results of the HSCSI benchmark using data integrity control

Figure 19.3: Results of the HSCSI benchmark using data encryption (128-bit AES)

Figure 19.4: Results of the HSCSI benchmark using a RedHat optimised kernel

19.6 Conclusion

Unfortunately, the iSCSI and HSCSI protocols can not be compared directly: while the iSCSI server is satisfactorily implemented only in a specialised hardware solution, the HSCSI server is available only as a software implementation. Nevertheless, the following conclusions can be made:

The HSCSI over Ethernet protocol in the Linux implementation seems to be a stable and usable solution for building local SANs. Its main disadvantage still remains the restriction to Ethernet and so limiting the overall reach of a concrete solution. In contrast to solutions based on the iSCSI or Fibre Channel, the HSCSI price can be more attractive. Compared to iSCSI, the HSCSI offers a better performance with smaller system load. It seems that it could be a useful complement to smaller Linux-based computer clusters.
The data transport security is very well designed and implemented but its system performance requirements are considerably higher (at least for the currently implemented AES algorithm variant). The data transport encryption will become more important with the IP variant.

The project ended with these conclusions and no further work is planned.

contents