HSI

Welcome to the HSI home page

The Hierarchical Storage Interface (HSI) is a friendly interface for users of the High Performance Storage System (HPSS). It It is intended to provide a familiar Unix-style environment for working within the HPSS environment, while automatically taking advantage of the power of HPSS (e.g. for high speed parallel file transfers) without requiring any special user interaction, where possible.


As of March 2017, HSI support is available to all HPSS sites with a current HPSS license at no extra change, while support transitions from the original developer (Michael Gleicher) to the HPSS Collaboration Developer laboratories.  Please see the  Bugs/Support page for more information, and to submit bug reports or questions, feature requests, and the like.


Features


HSI's many features include:

  • Various site-selectable authentication mechanisms, using OpenSSL encryption for transmittal of sensitive information.
  • Familiar Unix-style command interface, with commands such as "LS", "CD", etc.  Command aliases and abbreviations 
  • Ability to read command input from a file, and write log or command output to a file.
  • Interactive "history" and command line editing
  • Interactive, batch, or "one-liner" execution modes 
  • Commands for working in the local client filesystem space during an HPSS session, with commands such as “lcd”, “lls”, “lpwd”, etc.
  • Support for IPV4 and IPV6 protocols 
  • Automatically uses HPSS parallel I/O features for file transfer operations 
  • Ability to transfer files across firewalls by setting or clearing “firewall"  mode
  • Multiple concurrent get/put operations proceed in parallel, so operations such as "put * and “get *” can greatly reduce aggregate transfer times vs. serial transfers.
  • Ability to interactively pipe data into or out of HPSS, using filters such as "TAR" 
  • Recursive option is available for most commands; including the ability to copy an entire directory tree to or from HPSS with a single simple command 
  • Conditional put and get operations, including ability to update based on file timestamps 
  • Integrated with the HPSS Transfer Agent for high speed multinode transfers beween HPSS and cluster filesystems
  • Multi-threaded I/O within a single process space 
  • Support for the HPSS “End to End Data Integrity” (E2EDI) feature, which provides the ability to create, store and verify industry standard checksums that are maintained in HPSS metadata, such as md5, shaxxx, etc. and create or verify them during file transfers
  • Automatic optimization and background scheduling of tape mounts for file retrievals, while transfers proceed in the foreground
  • Administrator functions such as recursive chown, chmod, su, sudo, change COS and others.  
  • Administrators can also control, via a server-side configuration file, items such as the maximum number of background tape stages, the maximum number of concurrent tape drives to use per session, maximum concurrent file transfers to be allowed, and the ability to restrict particular HSI commands to certain users and/or groups.
  • Support for optionally creating file expiration times during file transfers, based on various criteria such as Class of Service (COS), user, group, account, file family and HPSS subsystem ID.  Other commands provide the ability to recursively set expiration times on existing files, find files with specific expiration times, etc.
  • Extensive listing options for "ls", with the ability to list tape and disk volume and position for files, and all of the hierarchy information for files
  • Ability to annotate files and directories using the ‘annotate’ command
  • HSI runs on most major Unix-based platforms, including Cygwin on Windows, and 32 or 64-bit Linux and FreeBSD platforms 
  • Provides the ability to connect to multiple HPSS systems and perform 3rd-party copies between the systems, using a "virtual drive" path notation.
  • Ability to run in "non-HPSS" mode, similar to an FTP server, to allow filesystem-to-filesystem transfers.



Caveats

Although HSI is freely available to the HPSS community, it is not supported by IBM


See the Support section of this web site for information on obtaining HSI and HTAR support from the HPSS Development team and Gleicher Enterprises..


The other usual legal caveats apply - while testing is performed prior to each release, and HSI is believed to transfer data reliably without inadvertently corrupting files, no guarantees are made, and no liability is assumed for any problems or damages that may accrue as a result of using HSI.


Use HSI at your own risk!


Acknowledgments


 Many of the early HSI improvements were funded by the National Energy Reseach Supercomputer Center (NERSC) and the PROBE project at Oak Ridge National Laboratory.