HSI COS Selection With/Without Piped Files

This page describes the way that HSI chooses a Class of Service when storing files that are passed in via a pipe command.


As described in the “Using PIPES for Input/Output” and in the “HSI Special Topics” pages, HSI supports the ability to read data from a pipe when writing to HPSS.


If the HPSS Class of Service has been set via ‘set cosid = cos_id” command or as a keyword on the put command, then HSI causes that COS ID to be used when the file is written.


However, many HPSS sites do not reveal the Class of Service IDs that have been defined to their user community, and instead rely on HSI (and HTAR) to choose the correct COS based on information that is defined by the site’s HPSS Administrator.  This is the case when automatic COS selection is in effect, which is the default when HSI is first started up.


The current class of service ID can be shown by using the “set” command with no options,

for example:


O:[/home/gleicher]: set

...

cosid = "auto"

...


When automatic Class of Service selection is in effect, usually the COS is chosen based primarily on the source file size and number of copies desired. The HPSS administrator has the ability to set other criteria to restrict COSs to particular users,groups or accounts, and to create sets of COSs called “Named COS Lists” that the user can specify via the “set coslist” command, or list via the “lscos -n” command.  In addition, starting with HSI 5.0.2.p11 (HPSS 7.4.3) and 6.0.0.p11/6.2.0.p1 (HPSS 7.5.2/7.5.3), the user can specify a default COS to be used when HSI is first started.

 

However, when the file size is unknown at the beginning of the transfer, HSI does one of the following:


  • Does nothing and lets HPSS choose the initial COS or
  • Chooses an initial COS that the site’s HPSS administrator has defined with the “default auto” setting in the server-side COS configuration file


There are two cases that can occur at the end of the transfer when the file size was initially unknown:

  • If the entire file fits within the first I/O buffer, before any data has been written to the file, HSI reselects the COS based on the amount of data in the buffer,  then writes the data to the sink HPSS file using the new COS that was just selected. The amount of data that can fit in the buffer is determined by the default buffer size that was chosen when HSI was built.  The release value is 8MiB (8388608), but this can be varied both at build time, or via the ‘set iobufsize’ keyword.
  • If the  file does not fit within a single I/O buffer, then after the transfer has successfully completed, using the COS that was initially chosen by HPSS or using the ‘default auto’ COS, HSI does the following:
    • reselects the COS based on the final sink file size in HPSS
    • internally issues a “Change Class of Service” to cause HPSS to move the file from the initial COS to the final COS. 


The amount of time that it takes for HPSS to internally move the file from the initial COS to the final COS varies.  There are a number of factors that influence the amount of time that this takes, such as number of Change-COS streams that are active, site I/O activity, size of the file, amount of free space on the final COS storage class, etc.  


The number of possible Change COS streams is defined by the site when HPSS is configured - there can currently be up to 32 active streams.  For each Change COS issued internally by HSI for piped files, stream 0 (the default) is used.   Within each stream, the files are processed in FIFO order.  See the HSI “chcos” command for more information on this topic.