R-HPC

R-HPC [6] offers R-Cloud, wherein clients can "rent" HPC resources. R-Cloud offers two distinct computing environments. The first is a Shared Cluster, which offers a login to shared nodes and a work queue. This environment offers a classic cluster environment and is essentially a "shared cluster in the sky." Users are billed by the job, creating a pay-as-you go HPC service. No support or administration services are provided. The second environment comprises virtual private clusters that are carved out of a shared configuration. Use can be on-demand with VLAN access. These systems are billed on a 24/7 basis.

R-HPC can provide new 3.4GHz quad core Sandy Bridge-based systems with 16GB of RAM/node (4GB/core), DDR 2:1 blocking InfiniBand, and 1TB of local disk. Additionally, they have dual-socket 2.6GHz eight-core Sandy Bridge with 128GB of RAM/node (8GB/core), QDR non-blocking InfiniBand, 1TB of local storage, and 1TB global storage. These offerings are rounded out by Magny-Cours, Nehalem, and Harpertown systems. GPU-based systems in beta test are provided for dedicated users.

Most applications can be set up and running within one day (although R-HPC notes that licensing issues can delay the process for some users). Similar to Penguin's products, all the interconnects, which include DDR, QDR, FDR, and GigE, are run on the wire with no OS virtualization layer. Storage options include 10 GigE attached NFS/SMB with Lustre over IB as a possible upgrade. If ultimate storage performance is needed, R-HPC also offers the Kove RAM disk storage array [7]. All dedicated systems have block storage for security, whereas the shared clusters use shared NFS (no private mounts).

R-HPC will make service level agreements on a case-by-case basis depending on the customers needs. In terms of workflow management, Torque/OpenPBS is the most common scheduler; however, Maui and Grid Engine (and derivatives) can be provided as needed. Interestingly, cloud bursting, although possible with R-HPC systems, is almost never requested by customers. Another interesting aspect of R-HPC offerings includes Windows HPC environments.

In terms of pricing, R-HPC pointed to several factors that make it difficult to offer a standard price for their HPC services. Generally, the length of commitment can factor heavily into the price. The basic "no commitment" starting price is competitive with other cloud providers offering shared systems. Pricing does include NFS over 10GigE. Dedicated systems can be substantially less (i.e., volume reduces cost). Dedicated storage nodes for parallel filesystems are generally twice the hourly rate of dedicated system nodes. R-HPC offers performance tuning and remote administration services as well. They have extensive experience in HPC and can provide "tuned" application-specific private clusters for clients.

Amazon EC2 HPC

Perhaps the most well-known cloud provider is Amazon. Inquiries to Amazon were not returned, so information was gleaned from their web page. Originally, the EC2 service was found not suitable for many HPC applications. Amazon has since created dedicated "cluster instances" that offer better performance to HPC users. Several virtualized HPC instances are available on the basis of users' needs. Their first offering is two Cluster Compute instances that provide a very large amount of CPU coupled with increased network performance (10GigE). Instances come in two sizes, a Nehalem-based "Quadruple Extra Large Instance" (eight cores/node, 23GB of RAM, 1.7TB of local storage) and a Sandy Bridge-based "Eight Extra Large Instance" (16 cores/node, 60.5GB of RAM, 3.4TB of local storage).

Additionally, Amazon offers two other specialized instances. The first is a Cluster GPU instance that provides two NVidia Tesla Fermi M2050 GPUs with proportionally high CPU and 10GigE network performance. The second is a high-I/O instance that provides two SSD-based volumes, each with 1024GB of storage.

Pricing can vary depending upon on-demand, scheduled, or spot purchase of resources. Generally, the cost for on-demand EC2 instances are as follows: Quadruple Extra Large Instance is US$ 1.3/hour (US$ 0.33/core per hour), Eight Extra Large Instance is US$ 2.4/hour (US$ 0.15/core per hour), Cluster GPU instance is US$ 2.1/hour, and the High I/O instance is US$ 3.1/hour.

Thus, using the small usage case (80 cores, 4GB of RAM per core, and basic storage of 500GB) would cost US$ 24.00/hour (10 Eight Extra Large Instances). The larger usage case (256 cores, 4GB of RAM per core, and 1TB of fast global storage) would cost US$ 38.4/hour (16 Eight Extra Large Instances).

Amazon does not charge for data transferred into EC2 but has a varying rate schedule for transfer out of the cloud; additionally, there are EC2 storage costs. Therefore, the total cost depends on compute time, total data storage, and transfer. Once created, the instances must be provisioned and configured to work as a cluster by the user.

Besides the Amazon cluster instance, users may select from one of several preconfigured clusters [8]. Several studies also compare Amazon Cluster instances to real clusters [9].

Univa

Univa [10] is known as the place Sun Grid Engine landed after the Oracle purchase. Not only does Univa continue to develop and support Grid Engine, they also offer "one-click" HPC computing with their partner RightScale [11]. The companies offer two distinct ways to build clouds. The first is an internal cloud infrastructure system that provides "raw" virtual machines using one of several methods, including VMware, KVM, Xen, or something more sophisticated like OpenStack or VMware vSphere.

Their UniCloud product uses an Infrastructure-as-a-Service (IaaS) layer to create the machines and then directs the IaaS to provision a raw machine on the fly. Once the machine is provisioned, UniCloud takes over as a Platform-as-a-Service (PaaS) layer and does additional software provisioning and configuration to turn the raw instances into nodes that can then be part of an HPC cluster.

The second offering is an external cloud infrastructure system, such as Amazon EC2 and AWS, and is used to provide the "raw" virtual machines for HPC use. Once those machines are provisioned by EC2, additional provisioning and configuration management is done by UniCloud (PaaS), which turns those raw instances into nodes that can operate as an HPC cluster. This process is automatic, and adding and removing nodes from this HPC cluster environment can be done on demand.

In addition to building HPC clouds, UniCloud can use a rules engine to dynamically change the HPC clouds – changing node roles and adding/removing nodes from the HPC cloud on the fly. This feature allows cloud administrators to tailor the cloud for specific users' needs. Virtualization is always used in Univa HPC clouds and can adapt to support any underlying type of IaaS layer.

Univa notes that launching an external cloud on Amazon EC2 is incredibly easy; you can just go to the Amazon Market Place [12], choose HPC from the list, and then launch your cluster. A similar feature is available from RightScale, where Univa Server template macros are available.

Once the cloud is built, which can take anywhere from 5 to 20 minutes (depending on how large you make your cluster), you can then install your application. Application installation works just like a regular local machine and requires a similar amount of time. Once your application is installed, Univa recommends taking a snapshot of the filesystem volume where the application is installed; otherwise, you might end up installing it again when you restart your cluster.

Univa offers CPU cloud virtualization right now and does not presently offer GPU-based clouds, although they can support it with UniCloud on customer request. In terms of interconnect, Univa offers the standard gigabit Ethernet (10GigE) that is available in many cloud deployments. Interconnects are virtualized, although they have shown some interest in using pass-through to improve performance. Storage options include NFS, and some customers have used other filesystems, such as Gluster.

As expected, Univa offers Grid Engine as part of their HPC cloud. Their UniCloud package provides prebuilt "kits" that include Univa Grid Engine automatically. Univa Grid Engine can easily request more cloud nodes from UniCloud; alternatively, UniCloud can monitor Univa Grid Engine and add/remove nodes as needed. UniCloud can burst from an internal cloud to an external public cloud or from one public cloud to another public cloud.

In terms of cost, Univa costs are additive to the AWS instance charges and range from US$ 0.02 up to a max of US$ 0.08/core per hour. Remember that these costs provide a "ready-to-run" HPC cloud, whereas buying AWS instances requires the user to configure the cluster. Univa proposed a Medium AWS instance for the small usage case (80 cores, 4GB of RAM per core, basic storage of 500GB) that would cost US$ 14.40/hour [(AWS @ US$ 0.16/hour + Univa @ US$ 0.02/hour)x80]. The larger usage case (256 cores, 4GB of RAM per core, 1TB of fast storage) would cost US$ 58.88/hour.

Univa also offers remote or on-site consulting that is related to the deployment, support, training, and management of Univa Grid Engine, UniCloud, and UniSight products.