Entries tagged with hpc - TechNet Edge

HPC Server 2008 High Availability - Part 4

gmarchetti — Tue, 08 Jul 2008 15:23:00 GMT

This is the last of a series of screencasts demonstrating how to set up HPC Server 2008 head nodes for high availability.
In this part we discuss the installation and configuration of HPC Pack 2008 on a fail-over cluster.

HPC Server 2008 High Availability - Part 3

gmarchetti — Tue, 24 Jun 2008 07:01:00 GMT

This is the third of a series of screencasts demonstrating how to set up HPC Server 2008 head nodes for high availability.
In this part we discuss the installation of SQL Server 2005 on a 2008 fail-over cluster.

HPC Server 2008 High Availability - Part 2

gmarchetti — Tue, 17 Jun 2008 07:01:00 GMT

This is the second of a series of screencasts demonstrating how to set up HPC Server 2008 head nodes for high availability.
In this part we discuss the creation of a 2008 fail-over cluster.

HPC Server 2008 High Availability - Part 1

gmarchetti — Fri, 30 May 2008 15:19:00 GMT

This is the first of a series of screencasts demonstrating how to set up HPC Server 2008 head nodes for high availability.
In this part we discuss the creation of a shared iSCSI storage device.

HPC in Media

gmarchetti — Thu, 24 Apr 2008 14:42:00 GMT

In my last post, I explained how HPC can be used to build UFOs. I have since learned about using HPC to make movies. I’d never have guessed, so let me share my surprise here.

Digital media production follows a complex workflows, from initial sketch to wireframe model, to rendered 3D images, to movies. HPC is typically used in rendering, encoding or transcoding.

Although Microsoft offers a solution to manage the workflow (Interactive Media Manager) and to encode (windows media encoder, expression encoder), we rely on partners for the rendering, most transcoding and non-windows media formats.

Rendering

Rendering is the process of transforming a 3-D model of an object (often displayed as a wireframe) into a 2-D image. Traditional non-interactive rendering (e.g. by ray-tracing) is a good example of an embarrassingly parallel problem. Several frames can be rendered at once, independently, one per computing node. The rendering software often also uses multi-threading on the node to compute several channels at once (e.g. diffuse light, reflected light, occlusions, refraction), then it composites those channels to generate the final image.

Partners that are active in this space are:

- Autodesk with Maya

- Softimage with XSI / Ray3

- Mental Images with Mental Ray

- Nvidia with Gelato Pro

All of those applications exist for Windows Server 2003 and are being ported to Server 2008. They offer varying levels of integration with our compute cluster scheduler. With built a simple demonstration with Softimage, for instance. We submitted a job with 25 frames to render on a small cluster with 1 head node and 3 compute nodes, each with 4 cores. The job was created as a parametric sweep with each task invoking the ray-tracing software installed on the node (Ray 3) on one of those frames. Ray 3 is multi-threaded and can render 8 channels, so we gave each task the full 4 cores per node.

Here’s what the rendered frame 1 looks like:

Here’s an example of one of the channels computed for the image above, i.e. the intensity of reflected light:

Encoding

Encoding a digital movie file (wmv, mpeg2 for instance) is the process of exploiting the redundancy in the sequence of frames making up that movie in such a way that fewer bits are necessary to represent them than in the uncompressed stream.

There are two kinds of redundancy:

- spatial, due to the correlation of neighboring pixels.

- temporal, due to the correlation of subsequent frames.

Thus, we could split a sequence of frames into a finite number of sections, then assign each section for computation to a separate node. We could further split each frame in the section into a number of tiles and distribute the tiles amongst cores within the node for encoding.

Microsoft has developed a VC-1 encoder (used for BluRay DVDs amongst others) and an SDK for it that enable parallel encoding. They are licensed on a commercial basis to ISVs and OEMs. An example of use is in Sonic’s Parallel Stream Encoder.

Another popular encoding format is MPEG2 (DVD, digital TV), for which several open source encoder implementations exist (Berkeley, Cornell).

The H264 standard is also widely supported and parallel encoder implementations are freely available for it (e.g. ELDER)

Turning a Mac Mini into a virtual Windows HPC cluster. 4 of 4

VolkerW — Wed, 26 Mar 2008 17:42:00 GMT

I am going prepare a Mac Mini in such a way that you will be able to install a Windows HPC cluster in virtual machines based on Hyper-V.

Once you have Windows Server 2008 running, it is just a matter of a few clicks and a reboot to get Hyper-V running.

After this final step you are ready to use Giovanni's document and prepare your Cluster-In-A-Box-Mac-Mini.

Check out my blog post if you are looking for more details.

Turning a Mac Mini into a virtual Windows HPC cluster. 3 of 4

VolkerW — Wed, 26 Mar 2008 17:42:00 GMT

I am going prepare a Mac Mini in such a way that you will be able to install a Windows HPC cluster in virtual machines based on Hyper-V.

In the third video we are going to install the Windows Server OS in the Boot Camp partition.

Check out my blog post if you are looking for more details.

Turning a Mac Mini into a virtual Windows HPC cluster. 2 of 4

VolkerW — Wed, 26 Mar 2008 17:41:00 GMT

I am going prepare a Mac Mini in such a way that you will be able to install a Windows HPC cluster in virtual machines based on Hyper-V.

In this second video we are going to prepare the Mac hard disk for the installation of Windows Server 2008 via Boot Camp. Boot Camp is the feature that Apple provides to build a partition on a Mac hard disk to install Windows or any other OS side-by-side with the original OS.

Check out my blog post if you are looking for more details.

Turning a Mac Mini into a virtual Windows HPC cluster. 1 of 4

VolkerW — Wed, 26 Mar 2008 17:41:00 GMT

I am going prepare a Mac Mini in such a way that you will be able to install a Windows HPC cluster in virtual machines based on Hyper-V. Once you followed the steps outlined in the videos you will be able to install a fully functional HPC cluster environment, literally a cluster-in-one-box.

The video series will not show how to install the HPC nodes. Giovanni has posted an excellent document describing how to do that. I just focus in these 4 videos on building a nice virtual host for HPC clusters.

This first video shows the reinstallation of the Mac OS.
I had to reinstall the Mac OS due to the fact that I screwed up the original installation the system shipped with. I did so much disk partitioning and repartitioning that the volume was in serious conditions and required reinstall. I also wanted to keep a minimal Mac OS installation on the hard drive. Just in case.

Check out my blog post if you are looking for more details.

HPC and UFOs Explained!

gmarchetti — Fri, 21 Mar 2008 07:01:00 GMT

I had several discussions about High Performance Computing over the past week or so with people in different job roles.
Most of them wondered: “What is HPC for and where does Microsoft fit into it?”. I thought I’d spend some time trying to answer it. Here we go:

HPC is for building UFOs

Now that Iʼve got your attention, let me try and explain: I would like to take a field of application - namely aeronautics - and take you through the design process that engineers in that field follow. Iʼll then try and explain where Microsoft technologies fit in that process. I am no expert in aeronautics - itʼs just a personal interest of mine; so, if you find any mistakes or just think I have smoked one too many, feel free to comment!

One of the hot topics in aeronautical engineering is how to overcome the design limitations of traditional airplanes. If you think about it, most commercial planes today look alike: a big pipe with arrow-shaped wings attached to it. This design is reaching its scalability limits: it is very difficult to make bigger planes that carry more people over longer distances, consuming less fuel than the current models. The recent problems and delays with both the Airbus A380, A350 and the Boeing 787 are just an example of this. One of the solutions being studied is a so called blended wing-body (BWB) aircraft.
Hereʼs an example of what Iʼm talking about:

Figure 1: An example of Blended Wing-Body aircraft.

Figure 2: A Manta ray, Natureʼs blended wing-body design

Aircraft designers typically use a variety of CAD / CAE programs to study the aircraft geometry on their workstations. A good example is IBM's CATIA.

The geometry thus generated is then discretized for finite-element analysis. This step is often called pre-processing or mesh generation. Material properties and boundary conditions (loads, constraints) are associated to the mesh elements. A variety of mathematical models are produced to study structural stress, pressure distribution, heat distribution, lift & drag, etc...

Figure 3: Example of finite-element meshes.

Software like Ansys is often used for mesh generation. Several engineering outfits write their own mesh generators as well. These packages may take advantage of parallel computing.

The models thus created are then computed by separate packages (or modules within the same commercial package) called solvers or processors. These applications definitely benefit from parallel computing and are often written for high-performance computing clusters. To put it simply, the meshes are partitioned amongst computing nodes. Each of those solves equations on the elements that pertain to it, then communicates with the neighboring nodes over low-latency links to pass boundary results. Once every node has finished, we have an overall picture of the approximated solution. Several solvers are available in the public domain. Again, part of the “secret sauce” of engineering companies is in their own solvers, their algorithms, speed and precision.

The last step of the simulation is visualization or post-processing. The results of the computation are collated and displayed in a human-intelligible form. Here's an example of it:

Figure 4: Pressure distribution over a BWB glider

The visualization step is as important as the computation itself, because it is on the visualized data that design decisions are made. The visualization process itself may benefit from parallel processing, depending largely on the quantity of data at hand.
Figure 5 summarizes the process.

Figure 5: FEM Process

What has Microsoft got to do with it?
Well, some answers are evident: We provide the operating system where the geometry designand visualization will most likely occur.

Our platform, though, can do much more than that.

First of all, the data-handling requirements of such process are huge. Not only the file sizes involved are typically in order of gigabytes, but also those files do not mean much unless the design and simulation parameters that were used are associated to them.
Here's an immediate opportunity to build a data and metadata repository centered around SQL Server. Engineers must be able to retrieve past simulation data to work on statistical models. Regulators impose that design documentation be stored for the life of the aircraft, just in case something goes wrong and decisions must be re-visited and corrected.

Secondly, the workflow that I have presented is rather simplistic and is part of a much more complex design process. Here's an example:

Figure 6: Structural, Aerodynamic and Aeroelastic Design Concept Model

Microsoft's Workflow Foundation (WF) offers the building blocks for the application logic that automates such complex workflows. It also exposes several services, like persistence and tracking, that are extremely useful to manage the complexity of such processes, making interaction with them much easier. With Visual Studio one can construct such application logic workflows and then host them as services on Windows Server. The communication amongst those services can be handled by Windows
Communication Foundation (WCF).

The compute cluster per se can also be conceived as a “computing” service exposed to the application logic via a WCF interface. In fact, thatʼs what weʼre building into HPC Server 2008.

Thirdly, HPC Server 2008 offers a computing platform for your solvers that is easy to integrate in your engineering workflows, thanks to the variety of interfaces it exposes. Its management can be mostly demanded to a traditional IT organization, thus leaving the engineers free to focus on the design. It is a relatively small and conceptually simple part of the overall solution.

Last but not least, Windows Presentation Foundation (WPF) is a powerful tool to build data visualizers relatively easily.
Sharepoint Server can be used as a presentation layer and as a WF host, masking the complexity of the underlying system. It has the advantage of enabling collaboration on design decisions as well. For instance, an engineer could use Office to produce a report on his latest simulation results, embedding links to a Sharepoint applet that
interfaces with the simulation system. Another engineer could then read the report and dig into the simulation data and parameters. If not satisfied, he could trigger another simulation from the portal and embed his results in the document as a comment. The document would then be checked back into Sharepoint.

Figure 7: Technical architecture of a computing solution

Summary
I hope that I haven't bored you too much. The morale of the story is relatively simple: Microsoft has a pretty complete platform, which you can use for high performance computing now. Projects may be long and complex compared to traditional Microsoft engagements, but their impact is huge - quite literally :-).

Figure 8: The B2 bomber.

References and Acknowledgements
Here are the sources that I have shamelessly plagiarized. A big thank you to:
C. Osterheld, W. Heinze, P. Horst, Preliminary Design of a Blended Wing Body Configuration using the Design Tool PrADO
M. Barton, S.D. Rajan, A Finite Element Primer for Engineers
Marc Holmes, Simon Cox, Delivering End-to-End High-Productivity Computing, Microsoft Architect Journal n.11

Using Powershell in HPC Server 2008

gmarchetti — Tue, 19 Feb 2008 08:01:00 GMT

This short video shows a few examples of how to use Powershell with HPC Server 2008 beta 1 to handle jobs. It also demonstrates exercise 7 of the HPC Server 2008 lab document previously mentioned here.

Compute Node Deployment with HPC Server 2008

gmarchetti — Mon, 18 Feb 2008 08:01:00 GMT

This short video shows an example of how to deploy compute nodes of a HPC Server 2008 beta 1 cluster. It corresponds to exercise 2 and 3 of the HPC Server 2008 lab document previously mentioned here.

Job Submission and Management in HPC Server 2008

gmarchetti — Tue, 12 Feb 2008 08:01:00 GMT

This short video shows an example of how to submit jobs and monitor their status on a HPC Server 2008 beta 1 cluster. It corresponds to exercise 4 of the HPC Server 2008 lab document previously mentioned here.

Deploying HPC Server 2008 head node

gmarchetti — Mon, 11 Feb 2008 08:01:00 GMT

This short video shows an example of how to set up a HPC Server 2008 beta 1 head node. It corresponds to exercise 1 of the HPC Server 2008 lab document previously mentioned here.

HPC Server 2008 webcast

gmarchetti — Wed, 23 Jan 2008 01:12:00 GMT

My introductory demo / webcast on HPC Server 2008 beta 1 has been recorded. It can now be viewed here.

I have received feedback from several people about it, requesting instructions to build the virtual machines I used and to repeat the demos. Hence, I have written a short document that will guide you through the set-up of a virtual HPC Server 2008 cluster on Hyper-V. It also provides you with some examples of what is possible with the new administration, diagnostic and scheduling tools. It is intended as a hands-on introduction to the salient features of HPC Server 2008 beta 1. It is not an official click-by-click manual and it must be considered work in progress. I will update it when beta 2 is released to cover new or changed features. No warranty is implied nor support promised. You try at your own risk ☺

The document is now available on https://windowshpc.net/Resources/Documents/2008b1_labs_and_demos.zip

HPC and Gaming

gmarchetti — Tue, 15 Jan 2008 06:38:00 GMT

Who'd have thought that HPC would be applicable to videogames? Well, my mind was changed for me when we worked with CCP Games, makers of the largest online multi-player game, Eve. Here their developers explain why they are looking at high-performance computing and what is behind Eve. Enjoy :-)

HPC in Action - 1

gmarchetti — Fri, 28 Dec 2007 21:48:00 GMT

Ever played with the "game of life"? Well, here's an example of the real thing: HPC used to study evolution and cellular dynamics. Bioinformatics is an interesting - if unusual - field of application. Computation and visualization go hand in hand, as one is required to make sense of the other. In this video you'll see examples of using WPF to display results computed by CCS v1 and v2. This is also one of the first examples of use of MPI.NET, a managed-code implementation of the message-passing libraries.

Linux / HPC Server 2008 dual-boot

gmarchetti — Fri, 14 Dec 2007 02:31:00 GMT

How to Build a Dual-Boot
Linux / Windows HPC Server 2008
System
Draft
Giovanni Marchetti, Technical Evangelist, Microsoft, gmarchet@microsoft.com
Overview
This document illustrates the steps necessary to build a dual-boot Linux / Windows HPC Server 2008 solution. Although it is intended for use on HPC clusters, the techniques illustrated are applicable with little modiﬁcation to deployment systems based on Windows Deployment Services, which is used in HPC Server 2008. The procedure can be summarized in:
1.Verify the existing Linux disk layout.
2.If necessary, modify the existing Linux partitions to free space for Windows installation.
3.Install Windows.
4.Set the active boot partition so that the desired o/s is started at the next reboot.

download the rest of this document from Windowshpc.net our HPC community site

An interview with Kyril Faenov on High Performance Computing (HPC) / Windows Compute Cluster Server

extreme — Thu, 15 Nov 2007 08:01:00 GMT

I chatted with Kyril Faenov, the general program manager of HPC, and asked things like why Microsoft even decided to get into the high performance computing environment, what can be done with the product, and why you might use Microsoft’s HPC solution over the others. My knowledge of HPC was pretty limited, so this is a great one to watch for an interesting intro. HPC for the stock market, video rendering, people already using 10s of thousands of nodes? If you care to listen over the edge, you’ll find elephants from Thailand, and some other interesting pieces from a known Russian artist.

Windows HPC Server 2008 overview screencast, previously known as compute cluster v2

gmarchetti — Thu, 15 Nov 2007 08:00:00 GMT

This video will give you a quick tour of the 1^st beta of Windows HPC Server 2008, introducing the many advancements made in the area of deployment, management and scheduling. It is the first of a series of "hands-on" examples. If you've used compute cluster server or equivalent products, you'll like what we have in store for v2

Click here to watch the video, or right-click the download link below to save the video to your local machine.