I see people talking about new super computers, new high performance computing (HPC) environments and even I am involved in managing a HPC environment. We talk about all this stuff without realizing that it’s all about managing data effectively that machines these days generate. If we can effectively manage data online or offline we won’t need these huge server and storage farms and will reduce the carbon footprint as a result.

So how can this be done is what I am trying to talk about in this blog. I have normally worked with Biomedical Datasets in last decade, so I will take examples from those but they can be easily applied to astronomy, geosciences, etc. which generate lot of data on daily basis.

Now let’s talk about what I regularly use on daily basis and then just compound the problem by adding thousands users like me. I have a high-end windows desktop for mundane tasks and I use a Windows HPC server and a Ubuntu server for high-end tasks. I usually process at least a whole or part of a genome as required by my fellow researchers or just for trying some new bioinformatics workflow and tools. I have huge amounts of storage attached to all these machines I daily use, but if I still fall short of storage I have access a national data fabric and to state’s peta scale data store. BTW the peta scale data store is currently being upgraded to handle huge amounts of SKA is going to generate because of Australia’s bid.

At home I have a complete Mac environment with one lone PC. I have a Mac OS X Lion Server controlling everything and two Mac Book Pros. I have about 3 TB of storage between these machines and I back up everything on a 3 TB NAS on daily basis. Now given the fact that I normally take a bit (stretching the semantics of bit here) of my work home and also that I use my NAS as mini Linux server, I m running out of space. Now that storage these days is so cheap I m in process of upgrading my NAS to 9 TB.

Now let’s talk about what happens when I m mobile. I normally access my data through the national data fabric and icloud. These cloud storage solutions normally give me around 200 GB of storage to move some of my data with me. I am always on a lookout for bigger storage solutions in cloud to give me mobility.

Let’s talk about cloud computing and options it has for storing and processing large amounts of data. Unfortunately for me apple is not big on high throughput data applications on cloud but I really like the advances Microsoft has made with windows azure. Also the slices of compute and storage they offer are competitive to amazon. And finally it’s Microsoft so it will seamlessly integrate with everything although we will have few bugs and quirks.

More about our use of Azure at Curtin for processing sequencing data:

http://www.itnews.com.au/News/271076,curtin-trials-dna-sequencing-in-azure.aspx

Somebody asked me a while back while preparing for a grant’s budget: what is the best hardware for running high throughput whole genome sequencing? And I said a budget laptop and put at least 50K for it. He looked puzzled, but what I meant was that we need the laptop to connect to our high-end server on which the compute will run and store data on the corporate SAN. We needed 50K to add storage to university’s corporate SAN.

I was also recently asked if bioinformatics on cloud was a good idea. I said yes and no and I have to get back to you on that. Basically we can use tablet devices like iPads running anything from Windows, iOS and Android to connect to cloud. The data can be stored in raw or processed form on the Cloud. There are numerous apps for seamlessly accessing data from the cloud. Even the compute can be done remotely using these thin clients like iPads and we use or build apps to run jobs on big HPC servers and clusters. The major issue with the cloud is network bottleneck and so the data and compute needs to be at one place or as close as possible. Also whether cloud in its current state can handle storing and processing Tera or Peta scale data is another question.

So I am working towards a HPC+Cloud Hybrid model where all the apps that you need are in cloud including the apps for storing data and doing computation on it with HPC. Researchers like me use thin clients like iPhone and iPads to use these apps.

Next we will see if such a framework is possible in current environment and what pieces of the puzzle are missing. Well Microsoft has done of work in developing Microsoft biology tools (MBT) for HPC server 2008 and they have NCBI BLAST on azure. Running BLAST is no easy task it is very resource intensive can consume huge amount of resources for big sequences. One of the best tools I like in MBT is the excel add-in for bioinformatics as a ribbon. It allows us do sequence alignment and BLAST on sequences. With Office 365 and BLAST already on Azure this app is the easiest for Microsoft to move to cloud. Well when we can put one of most computationally intensive tasks of sequence assembly on the cloud what stops us from putting rest of the tasks on the cloud and run everything from the cloud. Nothing right! Hold on to that thought and let me explain why we have been very slow in putting all the high data intensive tasks on the cloud.

I quote Fran Berman here, VP of Research at Rensselaer Polytechnic Institute and co-chair of the US Blue Ribbon Task Force on Sustainable Digital Preservation and Access. They released their report on Sustainable Economics for a Digital Planet: Ensuring Long Term Access to Digital Information. In the report Fran Berman said, “The data deluge is here. Ensuring that our most valuable information is available both today and tomorrow is not just a matter of finding funds, it’s about creating a ‘data economy’ in which those who care, those who will pay, and those who preserve are working in coordination”. The report is available at: Data Deluge Report

Now is it true, and yes it is very much a reality. I couldn’t think of better place than SDSC (San Diego Supercomputing Centre) to make the final report available. They crunch more data in a day, than all of us can ever generate collectively in a week. They manage many massive data sets but the one worth mentioning here from a biological prospective is Protein Data Bank (PDB). PDB contains structural information about all the known proteins in the world and links to more information about these proteins like their structure, classifications, and functions from other massive databases hosted elsewhere like sequence information from UniProt in European Bioinformatics Institute (EBI) Protein classification information from SCOP hosted at Cambridge, and function information from publications in biggest massive data set of all: MEDLINE.

Now I read somewhere in a recent Nature Article Janet Thornton Director of EBI saying: “we cannot manage to store all the we generate due to lack of funding”. Lack of Funding – if you haven’t realized by now they are the biggest bioinformatics institute in Europe and one thing they don’t lack is funding. She meant to say they lack storage for massive amounts of data EBI generates and they need to literally buy more every time new project starts. They need cooperation from their collaborators worldwide and share each other’s burden of storage. Well Worldwide Protein Data Bank (WWPDB) was recently created through global collaborations as massive data integration task and that can be done for every database. UniProt EBI’s own database was recently created in same fashion. The same can be applied for every other major data source – collaborate and share the burden.

I was the one who proposed the idea of integration of protein data sources in 2004 to bring some standardization. It went from having a semantic map to better interoperate heterogeneous data sources to a full-blown world’s first protein ontology (more at: Protein Ontology

Another example is GenBank recently announced that it will no longer accept submission of sequence reads generated from next generation sequencing (NGS) methods but will accept submissions of assembled genomes from these reads. Now this is essentially to save space – storing eventual output of assembly but not the intermediates. NGS these days generates gigabytes and terabytes of data that universities and institutions are not at all prepared to manage.

Also I was told that when the SKA bid gets decided in 2012 the first one-third of the project would generate more astronomy data than we ever generated.

So do we go for state and national massive server and storage that everyone uses or do we put everything on cloud and let the big IT companies do what they do best – manage infrastructure and storage. I will prefer a Hybrid HPC+Cloud model where all these stakeholders are involved in a collaborative network and we collectively share the burden of Big Data. Either ways we need centralize IT infrastructure to better mange the data and information and to save up electricity and maintenance costs, which put together are more than the cost of equipment.

———–
Dr. Amandeep S. Sidhu

This blog is first in the series, next will be on various cloud services provided by companies. I hope to cover Microsoft, Amazon, VMWare, EMC, NetApp, Dell, HP, IBM, Google and Apple in the discussion.

The comments in this blog are of my own and do not represent the companies I worked or the companies mentioned in the Blog.