Open Access Open Access  Restricted Access Subscription Access

THE CHALLENGE OF MANAGING AND ANALYZING BIG DATA

Hermann Heßling

Abstract


The amounts of data produced in science are growing exponentially. Traditional methods for storing and maintaining the enormous flood of data seem to be no longer sufficient anymore. The complexity of the data that will be distributed more and more worldwide, is going to constitute a considerable challenge for their analysis. According to Alex Szalay there soon will be produced so many data that they cannot even be stored and maintained anymore. The data have to be analyzed in real time in order to extract the relevant information. An outline of the project Large Scale Management and Analysis (LSDMA) is given. The status of our research group on distributed real-time computing is reviewed. Finally, a novel approach to time-dependent image processing based on local thermodynamical methods is presented.

Keywords


Big Data; Real-time computing; Time-dependent image processing.

Full Text:

PDF

References


T. Hey, S. Tansley, and K. Tolle (Eds.), The fourth paradigm, data-intensive scientific discovery, Microsoft Cooperation, 2009. Available at http://research.microsoft.com/en-us/collaboration/ fourthparadigm.

http://wlcg.web.cern.ch

P. E. Dewdney, SKA1 system baseline design, SKA-Tel-SKO-DD-001 (2013).

A. Szalay, Extreme data-intensive computing in science, at: 1st International LSDMA Symposium The Challenge of Big Data in Science, Karlsruhe, Germany (25 September 2012).

E. C. Friedberg, an Interview with Sydney Brenner, Nature Reviews Molecular Cell Biology, (9) (2008), pp. 8-9.

http://www.helmholtz-lsdma.de.

LSDMA IDM-Workshop, DESY, Hamburg, Germany (March 11, 2013).

http://www.dCache.org

J. Weschenfelder, CDMI for dCache, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German)

L. Blocher, and T. Schubert, private communication, 2013.

A. Kobitskiy, G. U. Nienhaus, J. C. Otte, M. Takamiya, U. Strahle, J. Stegmaier, and R. Mikut, Light Sheet Microscope (LSM), in: R. Stotzka (Ed.), Data Life Cycle Lab. Key Technologies, Big Data in Science, Status 2013. Available at http://digbib.ubka.uni-karlsruhe.de/volltexte/1000037134.

A. Barty, The coming deluge of data from XFEL sources, LSDMA Workshop, DESY, Hamburg, Germany, March 12, 2013.

H. He?ling, Real-time grid computing: recent results on a pilot job approach, DESY Computing Seminar, University Hamburg, Germany, February 7, 2011.

SIMON (Simple Invocation of Methods over Networks), http://dev.root1.de/projects/simon.

P. Eckert, Optimization in Java: Client-Server Communication, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

P. Eckert, Garbage Collection in Java, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

C. Lehmann, Studies on Using Peer-to-Peer Techniques in Grid Computing, Master Thesis, University of Applied Sciences (HTW), Berlin, 2013. (in German).

http://juxmem.gorge.inria.fr.

K. Kochan, Distributed Tree Search in GriScha, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

L. Bortfeld, Real-time Communication in Grid Computing based on XMPP, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

P. Stewart, Real-time Communication in Grid Computing based on XMPP, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

J. Erhard, and C. Lorenz (Eds.), 4D Modeling and Estimation of Respiratory Motion for Radiation Therapy, Springer, Heidelberg, 2013.

G. Dixit, J. M. Slowik, and R. Santra, Proposed imaging of the ultrafast electronic motion in samples using X-ray phase constrast, Physical Review Letters, (110) 13 (2013), 137403.

D. Kondepudi, and I. Prigogine, Modern Thermodynamics. From Heat Engines to Dissipative Structures, John Wiley, 1998.


Refbacks

  • There are currently no refbacks.