THE CHALLENGE OF MANAGING AND ANALYZING BIG DATA

Authors

  • Hermann Heßling

DOI:

https://doi.org/10.47839/ijc.12.3.600

Keywords:

Big Data, Real-time computing, Time-dependent image processing.

Abstract

The amounts of data produced in science are growing exponentially. Traditional methods for storing and maintaining the enormous flood of data seem to be no longer sufficient anymore. The complexity of the data that will be distributed more and more worldwide, is going to constitute a considerable challenge for their analysis. According to Alex Szalay there soon will be produced so many data that they cannot even be stored and maintained anymore. The data have to be analyzed in real time in order to extract the relevant information. An outline of the project Large Scale Management and Analysis (LSDMA) is given. The status of our research group on distributed real-time computing is reviewed. Finally, a novel approach to time-dependent image processing based on local thermodynamical methods is presented.

References

T. Hey, S. Tansley, and K. Tolle (Eds.), The fourth paradigm, data-intensive scientific discovery, Microsoft Cooperation, 2009. Available at http://research.microsoft.com/en-us/collaboration/ fourthparadigm.

http://wlcg.web.cern.ch

P. E. Dewdney, SKA1 system baseline design, SKA-Tel-SKO-DD-001 (2013).

A. Szalay, Extreme data-intensive computing in science, at: 1st International LSDMA Symposium The Challenge of Big Data in Science, Karlsruhe, Germany (25 September 2012).

E. C. Friedberg, an Interview with Sydney Brenner, Nature Reviews Molecular Cell Biology, (9) (2008), pp. 8-9.

http://www.helmholtz-lsdma.de.

LSDMA IDM-Workshop, DESY, Hamburg, Germany (March 11, 2013).

http://www.dCache.org

J. Weschenfelder, CDMI for dCache, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German)

L. Blocher, and T. Schubert, private communication, 2013.

A. Kobitskiy, G. U. Nienhaus, J. C. Otte, M. Takamiya, U. Strahle, J. Stegmaier, and R. Mikut, Light Sheet Microscope (LSM), in: R. Stotzka (Ed.), Data Life Cycle Lab. Key Technologies, Big Data in Science, Status 2013. Available at http://digbib.ubka.uni-karlsruhe.de/volltexte/1000037134.

A. Barty, The coming deluge of data from XFEL sources, LSDMA Workshop, DESY, Hamburg, Germany, March 12, 2013.

H. He?ling, Real-time grid computing: recent results on a pilot job approach, DESY Computing Seminar, University Hamburg, Germany, February 7, 2011.

SIMON (Simple Invocation of Methods over Networks), http://dev.root1.de/projects/simon.

P. Eckert, Optimization in Java: Client-Server Communication, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

P. Eckert, Garbage Collection in Java, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

C. Lehmann, Studies on Using Peer-to-Peer Techniques in Grid Computing, Master Thesis, University of Applied Sciences (HTW), Berlin, 2013. (in German).

http://juxmem.gorge.inria.fr.

K. Kochan, Distributed Tree Search in GriScha, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

L. Bortfeld, Real-time Communication in Grid Computing based on XMPP, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

P. Stewart, Real-time Communication in Grid Computing based on XMPP, Internal report, University of Applied Sciences (HTW), Berlin, 2013. (in German).

J. Erhard, and C. Lorenz (Eds.), 4D Modeling and Estimation of Respiratory Motion for Radiation Therapy, Springer, Heidelberg, 2013.

G. Dixit, J. M. Slowik, and R. Santra, Proposed imaging of the ultrafast electronic motion in samples using X-ray phase constrast, Physical Review Letters, (110) 13 (2013), 137403.

D. Kondepudi, and I. Prigogine, Modern Thermodynamics. From Heat Engines to Dissipative Structures, John Wiley, 1998.

Downloads

Published

2014-08-01

How to Cite

Heßling, H. (2014). THE CHALLENGE OF MANAGING AND ANALYZING BIG DATA. International Journal of Computing, 12(3), 204-209. https://doi.org/10.47839/ijc.12.3.600

Issue

2013, Volume 12, Issue 3

Section

Articles

License

International Journal of Computing is an open access journal. Authors who publish with this journal agree to the following terms:
•    Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
•    Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
•    Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.