PROVIDING ADITTIONAL COTENT TO PRINT MEDIA USING AUGMENTED REALITY

: While not a new concept, Augmented Reality (AR) has gained increasing momentum in the last few years through the widespread propagation of high-performance mobile devices. These powerful handheld devices allow users to see the real world enhanced with computer-generated content anywhere at anytime. This has led to a massive rise of AR technology; large companies and start-ups are adding affordable AR accessories to their product portfolios. In this paper we describe a joint project between our research group and the Konzerthaus Berlin to use AR to enrich the Konzerthaus’ printed media. With the aid of AR, the readers of the seasonal brochure of the Konzerthaus Berlin are given the opportunity to access additional content complementing the brochure. These contents include images, illustrations, audio, video, simple 2D and 3D animations, interactive media as well as 3D objects. The application was implemented using Vuforia and Unity. This is an extended version of the paper presented at the Conference on Culture and Computer Science 2017 [17]. Copyright © Research Institute for Intelligent Computer Systems, 2018.


INTRODUCTION
The paper presents the results of the APOLLO 1 project.The APOLLO project is a joint project between the Konzerthaus Berlin 2 and the research group INKA 3 at the University of Applied Sciences (HTW) 4 in Berlin.One of the projects goals is the development of various Augmented (AR) and Virtual Reality (VR) applications for the Konzerthaus Berlin.
Konzerthaus plus is the name of the first subproject being an AR system that aims to implement a "living concert hall magazine".In this subproject the printed seasonal magazine of the Konzerthaus Berlin will be augmented with digital content.The team of this subproject consists of musicologists, designers and computer scientists.
1 https://inka.htw-berlin.de/en/project/apollo 2 https://www.konzerthaus-berlin.de/en 3 https://inka.htw-berlin.de4 https://www.htw-berlin.de/en In cooperation with the Konzerthaus Berlin we are currently looking for innovative approaches to increase the customer relationship.While we considered many different areas, the print media published by the Konzerthaus Berlin has a vast reach and similar projects proved that the concepts are effective, e.g. the Städel Museum 5 has implemented a range of applications, mainly meant to be used inside the museum.Our approach is to provide a variety of additional content to users of the seasonal magazine for the Konzerthaus Berlin.In order to not disturb the layout and design of the magazine, markerless tracking was used.The corresponding page of the magazine is used in the same way as the traditional AR markers for the identification and registration of the corresponding contents.This has lead to a portfolio of barely visible AR markers embedded in the brochure with varying offers of visualization and interactivity.To inform the user that a page has additional AR content, every page has an additional small icon.This informs the user of the additional AR content, but otherwise these marker pages are not distinguishable from regular pages.Our long-term goal is to provide a single application that integrates a wide range of mixed reality applications that can be used outside as well as inside the physical building.In addition to the application described in this paper, AR modules for entrance tickets and a guided tour through the Konzerthaus Berlin will be integrated.A digital program booklet will also be part of this application.

RECENT WORK
The content in print media is based on the presentation of text and images.New media, such as audio, video and various interactive media formats, cannot be integrated directly into a printed newspaper or magazine.Classical print media consist of static content and offer poor or no interactivity [1].
Different approaches such as printing web links, printing QR codes or adding additional media such as CD and DVD were used to try and counter this problem in the past [18] [19].In principle, the desire of the reader to get further information can be fulfilled using these additions.However, the user has to switch from the actual medium, which provides the content and information, to a new medium to retrieve the additional information.The connection of the user to the print medium can quickly be lost.With the use of AR, print media can offer contextual information, make them interactive and give the possibility to include forms of media content that would not be possible by printed media alone.
AR allows the user to view the real world with superimposed computer-generated content.Therefore, AR complements the reality rather than replacing it completely [2].Since Ivan Sutherland's pioneering work of the 1960s and the development of the first Head-Mounted Display (HMD) up to the present time, the research and development in the field of AR have been significantly advanced [3].Advances were made in the main research areas of AR, tracking, visualization, and registration, as well as interaction.All these areas must be addressed by the usage of AR for print media.Tracking is necessary to identify the printed content to be expanded and to register the artificial content geometrically correct.Visualization techniques are required to present the additional contents to the user.Interactive access gives the user the possibility to control the additional content.
Registration is the process of overlaying the virtual objects in the real scene and is one of the most important research topics in AR [2], [4].The required information for registration is extracted from the characteristics or feature points of the real scene [5].The authors in [5] define two categories of registration approaches.
First, the sensor-based approach, where mechanical, magnetic, ultrasonic or optic sensors are used to gather the information for registration from the real scene.Often a calibration of the sensors is necessary and specialised sensors can be expensive or lack satisfactory levels of accuracy [5], especially when external sensors are used.Computer vision based approaches avoid the need to calibrate external sensors.They offer the potential for accurate tracking without the use of additional sensors.Different approaches for camera-based tracking are named in [5], which can be used without calibrating the camera.The necessary information for the registration is gained through the tracking.
The use of image recognition in AR for tracking is widespread.Due to their simplicity and accuracy, tracking techniques based on markers are some of the most commonly used techniques in AR [6].The simplicity and accuracy is based on the design of the markers.A predefined shape and the high contrast of the embedded pattern make them easily recognizable in most setups [6].The authors in [6] describe a system, which combines tracking of markers with the gyroscope sensor of the mobile device for pose estimation of a mobile AR device.Another example can be found in [7], where the authors use an ARToolkit 6 Marker in an interactive AR system to place furniture in a room.In this system, markers are used to define the spatial position of the furniture to be placed.Dibidogs and others' children's storybooks 7 contain also AR markers to enrich the user experience.
Markerless augmented reality techniques allow the use of natural images as targets or bases for the placement of superimposed virtual objects.The natural images correspond to parts of the real world, which are captured by the camera of the AR system and are examined for their natural features such as edges, corners or texture patches [8].The authors in [9] describe a markerless AR application for picture books with the usage of the scale-invariant feature transform (SIFT).The usage of the SIFT algorithm and associated SIFT descriptors to match the feature points of the images allows them to use the pictures included in the book included as AR markers and to enrich them with virtual 3D objects [9].The authors of [10] describe an approach to speed up the speeded up robust features (SURF) algorithm on mobile devices.
For a long time, most AR interfaces were based on the desktop metaphor or used designs from Virtual Environments Research [20].A major trend in interaction research, especially for AR systems, is the use of heterogeneous designs, tangible interfaces [4], and multimodal interaction methods.Heterogeneous approaches blur the boundaries between reality and virtuality and take parts from both worlds.Tangible interfaces emphasize the use of real, physical objects and tools.Similar to AR systems, the user sees the real world and will often interact with real objects; it is expedient that the AR interface has a real component [4].A major focus in interaction research lies in the use of mobile AR systems (MARS).With the increasing sales of smartphones and their increasing technical equipment and technical possibilities of these, smartphones seem to be the ideal platform for AR.Many AR applications for smartphones and tablets are limited to the visual superposition of reality with additional information.The interaction with the most important content is usually done via the touchpad [11].A classic example for this kind of application is the mobile AR browser.Interacting with virtual objects in the way one would be interacting with real objects is one of the biggest problems to solve.In order to allow interaction with virtual objects in such a way that the interaction with real objects is emulated, different approaches are pursued [11].One approach is tangible user interfaces for real object-based interaction.The targeted virtual objects are coordinated with a real object.They allow the user to handle, rotate or manipulate virtual objects by handling and rotating the coordinated real object [11].By using different tracking methods, changes on the real object in positioning or alignment can be transferred to the virtual object.
Another approach is to track the user's hand to interact with virtual objects [11].With the aid of computer vision, the user's hand is recognised by a camera-mapped AR system.The authors in [12] describe in their work a system based on a normal RGB camera to recognise the user's hand and gestures.The authors in [13] track fingers to realise in-air typing.In [14], hand-drawn shapes are used to create AR objects.In [15], a system is presented in which the user's fingertips are equipped with AR markers.This allows recognising the finger gestures of the user and using them for manipulating virtual objects.
The mobile device itself can also be used for interacting with virtual objects due to the configuration of the diverse sensors.The authors in [16] describe a system in which virtual objects can be manipulated depending on the orientation of the mobile device.By changing its position and orientation, the user gets the opportunity to move and rotate virtual objects.

APPLICATION CONCEPT
For every concert season the Konzerthaus Berlin publishes a magazine that focuses on highlights of the upcoming season and summarises the former season 8 .This brochure is the most important printed publication of the concert hall and has a far reach.Besides being handed out to visitors of concerts and other events at the concert hall, the seasonal magazine can be obtained free-of-charge at the location by passers-by.The cooperation with the Konzerthaus Berlin provided us with the opportunity to develop an AR application to enhance the contents of their seasonal magazine.However, as the reader base of the magazine is closely correlated with the visitors of the concert hall and the majority around the age of 50 to 70, the expected user base for our AR application will mostly be located in this age range as well.In addition, this application also offers the opportunity to inspire a younger audience for the Konzerhaus Berlin through the use of modern technologies and presentation forms.
This knowledge influenced the general concept of the AR application and led to a cautious approach.First of all, the AR application should only extend existing content of the brochure not replace it, so that users who do not want to use the AR application are not left with the feeling that they are missing out on essential material to understand the content of the brochure.Earlier experiments by the concert hall with modern technologies, e.g. when they published the accompanying information flyer for a concert only online and did not hand out printed versions, were met with distrust and were overall conceived negatively by the older concert guests.
Furthermore, the AR application's handling needed to be as intuitive as possible to enable less technically literate users to use the application.As a result, we discarded the use of any multi-touch gestures or hardware sensors like the accelerometer or the gyroscope.While widely used across mobile applications, we did not want to assume any of them as known nor did we want to congest the user interface by adding a lot of explanatory texts.Ideally any augmentation should only need to be viewed through the mobile device's camera and if interactivity is desired, can be touched using a simple tap.
For the content and the possibilities for interactivity this meant slight limitations, but we were able to use images, drawings, audio, video, simple 2D and 3D animations, and 3D models.For example, the Konzerthaus Berlin had a recent project titled #klangberlin (translated: the sound of berlin), where they used their orchestra to replicate the sound of typical everyday situations of the city of Berlin, such as the arrival of a train of the S-Bahn or the frying of a Currywurst.The videos where extensively edited and published through the concert hall's YouTube channel 9 .The project is included in the seasonal brochure and the AR application adds the actual videos to the experience (see Fig. 1).Finally, we decided to use a markerless AR approach as this enables us to use the nonaugmented content of the printed magazine to position our augmented content.The printed content is not obscured with typical QR-code-like markers.

APPLICATION DEVELOPMENT
The development process involved close collaboration with the existing team of musicians, musicologists, illustrators and designers of the Konzerthaus Berlin to convey the same look and feel through the augmented contents as with the physical magazine and therefore strengthen their conjunction.Originally the development of native applications for Android and iOS devices was planned.Due to a small team of developers and the resulting research to reduce the development effort, it was decided to develop the application using Unity 10 .Unity is a cross-platform game development engine and is used to develop video games for PC, consoles and mobile devices.Using Unity offers the advantage that only one application has to be developed, which can be deployed for different operating systems with 9 https://www.youtube.com/watch?v=Dd5BF48LMS0 10 https://unity3d.comonly a small increase in overhead.Therefore, the developers were able to focus on one implementation using Unity and did not have to work out multiple separate implementations, e.g. for Android and iOS.
For the augmentation the Vuforia11 library was used.The Vuforia platform supports a variety of AR features such as marker-based and markerless tracking and object recognition.It also supports extended tracking, which allows for the visualization of large objects, models and media even if the corresponding marker is no longer in the field of view of the camera and provides the possibility to view AR contents beyond large areas.
The extended tracking option was further experimented with to evaluate whether this function results in an added value for the end user.However, since only single-sided markers are used in the application and the augmented area extends only slightly beyond the marker page, this technology was discarded as not suitable for the application.Besides the missing benefit of using extended tracking, the algorithm also led to unwanted behaviours in tests, e.g.leaving the augmented content intact after users changed pages which could lead to unwanted overlapping of independent augmentations from different pages.Vuforia allows implementing AR applications using either cloud-or device-based marker recognition.The main benefit of cloud-based marker recognition is the related Vuforia Web Service (VWS) API, which allows developers to 12 https://play.google.com/store/apps/details? id=de.konzerthaus.khbexplorer13 https://itunes.apple.com/us/app/konzerthausplus/id1218227009?l=de&ls=1&mt=8 14 https://www.konzerthaus.de/media/filer_public/cd/2e/cd2e3866-7dd9-4186-8b53-8fe5749a9903/ khb_saisonbroschuere_17_18_einzelseiten_web.pdfdynamically change markers for published applications without the need to update the application itself.However, for the cloud-based recognition to work a permanent Internet connection is needed and the recognition time is highly dependent on the speed of the Internet connection.For device-based recognition all markers are present in the application and updates to the marker database are only possible through an update of the application.For regularly changing content this may be unfavourable, but the content of the seasonal magazine is fixed and we therefore decided to use device-based recognition for this project.However, looking forward to the addition of an AR indoor guide for the concert hall (see Future Work section below) this decision may have to be re-evaluated.

MEDIA, INTERACTION AND INTERACTIVE ELEMENTS
In order to access the AR content, the reader must interact with their smartphone and the magazine.The user has to view the printed page through their phone's camera.For support, a simple graphical user interface is displayed on the smartphone, highlighting the selection of the area graphically.In addition, the user is advised to centre the complete page in the marked area (see Fig. 2).If the marker (the corresponding side of the magazine) is detected, the additional elements of the AR application appear as an overlay of the size of the magazine on the display of the mobile device.Dynamic media such as audio and video are visualised by a graphical representation of the content with a superimposed play button.A gallery metaphor was chosen for the presentation.Three thumbnails of the corresponding videos overlay the printed content.The user must actively confirm the play by clicking on the play button of the corresponding video (see Fig. 1).
Here the conscious interaction concept of video portals or websites with video was emulated.The user is aware that the media can be started separately by pressing the play button.With acoustic contents of the magazine the procedure is equivalent.3D models support different themes of the magazine.To display the additional virtual content, the page in the app must be scanned again.Since it is not an interactive element, the 3D model is displayed directly on the display and is visualised perspectivecorrectly and geometrically upright on the page.
Overall the application visualises three 3D models on different pages of the magazine: the downscaled model of the concert hall itself, a stylised model of the organ in the large hall and the fish Melos (see Fig. 4).The latter is the mascot of the concert hall and can be found at around 200 different places inside the building.As one of the criteria for AR is embedding in the real world, a touchscreen-based interaction to the complete or detailed view of the models has been dispensed with.When viewing the models on the screen of the mobile device, it soon becomes clear that the perspective of the virtual representation also changes depending on the distance and the position of the mobile device to the magazine.The user will easily recognise that they can change the position of the virtual content by changing their position.Another part of the magazine is devoted to the subject of seasonal highlights and is supported by photographs.Due to the limited space of exactly one page and the layout of this page, not all images that the editorial team wished for can be displayed.In order to fulfil the desire to display more photos, a stack of photos was integrated as an interactive medium into the AR application.After recognition and identification of the corresponding page through the application, 12 pictures are superimposed onto the page.These are stacked above each other.If the user wants to see the individual pictures, they must press the displayed play button as known from the video and audio player.The pictures then spread over the page of the magazine in an animated way.The arrangement of the images takes place in such a way that no virtual picture superimposes an image of the printed page.The user can increase the size of any of the augmented images by clicking on them (see Fig. 5).This map corresponds to a respective article in the magazine.On one page of the printed magazine this year's partnership of the Konzerthaus Berlin with different musicians from the Baltic States, which include Estonia, Latvia and Lithuania are to be presented.The interactive virtual map expands the information offer and also presents the venues for the partnership.For this purpose, the printed map is superimposed with a virtual map on the screen of the mobile device.The different venues of the partnership are displayed on the virtual map as additional information.The venues are represented by a graphical representation of the participating houses and cultural institutions on the ground.These appear animated and graphically represented in a cardboard cut-out-like look.The graphical representation of the printed map is rotated upwards.The representation as cardboard holder was chosen to change from the two-dimensional representation of the printed card in the magazine to a threedimensional representation (see Fig. 6).In order to not overload the display on the mobile device with too much information, the user can select the different elements by clicking one of the elements on the display.For the selected items, information about the respective playground appears on the display.

TEST
Internal tests were carried out during the first test phase.In this test the functionality as well as the consistency of the presented contents was ensured.The first test also had to ensure that the application runs reliably and stable on different devices based on the iOS and Android operating system and that the pages of the magazine are reliably recognized as markers.The various devices of the participants were used as test devices (Samsung Galaxy Note 4, Samsung Galaxy S4, Huawei Nexus 6P, iPhone 6, iPhone 6s and iPad Air 2).With the selected devices, no problems were observed regarding the differences in hardware such as screen resolution, camera, processor and memory or with different operating systems and respective versions.All selected markers of the magazine were recognised quickly and reliably.The recognition of the markers, processing by the application and presentation of the content takes place during the period of a video frame.The markers were selected jointly with the concert hall being the driving force, so that they match in content but also meet the criteria for markerless AR.
In order to use the individual builds and versions of the test phase, the possibilities of iTunes Connect15 , Apple's Testflight 16 and Googles PlayStore 17 were used.Testflight is a mobile iOS Application and was used to load and run the application on the mobile device, as well as the capabilities of the iTunes Connect Web platform applications for iOS can be distributed very easily to a selected group of testers.150 test devices could be entered there.Testflight enables developers to perform tests with a maximum of 2000 external testers.Testflight and iTunes Connect make the test process easier and accelerate the ability to publish new builds and updates.Only the build is uploaded and the testers are defined for the test.After an examination of the application by Apple, each user can install the application independently with Testflight.
Nearly the same procedure was used to deploy beta versions for Google's PlayStore.The application bundle is uploaded using the Google Play Developer Console and is distributed to a predefined number of testers.Thus, changes and updates can be made available to the testers in rapid succession.The markers used for the development correspond to the layout and design of the final magazine.
Different tests were also carried out with regard to the Vuforia framework and the quality of the image recognition.In principle, a feature-based recognition of images should have certain invariance with respect to parameters such as, for example, scaling, rotation or inclination.To assess the quality of the algorithms implemented in Vuforia, the framework and the feature-based recognition of images regarding the parameters masking, scaling, rotation, exposure conditions and the Vuforiaspecific evaluation of the target images were carried out.10 test runs were selected for the test, all done with the same marker and on the same mobile device to create homogenous criteria for the execution.Each of the parameters to be checked was performed 10 times with a distance of 50 cm between the camera of the mobile device and the marker.If the marker was detected in under 0.05 seconds in the test the experiment was evaluated as a real-time detection.
The occlusion test was performed in four steps with a 0%, 25%, 50% and 75% occlusion of the markers.The results of the test runs for the occlusion are shown in Table 1 and Table 2.In order to test the dependency of the recognition rate on the size of the markers, identical markers with different dimensions were always printed in the same aspect ratio.The distance between device and markers were always 50 cm for each test series.Table 3 shows the test results.
In general the tests indicate that a marker has to be 10 cm or larger.
After the same experimental setup the rotation test was carried out.The markers were rotated in 45 degrees between 0 and 360 degrees.No problems were detected in the recognition of the markers.
In order to simulate different light conditions four different scenarios were used.These were categorized with darker and very dark or brighter and very bright.This division is based on a subjective observation.The test also examined how the detection speed changes in the different light situations.Due to very tight deadlines the printed version of the current magazine was not available in the test phase and so the pages that were used for the application had to be printed out on single pages.In order to check whether the glossy paper of the brochure leads to problems, a magazine from last year was used as test material.The results of this test confirmed that the glossy paper does not appear to influence the detection accuracy.
Finally, the test was carried out with the printed magazine.Differences in the recognition of the markers of the test phase compared to the final product could not be determined.Only the curvature at the binding of the individual pages of the final magazine led to jumps in the representation.This behaviour was only isolated and observed with strongly curved pages.

LOGGING AND INTERPRETATION OF DATA
As the AR application is a debut for the concert hall and they are unsure whether their customers will actually use it, we added a rudimentary, anonymous logging system, which collects usage statistics.Besides general information like the number of application downloads and the number of uses for the different markers, we put special emphasis on the interactive augmentations and are logging if users are experimenting with the interactivity.The logging is used to record how often the application is used, which media are used, how long they are used for, which videos are played for how long, which interactive elements are selected, and whether the application is terminated.The table shows the number of usages of the different markers in the seasonal brochure.At the current time it is difficult to draw a conclusion, but with additional users hopefully a noticeable trend of the most appealing type of media will emerge.This will help to improve the content for future releases of the magazine.
The average time of usage for different markers shows that the recognition of markers is working reliably.As losing the marker would reset the log timing and the high average times lead to the assumption that a marker can be scanned for a long time without being lost.Furthermore, the data suggests that all media types are used relatively equally.The high usage times of the interactive media types suggest that users are experimenting with the interactive content.
However, while developing the application we expected higher average usage times for interactive elements than for static elements.The logged data shows the opposite.This could mean that users are actively searching in the static elements for interactive content, which could lead to higher average times of usage.

FUTURE WORK
Following the publication of the first version of our AR application, we will continue the development by extending the application to further use cases.One of the long-time goals is to establish a content management system (CMS) for new AR content that the Konzerthaus Berlin can use to manage the AR application, add new, and alter existing content.One of the first new applications to be implemented is an indoor guide that will be used to explain the history of the building and inform about current events.The inside of the concert hall offers vast possibilities for exciting new concepts for AR.
The planned CMS will provide a further extension of the system.The aim of the CMS is to provide the content providers with a tool which allows them to maintain, add or remove the contents of the AR application independently of developers.For this, the architecture of the application, currently provided with static content, must be adapted to dynamic content.
Finally, we will be using the logged data of the current version of the application to evaluate the overall success of the application.The logged data will tell us how many readers of the magazine will download and use the app and if they use it only once or multiple times.Furthermore, we will use this data to determine the intuitiveness of the interactive components of the application and will use the result to decide if more complex interactive approaches can be introduced in the next seasonal magazine.
Various advantages would result from the change in the application architecture.Content can be loaded dynamically without the need to publish new content via the stores for mobile applications.The application can also be used for different use cases.Content can be delivered and downloaded according to the special use case.

CONCLUSION
The goal of the development of an interactive AR application for print media has been achieved.With the help of Unity and Vuforia, it was possible for the relatively small developer team to develop a crossplatform AR application in a short time.The application could easily be built for the operating systems iOS and Android.Additional adjustments to the code were not necessary.
The algorithms implemented in Vuforia for detecting the pages of the magazine work reliably, stable and with a high detection speed.All media, defined during the concept stage, could be implemented with the desired high demands on design and aesthetic.Unfortunately, no statement can yet be made regarding the use of the application.To be able to draw conclusions about the use of smartphone-based AR applications via the obtained data, the developer team hopes for a large number of users.
The "living concert hall magazine" application has been launched together with the new printed seasonal magazine of the Konzerthaus Berlin to the general public as well as to the press, radio and TV stations on 4th May 2017 with a public event in the Konzerthaus Berlin.

Figure 1 -
Figure 1 -The three augmented video players are placed on the page.In the upper part of the page the currently active video is displayed.The printed page displays a frame of each of the three videos (train station clock simulation is active).

Figure 2 -
Figure 2 -Two different sides and markers of the magazine.Test it with the application 12,13 .Additionally feel free to use the App with the markers in the magazine 14 or with Fig. 2 on screen.

Figure 3 -
Figure 3 -Pass-through camera UI to scan the pages of the magazine.You can use Fig. 3 as marker for the App.

Figure 5 -
Figure 5 -Stack of photos (left) and the photos distributed on the magazine (right).As the final interactive element, a Baltic map is displayed within the application.

Figure 6 -
Figure 6 -Augmented map with its interactive elements as cardboard holder

Figure 7 -
Figure 7 -Different light conditions that influence detection rate.