HoloPy³

“HoloPy³” is a cooperative game in and with virtuality, researches the expansion of theater with digital media and explores the applicability of holograms and Virtual Reality. In this respect, “HoloPy³” reflects on the added value of three-dimensional and interactive narrative forms in terms of content and art – especially in view of the non-existent fourth wall in these media.

The developer is Lena Biresch. She originally came from the theater where she worked as assistant, director and playwright. In 2017 she started with the big hype to deal more with Virtual Reality and then retrained as a game programmer with focus on VR. Besides that she experiments with hologram pyramids corresponding to the illusion technique “Pepper's ghost”.

“HoloPy³” brings theater, holograms and Virtual Reality together in a new way: First of all, theater as a place of embodied performance, ritual performance practice and social exchange; then holograms as a collectively perceivable augmented (extended) reality; and finally Virtual Reality as an experience that is currently mostly realized in the conscious perception of the individual. In this sense “HoloPy³” investigates how digital, interactive and three-dimensional theater can work. For this the form of a cooperative game is chosen: One player is in a virtual world via VR glasses (hereafter to be referred to as VRPlayer) and a second player can overlook this virtual world from the outside in a hologram pyramid (hereafter to be referred to as PyPlayer). This is above all a game with dimensions and perspectives: While the 1st player appears to the 2nd in the pyramid like a tiny figure, this world appears to the 1st player oversized, he even needs the 2nd player to orientate himself in it. And this is also how the gameplay is set up: Only if the two players support each other and complement their respective perspectives with those of the other player they can win the game.

Another point of interest is the combination of virtual and augmented reality. “HoloPy³” brings the virtual world out of the head of the VR Player as a hologram into “reality”. Furthermore is investigated the disappearance of the 4th wall in the new immersive media and the alternative forms of storytelling that result from this, particularly with regard to the role of the passive spectator that shifts towards the main character, the player. This paradigm shift is exemplified in the game by the fact that the players change roles after a round: The power of VR and gaming lies not least in the possibility to playfully take on the perspective of another.


Imagefilm


The research comprises 4 different branches:

1. Technical setup

• For example: How to channel the different outputs so that a different image appears on the VR glasses than in the pyramid?
  
• What camera orientation is needed to capture the scenario from a suitable perspective?
  
• And what game settings are necessary to ensure that a different image, in fact the correct one appears on each side of the pyramid?
  

2. Gameplay including programming

• For example: Which game forms are suitable for cooperation?
  
• How do you get the players to form a team?
  
• What is the best way to tell the change of perspective?
  

3. Content in the hologram pyramid

• For example: How must the hologram be created graphically to make the illusion work?
  
• Which worlds and dimensions are best suited for this?
  
• How must the playing field be designed so that it does not appear only two-dimensional?
  

4. Material and construction

• For example: What material must the hologram pyramid be made of to get the best possible image?
  
• How to stabilize a pyramid of this size? (i. e. at least 900x900mm ground)
  
• How could a version of the entire setting be build to easily take it away on a tour?
  

The first month of the fellowship was spent on the basic technical setup and material questions. Thanks to the generous loan of a professional hologram pyramid by the Puppeteers, the 2nd and 3rd months were used to focus entirely on the content. In doing so, it was proceeded in developing many very different mini-games and learning much about what works and what doesn't. In the 4th months what has been learned was applied by developing a final game. Unfortunately, there was not enough time to develop the finished game, but only the first few scenes. The last month was mainly used to prepare the final event, which could only take place virtually on Mozilla Hubs because of Corona. This swallowed a lot of time away from the actual project, but in the end was a complete success as part of the annual conference of the dramaturgische Gesellschaft. And finally, the project had to be documented and the workspace had to be cleared.

The project is made with Unity 2020.1.4f1.
The concrete scene plays around the maps origin. Around it are positioned 4 cameras on the 4 different sides (right, left, front, back), which have to be adjusted correctly for the respective scene, depending on how extensive it is and how large its image in the hologram pyramid should be seen. The images of the 4 cameras are targeted via target texture to 4 raw images on the UI. The UI is ultimately what is projected into the hologram pyramid: A different image on every side of the pyramid (see photo of “HoloFire” on the right). To prevent the single images from overlapping, mask objects must be parented to the raw images. These consist of the respective sections that are meant to be visible as a raw image plus the mask component (see photo of the hierarchy on the left).

In the project a HTC Vive Pro and the player prefab of Steam VR are used. Because the teleporting integrated in Steam VR is not suitable for this gameplay, since the PyPlayer has to see from where to where and how fast the VRPlayer moves, an extra control was implemented. This allows the player to move classically with the controller pad depending on in which direction he is facing and 1. only if the pad is also clicked and 2. if no NoGoLayer is raycasted in the direction of movement. In addition, the player controller performs a ground check via raycast and ground layer, so that the player always remains on the ground.
Since with the VR glasses and the projection into the hologram pyramid 2 different video outputs have to take place for 2 different monitors and Unity’s default display mode is one monitor only, it is used Unity Multi-display (implemented only since Unity 2020) as it is well described on https://docs.unity3d.com/Manual/MultiDisplay.html

Formally, augmented and virtual reality are combined in “HoloPy³”. For this purpose, the form of cooperative play was chosen from the outset, in which no one can win alone, but only both together. Such forms of games have a team-building effect and achieve a shared experience. Thus, not only the two realities, but also the players standing up for them are connected with each other. Cooperative games are known from Escape Rooms, for example, where several players work together to solve puzzles of varying difficulty. This form of game serves as a template for “HoloPy³”.
Numerous game settings were tried out, many of which worked, but many of which did not. Due to the number, this section will focus on the central results.
The central question in game design is: What should the game tell? And what should happen in the player's head while playing? Therefore, it is not so much about the individual quests that occur. And besides, the puzzles must not be too difficult, as this would lead to frustration - a less desirable narrative content. So the player is taken by the hand and guided through the game, so that he has just the feeling of having solved the puzzles himself - or rather in this case together. So much more than stringing together complicated puzzles, a story needs to be told that conveys the content achieved.The central themes of “HoloPy³” that are to be narrated are 1. the cooperation between the two players that connects them and 2. the play with dimensions as well as the subsequent change of perspective. In order to narrate the 1st theme, it is especially important to divide the game responsibility equally between the two. And for the 2nd theme, the difference of the two dimensions must be emphasized and also very different worlds of experience must be created. On the other hand, how well the PyPlayer can recognize the VRPlayer in the hologram pyramid makes a big difference in how connected the two players are. It turns out that it works much better if the VRPlayer is so big that you can recognize his face. However, this contradicts the goal of having as many different dimensions as possible. A good middle ground must be found on this point. Furthermore, both of them depend on the size of the hologram pyramid, of course.

In this, the Puppeteers from Dortmund were a great contact. With their many years of experience with hologram pyramids, they were able to give numerous tips on how the image projected into the hologram pyramid must be:

1. Movement is the be-all and end-all. Pepper's Ghost is an illusion trick where you have to distract from the illusion like a magician. Because actually the image is only projected as a two-dimensional reflection plane in the hologram pyramid. As soon as movement comes into play, this two-dimensionality is no longer perceptible to the human brain and the illusion of a three-dimensional body in the pyramid is created.
   
2. The image must look exaggeratedly three-dimensional. The more spatiality is created with lighting and contrasts, the more spatiality/physicality is created in the hologram pyramid.
   
3. Particle systems work! They fake movement as well as spatiality. (Example: The VRPlayer flies “through the pyramid” with a Spaceship that has a particle system to the exhaust. Thus, the particle system has a quasi double effect, because it moves itself and is in motion.)
   
4. Force the PyPlayer to move around the hologram pyramid: This introduces an immersion break at the edges of the pyramid, but enhances the overall 3D experience.
   

The dimensions of the pyramid must give a 45° angle between the projection screen and the sides. This gives a ratio of the side width to the side height of 1.4142 : 1, rounded up.
The professional devices are made of nano-coated plastic, which the manufacturers themselves are researching; the panes are mirrored on one side and mutually anti-reflective to avoid the double image that results from the thickness of the glass; the glass that comes closest is “Schott Beamsplitter” (50% reflection: 50% light gets through) = semi-transparent mirror according to Schüfta; Of course, you can also achieve the effect with film, but the image is much clearer with glass, which greatly enhances the immersion. Besides, foil is much more delicate and the pyramid construction would be hard to transport. But for “HoloPy³” it is planned to play at game festivals or other theaters. That's why “HoloPy³” relies on the beamsplitter variant. A specialized glass company cuts the glass to size. After that, the locksmith's shop will build 4 triangular frames into which the glass panes will be inserted. These 4 triangles can finally be put together with built-in magnets. Thus, the pyramid can be disassembled and easily transported to guest performances.