All posts by Jean

Some fun with multiview rendering on 3D screens

⚠ The content of this page may be outdated.
Please refer to this page for more up to date information: https://github.com/gpac/gpac/wiki/Some-fun-with-multiview-rendering-on-3D-screens

 

 

In the past months I have played with multi-views auto-stereoscopic display, and thought that it could be a nice feature for GPAC. This is still an on-going work, but results are already interesting. GPAC can now support various output type and view number, but requires OpenGL to do so.

Warning, multi-view rendering is a costly operation. The scene gets rendered multiple times, and you will need a decent graphic card to try that, especially support for VBOs to avoid sending the geometry data repeatedly, and OpenGL shaders for most screen configurations.

The output type is selected in GPAC configuration file by the following keys

[Compositor]
NumViews=2
ReverseViews=no
StereoType=None
InterleaverShader=/path/to/fragmentshader_sourcecode.glsl
ViewDistance=180
CameraLayout=Circular

NumViews specify the number of target views. It may be ignored depending on the stereo output type. ReverseViews instruct GPAC to inverse the views, e.g. ViewX becomes NumViews-ViewX.

StereoType can be one of the follwing:

  • None: regular single-view display
  • SideBySide: views are rendered side-by-side, from left to right
  • TopToBottom: views are rendered top-to-bottom, starting with the left
  • Anaglyph (*): views are rendered as anaglyph. Number of views in this mode is forced to 2.
  • Columns (*): views are rendered interleaved columns by columns. Number of views in this mode is forced to 2. Left view is mapped to even pixel X coordinate on the display window/screen.
  • Rows (*): views are rendered interleaved rows by rows by. Number of views in this mode is forced to 2. Left view is mapped to even pixel Y coordinate on the display window/screen.
  • Custom (*): views are rendered by the GLSL fragment shader source code specified by the InterleaverShader option.

(*): Requires OpenGL GLSL shader support.

 

When shaders are used, each view X in [1, NumViews] is exposed to the shader as:

 uniform sampler2D gfViewX

For example, the column interleaving in GPAC is done with a built-in shader whose code is:

uniform sampler2D gfView1;
uniform sampler2D gfView2;

void main(void) {
 if ( int( mod(gl_FragCoord.x, 2.0) ) == 0)
  gl_FragColor = texture2D(gfView1, gl_TexCoord[0].st);
 else
  gl_FragColor = texture2D(gfView2, gl_TexCoord[0].st);
}

CameraLayout defines how the camera is positioned in the 3D world. It can take the value OffAxis (default), Circular or Linear – you can get pretty good overviews of this on the web. ViewDistance specifies the nominal viewing distance of the autostereo display. GPAC will use that to compute each view’s camera position in the virtual world. I have started some experiments on automatic calibration of 3D models, but it is not yet fully tested and may results in too high disparity. Don’t worry, this can be adjusted at run-time: in MP4Client, use Alt+shift+UP/Down to modify the inter-occular distance (by default 6.8 cm)  and Alt+shift+UP/Down to modify the view distance.

 

Here are some videos showing some of these modes.

[youtube]http://www.youtube.com/watch?v=_8sZc3dL9ds[/youtube]

Original scene designed during the Triscope project, in which we built an auto-stereoscopic mobile prototype.

[youtube]http://www.youtube.com/watch?v=7bNHkuFBg0c[/youtube]

Anaglyph version of the Triscope demo.

[youtube]http://www.youtube.com/watch?v=6kzK54XiRXw[/youtube]

Side-by-Side version of the Triscope Demo.

Triscope Demo 5 views interleaved

Custom interleaving version of the Triscope Demo for a 5 views display.

The demo is written in BIFS, and uses a hardcoded proto (e.g. gpac-specific) node called DepthGroup for the 2D menu. This node allows to specify depth translation and scaling factors to a subtree, which are then cumulated when going down the subtree. We could also have used Transform3D nodes, but we didn’t want to have too many 3D matrices to handle (the original prototype platform was not that powerfull). To declare such a node in a BT file, you need to insert the following code before your scene:

EXTERNPROTO DepthGroup [
exposedField MFNode children []
exposedField SFInt32 _3d_type 4
exposedField SFFloat depth_gain 1.0
exposedField SFFloat depth_offset 0.0
 ] "urn:inet:gpac:builtin:DepthGroup"

You can ignore the _3d_type field, which was specific to the hardware part of the prototype.

The resulting depth value is scaled by a constant defined in GPAC configuration file:

[Compositor]
DepthScale=100

This is still very preliminary work, automatic scaling is planed in the near future.

Support for SVG and depth has also been investigated, but this work is not yet integrated in GPAC.

For images, we defined a new image format called PNGD. These are regular PNG files where the alpha channel is interpreted as the depth map of the RGB channels. These files can be played by GPAC whether in stereo-mode or not.

For video, we defined a new (not standard yet) MPEG-4 descriptor defining a video stream as carrying the depth map of another video stream. This is in-sync with approaches based on MPEG-C Part 3. You can find a sample BT file showing this here, then use MP4Box to produce the mp4 file.

The playback of video+depth or image+depth is in its early stage in GPAC, you will need a BIG machine and a recent graphics card to get good results. Don’t expect too much from high resolution video yet, they will likely overkill the player. The reason for this ? The video data is sent as as a point sprite, triangle strip elevation grid or vertex buffer, and this costs a lot (this will have to be improved in the future).

The way depth video is rendered is defined by the following option:

[Compositor]
DepthType=None

DepthType take the following value:

  • None: video is rendered as a flat rectangle
  • Points: video is rendered as a point sprite if GL_ARB_point_parameters extension is supported
  • Strips: video is rendered as a triangle strip. You may specify “Strips=V” as an option; this will perform two-pass rendering and V the cut-off value for the depth (expressed between 0 and 255)
  • VertexArray: video is rendered as a full vertex array, potentially cached if VBOs are supported.

The height of each vertex is taken from the depth map and multiplied by the DepthScale value.

Note that you don’t need stereo-rendering to play with this:

[youtube]http://www.youtube.com/watch?v=-AuaU1Eq5Xg[/youtube]

When playing depth+video or depth+image, GPAC will automatically switch to OpenGL rendering if the DepthType option is not None.

Thanks for reading, and, happy 3D’ing !!!

 

Compiling GPAC for MacOS X

⚠ The content of this page may be outdated.
Please refer to this page for more up to date information: https://github.com/gpac/gpac/wiki/GPAC-Build-Guide-for-OSX

 


This page describes how to setup a complete build environment for GPAC using Macports (update: Brew command-line below tested on Mac OS 10.11) in order to generate DMG installer images for both 10.5 and 10.6 versions of MacOS X.

Getting GPAC source code

Using a git client, checkout GPAC from the repository:

git clone https://github.com/gpac/gpac.git

Configuring macports

(Re-)Installing

If macports is installed, uninstall it:

sudo port -f uninstall installed
sudo rm -rf /opt/local /Applications/DarwinPorts /Applications/MacPorts /Library/LaunchDaemons/org.macports.* /Library/Receipts/DarwinPorts*.pkg /Library/Receipts/MacPorts*.pkg Library/StartupItems/DarwinPortsStartup /Library/Tcl/darwinports1.0 /Library/Tcl/macports1.0 ~/.macports

Install macport. You MUST install a version >=1.9.x

Custom flags for OSX 10.5 and 10.6 compatibility

If you plan to generate GPAC binaries compatible with 10.5, add the following lines at the end of /opt/local/etc/macports/macports.conf:

macosx_deployment_target 10.4
sdkroot /Developer/SDKs/MacOSX10.5.sdk

This allows compiling a 10.5 binary with a 10.4 compatible dyld (dyn. loader). Furthermore, you will need to install the 10.5 development SDK. Such binaries should work on any 32 bits x86 versions of OS X.

Custom flags for OSX 10.6 32 bits only

If you are on a 10.6 platform with default targetting to x86_64 (ie x86_64 compatible cpu, whatever is your kernel configuration), you still may want to generate i386 binaries. Uncomment the build_arch i386 line of /opt/local/etc/macports/macports.conf. Also comment the universal_archs feature as MacPorts may end up in an unpredictable state and you won’t be able to compile GPAC with most features:

build_arch i386
# CPU architectures to use for Universal Binaries (+universal variant)
#universal_archs i386

Installing GPAC extra libs

MacPorts packages

To install a package, type:

sudo port install my_package_name

The command-line to install all the packages (compulsory and optional) can be found below in this chapter.
The zlib package is required to build GPAC.

To install all required packages, type:

sudo port install pkgconfig freetype libpng jpeg spidermonkey185 libsdl-devel ffmpeg faad2 libmad xvid libogg libvorbis libtheora a52dec openjpeg

WARNING: it is possible that some ports are not working for your system. In this case, you should get  the latest version of the source code of the package, and recompile it locally.

If you have configured macports to generate binaries compatible with 10.5, you must edit /opt/local/bin/sdl-config and replace the line below --cflags with the following:

echo -I${prefix}/include/SDL -D_THREAD_SAFE -arch i386

If you want to cross-compile from a 64 bits environment to a 32 bits target, you may encounter this bug with the ffmpeg package (old versions), or this bug with newer ffmpeg versions.

Brew packages

Alternately to Ports, you may want to use Brew:

brew install jpeg libpng faad2 sdl pkgconfig freetype libvorbis theora openjpeg libmad xvid libogg spidermonkey ffmpeg

Setting up UPnP (Optional)

Install Java developer package if your OSX version is > 10.6.3.

Install the Scons build system:

sudo port install scons

Get the platinum source code patched for gpac here: gpac.sourceforge.net/downloads/platinum_sdk_0.4.5.zip, and extract it.
Edit file Platinum/Platinum/Build/Targets/universal-apple-macosx/Config.scons as follows:

If building for 10.6 64bits, keep the following flags:

universal_flags = [('-arch', 'x86_64'), ('-isysroot', '/Developer/SDKs/MacOSX10.6.sdk'), '-mmacosx-version-min=10.6']

Otherwise, set the flags to:

universal_flags = [('-arch', 'i386'), ('-isysroot', '/Developer/SDKs/MacOSX10.5.sdk'), '-mmacosx-version-min=10.5']

Depending on your GCC version, you may need to edit the file Platinum/Platinum/Build/Boot.scons and replace the line

BoolVariable?('stop_on_warning', 'Stop the build on warnings', True),

with the following:

BoolVariable?('stop_on_warning', 'Stop the build on warnings', False),

then go to Platinum/Platinum and type:

scons
cp Build/Targets/universal-apple-macosx/Debug/*.a gpac/extra_lib/lib/gcc

The Platinum version used in GPAC is 0-4-5 patched for compilation under various platforms, and is unstable on Mac OS X. It may cause GPAC to crash from time to time, and will likely hang out the player upon exit. By default, UPnP module will be disabled on OS X when starting the client for the first time. You can enable it by setting the following key:

[UPnP]
Enabled=yes

or at prompt with MP4Client:
MP4Client -opt UPnP:Enabled=yes

Setting up OpenSVCDecoder (Optional)

Get the latest GPAC extra libs package.
Unzip and go to ./opensvcdecoder
Build using
cmake .
then copy the library to /opt/local/lib (or any place included in your link settings):
sudo cp ./CommonFiles/src/libOpenSVCDec.dylib /opt/local/lib

The OpenSVCDecoder in gpac_extra_libs package is version 1.11 (latest) patched for MSVC and OS X compilation.

Building gpac

Configure gpac

If you plan to generate GPAC binaries compatible with 10.5, type the following configure:

./configure --extra-cflags="-arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk" --extra-ldflags="-mmacosx-version-min=10.4 -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk"

This allows compiling a 10.5 binary with a 10.4 compatible dyld (dynamic loader).

If you don’t plan to generate GPAC binaries compatible with 10.5, type the following configure:

./configure

Build GPAC by typing

make

If you have setup UPnP, type

make -C modules/platinum

If you have setup OpenSVC decoder, type

make -C modules/opensvc_dec

Installing gpac

If you want to generate a DMG package for GPAC, regardless of 10.5 compatibility, type

./mkdmg

or

make dmg

This will produce a file called GPAC-$VERSION-r$REVISION.dmg

If you want to install GPAC directly on your system, type

sudo make install

NOTE: DO NOT RUN ./mkdmg AFTER make install, THE BINARIES WON’T BE USEABLE THROUGH THE DMG ARCHIVE. INSTEAD USE make dmg

Now you deserve a coffee, enjoy it 🙂

GPAC and Mac OS X

As some of you may have noticed on the SVN, we have been working on a Max OS X installer for GPAC. Only OS X 10.5 and 10.6 are supported, and this is still experimental. Snapshot installers are available here, and we warmly welcome any feedback on them.

Note that the installer also contains MP4Box under Osmo4.app/Contents/MacOS

Enjoy!

GPAC @ 95th MPEG Meeting

Some GPAC Developers (Jean and Cyril) are attending the 95th MPEG Meeting in Daegu, Korea (24/01-28/01). New GPAC features will be demonstrated, especially preliminary DASH support. Come and meet them!