3D Rendering Pipeline for GeoTools

A Google Summer of Code 2007 project.

Abstract

The 3D Renderer provides a three dimensional view of GeoTools geographical data.

It will uses the normal 2D renderer for rendering the surface texture.

It implements a level of detail based loading and caching system for the geographical data to speed up rendering, and allow perspective views showing both nearby and far away features at the same time.

Possible future improvements are rendering elevation data based on height coverage data. In addition, there could be support for some common 3D rendering styles that can be used for features consisting of points, lines, and areas.

Overview

I created a powerpoint presentation for handouts and similar for the FOSS4G conference. It gives a short overview of the 3D Map Renderer, it's current status, and future plans.

Contact Info

Student: Hans Häggström
Handle: zzorn
Email: zzorn at iki.fi
Mentor 1: Jody Garnett
Mentor 2: Jesse Eichar
Daily IRC: 8pm Helsinki / 10 am Victoria

3D Rendering Pipeline Status

3D Rendering Pipeline Status

2007-09-28 Tiled Map Renderer

The FOSS4G conference has been great so far, and has given me a lot of ideas for how to continue the 3D renderer, as well as an idea of what other open source solutions exists within the same area. I'll get back to those in more depth after the conference.

For now, on the right is a screenshot of a tiled map, using multi-detail level satellite image tiles. The tiles are generated by GDAL2Tiles, the project of Petr Klokan, another of the summer of code programmers. The implementation is a relatively quick hack for now, just to demonstrate that loading tiles from disk is also a possible way to create the ground raster.

2007-09-21 Going to FOSS4G

I'm going to be present on the FOSS4G GIS conference in Victoria, Canada, to demonstrate the 3D Map Renderer. I'll be in the Google booth, along with other summer of code students.

I created a short powerpoint presentation of the 3D Map Renderer, for handouts and such.

The performance of the 3D renderer is really not that good with large maps, as the used 2D map renderer is not good at efficiently rendering only a small part of a map at a time. Maybe I'll take a stab at implementing a custom random map generator on the plane, just to demonstrate the ideal speed of the renderer ...

Another problem with the current approach of rendering a textured version of the 2D map on the ground plane of the 3D map is that it requires a lot of memory. It also doesn't give a very realistic end result.

I'm planning to experiment with using pre-defined tiled, terrain type specific textures (e.g. different ones for woods, suburbs, cities, fields, etc). Roads, borders, lakes, and other similar features would then have to be presented as 3D objects, as they can no longer be rendered on the ground. This approach could look better and would be faster, at least as long as the 3D objects don't get too intricate.

2007-08-25 It's ready!

Today I finally fixed the last major bug, and switched to a larger dataset. It's now ready for review.

In the end, some major features like elevation are left to do, as well as some of the more fancy planned features like 3D markers. Also, the resolution is not that high in the current map, due to memory constraints - increasing the amount of memory for the JVM will allow using higher resolutions though. The rendering order of tiles could be optimized, to give it a more responsive feel (now it starts off by rendering all the large tiles that are invisible, so the user sees no change for a while after starting up). Also, it seems the size of the map affects the rendering performance of the 2D StreamingRenderer used, even when a tile doesn't contain any actual features. This will make using the 3D renderer for large maps unpractical, until the StreamingRenderer is optimized.

But all in all, it works, it's a relatively solid foundation for future work, and it was a challenging and fun summer project to do. Thanks to Jody for mentoring me, GeoTools for accepting my application, and Google for the Summer of Code program!

2007-08-18 Bugs killed, now to apply some polish

I had some hard-to-track-down bugs involving the terrain blocks being layered on top of each other, and the textures not being updated when terrain blocks were re-used. Now they are both fixed. My laptop also shows a pretty consistent 90 FPS when moving around the terrain (I capped the max rendering speed to that IIRC), so it looks like I don't have to be concerned about performance issues.

Recently I also implemented rendering of the map texture in a separate thread, so it doesn't slow down rendering.

Next I'll have terrain blocks that still are rendering their texture re-use a part of their parent texture instead.
Another outstanding thing that needs to be fixed is to enforce that no quad tree node can differ more than one subdivision level from a neighbour. If I have time after that tomorrow I might take a stab at implementing elevation support (it will be retrieved in basically the same way as the textures, so I have the pattern designed already).

For ongoing project status see this page.

Build Instructions

NOTE: These build instructions assume that you have the 2.4-SNAPHSOT version of GeoTools installed - the build file needs to be updated to work with the latest stable release instead. I will do that as soon as I have time, but it may take until after FOSS4G (so start of October 2007).

Check out the code
Install third party libraries in the local repository
- On Windows:
- On Linux:
Build using Maven
You can run the example with:

Screenshots

A tiled map using multi-detail level tiles (thanks Petr!) that I hacked together in two days during the FOSS4G 2007 conference.

Screenshots from the final version at the end of the summer. The left picture shows a normal view, and the right picture shows how a low resolution picture is used as a preview while a tile is being rendered.

The left picture shows the quad tree structure of the terrain, and the right one shows the first hand generated test map.

Work Plan / Task List

Contributor setup (mailing lists, access rights, etc)
Spikes
- 2D Spike - Loading and rendering a shapefile
- 3D Spike - Show 3D map in a swing window
Test Data
- Getting Test Data (I need a shapefile and a matching SLD file so that I can render a map with StreamingRenderer) - GeoServer has some shapefiles with correct SLD files, use them
Navigation on the 3D map
- Pan with left mouse button drag use side arrow keys and pgup & pgdown instead, or maybe middle button drag (will move camera along the left/right, up/down axes of the camera). (A single left mouse button click will select / click on an item on map). The drag has some inertia, so a quick drag will move the camera faster than a slow one. The camera is kept above the ground level at all times though.
- Move across map with left mouse button drag. The camera moves along the ground plane (at a constant altitude?), relative to the direction the camera is facing. The drag has some inertia, so a quick drag will move the camera faster than a slow one.
- Turn camera using right mouse button drag (change yaw and pitch, pitch locked to avoid rolling to back, there could be a maximum angle to avoid getting lost looking at the sky) (cursor disappears while mouse pressed, relative movement is measured. Similar to first person view games). (A right mouse button click without drag will typically show a context menu for the feature under the mouse).
- Move camera along forward axis with W,D or up/down arrow keys or scroll wheel (acceleration is enabled (and a bit of inertia too for effect)). With scroll wheel, a few scrolls gives a small thrust, many scrolls leaves the thrust on. Scroll back to turn the thrust off again (and to thrust backwards).
- Use middle click to reset view to standard view that the camera currently is closest to - N,S,W,E or straight down, with up on screen being north (maybe also NW,NE,SW, and SE directions, and 45 degree views). Also stops movement.
Build setup
- Include the required dependencies (JME & LWJGL) and provide an install script for them
- Change the maven project file to a Java 1.5 one
3D Terrain
- Quadtree Code
- Generate 3D Mesh for a block of terrain
- Make 3D Mesh have a down turned 'skirt' around its edges to avoid 1 pixel cracks in the terrain caused by rounding errors.
- Assign a texture to a terrain mesh from a BufferedImage
- Define low level interfaces for how the renderer gets height and texture information, and implement them.
- Render 2D Map to an Image and from there to a terrain mesh
- Map Block Management - subdivide and merge as camera moves
- Fix growing of the quad tree so that the map blocks always extend to the horizon
- Set up textures to use smooth rendering, to avoid the pixelated effect visible in the second screenshot.
- Implement pooling of BufferedImages used for generating the ground textures
- Implement pooling of TerrainBlocks.
- Use a low resolution preview taken from a parent block for textures that are waiting to be rendered.
- Fix y-flipping bug (maybe turn around whole terrain?)
- Split map blocks so that neighboring ones differ at most by one subdivision
- Reposition the 3D scene to origo when it gets too far away - floats that the geometry is represented in have accuracy problems with too large values.
Investigate Issues
- Check if there is a memory leak in reserving texture memory on the 3D card, and how to free it when it is no longer used.
- Investigate projections - what coordinate system to use in the 3D view, and how to translate coordinates
- Find out if there is a way to force Swing menus and tooltips to use heavyweight awt rendering, so that they are visible on top of the 3D canvas.
Elevation Support elevation data, use coverages or other data sources to get it.
3D Markers (3D models on the map) and Labels
Point and click selection of a marker or point on the ground
Fancy features - buildings, forest, 3D lines, globe view

Legend

= Done = Not started = In progress = Waiting for help

Dependencies

The 3D rendering pipeline requires the Java Monkey Engine (JME) version 0.10, which in turn requires LWJGL version 0.99.

The java.library.path needs to point to the LWJGL native code libraries for the correct platform. It can be set on the java command line like this: java -Djava.library.path=C:/Libs/LWJGL/Win32 foo.class

The mvn exec:exec goal does this when running the example program.

Usage Instructions

Using the 3D Renderer in your own projects is easy. Just make sure you have the necessary dependencies available (and remember to set the java library path to point to the LWJGL native lib directory), then just:

Create a map context that should be rendered:

Create a 3D renderer:

Get a 3D view UI component from the renderer:

The component is a normal AWT UI component, and can be added to any Swing or AWT UI:

See Show3DMapExample.java for a working example program.

Issues

Lack of multithreaded UI support

The Swing support in JME version 0.10 does not manage multithreaded UI:s very well - JME 0.11 has better support for multithreading, but requires Java version 1.5.

For now I decided to just use the Java 1.4 compatible version. This may limit the kind of UI:s that can be built around it, but basic Swing widgets seem to work. If we later start to use Java 1.5 in GeoTools it should be easy to port the code to the newer version of JME.

Java 1.5 modules seem to be acceptable, so I'll be using Java 1.5 and the latest version of JME.

Lightweight Swing components do not render on top of the 3D Canvas.

The 3D canvas is an AWT component, and is updated by the 3D card/library, so it is not possible to draw lightweight swing components on top of it. This causes problems especially with menus, context menus, and tooltips. Maybe there is some way to force them to be heavy weight (=own windows at the window system level)? I seem to recall that something like that was possible to do.

A workaround is to use AWT when implementing the UI, it should work (to be checked). But that is a major disadvantage.

UPDATE: It's possible to use JPopupMenu.setDefaultLightWeightPopupEnabled( false ); to disable lightweight rendering for popups, this should help make them overlap the 3D window (thanks Eclesia!).

Design Notes

Data Model

The map could be based around a multi-resolution quad tree - each leaf in the quad tree represents a square area of some size - a MapBlock.
The quad tree is subdivided into smaller grids near the camera, to provide higher resolution for nearby terrain.

Each MapBlock may have a texture associated with it, to be used for texturing the terrain. It may also have an array of elevation data, and a set of geometrical features.

Interfaces

The interface towards data sources should be such that features can be queried for (rectangular) areas using a significance threshold (where the significance is calculated from feature size and/or how important it is to show - small roads are not important in a large overview of a country, while individual Google SoC developer locations should be visible even in a global view when showing the developer distribution).

3D styles may be implemented by extending the current style system in some way if possible, or otherwise by specifying datasources and the 3D style to use for them separately.

In addition, the 3D renderer provides an interface with methods to get a Swing component containing the 3D view, methods for moving the camera, and constructor parameters / hints for specifying the resolutions to use for the textures and elevation density on MapBlocks.

Rendering

Features will normally be rendered directly to the ground texture, but optionally 3D styles can be used for features, creating 3D geometry for them instead. Examples of 3D styles could be abstract shapes used for markers (pyramids, spheres), slightly elevated roads, buildings, and imported custom 3D landmarks (Eiffel tower in a map of Paris, etc).

Ground Rendering

The ground textures will be rendered using the normal 2D rendering pipeline (although the 3D renderer could offer an extension point for custom texture renderers).

The 2D ground rendering should take place in a background thread. While it is ongoing, a rougher version can be displayed by using previously rendered ground textures from a higher quadtree level, or a placeholder image can be used.

Soft transitions in the texture may be needed at the edges between MapBlocks of different resolutions. This could be done in the graphics card, using shaders.

Label Rendering

Different features may have labels that should be shown. They are best rendered to a texture, and shown as a billboard (3D polygon always turned towards the camera)

Marker Rendering

Markers can be either 2D or 3D.

2D markers are simply icons / pictures rendered to a texture and shown on a billboard, like labels.

3D markers use some built in shape (sphere, pyramid, cube) in a solid color or textured with some texture. Alternatively they can use a custom 3D model (e.g. Eiffel tower, map pin, etc ).

Building Rendering (optional)

Buildings or city blocks may be represented by arbitrary polygons on some maps. They can be rendered as 3D by simply extending the polygon upwards, giving it a height. It might also be textured, and depending on the style of building, the roof could be sloped. The style how to render buildings would probably be specified as some kind of metadata for a larger area (e.g. downtown has taller buildings with flat roofs, while suburbs have lower buildings with slanted roofs).

Initially just a simple fixed height, flat roof, untextured building style could be implemented.

See Instant Architecture (pdf) for one approach on building texture generation.

See Automatically Generating Roof Models from Building Footprints for a paper on generating building roofs.

There is also a lot of research on automatically creating 3D buildings from satellite photos (identifying building shape and height from the shadows that they cast), but that is out of scope for this project.

Forest Rendering (optional)

Optionally, forest and vegetation could be rendered using multiple textured layers for each MapBlock, as described in Forest Scenes in Real-Time . The extent and type of a forest could be specified by polygonal features or coverage (which one is normally used?).

Globe View (optional)

It would be nice to be able to support a globe view, where the earth is shown as a sphere, in addition to a detailed low altitude view, and provide seamless transitions between them. However, for now it remains a lower priority.

Optimizations

To optimize rendering, features could be classified by size and/or importance (significance). Only features with a significance over a certain threshold would be included on a MapBlock. The significance threshold would be lower for small, detailed MapBlocks close to the camera, and higher for large MapBlocks far away from the camera.

Optionally the rendering speed of 3D markers, buildings, landmarks, and such can be sped up by rendering them to impostor textures, which are always turned towards the camera, and only updated when the camera moves significantly in relation to them.

Point And Click

Another problem that needs to be solved is picking - determining which feature or map coordinate has been clicked when the user presses a mouse button over the 3D view. At least for 3D features, picking could be solved by using an existing 3D scenegraph, such as JME.

The 3D view should include at least some rudimentary navigation support based on mouse and keyboard input, but allow clients of the library to override those.

Used Libraries

I'm leaning towards using the Java Monkey Engine (JME). It is a scenegraph library built on top of LWJGL, and is actively developed. It would provide a number of useful features, such as visibility culling, picking, impostor support, and Swing integration (it's terrain support is not very advanced though, but that is what I'm working on here).

Hardware Requirements

Shaders may be utilized, although the 3D renderer should also work without them.

The 3D renderer might be using a lot of texture memory too, depending on the resolution settings specified by clients.