The roads I take...

As I mentioned previously, the Mixed Reality "virus" has caught me recently and I spend a good portion of my Mozilla contribution time with presenting and writing demos for WebVR/XR nowadays.

The prime driver for writing my first such demo was that I wanted to do something meaningful with A-Frame. Previously, I had only played around with the Hello WebVR example and some small alterations around the basic elements seen in that one, which is also pretty much what I taught to others in the WebVR workshops I held in Vienna last year. Now, it was time to go beyond that, and as I had recently bought a HTC Vive, I wanted something where the controllers could be used - but still something that would fall back nicely and be usable in 2D mode on a desktop browser or even mobile screens.

While I was thinking about what I could work on in that area, another long-standing thought crossed my mind: How feasible is it to render OpenStreetMap (OSM) data in 3D using WebVR and A-Frame? I decided to try and find out.



First, I built on my knowledge from Lantea Maps and the fact that I had a tile cache server set up for that, and created a layer of a certain set of tiles on the ground to for the base. That brought me to a number of issue to think about and make decisions on: First, should I respect the curvature of the earth, possibly put the tiles and the viewer on a certain place on a virtual globe? Should I respect the terrain, especially the elevation of different points on the map? Also, as the VR scene relates to real-world sizes of objects, how large is a map tile actually in reality? After a lot of thinking, I decided that this would be a simple demo so I would assume the earth is flat - both in terms of curvature or "the globe" and terrain, and the viewer would start off at coordinates 0/0/0 with x and z coordinates being horizontal and y the vertical component, as usual in A-Frame scenes. For the tile size, I found that with OpenStreetMap using Mercator projection, the tiles always stayed squares, with different sizes based on the latitude (and zoom level, but I always use the same high zoom there). In this respect, I still had to take account of the real world being a globe.

Once I had those tiles rendering on the ground, I could think about navigation and I added teleport controls, later also movement controls to fly through the scene. With W/A/S/D keys on the desktop (and later the fly controls), it was possible to "fly" underneath the ground, which was awkward, so I wrote a very simple "position-limit" A-Frame control later on, which prohibits that and also is a very nice example for how to build a component, because it's short and easy to understand.

All this isn't using OSM data per se, but just the pre-rendered tiles, so it was time to go one step further and dig into the Overpass API, which allows to query and retrieve raw geo data from OSM. With Overpass Turbo I could try out and adjust the queries I wanted to use ad then move those into my code. I decided the first exercise would be to get something that is a point on the map, a single "node" in OSM speak, and when looking at rendered maps, I found that trees seemed to fit that requirement very well. An Overpass query for "node[natural=tree]" later and some massaging the result into a format that JavaScript can nicely work with, I was able to place three-dimensional A-Frame entities in the places where the tiles had the symbols for trees! I started with simple brown cylinders for the trunks, then placed a sphere on top of them as the crown, later got fancy by evaluating various "tags" in the data to render accurate height, crown diameter, trunk circumference and even a different base model for needle-leaved trees, using a cone for the crown.

But to make the demo really look like a map, it of course needed buildings to be rendered as well. Those are more complex, as even the simpler buildings are "ways" with a variable amount of "nodes", and the more complex ones have holes in their base shape and therefore require a compound (or "relation" in OSM speak) of multiple "ways", for the outer shape and the inner holes. And then, the 2D shape given by those properties needs to be extruded to a certain height to form an actual 3D building. After finding the right Overpass query, I realized it would be best to create my own "building" geometry in A-Frame, which would get the inner and outer paths as well as the height as parameters. In the code for that, I used the THREE.js library underlying A-Frame to create a shape (potentially with holes), extrude it to the right height and rotate it to actually stand on the ground. Then I used code similar to what I had for trees to actually create A-Frame entities that had that custom geometry. For the height, I would use the explicit tags in the OSM database, estimate from its levels/floors if given or else fall back to a default. And I would even respect the color of the building if there was a tag specifying it.

With that in place, I had a pretty nice demo that uses data directly from OpenStreetMap to render Virtual Reality scenes that could be viewed in the desktop or mobile browser, or even in a full VR headset!

It's available under the name of "VR Map" at vrmap.kairo.at, and of course the source code can also be expected, copied and forked on GitHub.



Again, this is intended as a demo, not a full-featured product, and e.g. does at this time only render an area of a defined size and does not include any code to load additional scenery as you are moving around. Also, it does not support "building parts", which are the way to specify in OSM that a different pieces of a building have e.g. different heights or colors. It could also be extended to actually render models of the buildings when they exist and are referred in the database (so e.g. the Eiffel Tower would look less weird when going to the Paris preset). There are a lot of things that still can be done to improve on this demo for sure, but as it stands, it's a pretty simple piece of code that shows the power of both A-Frame and the OpenStreetMap data, and that's what I set out to do, after all.

My plan is to take this to multiple meetups and conferences to promote both underlying projects and get people inspired to think about what they can do with those ideas. Please let me know if you know of a good event where I can present this work. The first of those presentations happened a at the ViennaJS May Meetup, see the slides and video.
I'm also in an email conversation with another OSM contributor who is using this demo as a base for some of his work, e.g. on rendering building models in 3D and VR and allowing people to correct their position data.



I hope that this demo spawns more ideas of what people can do with this toolset, and I'll also be looking into more demos that will probably move into different directions.

After my last post on Lantea Maps (my web app to record GPS tracks), I started working on some improvements to its code.

First, I created a new backend for storing GPS tracks on my servers and integrated it into the web app. You need to log in via my own OAuth2 server, and then you can upload tracks fairly seamlessly and nicely.
The UI for uploading is now also fully integrated into the track "drawer" which should make uploading tracks a smoother experience than previously. And as a helpful feature for people who use Lantea Maps on multiple devices, a device name can be configured via the settings "drawer".



The saved tracks are listed in the new library view (also accessible for the track "drawer" when logged in) and linked to a GPX file to download download - that way the recorded and uploaded tracks can be accessed from a different device and downloaded to there. The library UI has a lot of potential for improvement but this first version has been working decently for me for a while now in testing.

In addition, the first piece of new PWA (Progressive Web Apps) technology has been integrated: Due to the W3C Manifest, you can now add Lantea Maps to your home screen from browsers like Firefox for Android.



Even more, I optimized the code drawing the GPS tracks so that off-screen segments aren't drawn, even though I'm unsure how to measure drawing and panning speed, so I can't put actual numbers behind what that work may have helped or not - but I hope it improved performance when large tracks are loaded.

To round up all the work, I added a welcome and an update information screen to be able to tell people both how to initially use the app and what changed on updates.

This is a spare time project so I'm doing updates very irregularly but I'm using the app myself almost daily so it should continue to be maintained in the future as time and motivation allow.

During my holidays, when I was using Lantea Maps daily to record my GPS tracks, I suddenly found out one day that upload of the tracks to OpenStreetMap was broken.

I had added that functionality so that people (including myself) could get their GPS tracks out of their mobile devices and into a place from which they can download them anywhere. A bonus was that the tracks were available to the OpenStreetMap project as guides to improve the maps.

After I had wasted about EUR 50 of data roaming costs to verify that it was not only broken on hotel networks but also my mobile network that usually worked, I tried on a desktop Nightly and used the Firefox devtools to find out the actual error message, which was a CORS issue. I filed a GitHub issue but apparently it was an intentional change and OpenStreetMap doesn't support GPS track uploads any more in a way that is simple for pure web apps and also doesn't want to re-add support for that. Find more details in the GitHub issue.

Because of that, I think that this will mark the end of uploading tracks from Lantea Maps to OpenStreetMap. When I have time, I will probably add a GPS track store on my server instead, where third-party changes can't break stuff while I'm on vacation. If any Lantea Maps user wants their tracks on OpenStreetMap in the future, they'll need to manually upload the tracks themselves.

In the last weeks, I spent some time on developer events, which I always embrace a lot because the "hallway tracks" are a constant great flow of interesting information and discussions - and of course the official talks are often quite interesting as well!



First up was the Mozilla Hacking Day in Berlin on Saturday, April 26. I met Arpad Borsos, fellow Mozillian from Vienna, already at Vienna airport on Friday, as we incidentally were hopping over from the Austrian to the German capital in the same plane. Later in Berlin, we met more Mozillians at the hotel and went out for dinner together.
At the actual Hacking Day, we had ~70 people streaming in for some introductory talks, and then people split up into hacking sessions and some more in-depth talks (like the one in the picture, where fellow "Desktop QA" team member Henrik talked about automated testing), which were followed by even more hands-on hacking. I spent most of the time talking to various people about different topics, showing off Firefox OS a bit (after I had reflashed my Geeksphone Peak and got it into a working stage again during the talks) and my actual "hacking" achievement: Together with Georg Fritzsche, we could find out that an issue with my Lantea Maps app freezing was a WebGL-related bug in Gecko, found a stack, reported it as a new bug and set a needinfo flag to a gfx developer. (Yesterday, I found a hacky workaround so Lantea Maps will soon work again despite that bug.)
On Sunday, I also was able to squeeze in some sightseeing and photo-taking, marching through Berlin for ~3.5h before returning to the airport.



And last week, May 8-10, it was time for the yearly Linuxwochen conference in Vienna. As we do not have an organized group of Mozillians here, we didn't have a Mozilla booth, but I usually share the OpenStreetMap booth (that's why you'll spot that in the pictures), as I'm pretty active in the local community there, and just run around with Mozilla T-shirts and talk about Mozilla stuff in addition to OSM things. I also had submitted talks about Firefox OS and the Makey Makey, but as you'll see in my speaker profile, I was additionally "coerced" into a privacy and security panel discussion last-minute as well. I was put there as more or less a representative of Mozilla and more in general those doing end-user software, and of course also tried to drum home a few points of how Mozilla strongly works for respecting user privacy but also pointed out how it is an interesting field that sometimes has tradeoffs with enabling features that people out there really want, and finding a balance can be hard.
The Firefox OS talk was quite appreciated by people attending, and we had quite a few good questions asked there as well, and interest from people afterwards to actually see and try my FxOS phones - I remember how some hard a hard time believing how well the ZTE Open worked with just 256MB of RAM and an all-web UI.
In the "hallway track", I did get a lot of questions about Mozilla on all three days, given my T-shirts clearly pointed out my affiliation with the project. Some of those were Firefox OS, mostly about strategy and availability, where I had to explain a few times how we target features phone converts in emerging markets for now. A number of the concerns and questions were around Australis, with quite a few of the very technical folks there not liking how we messed up their customizations and preferences, including tops-on-bottom or the add-ons bar, or how esp. our tabs looked "just like Chrome" now, but I even heard one or two comments on how awesome the new design was. Some of those I could ease with explanation and pointing to the Classic Theme Restorer add-on, but some of them are just unhappy (and it's not helpful that the very helpful Tour does not run when NoScript is installed and active). And then there were questions about our finances (the "Google dependency" issue) and about the fact that new Sync clashes with Master Password (which also protects casual "friends" from reading through passwords in your password manager), among others. Overall, a lot of talk about Mozilla, and I hope I could make most of those people feel better about us than before, wherever they stood then.

Both events cost me a lot of sleep and energy right there, but feel like they were worth the investment for sure, and the meeting and talking to people also gives me energy of a different form.

I blogged about Lantea Maps 18 months ago. As its marketplace listing describes, the app's purpose is displaying maps as well as recording and displaying GPS tracks.

I wrote this app both to scratch an itch (I wanted an OpenStreetMap-based app to record GPS tracks) and to learn a bit more of JavaScript and web app development. As maps are a 2D problem and the track display requires drawing various lines and possibly other location indicators, I wrote this app based on 2D canvas. I started off with some base code on drawing bitmap tile maps to canvas, and wrote the app around that, doing quite some rewriting on the little bit of code I started from. I also ended up splitting map and track into separate canvases so I wouldn't need to redraw everything when deleting the track or when moving the indicator of the last location or similar. Over time, I did a lot of improvements in various areas of the app, from the tile cache in IndexedDB via OpenStreetMap upload of tracks to pinch zooming on touch screens.

Still, performance of the map canvas was not good - on phones (esp. the small 320x480 screens like the ZTE Open), where you only have a handful of 256x256 map tiles to draw, panning was slightly chunky, but on larger screens, like my Android tablet or even my pretty fast desktop, it ranged from bad to awful (like, noticeably waiting from any movement until you saw any drawing of a move map). Also, as it takes until images are loaded (cached from IndexedDB or out from the web) and that's all called asynchronously, the positions the images ended up being drawn often weren't completely correct any more at the time of drawing them. I tried some optimizations with actually grepping the pixels from the canvas, setting them in the new positions and only actually redrawing the images on the borders, but that only helped slightly on small screens while making large ones even worse in performance.

Given what I read and heard about how today's graphics chips and pipelines work, I figured that the problem was with the drawImage() calls to draw the tiles to the canvas as well as the getImageData()/putImageData() calls to move the pixels in the optimizations. All those copy image data between JS and graphics memory, which is slow, and doing it a lot doesn't really fit well with how graphics stacks work nowadays. The only way I heard that should improve that a lot would be to switch from 2D canvas to WebGL (or go to the image-based tile maps that many others are using, but that wouldn't be as much fun). I don't remember all sources for that, but just did get another pointer to a Mozilla Hacks post that explains some of it. And as Google also seems to being moving their Maps site to WebGL (from image-based tiles, mind you), it can't be a really wrong move.

So, I set out to try and learn the pieces of WebGL I needed for this app. You'd guess that Mozilla, who invented that API together with Khronos, would have ample docs on it, but the WebGL MDN page does only have one tutorial for an animated 3D cube and a list of external links. I meanwhile have filed a bug on a WebGL reference so may improve this further in the future, but I started off first trying with the tutorial that MDN has. I didn't get a lot to work there except some basics, and a lot of the commands in there were not very well explained, but the html5rocks tutorial helped me to get things into a better shape, and some amount of trying around and the MSDN WebGL reference helped to understand more and get things actually right.
One thing that was pretty useful there as well was separating the determination of what tiles should be visible and loading them into textures from the actual drawing of the textures to the canvas. By doing the drawing itself on requestAnimationFrame and this being the only thing done when we pan as long as I have all tiles loaded into textures, I save work and should improve performance.


2D Canvas (left) and WebGL (right) version of Lantea Maps on the ZTE Open

As you can see from the images, the 2D canvas and WebGL versions of Lantea Maps do not look different - but then, that was not intended, as the map is the map after all. Right now, you can actually test both versions, though: I have not moved the WebGL to production yet, so lantea.kairo.at still uses 2D canvas, while the staging version lantea-dev.kairo.at already is WebGL. You'll notice that panning the map is way more fluid in the new version and the tile distortions that could happen with delayed loading in the old one do not happen. I still wonder though why it sometimes happens that you have to wait very long for tiles to load, esp. after zooming. I still need to figure that out at some point, but things render after waiting, so I found it OK for now. Also, I found the WebGL app to work fine on Firefox desktop (Linux), Firefox for Android, as well as Firefox OS (1.1, 1.2, and 1.5/Nightly).

So, I'm happy I did manage the conversion and learn some WebGL, though there's still a lot to be done. And as always, the code to Lantea Maps is available in my public git as well as GitHub if you want to learn or help.

The OpenStreetMap project has launched an all-in-JavaScript map editor called "iD" this week:



While we at Mozilla know you can do a lot of good things in JS these days - after all, we're even launching our own phone OS building fully on HTML+JS, and we have been using more and more JS code to power key functionality in our browsers and other products over the years - it's great to see that complex things like editing maps can be done fully in JS and available for all platforms now, while previously it took proprietary and availability-limited technologies like Flash or Java to do the same thing.

Great work, OpenStreetMap guys!


(And yes, as a contributor to OpenStreetMap and even OSMF member, I am biased, but free and open map data on the web fits Mozilla philosophy pretty well anyhow...)

I think I probably became aware of the MaKey MaKey when Chris Heilmann blogged about it, but I might have heard about it even before. I surely saw it when Chris had his WebRTC photo booth running with it at MozFest 2012 in London.



When I ran across it again on ThinkGeek, I put it on my wish list - and finally ordered one this month. Now, after I had wrapped up this week of work, I finally found some time to play with it, and an interesting and very geeky Friday night ensued. Here's a bit more about that - and about Saturday, and further plans/ideas.

So, for one thing, I wanted to use this device with actual Open Web stuff, and not with Flash or other proprietary software. After all, this is Open Hardware (yay!) and I'm entirely entrenched in Open Source / Free Software, from using Linux on desktop, laptop and server, via working for Mozilla/Firefox, to doing some web apps under the MPL2 license in my free time. So, given the latter, I decided it would be nice if I could navigate the OSM world with my Lantea Maps app/site (source) using the MaKey Makey. For that, I had to put some keyboard accessibility into Lantea Maps itself, which is a good idea for accessibility, among other things, anyhow. So I did that, looking at Chris' testy-testy and MDN to find out how to achieve that best. I ended up implementing methods to move the map with the arrow keys, hooked up zoom in/out to +/- keys as well as w/s (the latter are supported by MaKey MaKey out of the box), and then also created direct shortcuts to certain zoom levels with the 0-9 numeric keys (not supported by MaKey Makey, but convenient for keyboard users).

OK, then it was time to actually bring in the MaKey MaKey. I really want to do some fruit stuff at some point, but I only had a few apples around, and I thought it actually would be nice to create some kind of navigation pad that can be used with Lantea Maps at full screen when having an OpenStreetMap booth at Linuxwochen in Vienna. I figured that with some cardboard from the back of an old note pad, and some tinfoil, that should be doable. I added some plastic wrap for insulation, glue of course, and some paper clips so the crocodile clips to connect to the MaKey MaKey wouldn't scratch the tinfoil too much (as well as some temporary applied ones to hold things together while allowing the glue to dry). Here's some photos from production:



Note that the back side as well as the right rim of the pad is covered with a single sheet of tinfoil that makes the earth connection quite naturally when you hold the pad in your hands.
As of the last photo, while the glue was still drying, it was ready to use for some map navigation (and after the night, I removed the temporary paper clips and took another "promotional" picture):



Even while getting to bed that evening, the ideas for my next project were flying around in my mind already. On one hand, I saw that MaKey MaKey had connectors for mouse up/down/left/right, on the other hand, ever since trying the original BananaBread demo as someone who's usually not doing any first person shooter games, I wondered if there was a nicer or more obvious way to operate this, rather than using w/a/s/d keys for movement, space/click for jump/fire, and mouse for turning. Well, now that I had done this first custom pad for MaKey MaKey, would there be a handy solution for that as well? In any case, it would be fun. So I took a smaller piece of cardboard that would make this thing fit nicely into my hands (just like those professional game pads), and decided this time I would try something slightly different by using coins as the actual "buttons" on the pad. One-cent coins looked like the right size, and I had a 10-pin cable around from a different project, which would fit for the 10 "keys" pretty well (just that I needed one more for earthing, which I again did with a sheet of tinfoil at the back of the pad, so I added yet another single cable in the end). Also, this time I used some double-sided tape instead of glue for many cases, as that works better with the cable and coins:



And then I was ready to play some BananaBread, now with both the awesomeness of running a 3D first person shooter seamlessly in the browser AND using a special game pad for playing!



If you're interested, not only are those pictures all linked to the gallery where you can go up to "big" versions of those, there's a few more steps of building visible in this photo gallery.

Given all that and the fact that Linuxwochen Wien in the first days of May has an additional focus on Open Hardware this year, I decided to hand in a proposal for a talk on MaKey MaKey there. I intend to show off those pads as well as Chris' photo booth and any other MaKey MaKey experiments that I can fit, preferably ones that run as web pages/apps (let me know if there are any nice ones).

I'm thinking that it could be nice to have an app that shows you on screen in a web site which kind of fruit/item you touched (configurable with key <-> item entries), and I'd love a web (not Flash) piano and/or drumset (using ogg or even opus files with HTML5 audio!) app to present, maybe I can hack something up if there's nothing around.

If this has caught your interest, it's easy to get your own MaKey MaKey, and if you're in or around Vienna in the first days of May, I'd be happy to meet you at my talk (there will be a Firefox OS App workshop as well, probably!) - oh, and if you have any nice, open web apps/pages that show off this device, let me know!

I just split my Lantea Maps web app out of my main git repo and into its own small one, one main reason being that I can place it on GitHub as well. This is not meaning that I suddenly love the GitHub site, but having the repo there lowers the entry barrier for potential contributors and I'd be very happy to have people help me with this app.

Lantea Maps is a prototype web app to display maps and record (GPS) tracks, and I found it works pretty well with Firefox OS, actually. It's one of the apps I keep installing on my test device for that system, of course from its Mozilla Marketplace listing.

The app is based on a canvas that displays a map (from OpenStreetMap, multiple styles can be selected) and on which the recorded GPS tracks are displayed. The start of the app is there and working, but it could need quite some improvement - like making zoom switches and probably map panning smoother by using some canvas magic, supporting pinch gestures, caching tiles not just during a session but also across sessions (using IndexedDB), and more.

As mentioned on the original Lantea idea wiki page, there's a quite nice app for the Nokia N900 called Mappero that is a good source for feature and UI ideas, for me it's one the the best apps in that space and I'd love to have something similar to it as a web app - which is basically why I started Lantea at all.

That said, I'm open for all kinds of ideas for it - and even more, to patches and pull requests!

The "State of the Map - Europe" conference in Vienna was a full success, 200 members of the OpenStreetMap community had inspiring talks and discussions here and everyone had a lot of fun and inspiration. I also couldn't be more proud of acting as a Mozilla representative on the conference and I had a lot of interesting discussions in that role, including people being thrilled that Mozilla sponsors such and event "just" to move openness and innovation on the web forward. But that's not what this post is about, I actually want to highlight one thing I heard about in a talk there.

Martin Christen from Switzerland had a really great talk on the OpenWebGlobe SDK today, and I was completely thrilled to see this as it does a lot of things I had wished for in the days before. When there I dreamed about "something similar in spirit to Google Earth, but completely in the browser", I didn't know yet that the University of Applied Sciences Northwestern Switzerland was already working on the basis for all that and had demos pointing in that direction.



Doesn't that look cool? Remember that this is actually running as a 3D environment in the browser - and directly in HTML, not needing any plugin!
Oh, and all its code is open source software (the web viewer is available via github under MIT license, there's a download for the SDK, I didn't check its license but I think it's MIT as well)!

OpenWebGlobe for WebGL is in an alpha stage, but it's usable under Firefox now (other WebGL-capable browsers might work but haven't been tested enough). Martin said they found that JavaScript is still not as fast as they'd like, esp. when compared to their tests on native C++ code using the same base SDK. Maybe Mozilla developers can help there and make JavaScript even faster than it is now.

In any case, check out the work of his team, it's absolutely awesome. If you don't believe me yet, there's even an alpha preview you can test yourself right now!

And a video of his talk will become available very soon (watch out for a camera icon in the box for his talk on the SotM-EU schedule), if you're interested in what he presented in Vienna.

I've recently enlisted into the Mozilla Reps ("ReMo") program, mainly to procure sponsoring by Mozilla for the 1st European OpenStreetMap conference, known as "State of the Map - Europe" (SotM-EU).



I'll be the sole representative this weekend of Mozilla being one of the major sponsors of this event, and I'm also helping somewhat with organizational matters, as my colleagues from the "OpenStreetMap Austria" association are the organizers of the conference.

It will be an interesting role to represent Mozilla, and I'm very proud of that, as this gives me a chance to talk a lot about the one thing in Mozilla I'm most passionate about: our mission.
Supporting a conference on an open, innovative project that creates opportunities for everyone on the web and beyond goes to the heart of what Mozilla is, and I couldn't be happier about being present there.



Things start off with a public pre-event discussion tomorrow evening and the conference itself is taking place from Friday to Sunday, so don't expect to see me online a lot in the next few days, I'm exploring this strange place called "real life" with other geeks - and probably mapping it out some more.