jata development issues

Issues of general interest that do not reveal the secret sauce recipe of jata.

Wifi hotspots, a bad keyboard, Android SDK Build-tools problems.

12/19/2013  I need to do some mobile troubleshooting.  The local bus company WRTA has a new hub building and they have public wifi there.  Jata has built in handling for a wifi redirect.  A redirect occurs when an app tries to access an internet site after it has joined a wifi hotspot network, and your app requests are redirected to a Captive Portal.  If you access the internet with a browser, it’s obvious what to do, but when jata is re-directed, code detects this and alerts you, and allows you to adjust wifi settings, or simply sign on to the portal.   I have tested this at a variety of hotspots and thought the logic was good, but at the WRTA wifi, the redirect is not detected and the html of the agreement page is passed to an XMLPullParser object which expects XML input, and it the code reports an error about bad XML, but does not alert you about a wifi redirect.

Ok, so that’s why I need to take a laptop on the road.  And I formerly took a nice small gateway netbook on the road when needed.  It usually sits on a shelf running as a LAMP server.   And, after the last road trip, it now has a bad keyboard with the spacebar and other keys not working.  Normally I’d have fixed that by now, but finances are tight, and I am planning to use an Acer Travelmate for the mobile troubleshooting, so I updated it’s Android Development tools to V19 and imported the jata project from the desktop.  But Eclipse reported java.nio.BufferOverflowException when I tried to build it.  I thought it might be due to a mistake in the import process (which it was, but even after correcting that, Eclipse told me there was an error building the app).  I finally traced it down to an issue where Android SDK Build-tools V19 has a bug.   Stackoverflow was the first hint, and the Android Open Source Project – Issue Tracker confirms it.

It turns out that my normal dev machine was not yet up to date with Build-tools V19, so this is my first encounter with this.   Moving the project to the laptop compounded the problem.  The bad keyboard on the netbook compounded it even more.

UPDATE I went to the WRTA hub with the laptop, plugged in the phone to debug the wifi issue, and the WRTA Free Wifi simply would not let my phone (or the laptop) join the network.  Let me qualify that by adding “while inside”.  Outdoors, before I went inside, I checked and it seemed that the phone did connect to the portal, and the same after I left.  But it was a little cold to be setting up the laptop outside.  I wonder if the portal is set up to not connect indoors in order to discourage people that might stay there for hours with free wifi inside out of the weather.

Application not installed.

ApplicationNotInstalled

Android is a bit wonky about the Install location.
It turns out (correct me if I’m wrong) that if you install jata on a device the Package Manager (on V2.2.2) does not ask where you want to install it.  Assuming it installed to the device, then a user does “Move to SD Card” in the Settings dialog, and later tries re-installing a new version will fail.  The Android won’t say why, it will just say “Application not installed”.

A little digging and adb told me:

pkg: /data/local/tmp/jata.apk
Failure [INSTALL_FAILED_CONTAINER_ERROR]

Google mumbles about “The unique container in which your application is stored is encrypted with a randomly generated key that can be decrypted only by the device that originally installed it. Thus, an application installed on an SD card works for only one device.”, but it is not clear whether that applies to an application moved to SD Card, rather than installed there.

A solution is to use Android Settings to temporarily move it back to the phone, do the install, then use Android Settings to again move it to SD card.  Another is to use jata’s Settings Export to save cached data and favorites, un-install jata, install the new version, then use jata’s Settings Restore to re-import cached data and favorites.

Why is this an issue?  Well, the database maintained by jata can grow pretty large, for instance if a user actually retrieved all the 11,753 stops for all of the 140 CTA routes.  I originally included an option to move data to SD Card, but there were some complications, and I thought that simply moving the app and it’s data was easier.   Well easier for me, but not for a user.

Descriptive route names and gtfs files

Jata started out using the  Clever Devices BusTime API.  This is not as complete as I would like it to be, and I considered using the General Transit Feed Specification, gtfs to supplement BusTime info.  One supplemental content item lacking is descriptive route names.

OK, so the descriptive route names might be in the gtfs files.  Simply have jata download the gtfs, pull out the descriptive names, and shove them into the routes table in the SQLite database.  What’s the problem?

No problem.  It can be done.  But on a phone, with sketchy network connections, downloading the Chicago Transit Authority’s 62.8 MB file might be a challenge.  When I downloaded it on my 1 Mbps DSL connection, it took 10 minutes (after one  “network error” failed attempt).  I would rather keep the phone connection time to a minimum.

I can only think of one way to ease the load on the phone, and that is to pass the job off to an intermediate server.

But, even with that, we may or may not get the info we want.  We need to standardize the gtfs implementation.  The field I want in routes.txt within the gtfs file is “route_desc“.  Here is how that is handled by jata’s current three agencies:

  • CTA – route_desc not included
  • PSTA – route_desc included but empty
  • WRTA – route_desc included and has data

So, three agencies yields three implementations of gtfs, and only WRTA has what we want.  It would be a pretty big disappointment to wait for the 63 MB CTA gtfs file download and to get no useful (for this purpose) information.

And there are tools to validate gtfs files.  Here are some result summaries from FeedValidator:

CTA

Agencies: Chicago Transit Authority
Routes: 140
 Stops: 11753
 Trips: 90924
 Shapes: 1643
 Effective: June 06, 2013 to August 31, 2013
During the upcoming service dates Tue Jun 18 to Fri Aug 16:
 Average trips per date: 19898
 Most trips on a date: 22418, on 1 service date (Wed Jun 19)
 Least trips on a date: 13389, on 8 service dates (Sun Jun 23, Sun Jun 30, Sun Jul 07, ...)
Found these problems:
 982 warnings
 459 Invalid Values
 26 Overlapping Trips In Same Blocks
 4 Stop Too Far From Parent Stations
 156 Stops Too Closes
 333 Too Fast Travels
 1 Unknown File
 3 Unrecognized Columns

Note that CTA took quite a while (15-20 minutes) to process on a HP P6000 with 4GB mem, a AMD Athlon II X4 3 GHz running Win7 64 bit.  I don’t think you want the equivalent work done on your phone.

PSTA

Agencies: PSTA
 Routes: 87
 Stops: 5122
 Trips: 8262
 Shapes: 335
 Effective: February 10, 2013 to October 05, 2013
During the upcoming service dates Tue Jun 18 to Fri Aug 16:
 Average trips per date: 1633
 Most trips on a date: 1896, on 9 service dates (Fri Jun 21, Fri Jun 28, Fri Jul 05, ...)
 Least trips on a date: 858, on 9 service dates (Sun Jun 23, Sun Jun 30, Thu Jul 04, ...)
Found these problems:
 704 warnings
 43 Invalid Values
 87 Stops Too Closes
 574 Too Many Consecutive Stop Times With Same Times

WRTA

Agencies: Worcester Regional Transit Authority
 Routes: 26
 Stops: 81
 Trips: 229
 Shapes: 51
 Effective: May 20, 2012 to December 31, 2013
During the upcoming service dates Tue Jun 18 to Fri Aug 16:
 Average trips per date: 688
 Most trips on a date: 857, on 43 service dates (Tue Jun 18, Wed Jun 19, Thu Jun 20, ...)
 Least trips on a date: 0, on 1 service date (Thu Jul 04)
feed validated successfully
Generated by FeedValidator version 1.2.12 on June 18, 2013 at 01:50 PM Eastern Daylight Time.

Also note that the number of stops in the gtfs file can be drastically different from what the agency and BusTime will actually report.  I compared jata’s internal database against the 81 stops claimed in the gtfs file, and it is actually 1,358 stops.  On the other hand, CTA’s gtfs reports 11,753 stops which is more in line with a test I did with jata attempting to download all the stops for CTA.  I thought it was more like 14,000, but who’s counting?