26 August 2013

Preserving stack trace when catching and re-raising runtime exceptions

(This is a repost of an article of mine on Real Software's, now Xojo Inc's, former blog)

This post offers an introduction to the usefulness of exception handling in general, while also explaining how to deal with a particular shortcoming when you use debugging code and the UnhandledException event handler in built programs you deliver to people. I hope this article will be helpful to both newcomers and veterans alike.

Background (UnhandledException and the Stack Trace)

You probably know the UnhandledException event handler in the App class.

It lets you catch any unhandled exceptions in your program, preventing your program from getting terminated unexpectedly, and allowing you to save any interim data that the user may have not saved yet, and other defensive fallback procedures. 

One other use of this handler is to record the location where the exception occurred in order to use that for later debugging. For instance, if you send your built application to others who do not have the Xojo debugger, your program could still record the so-called stack trace from the exception, show that to the user, and ask him to forward that information to you so you can hopefully deduce what went wrong and how to avoid it in the future. 

You get this stack trace by invoking the Stack() method of the RuntimeException object you get passed in UnhandledException. It returns an array of strings, identifiying the methods that were called up to the one that caused the exception. 

For instance, if your program causes an exception in a Window's Open event, the stack trace might look like this: 

1. RaiseOutOfBoundsException
2. Window1.Window1.Event_Open%%o<Window1.Window1>
3. FireWindowOpenEvents
4. Window.Constructor%%o<Window>
5. Window1.Window1%o<Window1.Window1>%
6. _MakeDefaultView
7. _Startup
8. RunFrameworkInitialization
9. REALbasic._RunFrameworkInitialization%%p
10. _Main
11. % main

The topmost line shows where the exception occured, the bottommost is the topmost caller at that time. 

It tells several things:

  • The exception, obviously an OutOfBounds exception, occured in Window1's Open event (line #2)
  • The Open event was invoked by the Window1's Constructor (lines 4 and 5).
  • The Window1 got constructed right at start of the program (line 7). Otherwise, we'd usually see the method "RuntimeRun" up in the stack trace, indicating that the startup has finished and the program is now processing events.

Raising exceptions 

Whenever an exception occurs, we say that it gets raised. Xojo even offers a custom statement for that: The Raise command. With that, you can make your own exceptions occur. An example: 

  raise new OutOfBoundsException 

is equivalent to: 

  dim anArray(0) as Integer
  anArray(1) = 0 // this will cause an OutOfBoundsException 

(Of course, you can create your very own exception types by subclassing them from RuntimeException. See the Language Reference to learn more about that.) 

Now, here's an imporant part: The stack trace that gets attached to any RuntimeException object is created when this raise statement is invoked. And the stack trace is then created from the very calling stack that's in effect at the time of invocation. Logical, right? 

Catching exceptions 

You can catch exceptions in your methods using the try ... catch ... end and the exception statements.

There are two common cases why and how you'd catch exceptions: 

1. Handling the exception 

You expect a particular exception to occur and are prepared to handle it. This is, for instance, the case when you use file I/O operations and which you cannot otherwise predict to avoid them, such as that a file cannot be created, causing an IOException. 

2. Cleanup 

You do not know that any particular exceptions might occur in some subroutines you call but you like to be prepared and do some cleanup if that should happen. In this case, you'll pass on the exception to the caller because you don't know have reason way to handle the exception at that particular point. 

Such cleanup handling code usually looks like this: 

   ... // your operations
 catch exc as RuntimeException
   db.Rollback // e.g. abort a database transaction
   raise exc // pass the exception on

The complication (catching and re-raising) 

Let's assume you have a UnhandledException event handler in the App class, which reads the Stack array and for later analysis (e.g. by writing to a file, creating an email to you or even uploading it to your web server). 

So, if any part of your program raised an exception for which you don't have particular handling code, it'll end up in App.UnhandledException, which will eventually allow you to learn where the crash ocurred. That's the general idea, at least. 


Consider what happens if an exception occurs inside a subroutine that's called by code using the Cleanup exception handler? 

Remember how the stack trace gets created: During the invocation of the raise statement. 

This means that if an exception occurs deeper down, then gets caught by the Cleanup handler code, which then re-raises the exception to end up finally in the App.UnhandledException handler, you'll get to see the wrong Stack trace. You won't see the methods that were actually the cause of the exception but only the stack trace from the upmost raise call.

The solution 

Finally, we're getting to the root of this article's purpose. 

I'm suggesting that whenever you have a raise exc call in your code, you change that to: 

  raise new CaughtException (exc) 

And, of course, you'll need a new class CaughtException which I'll show you here: 

First, create a new Class, name it CaughtException and set its Super class to RuntimeException.

Then add the following two methods to it: 

  Sub Constructor(exc as RuntimeException)
    if exc isA CaughtException then
      exc = CaughtException(exc).OrigExc
    self.OrigExc = exc
  End Sub
  Function Stack() As String()
    return self.OrigExc.Stack
  End Function 

Finally, add this private property: 

  OrigExc As RuntimeException

That's all you need to care about. 

The trick here is that this class overwrites the Stack() method of RuntimeException, providing the original stack trace instead of the (useless) one created by the raise command. Therefore, your App.UnhandledException handler won't have to be changed at all to make this work. 

Even if you do not fully comprehend what this is doing, simply follow my advice of always using this construct wherever you catch an exception and raise it right after again. It'll magically make your debugging efforts easier in the future.

17 August 2013

Debugging Tips (using local objects to your advantage)

(This is a repost of an article of mine on Real Software's, now Xojo Inc's, former blog)

In this article I am going to share a nifty way to help with debugging your code. There are times when you need to figure out which methods of your code are called when. A common solution for this is to add lines like this to your code:

Sub myMethod()
System.DebugLog CurrentMethodName + " Entered"

... your code ...

System.DebugLog CurrentMethodName + " Exited"
End Sub

This can quickly get overwhelming if the methods you want to test have lots of exit points, i.e. return statements, as you'll have to spot all of those and add a line before each of the return statements. It gets even worse if the return statement calls a function, like this:

return mySubroutine()

To get this right, you'd have to modify your code like this:

dim tmpResult as String = mySubroutine()
System.DebugLog CurrentMethodName + " Exited"
return tmpResult

But there's a much simpler solution which brings along a few other enhancements as well: 

Use an object that exists for the time the method is running, like this:

Sub myMethod()
dim myTmpObject as new MethodProfiler(CurrentMethodName)

With this construct, you create an object of class MethodProfiler at the start of the method, and it'll be destructed automatically when the method is exited. That's all thanks to RB's method of reference-counted object management (it wouldn't work with the garbage disposal management as it's used by Java and Objective C).

The MethodProfiler class just needs its constructor and destructor methods implemented, like this:

Sub Constructor(name as String)
System.DebugLog "<"+name+"> Entered"
mName = name
End Sub

Sub Destructor()
LogMsg "<"+mName+"> Exited"
End Sub

No more need to add numerous DebugLog calls before return statements.

But wait, there's more!

With that single class taking care of all the tracking, you can easily add code to have it measure the time that the method takes to execute. And it could count the depth, allowing you to indent the output, making it more readable if you have nested calls you want to trace.

For your convenience, here is a demo project with a ready-to-use MethodProfiler class: http://files.tempel.org/RB/MethodProfiler.rbp.zip

Also, if you have at least a Basic license for my tool Arbed, you can use Arbed to add these MethodProfiler invocations to all methods of your project in one easy step. See this Xojo forum article for instructions.


16 August 2013

Updates for Arbed, Zip Classes and CustomEditField

My Zip Classes for Xojo / REALbasic have been updated to v3.3.2, fixing a bug on Linux and improving the demo to handle the case when the destination zip can't be created, letting the user choose a new destination folder.

Arbed, my project editor for REALbasic projects, is currently in beta for supporting Xojo projects. I've just released v1.7.0b7 that fixes a critical issue when saving modified Xojo projects in VCP (textual) format (using extensions .rbvcp or .xojo_project): With previous versions, it could happen that colors of controls and windows would be reset to black (i.e. when a color property uses &c instead of an integer or hex (&h) number for the color code).

Oh, and CustomEditField was recently updated as well, mainly adding improvements on syntax highlighting for REALbasic code.

07 August 2013

When I use too many programs that I also maintain

This post has nothing particular of interest, I'm just venting because I need a break. Read on to understand why:

I'm very detail oriented when I program. And when I see a critical bug, I need to fix it right away. I get so hooked on it that I often go into long sessions, and I'll be miserable if I have to stop in the middle of it.

Now, here's an example of what often happens then:

I am working on a paid project for a client. It has to do with recovering data from a complex file structure.

The data structures are in binary format, so I need to be able to read the data inside.

Since I have my own disk editor, iBored, I start writing a template for this file format. Since iBored's template system is a work in progress, it naturally leads to me having to add new code to iBored to suit a particular new construct in the template syntax. So I add new code to iBored.

The template system uses RbScript so that I can perform calculations to decypher complex data structures. RbScript is fairly limited by default, though. For instance, there is no sort function.

Which means that I have to write a sort function in RbScript. I need a RbScript editor. Real Studio's own Script editor sucks enormously - it doesn't even have Undo. Fortunately, there is Arbed. It has a better RbScript editor.

While writing my script in Arbed, I notice that its syntax parser doesn't indent Interfaces correctly. I like to fix that. I can, because Arbed is another tool of mine.

The RbScript editor and syntax highlighter is coming from the open source class "CustomEditField", written by Alex Restrepo. He has stopped working on it. Coincidentally, I took over.

Thus I am working several hours on the CustomEditField open source project to fix its indentation code, which is fairly convoluted (partly my fault). Eventually I get this done.

Next I need to merge the fixes of the CustomEditField project into the Arbed project file. Naturally, I use Arbed for this.

Merging takes a while because I had recently added new features to CustomEditField directly in Arbed, without merging those improvements back into the open source version. Meaning that I have to merge some code from CEF to Arbed, and other code in the other direction. I have to do this carefully. But eventually, I get all the changes merged into both projects.

When I try to save the updated projects, Arbed gives me an error message: wrong id in block header.

Great. So I have to find this bug. Takes me another 2 hours. It was very very well hidden.

That's where I am as of writing this blog post.

Now, I can go back to merge the changes between the projects once again. Then I can hopefully continue writing the Sort function in RbScript, after which I can finish the iBored template to view the data so that I can write the code I'm getting paid for.

Programming is fun. But so exhausting when you care too much.

06 August 2013

Xojo - Using MsgBox from a thread

I recently needed to debug some threaded REALbasic code. I could not use the IDE debugger because the issue I had occured only in built apps. Even worse, I needed to stop the thread at certain places so that I could investigate what it had done so far (i.e. writing a file, and looking at the data written).

An easy way to do this usually is to use MsgBox - it stops the code execution until you press the OK button.

However, calling MsgBox from within a thread is not working, at least not when building for Cocoa.

The solution is to put the thread to sleep temporarily, then have non-threaded code call MsgBox, and when that's done, wake up the thread again.

To put your own thread to sleep, call App.CurrentThread.Suspend.
To wake it up again, call its Resume function.

Here's a code example of how I solved this. Place both methods into a Module, and invoke MsgBoxFromThread from your thread.

Sub MsgBoxFromThread(msg as String)
  // Invoke this method to use MsgBox from within a thread.
  // Note: This is not multi-thread-safe! I.e, if you have more
  // than one thread from which you invoke this function, the
  // code needs to move away from single global/static variables
  // and instead maintain a queue for the threads and their
  // respective messages to be shown.
  // (See the "Tasks" project from the Xojo Examples to learn
  // how to maintain such a queue).
  dim currentThread as Thread = App.CurrentThread
  if currentThread = nil then
    // the main thread is active - we can simply call MsgBox here
    MsgBox msg
  end if
  // Create a timer for executing the MsgBox instruction
  // and waking up the thread afterwards again
  static t as Timer
  if t = nil then
    t = new Timer
    AddHandler t.Action, AddressOf MsgBoxFromThreadShow
    t.Period = 0
  end if
  // Store the msg and thread for the Timer
  mMsg = msg
  mThread = currentThread
  // Start the timer and put this thread to sleep
  t.Mode = 1
End Sub

Private Sub MsgBoxFromThreadShow(t as Timer)
  MsgBox mMsg
  // Once MsgBox is finished, wake up the thread again:
End Sub

Xojo's Build Automation: Beware of CopyFiles

If you use the Xojo IDE's CopyFiles build step to add files to your built application, beware of this long-existing bug.

Maybe you've had this happen yourself: You're pretty sure you had added some entries to a CopyFiles step, but they're not there any more. And you can't remember that you've removed them, either.

Here's what happens:

If you open a Xojo project, the IDE checks if the files referenced in the CopyFiles step do actually exist. And if they don't exist, the IDE deletes these entries - without a warning. Then, if you save your project, the entries are gone forever, even if you make the missing files available again.

And it's not just missing files (something that could happen if you copied your project to another computer, e.g. using a version control system, and forgot to also copy these files along)!

The same happens if you just move your project file without also moving the referenced files relatively to it, or if you move or rename the referenced file temporarily. This is surprising because doing this is not an issue with regular external items added to the IDE!

I plan to soon add a check to Arbed so that it will warn me when it notices that a file that's referenced in a CopyFiles step is not available. Since I do use Arbed with git, it's likely to notice this issue as soon as I verify the changes on a computer that's missing the CopyFiles step files.

I've filed the issue as Feedback #28676. Though, this only describes the issue that the IDE drops the entries if the files are not found. The other issue is that the IDE doesn't find the files if the project has been moved to another folder: Feedback #28678.

Until these issues are fixed, I recomment using a IDE Script instead, issuing a shell command to copy the files. That way, you can check if the shell command fails and report the issue, e.g. like this (Mac and Linux only, for Windows you'll need different code):

  dim cmds(), s, dst as String

  // The path to the built app's Resources folder (Mac):
  dst = CurrentBuildLocation+"/"""+CurrentBuildAppName+""".app"
  dst = dst+"/Contents/Resources/"

  // prepare the commands
  cmds.Append "cd  ""$PROJECT_PATH""" // Move to the project folder
  cmds.Append "cp myfile "+dst // Copy "myfile" to app's Resources folder

  s = DoShellCommand (Join (cmds, ";"))
  if s <> "" then
    print "Copy failed: "+s

BTW, this issue wouldn't be one if the IDE would handle these CopyFiles references the same way as it does for general external files and classes added to a project. Because, if such files are temporarily missing, the project doesn't simply drop them. If you look into the project file, e.g. by saving it in XML format, you'll see that external files added to the project are referenced in three ways: With an absolute path, with a path relative to the project files, and with a "SaveInfo", which uses the special Finder Alias format, allowing it to locate files even if they've been renamed or moved, as long as they're still on the same volume. On the other hand, CopyFiles uses only use a relative path (which is, unfortunately, titled "FileAlias", misleadingly suggesting it being a powerful Finder Alias).

In my opinion, the implementation of CopyFiles was done quite badly (it's not the only bug with this seemingly simple feature), where the programmer clearly did not understand how the IDE usually references external files, instead making up its own solution, which isn't only inferior but even leads to data loss.