New: OS X Prefs Editor for editing app defaults

Mac OS X 10.8 introduced a caching system for app preferences ("cfprefsd"). While this probably increases performances for apps, it makes it harder for developers to manipulate preference values quickly for testing, because making changes directly to the plist files in the ~/Library/Preferences folder does not work any more with editors such as "Property List Editor.app" and the similar one in Xcode.

Therefore, I've just written a new tool that works like the plist editor in Xcode but uses the CFPreferences functions to modify the values instead of editing the plist files directly.

If you've worked with "Property List Editor" or Xcode, you should have no trouble using my tool. It supports even Copy + Paste the same way. The only things I have not implemented (yet) are Undo and changing the type of existing elements (the reason for the latter is that REALbasic's framework does not offer a way to show a popup menu inside a listbox cell, and so I was too lazy finding a work-around for this to imitate Xcode's UI).

It even has one feature that Xcode's editor doesn't: You can delete all entries at once with Select All, then delete.

Also note that due to the use of the CFPreferences API, any changes you make are immediately committed to the preferences as seen by other apps, just like when you use the "defaults" command. For the same reason, changes made to the prefs by other apps (or with the "defaults" tool) will be seen in this tool after a change, a front switch or by pressing the "Sync" button. Hence, there is no Save command - updates are instant. For that reason Undo would be quite useful. Alas, I can live without it for now.

The tool is free for your use. Get it here:

http://files.tempel.org/Various/OSX_Prefs_Editor/PrefsEditor.zip (Version 1.1.1)

It includes the REAL Studio (Xojo) source code as well. But I retain the copyright and distribution rights to the program, so please do not distribute the app yourself without getting my permission first!

To learn of new updates, use the RSS feed on this page or watch it on MacUpdate.

(Shameless plug: If you'd like to have a customized version, let me know, I also do work for money ;) )

iTunes 11 modal alert dialog overflow

Do you tend to show modal dialogs to show issues to the user when processing his data? Well, think twice, or you might end up upsetting your users like Apple manages with iTunes 11.

See here for a screen recording I created when trying to update my iOS apps after about a week from the last update.

Blogger was supposed to show a video box here,

but that doesn't seem to work.

Instead, watch the video here (Youtube)

Or here (Quicktime)

Admittedly I am suffering from iTunes 11 more than most users because I have two iTunes accounts: A German and a U.S. account. And I have downloaded iOS apps from both accounts. This leads to iTunes wanting to update all apps for both accounts at the same time, even if it only shows me the German updates.

However, before iTunes 11, this was still manageable. iTunes 11 made this much worse.

Also, there are other alerts in there that even the usual Apple customers with just one iTunes account might experience, such as the late-appearing "age restriction" notice that pauses downloading, and messages about problems with particular apps without naming those apps.

Overall, iTunes 11 does this all plain terribly. Apple used to excel at good user interfaces. Nowadays, I get the impression they take the kids right from college, let them work on these programs with no supervision from anyone who still has a clue about how to avoid such bad user experiences.

(I've also reported this to Apple via their bugreporter, see http://www.openradar.me/15350892)

Understanding the Xojo (REALbasic) language

This is a small assortment of tips and tricks to make you understand the Xojo language better.

The differences between identifiers, keywords and reserved words

Even though the IDE paints them all the same color, they're not the same in the way you can use them in code code:

• Identifiers are words you can use to name your methods, properties, variables etc.
• Reserved Words (or reserved names) are all those that you can never use as identifiers. This includes: if, then, to, function, end, attribute
• Keywords are a mix of the above. They are all those words that are pre-defined by the compiler. This not only includes the Reserved Words but also standard types such as String, Integer, Boolean and Color. But it does not include Date, for instance - that's a name from the framework, i.e. it's practically code written in Xojo that you do not see but that the compiler secretly adds to your source code when building your program. You can tell the difference if you use XojoScript (RbScript), which is a bare bones access to the compiler without most of the framework: While you can use Color there, the Date class is not available. (Note, however, that even XojoScript gives you a small framework to work with that includes some strings operations and the Print function, for instance).

Now, here's the interesting part:

Whereas you cannot declare a variable with the name attribute, you can declare one with the name boolean!

Here is an example that actually works:

  dim boolean as integer
  boolean = 2
  MsgBox "The value of the variable 'boolean' is "+Str(boolean)

While the above is quite some nonsense, it can be used in other ways, too, for instance in an enumeration. You can declare an Enum named Types and add the values String, Integer, Boolean to it. Then you can write:

  dim t as Types
  t = Types.Integer

BTW, for that very reason that reserved words (if, then etc.) are not the same as predefined types (String, Integer etc.), I tend to write the former always in lower case while I write types (both predefined and self-declared ones) always in upper case. So, while some may think my mixed case code writing is inconsistent, it actually follows rules based on the understanding of the language.

Boolean logic

I often see beginners not making best use of Boolean types.

For instance:

  if found = True then

could also be written as:

  if (not (found <> true)) <> false then

The second example is clearly unreadable and therefore quite counterproductive. The first one is, to some lesser extent, similar in that nature.

The recommended variant would be:

  if found then

Read it aloud. Doesn't that sound better than the first example?

In the same way,

  if found = False then

should be written as:

  if not found then

Think about this the next time when you choose a name for a Boolean property or variable. Name it so that it reads well in a "if ... then" sentence, avoiding "= false" and "= true" constructs. It clutters your code and makes it often less readable.

Similarly, you may want to avoid constructs like these:

  if me.Value = true then
    button1.Enabled = true
  else
    button1.Enabled = false
  end

Why not simply:

  // enable button1 only if this checkbox is checked
  button1.Enabled = me.Value

Though, some may argue that "me.Value" is not self-explanatory, thus "me.Value = true" is adding some context. Here's an alternative that's even more self-explanatory:

  dim checked as Boolean = me.Value
  if checked then

and, analogously:

  dim checkboxChecked as Boolean = me.Value
  button1.Enabled = checkboxChecked

Using Xojo or Real Studio? Please help me test Arbed

I am close to releasing the next version of my tool "Arbed" for Xojo and Real Studio project files.

As explained in my previous post, I have added code that performs a self-check to make sure it doesn't miss anything when reading and writing a Xojo project in textual (VCP) format.

If you haven't yet, please download the latest beta from http://files.tempel.org/Arbed/beta/1.7.0/b13/ and launch it, then open your .rbvcp and .xojo_project projects with it. (Either use the "Open" command from the File menu, or drag and drop your projects onto its main "Arbed Drop Pad" window, into the "Project Editor" group box.)

If you get an error or verification failure message, please let me know and I'll fix the issue within a day or two.

Lessons in working with file formats in flux

As you may know, my tool Arbed for Xojo (formerly known as Real Studio) is capable of reading and modifying Xojo's project files. This ability is used to perform often needed operations that the Xojo IDE doesn't offer (yet), such as comparing projects and single classes, scripted code replacement, code obfuscation, preparing code for localization and more.

I'm going to tell you a little about the challenges I have to deal with.

There are 3 different project formats supported by Xojo:

1. Binary, aka RBP, aka RbBF, using extensions .rbp and .xojo_binary_project (sheesh!)
2. XML, using extensions .xml and .xojo_xml_project
3. Textual, aka VCP, using extensions .rbvcp and .xojo_project

Dealing with RBP and XML

The first two formats are structurally identical: The RBP format is nothing more than a more compact version of the XML format, using a binary representation of the XML by using 4-letter-codes for the tags and 4 byte length fields to indicate the size of the elements that follow. (Xojo engineers informed me that actually RBP was designed first, and XML is a translation of the RBP format.)

Arbed originally only supported these first two formats, and it was pretty easy to handle. The data was well structured and repetitive in a way that gives little room for (problematic) surprises. If you've ever looked at a xml (or html) file, you probably understand what I mean.

When Arbed reads a project in the RBP and XML formats, it modifies only the parts it well understands while leaving all the other data untouched. For instance, when it changes some source code inside a function, it only modifies the affected <SourceLine> elements. That way, it can safely modify the project even if Arbed doesn't understand everything that's inside the project. And believe me, even though XML is apparently self-explanatory, there are a lot of unexplained things in there that just leaves one guessing.

The blame for this can be put with Xojo for not documenting this. I guess they do not even have an internal documentation. As so often in rushed software engineering, the code is the only documentation (for that, it doesn't even deserve the term "engineering"). When I am about to write complex code, I usually start with documenting and specifying it first, so that I (and others that might join the project) can later reder to that. It really helps, and should be obvious. However, most people do not follow this simple rule. Many even never learn. It even starts with documenting your single subroutines. See my little article on coding guidelines.

Converting between RBP and XML formats.

It gets a little more complicated when using Arbed's Convert operations. They's available in the main window (titled Arbed Drop Pad). For instance, to convert a RBP to a XML file, it has to know how the 4-letter-codes in the RBP file translate to the XML tag names.

It happens every once in a while that Xojo adds new features to the project file format that lead to using newly names element tags. For instance, when the Web Edition was added, a new XML tag named WebApp got added. Now, while Arbed doesn't need to know what this value means, it needs to know how to translate it between the XML tag name and the RBP tag code. Therefore, when you're using an Arbed version that doesn't know this translation yet, it'll tell you about it if you ask it to convert a newer project file with yet-unknown tags in it. I will then have to update Arbed with the new code (which is easy to do, I just have to save the same project in both XML and RBP format in the IDE and see look for the code in question in both files).

Overall, working with RBP and XML projects in Arbed is therefore fairly safe and fast.

Arbed only rarely needs updates, e.g. when a new tag code gets added, and then only to make the Convert functions work (which are not even really necessary because you can just use the IDE to save a project in a different format).

Dealing with the VCP format

The VCP is a different and much more scary beast.

In theory, it's just plain RbScript code, which is fairly well specified (even though Xojo has failed to present a syntax/grammar spec for their language for 15 years now, it's fairly well understood by me by now, and my background in compiler design helps there, too).

I even have a fully working RbScript parser that I hope to use soon to implement some great features such as method name obfuscation (i.e. rename all custom method names in your source in order to foil reverse engineering attempts), automatic code reformatting and dead code identification and removal.

But the reality is harsh. The VCP format has a lot of inconsistencies and hard-to-understand behaviors that make parsing it challenging.

Some examples:

Properties of Controls vary in representation

Recent Real Studio versions tended to write some integer values of Control properties as floating point numbers. If you're using version control you may have noticed that a control's Width and Height were sometimes shown as 1.6e2 when you did input 160. Recent Xojo version seem to have finally fixed this.

Odd spacing in method declarations

This is how a normal Method declaration looks like when using qualified type identifiers:

Sub Foo(x as a.b)

And this is how an External Method looks like:

Declare Sub extmethod Lib ""  Foo(x as a . b)

Note the inserted blanks around the period. They're syntactically allowed but I didn't type them - the VCP format inserted them automagically.

Troubles with Attributes

Let's enter 3 attributes with, admittedly, some unusual values:

• test: a
• t2: \x22 (which gets automatically quoted into: "\x22")
• t3: b=," (which gets automatically quoted into: "b=,""")

Now let's see how they look like in the VCP format.

This is how a normal Method looks like:

Attributes( test = a, t2 = "\x22", t3 = "b=,""" )  Sub Foo()

Alright. That's fairly readable. (Though what's with those spaces inside of the parentheses? Also note the double space before "Sub".)

Here's a similar Delegate declaration:

Attributes( test = a, t2 = "\x22", t3 = "b=,""" ) Delegate Sub Foo()

Looks pretty identical, doesn't it? And now we can also understand that double space: It makes room for plugging a "Delegate" word in there :)

But wait. Method declarations are surrounded by #tag markers, providing extra information that the RBP and XML formats usually store in extra elements or attribute fields. Here's the one for a normal Method:

#tag Method, Flags = &h0

Okay. Now the one for a Declare:

#tag DelegateDeclaration, Flags = &h0, Attributes = \"test \x3D a\x2C t2 \x3D "\x22"\x2C t3 \x3D "b\x3D\x2C""""

Uh, what? Not only it includes the Attributes that are already present - and more much readable - in the declaration source code line, but it's only appearing in this special "delegate" method declaration but not in a normal method - even though they have the same syntax and should thus be created the same way.

But it gets even weirder: The above was created in the latest Xojo IDE. When doing the same with Real Studio 2012r2.1, the above line does not include the Attributes in the #tag line. So, someone must have accidentally added this nonsense just recently, and only to Declares, not to normal functions. Or maybe it wasn't an accident. In any case, it's quite a mess.

Encoding horrors

As a final example, let's focus on the odd encoding of attributes in the #tag line. It's obvious that it tries to escape some codes so that they may also be used inside the attribute values. It looks like a homemade algorithm, but appears to work pretty well. As you can see above, I've used characters as values that it also uses for escaping to see if I can break it. But I couldn't. That is, almost. When I set the value of an attribute to something containing line feeds (returns), then the IDE crashes hard or the compiler spills out inexplicable error messages. Oh, and later I found that if you put \x22 into an attribute's value, it'll be converted to a comma next time you open the project. Doesn't happen in RBP format, of course.

Sure, the test with the return in the attribute value was something that's unlikely to happen, but it shows that the whole thing is put together with little understanding how escaping random data should be done. Heck, since the Xojo IDE is programmed in Xojo, they could have have just base64-encoded the string, or used quoted-printable encoding. That's what they're meant for, and both are functions that Xojo provides anyway! But no, a new technique had to be invented: Something unreadable and prone to crashing.

What this all means

Why do I even bother with this painful stuff?

Well, here's the thing: In 2012, I enhanced Arbed to read and write the VCP format.

It was one of the worst decisions I had made in past years, because it took me several months of intensive work to get even close to being usable. And since then I spent many more days working out all the kinks. These kinks are what you see above: Hardly anything that looks similar also behaves similarly. Behaviors change in different RS and Xojo versions, sometimes without making sense. I had to identify and special-case them all, for each single IDE release, so that Arbed generates the exact same output that the IDE does.

I am still not sure that it was worth the effort, because I do not use the VCP format for myself, and until then, all the features I added to Arbed were needed by myself. But when I started selling Arbed, I believed I had to add this feature because many expected it to just work.

Now, every time Xojo changes the output format of the VCP files only slightly, it might mean that my own code could miss it. For instance, some files use undocument flag values, such as &h1000. No idea what it's for. Formerly, I had just generated these flags from the declaration source line. But now, sometimes, the Flag in the #tag line contains more information and I have to preserve it. Special handling galore!

Arbed's VCP output needs to match that of the IDE exactly

The issue is that, contrary to how Arbed can edit just specific parts of a method's source code in the RBP and XML formats, it can't do this with VCP files. Arbed's project modification code was written to work on the RBP/XML format directly, in order to keep anything unmodified that the user doesn't directly alter. For this functionality to work with VCP files, Arbed has to convert it internally into RBP format, so that my existing code can operate on it. Then, when the user saves his changes, Arbed recreate the VCP file from the updated RBP code (Arbed only rewrites the files that were affected by the changes). But that means that I have to recreate the entire VCP file from RBP data, instead of just modifying the source lines or whatever else was changed by the user as I did with RBP/XML files.

All this requires that Arbed understands every detail of a VCP file in order to properly convert it to RBP and back, internally.

Potential for damage

And that's what bit me a few times already in the past. I would miss small changes in the format, such as that color constants were originally written as hex values (&h...), but recently Xojo changed this to using &c... codes instead. My code was not prepared for this and would then write any color back as a 0 (black).

Basically, Arbed could, unknowingly, damage your VCP files if you edited and saved them in Arbed. You'd notice eventually, but Arbed should notice this before it does such damage.

So, how do you make sure that your code that reads and writes an external file format doesn't damage it just because the file format introduces changes that you are not aware of, yet?

The solution is: When reading the input, recreate the would-be output from it right away and then verify that both match. If there's a mismatch, your code is prone to damaging the file when writing it back.

Simple as that.

Arbed 1.7 is made safer by performing a self-test on any VCP file it reads.

Therefore, from Version 1.7b9 on (which I'll release shortly), Arbed will verify its own VCP read/write code to make sure that it entirely understands the project it reads and (optinally) writes.

So that, when you save a project, anything you didn't specifically change, will remain the same in the rewritten VCP file. So that when you use version control (git, mercurial, subversion) with your VCP files, it won't show lots of unrelated changes after Arbed wrote the file back (even if it's still valid to Xojo). Something Arbed may do even better than the Xojo IDE itself (ever seen those TabStops disappear and reappear randomly?)

Arbed accomplishes this self-check by converting the read project into its internal RBP format, then back the VCP representation, then comparing its VCP output with the original file. If it finds a mismatch, it warns the user that the data was not fully understood. It also creates a file on the user's Desktop containing the specifics of the rendering differences, so that the user can tell what's gone wrong (allowing the more experienced user to decide if the changes are benign), and if the user is sending the file to me, I can quickly fix Arbed and release a new version that deals with it.

Of course, if Arbed reports a problem with its VCP conversion, one can always just use the Xojo IDE to make the conversion: Open your project in Xojo, and use the Save As... menu command to save the project in the binary or XML format. Then use that file with Arbed, and if you've made changes in Arbed and saved them back to the project file, open that again in Xojo and use again Save As... to save it in VCP format. It's tedious but that's the way it's been working even before Arbed added support for VCP projects.

Preserving stack trace when catching and re-raising runtime exceptions

(This is a repost of an article of mine on Real Software's, now Xojo Inc's, former blog)


This post offers an introduction to the usefulness of exception handling in general, while also explaining how to deal with a particular shortcoming when you use debugging code and the UnhandledException event handler in built programs you deliver to people. I hope this article will be helpful to both newcomers and veterans alike.

Background (UnhandledException and the Stack Trace)


You probably know the UnhandledException event handler in the App class.


It lets you catch any unhandled exceptions in your program, preventing your program from getting terminated unexpectedly, and allowing you to save any interim data that the user may have not saved yet, and other defensive fallback procedures. 


One other use of this handler is to record the location where the exception occurred in order to use that for later debugging. For instance, if you send your built application to others who do not have the Xojo debugger, your program could still record the so-called stack trace from the exception, show that to the user, and ask him to forward that information to you so you can hopefully deduce what went wrong and how to avoid it in the future. 


You get this stack trace by invoking the Stack() method of the RuntimeException object you get passed in UnhandledException. It returns an array of strings, identifiying the methods that were called up to the one that caused the exception. 


For instance, if your program causes an exception in a Window's Open event, the stack trace might look like this: 


1. RaiseOutOfBoundsException
2. Window1.Window1.Event_Open%%o<Window1.Window1>
3. FireWindowOpenEvents
4. Window.Constructor%%o<Window>
5. Window1.Window1%o<Window1.Window1>%
6. _MakeDefaultView
7. _Startup
8. RunFrameworkInitialization
9. REALbasic._RunFrameworkInitialization%%p
10. _Main
11. % main

The topmost line shows where the exception occured, the bottommost is the topmost caller at that time. 


It tells several things:

• The exception, obviously an OutOfBounds exception, occured in Window1's Open event (line #2)
• The Open event was invoked by the Window1's Constructor (lines 4 and 5).
• The Window1 got constructed right at start of the program (line 7). Otherwise, we'd usually see the method "RuntimeRun" up in the stack trace, indicating that the startup has finished and the program is now processing events.

Raising exceptions 


Whenever an exception occurs, we say that it gets raised. Xojo even offers a custom statement for that: The Raise command. With that, you can make your own exceptions occur. An example: 


  raise new OutOfBoundsException 


is equivalent to: 


  dim anArray(0) as Integer
  anArray(1) = 0 // this will cause an OutOfBoundsException 


(Of course, you can create your very own exception types by subclassing them from RuntimeException. See the Language Reference to learn more about that.) 


Now, here's an imporant part: The stack trace that gets attached to any RuntimeException object is created when this raise statement is invoked. And the stack trace is then created from the very calling stack that's in effect at the time of invocation. Logical, right? 



Catching exceptions 


You can catch exceptions in your methods using the try ... catch ... end and the exception statements.

There are two common cases why and how you'd catch exceptions: 


1. Handling the exception 


You expect a particular exception to occur and are prepared to handle it. This is, for instance, the case when you use file I/O operations and which you cannot otherwise predict to avoid them, such as that a file cannot be created, causing an IOException. 


2. Cleanup 


You do not know that any particular exceptions might occur in some subroutines you call but you like to be prepared and do some cleanup if that should happen. In this case, you'll pass on the exception to the caller because you don't know have reason way to handle the exception at that particular point. 


Such cleanup handling code usually looks like this: 


 try
   ... // your operations
 catch exc as RuntimeException
   db.Rollback // e.g. abort a database transaction
   raise exc // pass the exception on
 end

The complication (catching and re-raising) 


Let's assume you have a UnhandledException event handler in the App class, which reads the Stack array and for later analysis (e.g. by writing to a file, creating an email to you or even uploading it to your web server). 


So, if any part of your program raised an exception for which you don't have particular handling code, it'll end up in App.UnhandledException, which will eventually allow you to learn where the crash ocurred. That's the general idea, at least. 


However:

Consider what happens if an exception occurs inside a subroutine that's called by code using the Cleanup exception handler? 


Remember how the stack trace gets created: During the invocation of the raise statement. 


This means that if an exception occurs deeper down, then gets caught by the Cleanup handler code, which then re-raises the exception to end up finally in the App.UnhandledException handler, you'll get to see the wrong Stack trace. You won't see the methods that were actually the cause of the exception but only the stack trace from the upmost raise call.



The solution 


Finally, we're getting to the root of this article's purpose. 


I'm suggesting that whenever you have a raise exc call in your code, you change that to: 


  raise new CaughtException (exc) 


And, of course, you'll need a new class CaughtException which I'll show you here: 


First, create a new Class, name it CaughtException and set its Super class to RuntimeException.

Then add the following two methods to it: 


  Sub Constructor(exc as RuntimeException)
    if exc isA CaughtException then
      exc = CaughtException(exc).OrigExc
    end
    self.OrigExc = exc
  End Sub
  
  Function Stack() As String()
    return self.OrigExc.Stack
  End Function 


Finally, add this private property: 


  OrigExc As RuntimeException

That's all you need to care about. 


The trick here is that this class overwrites the Stack() method of RuntimeException, providing the original stack trace instead of the (useless) one created by the raise command. Therefore, your App.UnhandledException handler won't have to be changed at all to make this work. 


Even if you do not fully comprehend what this is doing, simply follow my advice of always using this construct wherever you catch an exception and raise it right after again. It'll magically make your debugging efforts easier in the future.

Debugging Tips (using local objects to your advantage)

(This is a repost of an article of mine on Real Software's, now Xojo Inc's, former blog)

In this article I am going to share a nifty way to help with debugging your code. There are times when you need to figure out which methods of your code are called when. A common solution for this is to add lines like this to your code:

Sub myMethod()
System.DebugLog CurrentMethodName + " Entered"

... your code ...

System.DebugLog CurrentMethodName + " Exited"
End Sub

This can quickly get overwhelming if the methods you want to test have lots of exit points, i.e. return statements, as you'll have to spot all of those and add a line before each of the return statements. It gets even worse if the return statement calls a function, like this:

return mySubroutine()

To get this right, you'd have to modify your code like this:

dim tmpResult as String = mySubroutine()
System.DebugLog CurrentMethodName + " Exited"
return tmpResult

But there's a much simpler solution which brings along a few other enhancements as well: 

Use an object that exists for the time the method is running, like this:

Sub myMethod()
dim myTmpObject as new MethodProfiler(CurrentMethodName)

With this construct, you create an object of class MethodProfiler at the start of the method, and it'll be destructed automatically when the method is exited. That's all thanks to RB's method of reference-counted object management (it wouldn't work with the garbage disposal management as it's used by Java and Objective C).

The MethodProfiler class just needs its constructor and destructor methods implemented, like this:

Sub Constructor(name as String)
System.DebugLog "<"+name+"> Entered"
mName = name
End Sub

Sub Destructor()
LogMsg "<"+mName+"> Exited"
End Sub

No more need to add numerous DebugLog calls before return statements.

But wait, there's more!

With that single class taking care of all the tracking, you can easily add code to have it measure the time that the method takes to execute. And it could count the depth, allowing you to indent the output, making it more readable if you have nested calls you want to trace.

For your convenience, here is a demo project with a ready-to-use MethodProfiler class: http://files.tempel.org/RB/MethodProfiler.rbp.zip

Also, if you have at least a Basic license for my tool Arbed, you can use Arbed to add these MethodProfiler invocations to all methods of your project in one easy step. See this Xojo forum article for instructions.

Enjoy.

Updates for Arbed, Zip Classes and CustomEditField

My Zip Classes for Xojo / REALbasic have been updated to v3.3.2, fixing a bug on Linux and improving the demo to handle the case when the destination zip can't be created, letting the user choose a new destination folder.

Arbed, my project editor for REALbasic projects, is currently in beta for supporting Xojo projects. I've just released v1.7.0b7 that fixes a critical issue when saving modified Xojo projects in VCP (textual) format (using extensions .rbvcp or .xojo_project): With previous versions, it could happen that colors of controls and windows would be reset to black (i.e. when a color property uses &c instead of an integer or hex (&h) number for the color code).

Oh, and CustomEditField was recently updated as well, mainly adding improvements on syntax highlighting for REALbasic code.

When I use too many programs that I also maintain

This post has nothing particular of interest, I'm just venting because I need a break. Read on to understand why:

I'm very detail oriented when I program. And when I see a critical bug, I need to fix it right away. I get so hooked on it that I often go into long sessions, and I'll be miserable if I have to stop in the middle of it.

Now, here's an example of what often happens then:

I am working on a paid project for a client. It has to do with recovering data from a complex file structure.

The data structures are in binary format, so I need to be able to read the data inside.

Since I have my own disk editor, iBored, I start writing a template for this file format. Since iBored's template system is a work in progress, it naturally leads to me having to add new code to iBored to suit a particular new construct in the template syntax. So I add new code to iBored.

The template system uses RbScript so that I can perform calculations to decypher complex data structures. RbScript is fairly limited by default, though. For instance, there is no sort function.

Which means that I have to write a sort function in RbScript. I need a RbScript editor. Real Studio's own Script editor sucks enormously - it doesn't even have Undo. Fortunately, there is Arbed. It has a better RbScript editor.

While writing my script in Arbed, I notice that its syntax parser doesn't indent Interfaces correctly. I like to fix that. I can, because Arbed is another tool of mine.

The RbScript editor and syntax highlighter is coming from the open source class "CustomEditField", written by Alex Restrepo. He has stopped working on it. Coincidentally, I took over.

Thus I am working several hours on the CustomEditField open source project to fix its indentation code, which is fairly convoluted (partly my fault). Eventually I get this done.

Next I need to merge the fixes of the CustomEditField project into the Arbed project file. Naturally, I use Arbed for this.

Merging takes a while because I had recently added new features to CustomEditField directly in Arbed, without merging those improvements back into the open source version. Meaning that I have to merge some code from CEF to Arbed, and other code in the other direction. I have to do this carefully. But eventually, I get all the changes merged into both projects.

When I try to save the updated projects, Arbed gives me an error message: wrong id in block header.

Great. So I have to find this bug. Takes me another 2 hours. It was very very well hidden.

That's where I am as of writing this blog post.

Now, I can go back to merge the changes between the projects once again. Then I can hopefully continue writing the Sort function in RbScript, after which I can finish the iBored template to view the data so that I can write the code I'm getting paid for.

Programming is fun. But so exhausting when you care too much.

Xojo - Using MsgBox from a thread

I recently needed to debug some threaded REALbasic code. I could not use the IDE debugger because the issue I had occured only in built apps. Even worse, I needed to stop the thread at certain places so that I could investigate what it had done so far (i.e. writing a file, and looking at the data written).

An easy way to do this usually is to use MsgBox - it stops the code execution until you press the OK button.

However, calling MsgBox from within a thread is not working, at least not when building for Cocoa.

The solution is to put the thread to sleep temporarily, then have non-threaded code call MsgBox, and when that's done, wake up the thread again.

To put your own thread to sleep, call App.CurrentThread.Suspend.
To wake it up again, call its Resume function.

Here's a code example of how I solved this. Place both methods into a Module, and invoke MsgBoxFromThread from your thread.

Sub MsgBoxFromThread(msg as String)
  // Invoke this method to use MsgBox from within a thread.
  
  // Note: This is not multi-thread-safe! I.e, if you have more
  // than one thread from which you invoke this function, the
  // code needs to move away from single global/static variables
  // and instead maintain a queue for the threads and their
  // respective messages to be shown.

  // (See the "Tasks" project from the Xojo Examples to learn

  // how to maintain such a queue).
  
  dim currentThread as Thread = App.CurrentThread
  if currentThread = nil then
    // the main thread is active - we can simply call MsgBox here
    MsgBox msg
    return
  end if
  
  // Create a timer for executing the MsgBox instruction
  // and waking up the thread afterwards again
  static t as Timer
  if t = nil then
    t = new Timer
    AddHandler t.Action, AddressOf MsgBoxFromThreadShow
    t.Period = 0
  end if
  
  // Store the msg and thread for the Timer
  mMsg = msg
  mThread = currentThread
  
  // Start the timer and put this thread to sleep
  t.Mode = 1
  currentThread.Suspend
End Sub

Private Sub MsgBoxFromThreadShow(t as Timer)
  MsgBox mMsg
  // Once MsgBox is finished, wake up the thread again:
  mThread.Resume
End Sub

Xojo's Build Automation: Beware of CopyFiles

If you use the Xojo IDE's CopyFiles build step to add files to your built application, beware of this long-existing bug.

Maybe you've had this happen yourself: You're pretty sure you had added some entries to a CopyFiles step, but they're not there any more. And you can't remember that you've removed them, either.

Here's what happens:

If you open a Xojo project, the IDE checks if the files referenced in the CopyFiles step do actually exist. And if they don't exist, the IDE deletes these entries - without a warning. Then, if you save your project, the entries are gone forever, even if you make the missing files available again.

And it's not just missing files (something that could happen if you copied your project to another computer, e.g. using a version control system, and forgot to also copy these files along)!

The same happens if you just move your project file without also moving the referenced files relatively to it, or if you move or rename the referenced file temporarily. This is surprising because doing this is not an issue with regular external items added to the IDE!

I plan to soon add a check to Arbed so that it will warn me when it notices that a file that's referenced in a CopyFiles step is not available. Since I do use Arbed with git, it's likely to notice this issue as soon as I verify the changes on a computer that's missing the CopyFiles step files.

I've filed the issue as Feedback #28676. Though, this only describes the issue that the IDE drops the entries if the files are not found. The other issue is that the IDE doesn't find the files if the project has been moved to another folder: Feedback #28678.

Until these issues are fixed, I recomment using a IDE Script instead, issuing a shell command to copy the files. That way, you can check if the shell command fails and report the issue, e.g. like this (Mac and Linux only, for Windows you'll need different code):

  dim cmds(), s, dst as String

  // The path to the built app's Resources folder (Mac):
  dst = CurrentBuildLocation+"/"""+CurrentBuildAppName+""".app"
  dst = dst+"/Contents/Resources/"

  // prepare the commands
  cmds.Append "cd  ""$PROJECT_PATH""" // Move to the project folder
  cmds.Append "cp myfile "+dst // Copy "myfile" to app's Resources folder

  s = DoShellCommand (Join (cmds, ";"))
  if s <> "" then
    print "Copy failed: "+s
  end

---

BTW, this issue wouldn't be one if the IDE would handle these CopyFiles references the same way as it does for general external files and classes added to a project. Because, if such files are temporarily missing, the project doesn't simply drop them. If you look into the project file, e.g. by saving it in XML format, you'll see that external files added to the project are referenced in three ways: With an absolute path, with a path relative to the project files, and with a "SaveInfo", which uses the special Finder Alias format, allowing it to locate files even if they've been renamed or moved, as long as they're still on the same volume. On the other hand, CopyFiles uses only use a relative path (which is, unfortunately, titled "FileAlias", misleadingly suggesting it being a powerful Finder Alias).

In my opinion, the implementation of CopyFiles was done quite badly (it's not the only bug with this seemingly simple feature), where the programmer clearly did not understand how the IDE usually references external files, instead making up its own solution, which isn't only inferior but even leads to data loss.

Beware of a pitfall when reading the OS X clipboard with the Carbon API

This is a heads-up for all those who read the Clipboard with the Carbon API (e.g. PasteboardCreate).

This is about rare but possible issue where you might be reading the clipboard contents incorrectly.

A very simple demo for this is the following:

1. Open Automator to create a new Workflow.
2. Add a Run Shell Script, with the command: ls -l /
3. Add a Copy to Clipboard.
4. Run it.

Launch TextEdit and Paste into a document. You should be seeing the listing of your root folder.
That's what is expected.

Now try your Carbon app and see what it takes from the same clipboard.
Does it only see one line of the clipboard ("total NNN")? Then you're subject to this issue.

Well, here's the explanation: That darn automator operation puts each line as a separate item into the pasteboard instead of putting all lines into a single text item. (The Finder does that, too, if you Copy a multi-selection of files.)

The fact that this Automator operation does it this way is suggesting that other NeXT (Cocoa) based apps may do the same, so you better be prepared for it.

To deal with this, you'll have to iterate over all items (whose count you find via PasteboardGetItemCount), and concatenate them with a line delimiter in between.
Or use NSPasteboard and get the NSStringPboardType, which contains all in one, with the proper delimiters.

Note: You can access the pasteboard using any of the 3 APIs (Scrap Mgr, Carbon Pastebaord, NSPasteboard) at the same time. E.g., if you put a new item into the Carbon Pasteboard, it becomes instantly available to the other APIs as well. Meaning that the 3 pasteboards are fully synchronized.

Therefore, you could just handle this special case like this, provided you're interested in text:

 if PasteboardGetItemCount() > 1 then
   txt = [[NSPasteboard pasteboardWithName:NSGeneralPboard] dataForType:NSStringPboardType]
 end

Finally, a blog from a coder!

Hello visitor,
This is a short intro about my background and what you can expect from this blog in the future.

I've been programming for over 30 years now. Started with programmable calculators, then chose to go to the school in my town that had the first computer, i.e. a CBM 3032, eventually could afford a VIC-20, C-64, Atari ST, and many Macs since. Had a few Windows PCs, too, mainly for gaming.

I tried earning my money with designing my own software from early on. Had and still have to do contract work to get by, though.

My passion (in programming) is with tools. My earliest programs were a machine level debugger (for 6502, later the same for the Gepard and Atari ST computers) and a disk copy program (F-Copy 64). I also worked intensively on a 68000 OS, later maintained a complete Modula-2 development system. For a while, I also worked on low level software and drivers (e.g. Joliet and UDF file systems for Mac OS, HFS drivers for iPodLinux), until I discovered the programming system REALbasic (formerly Crossbasic, then Real Studio, now Xojo), short RB. REALbasic is my preferred programming system for several reasons:

• It has all the basic building blocks of a modern programming languages (well, I'm still waiting for lambda expressions and/or ObjectiveC-like blocks).
• It lets me create a program with a simple user interface very quickly.
• It is cross-platform (Mac, Windows, even Linux).
• It supplies its own scripting engine which I've used many times to make complex development tasks much simpler.
• It's quite flexible, allowing me to add native C/C++ code via external libs or plugins.

There's also a few things I strongly dislike about Xojo, and not because I just am a hater but because I feel that Xojo has made a few bad design decisions that they could have avoided had they been giving it more thought or just asked someone for advice who had more experience in this field, i.e. me, for example). You'll most probably learn about a few of them if you keep reading this blog.

Because of my long experience with programming tools I also have a quite good understanding of how RB works internally. I understand quite well how both the framework and the compiler work. I've been able to identify or even fix bugs in RB that even RB's own developers were unable to find, in some cases.

Based on this knowledge I started writing tools for RB, which eventually evolved into Arbed, a editor that offers a lot of functionality that Xojo's IDE does not offer. My goal for this blog is to frequently give tips for using Xojo more effectively with the help of Arbed.

I suck at visual design, so I don't even try. You'll probably know this already if you had a look at my website or at my programs.

However, I care for usability and elegance, and I can get quite annoyed if I have to use a program that doesn't follow standards or makes using it unnecessarily cumbersome. Xojo's IDE is a recurring example of that, unfortunately.

I am one of those who chose the Mac platform over Windows for its clean usability concepts and for its rather smart OS technologies (back in the 80s, Mac OS was far superior over DOS, technically).

My native language is German, BTW. I learned a little English in school and only got fairly good at it after living around the turn of the millenium for five years in Carlifornia, USA. Still, I keep making grammatical mistakes or use incorrect idioms. Feel free to let me know about them, I don't take offense at being corrected, I rather try to welcome it.

See my website for more about my work and musings: http://www.tempel.org/