08 April 2017

Adding fast external disks to various Mac models, with benchmarks

2nd Update on 9 Apr 17: See end of article
3rd Update on 18 Apr 17: Added more adapters

I wanted a fast external disk for my three Macs, which are now all 5 years or more old:
  • MacBook Pro Mid 2012 ("MacBookPro10,1"), 2.6 GHz Core i7
  • iMac 27" Mid 2011 ("iMac12,1"), 3.4 GHz Core i7
  • Mac Pro Late 2008 ("MacPro3,1"), 8 Core-Xeon 2.8 GHz

The Mac Pro has no Thunderbolt ports but I've installed a USB 3.0 PCI card from Inateck.

The iMac has Thunderbolt but only USB 2 built-in. I've added USB 3.0 support with the Elgato Thunderbolt 2 Dock (230 €).

I was interested to find out whether I should just go with a cheap USB 3 disk enclosure or use a significantly more expensive Thunderbolt enclosure.

The short answer is: If you have USB 3 ports on your Mac(s), you'll be fine with a USB 3 enclosure from a performance standpoint. But you won't get TRIM support, which is especially important if you plan to use the SSD as a faster boot volume - in that case, I recommend Thunderbolt instead.

For testing, I was using a new SanDisk Ultra II SSD 960GB (SATA III). I connected it to these adapters / enclosures:

Note: Even though the ICY BOX supports USB 3.1, which allows up to 10 Gbit/s, it connects to my Macs only at USB 3.0 speeds (5 Gbit/s) as they do not support 3.1. The provided USB cables are compatible with these Macs' "standard" USB ports.

The Thunderbolt enclosure by Delock requires an external power supply, and the AKiTiO comes with a second cable that needs to be plugged into a regular USB port to provide power to the disk. The USB adapters, on the other hand, do not have nor provide an extra power supply - they pull all the needed power for the disk from the USB port they're connected to.

Test Procedures

The testing was done in various ways, but I always only attempted to figure out the maximum possible transfer speed, such as for copying large single and unfragmented files, as that's the main use case for me (I often transfer entire disks, sector by sector, to another disk, e.g. for migration to a newer Mac, data analysis or quick backups).

For testing read performance, the easiest way is to use the dd command in Terminal.app:

sudo dd if=/dev/rdiskN of=/dev/null bs=512000

N has to be replaced by the disk number, which can be learned by using this command:

diskutil list

Note that this command lists the disk names with the leading "r", but when doing the test, the "r" is important for best speed results.

Write performance testing is a bit more difficult, because you need some data to copy from. You could use a file or another disk, but then you get a result that also involves the reading speed from the other disk. If that disk is an internal SSD, this may not affect the overall result too much, but in my case, it would, at least on the Mac Pro, despite having installed a Samsung SSD 850 EVO 1TB on a Velocity Solo x1 PCI card with a SATA III socket.

Copying with dd from /dev/zero may be a valid method for hard disks, but is not entirely honest for SSDs, as that would indeed test the transfer speed over the USB or TB bus, but since SSDs treat empty sectors specially, by not writing them to their flash memory, that would be cheating. I wanted to measure the true effective write speed, and therefore I'd need to write random (non-zero) data.

Copying from /dev/randon is no solution either, because generating the random data is very slow. I ended up getting unrealistic write speeds around 10 MB/s.

Being the author of a versatile disk editor, iBored, I have instead added a feature to it: Random data is generated once in memory, and then written repeatedly to disk, to consecutive sectors. That gives me realistic results that also correlated fairly well to copying from /dev/zero (i.e. it's a bit slower as expected). The version I used has not been officially released yet, but it's the 1.2b5 in the downloads. The Read/Write Speed Test commands are found in the Disk menu. If you try them yourself, be aware that this tool is not fool-proof. If you accidentally choose the wrong disk, and you could erase your precious files! So, be careful and always have an external backup, just in case.

Due to caching done by the SSD, I also had to make sure to write a large amount of data, to avoid reading and writing just through the faster cache.

Performance Test Results

The graphic does not include the Enateck, Teorder and AKiTiO adapters, as I got them later. The Teorder performed similarly to the Delock 42486, and the AKiTiO similarly to the Delock 42510. The Inateck is about 15 MB/s faster than the ICY BOX on the MacBook (didn't test on the other Macs, yet).

I also made a test on a PC with USB 3 ports, using iBored again. There, read speed was 240 MB/s with the Delock 42486 and the other USB3-SATA II adapter, and 320 MB/s with the ICY BOX. Write speed was only getting to 150 MB/s, though. That's odd, but a test with a regular Windows benchmark tool showed similar results.


  • The USB 3 performance with the Thunderbolt Dock is not as good as with the built-in USB 3 port on the MacBook Pro. That's not a weakness of the Dock, though, as far as I can tell, but a weakness of the iMac. I suspect its memory bus isn't up to it. When I connected the Dock to the MacBook Pro and then connected the ICY BOX to the Dock's USB 3 port instead of the MacBook Pro's port, I read at 330 MB/s and wrote at 300 MB/s, which is much closer to the built-in USB port.
  • The Delock 42486 USB 3 and the Teorder adapters are mislabelled in my opinion. Both supposedly support SATA III (i.e. 6Gbit/s) but perform like a SATA II (3Gbit/s) adapter.
  • My  Delock 42486 has a problem reporting incorrect power needs over USB, often just 24mA, when it needs 10 times or more, and that causes intermittend problems with some Macs but not with others. Delock support has not been helpful resolving this, insisting that it's not their fault, not even offering a replacement. This problem did not affect the performance testing, though.
  • The Thunderbolt adapters perform about as well or better as a good USB 3 adapter.
  • All the USB adapters have trouble reporting disk information such as name and serial number. While most at least report most of the disk's name ("Ultra II 960GB"), the ICY BOX only reports "ICY BOX" for any disk attached to it, the Inateck even just "2115". Only the Thunderbolt adapters are able to report the complete name ("Sandisk Ultra II 960GB") and the SDD's Serial Number to the Mac. These limitations with the USB adapters are minor, though, and will not affect performance or usability.
  • The Macs are also unable to report S.M.A.R.T. status for disks attached via USB - that's a known limitation of OS X / macOS. Again, with Thunderbolt there is no such issue.
  • UASP support may or may not be having a positive effect on performance. Obviously, the Teorder adapter, claiming UASP support, does not perform well at all. OTOH, the Inateck adapter is slightly faster than the ICY BOX, which may be due to UASP.


If you have a Mac that supports USB 3, there's hardly a need for the Delock Thunderbolt adapter.

The disadvantages of the Thunderbolt adapter (much higher price and having to use an extra power supply) have no advantages I can see.

The only exception I can think of is when you use disk in with that requires more power than USB 3 can supply. Though, my MacBook Pro, in System Profiler, under USB, tells me it can supply 1800mA, while the ICY BOX adapter with the SSD installed only requests 224mA.

It remains to be seen if faster disks (i.e. SSDs) are able to provide even more throughput, and if that would give the Thunderbolt adapter an advantage. However, I believe that SATA III, which is used by all three adapters as the interface to the disk, is the bottleneck here, though there is still a bit of room to fill that out (nominally, SATA III can transmit about 500 MB/s, and suppedly the SanDisk SSD can achieve that, but I have currently no way to verify this).

(9 Apr 17) After considering TRIM support, I've changed my mind on this: The lack of TRIM support in the tested USB 3 makes them not well suited for external SSDs as a Thunderbolt adapter that invariably supports TRIM. See below for more details.

If you have a Mac without USB 3 but with Thunderbolt, you still have two choices:

  • Either purchase a AKiTiO Thunderbolt adapter at about 125 € (as of 12 April 2017 at Amazon). That's probably the cheapest option in the short run.
  • Or get the Thunderbolt 2 Dock from Elgato at about twice the price, plus a cheap USB 3 enclosure for your disk. The Dock has the advantage that it equippes your Mac not only with three USB 3 ports but also with an HDMI port to connect a second monitor, an easier-to-access Audio port (which apparently is also higher-quality than that of some Macs) and even a microphone port. It is more bulky and requires a power supply, too (and here it makes actual sense). That leaves you with a slightly worse performance than with the Thunderbolt case, but therefore your external disk can be used even in older Macs that have no Thunderbolt, and even in regular PCs.

1. Update on 9 Apr 17:

@felix_schwarz pointed out to me that there's an advanced protocol for external disks called UAS (aka UASP). UAS can provide better performance USB (and, according to AKiTiO, apparently also over Thunderbolt), and Mac OS X supports UAS since 10.8, according to WP.

Of the adapters I tried, some claim UASP support, others do not. The results are inconclusive, as one adapter with UASP (Teorder) was very slow, whereas the one with it (Inateck) was slightly faster than the one without claiming UASP support (ICY BOX).

I just learned that AKiTiO makes various Thunderbolt products of interest. There is a Thunderbolt Dock with USB 3 ports as well as an External Thunderbolt + USB SSD that supports UASP. AKiTiO claims that this gives over 500 MB/s effective read speed with their 1TB SSD model, though their benchmark makes me wonder why the effective write speed is below 200 MB/s. Maybe they're using an older type of SSD that is not up to it. The Neutrino model seems to be better in this regard, but it does not offer a 1 TB SDD version. Also, the price is quite high compared to the setups I described above, although actual prices seem to be significantly lower that the given MSRP. If you need maximum performance, these SSDs may still be worth considering. (Update 18 Apr 17: I've now received and tested the AKiTiO Thunder SATA Go Adapter, see above).

2. Update on 9 Apr 17:

I had so far overlooked one important property that affects SSDs: Trim support.

It turns out that the tested USB 3 adapters provide no TRIM support. It's unclear why that is - I suspect that it's simply lack of support for it in OS X, similar to the missing S.M.A.R.T. support.

Missing TRIM support can will quickly lead to performance degradation of the installed SSDs.

If you only use hard disks with these adapters, that won't matter, but for SSDs, especially if they're used frequently for adding and deleting files, TRIM support is fairly significant to the performance. However, if the SSD is mainly used as a permanent backup device, e.g. as a Time Machine backup destination, TRIM support is not that important, even if files get replaced by TM once the disk is full.

Only the Thunderbolt adapters support TRIM. I've verified this by first checking in System Profiler whether the disk is even listed as "TRIM Support: Yes", and then by using the technique described in this Ars Technica article.

This realization inverts my earlier recommendation: Of the tested adapters, the ones with Thunderbolt are the best choice if you want lasting optimal performance, especially if you plan to boot from it regularly (which is a scenario for iMacs with only a HD installed, to speed up their system with the permanently connected SSD that'll host the system software). Of course, that still requires that you enable system-wide TRIM support in OS X / macOS, e.g. by using the "trimforce" command as explained in the Ars Technica article.

There is even an awkward work-around for missing TRIM support, in case you decide to go with an SSD and USB 3 regardless: Regularly fill the free space on the disk with a file that just contains zeros, and then delete the file again, as that'll tell the SSD that this space is actually unused, and should make the disk perform better again for a while. This can be done using iBored (menu Tools, Erase Unused Disk Space...) or with this Terminal command:

cp /dev/zero "/Volumes/YourSSDsName"

Once the copy has finished, filling up the entire disk, delete the file again:

rm "/Volumes/YourSSDsName/zero"

(Update 18 Apr 17): BTW, I now learned how to issue the TRIM command (only for supporting adapters, i.e. Thunderbolt) and will soon add this feature to my disk tool iBored.

28 March 2017

Xojo: How to improve performance when using WeakRefs

If you're building tree structures with leaves (children) that shall keep a reference to their branch (ancestor, parent) they're attached to, the simplest (but not smart) solution is to store a direct references to the parent object in the child object.

Imagine this node class:

class Person
  property name as String
  property parent as Person
  property children() as Person
end class

And then you'd have a Constructor like this:

sub Constructor (parentIn as Person, nameIn as String)
  self.parent = parentIn
  self.name = nameIn
end sub

and create new children like this:

sub AddAChildNamed (name as String)
  dim newChild as new Person (self, name)
  self.children.Append newChild
end sub

This is not optimal because you'd get circular references (parent references children who in turn reference back their parent). And since Xojo uses ref-counting for keeping its objects alive, and does not have a separate garbage collection task, removing all references from your main code to the top parent won't free all those Person objects, because they still keep referencing each others, thereby keeping themselves alive.

One solution would be to add a "FreeAllChildren" method that you'd have to call right before are ready to release the entire tree, but that's not very elegant (it's the fatest solution, though, if you get this right).

A more natural and fool-proof way is to use weak references for the child-to-parent connections, like this:

class Person
  property name as String
  property parent as WeakRef // **changed**
  property children() as Person
end class

sub Constructor (parentIn as Person, nameIn as String)
  self.parent = new WeakRef (parentIn)
  self.name = nameIn
end sub

To access the actual parent, you'd use code like this to temporarily re-create a hard reference to the parent object:

  dim myParent as Parent = Parent (self.parent.Value)

With the above changes, you won't have circular references any more.

However, it's not very efficient. If you have many 1000s of parent objects, you'll get as many WeakRef objects. And the downside of WeakRef is that it involves a search operation to locate the actual parent object when you use the WeakRef.Value function. And the more WeakRef objects you have, the more time it takes to look up these values.

Now consider this: Usually, with these parent-child relationships, there are more children to a parent than vice versa. In the above code, every child creates a WeakRef, often for the same Parent. And that's what we can optimize for:

Instead of having the child create its own WeakRef, let us have the parent provide it, as it'll remain constant.

Change the constructor to accept a WeakRef:

sub Constructor (parentRefIn as WeakRef, nameIn as String)
  self.parent = parentRefIn
  self.name = nameIn
end sub

add a new property to the class:

class Person
  property selfRef as WeakRef

and update the way a child gets added accordingly:

sub AddAChildNamed (name as String)
  if selfRef = nil then selfRef = new WeakRef (self)
  dim newChild as new Person (selfRef, name)
  self.children.Append newChild
end sub

And that's all.


If you use WeakRefs, and if you create many WeakRef for the same object, consider creating the WeakRef only once and cache it, to gain runtime performance.

24 March 2016

Using USB barcode scanners on Macs with Xojo

Barcode scanners translate 1D and 2D bar codes, such as Code128 and QR Code, into readable characters.

Usually, handheld bar code scanners with a USB interface simulate a keyboard by default. With that, they work out of the box: If you scan a code with such a reader, it'll send keyboard strokes to the computer.

This makes them easy to use, but it also causes some issues:
  • If you have a program that expects input from a scanner, you have to make sure the typed characters are properly caught by your program and not go somewhere they don't belong. E.g, if the user switches accidentally to another program and then scans a bar code, not only will your program not be able to get the code from the scanner, but the keystrokes may cause odd effects in the the program that's active then. It's like your cat walking over your keyboard and wonder what'll go wrong.
  • Even worse, it can pose a security threat. Hackers have been able to show scanners customized bar codes that contained key strokes that would lead to remote controlling computers that the user was not supposed to access that way. Imagine a so-called "kiosk" terminal that's supposed to run only a particular application but gets quit by a alt-F4 or cmd-Q keystroke, then opening another program such as a Terminal to mess with the computer. In other words, using a bar code scanner in the "keyboard emulation" mode opens the computer to all kinds of abuse that you may have wanted to prevent by not giving the user a keyboard and mouse.

One way to prevent these issues is to put the scanner into a mode where it does not emulate a keyboard any more. That mode is usually called a "HID POS" or "USB OEM" mode. In such a mode,  the software needs to run special code to communicate with the scanner directly. Another option would be to not use the scanner's USB interface but rather use a serial interface (many scanners can be used with either) and then use a Serial-to-USB adapter to connect the serially operated scanner to the Mac or PC.

In case you want to use the USB HID POS mode with a scanner (e.g. GoDEX models), I've posted some details and a demo project on the Xojo forum a while ago: https://forum.xojo.com/24557

In the meantime, I've also managed to scanners from Datalogic (Gryphon 44xx models) and similar scanners using the sparely documented IBM / Toshiba "OEM" USB protocol. If you are interested in supporting those, contact me directly.

31 October 2015

Searching multiple Xojo projects

Finding that useful method or class in one of your numerous Xojo projects

If you've used Xojo (Real Studio) for a while, you'll probably have collected more than just one project.

And each of these projects contains unique code, and some of that may even be re-usable for other projects.

How do you keep track of all the code and methods you've written in the past? E.g, you do remember you've once written that nifty function to count the words in a string, but where is it?

I am going to show you several ways how you can find code across multiple projects.

Mac OS X only: Use Spotlight

For Spotlight to be able to search Xojo projects, it needs to be able to understand their content so that it can extract any text from it and add it to its searchable database. By default, textual project formats (.xojo_project, .xojo_xml_project, .xojo_script, .xml, .rbvcp, .rbs) are regarded as text files and thus be automatically scanned by Spotlight. However, it won't scan binary file formats such as .xojo_project and .rbp - to make them usable by Spotlight, a so-called Spotlight Importer is needed.

Real Studio and early Xojo IDEs did include such an importer, but recent versions don't any more. However, as I wrote this importer originally, I've now made the effort and updated it for the latest Xojo file format (as of Xojo 2015r3). So, if you're on a Mac I suggest you install this latest importer for Xojo if you like to use Spotlight:

This Spotlight importer will remember all the class and function names of all your projects, along with any other text it they contain.

Let's assume you have a method names "CountWords". If you enter "countwords" into the Spotlight search field, you should see the project file listed in the results.

However, the results may be cluttered with a lot of other types of files not related to Xojo.

So, here's how you tell Spotlight to only show Xojo project files:
  • Open the Spotlight Finder window (e.g. by pressing cmd+F in the Finder).
  • If you do not see a popup menu under the Search: row, click the [+] button on the right.
  • Click on the leftmost popup menu (which probably reads Kind) and choose Other...
  • A new sheet dialog appears in which you can select a search attribute.
  • Find the "File extension" attribute. Tick its checkbox under the In Menu column.
  • While you're at it, also find the "Xojo class" attribute and tick its checkbox as well.
  • Then click OK to dismiss the dialog.
Now, whenever you use this Find window, you can choose File extension from the popup menu to limit the found items to file types that your Real Studio projects use: rbprbbas, xojo_binary_project etc.

Furthermore, if you are looking for a particular class, you can choose the Xojo class attribute and enter the (partial) class name there, leaving the main search field empty. That way, you'll see all the projects that contain that class.

All platforms: Searching inside VCP and XML projects

If you are used to saving your projects in the textual VCP (.rbvcp or .xojo_project and related class files) or XML (.xml or .xojo_xml_project) format, then you may have success performing simple searches using Spotlight on OS X or Windows Search on Windows.

If you need more control over what files are found, so that you don't get lots of false results from non-RB files, here are a few 3rd party programs you could try:


  • TextWrangler is free and has a Multi-File search in which you can set up filters with the extensions you want to search.
  • EasyFind mainly finds files by names, but if you specify to search files ending in .rbbas, .xojo_class etc., then you can also search for their content. (I'd have liked to tell you that my program Find Any File could also be of help here, but it doesn't search file contents - yet.)


  • UltraFileSearch does a good job. You'd use the Wildcards search in the Files and Folders tab, searching for "*. xojo_*", and then enter the name of the function or code under the Containing Text tab.
Unfortunately, most of these methods (apart from TextWrangler), including Spotlight, won't show you the actual code snippets, and the Xojo IDE also won't open ".xojo_window" and similar files alone, so you'll have to find their main project file and double click that, then use the IDE again to search for the function name.

All platforms: Arbed

Arbed is a tool that was developed mainly for dealing with larger projects and the tasks around them. One of its many features lets you search inside project files within a folder and all its sub folders (and while some features require a license purchase, this is a free feature).

Contrary to Spotlight, Arbed runs on Windows and Linux as well, so you can use this even if you do not use a Mac.

The advantage of Arbed over the above methods is that Arbed can show you the results conveniently, so you don't have to re-enter the search in the IDE before you know if the results contain what you were searching for.

Launch Arbed and then either drop a folder onto its Search Multiple Projects box or choose Find on Disk... from the File menu:

You can then enter the search text, even as a Regex formula, and also choose the folder in which you want to search:

Arbed then lists all found projects:

Double clicking any of the found items opens a new window showing that project, along with a window listing all occurances. Clicking on any of the occurances will show you the related project item (class, function etc.):

Right-Clicking on an item in the project list gives you the option to reveal the file on disk or open it in the Xojo / Real Studio IDE.

21 September 2015

New: OS X Automator Action plugin for invoking Services

When I started adding Services to my apps (Find Any File, iClip), I realized that this was not enough for users who want to write workflows with Automator or Applescript because there seems to be no way to invoke Services from Automator nor AppleScript.

So I wrote an Automator Action that allows you to run any Service that operates on Text or Files & Folders:

Find instructions and the downloads on github (including Xcode source).

License is unrestricted, i.e. totally free.

17 September 2015

Xojo SQLite databases: Enabling ICU for full Unicode support and adding custom functions

Unicode Support (ICU)

Xojo's default Database class REALSQLDatabase (or its replacement SQLiteDatabase), which is based on sqlite, does not support international (non-english) characters. For instance, if you try to use the SQL function upper on a french phrase such as Tête-à-Tête, you will get TêTE-à-TêTE, i.e. only the plain ASCII chars (a-z) will get uppercased but not the special french chars. This missing unicode support also affects sorting and related comparison functions.

The usual solution to this is to compile the sqlite3 library with ICU support and then include this lib in your program. This is not a viable option with Xojo because you'd also have to write your own Database plugin to make the sqlite lib usable in Xojo (while the sqlite lib is open source, Xojo's plugin is not, or this would be less of an issue). Furtunately, though, Xojo's plugin allows dynamic loading of extensions (like plugins) into the sqlite engine.

Still, you need to have a readily built ICU lib for sqlite before you can load it. And unfortunately, there is none publically available as far as I could see, at least not for OS X. Therefore, I'm making my own built version available, along with the source code. Mind you, only for OS X, though, as I do not have the time to make and test versions for Windows and Linux as well. I believe, however, that you can more easily find .dll and .so files of the ICU-sqlite lib on the web and use them with these instructions.

Custom SQL Functions

I recently also started using sqlite's FTS (Fast Text Search) functionality. At the end of the FTS page, examples are shown on how to implement a smart ranking feature when searching for text. To implement it, a custom ranking function needs to be added to the SQL engine. Again, this is usually done by writing C code, and then load that as an extension. But since my code that controls what needs ranking is written in Xojo, I'd need to have this ranking function written in Xojo as well, in order to have access to the other data I've collected in my program relevant to performing the ranking.

Adding a custom functions to sqlite is fairly easy: Invoke the SQL command "SELECT load_extension('libfile_path', 'init_function_name')" is all it takes.

It would be great if we could simply pass the path to own code that's been compiled by Xojo, along with a custom init function we've prepared for this. Unfortunately, that doesn't work because all methods in Xojo are local (non-exported), making it impossible for sqlite to find any functions in our code.

Extension Stub

We need to use a small helper code file that provides the init function we need to implement. Here's the C code (I've removed some of the non-essential parts):

    #include <sqlite3ext.h>

    typedef __cdecl int (*callback_func) (const void *pApi, void *db);

    static callback_func callback;

    void SetCallback(void *p) {
        callback = p;

    int sqlite3_extension_init (
      sqlite3 *db,
      char **pzErrMsg,
      const sqlite3_api_routines *pApi)
        if (callback) {
            callback (pApi, db);
        return SQLITE_OK;

The first relevant part here is the SetCallback function. This is a function we'll call from Xojo before loading this extension into the sqlite engine. We pass it the address of a Xojo function in which we'll handle the installation of our custom functions. The SetCallback function stores the address of our Xojo function for later use. Then, once we load this extension using the "SELECT load_extension" command, sqlite3_extension_init will get called. Its code then invokes our previously set Xojo function through the stored callback function pointer. That's basically all it does.

(This code should be portable for Windows and Linux as well - the actual source code contains some extra instructions for Windows DLLs).

Once the extension gets loaded and our callback function gets called, we perform the registration of the custom functions. How that's done is documented in the SQLite docs.

Using the sqlite API

There is another challenge, though. Our custom function certainly wants to be able to receive parameters, return values and maybe even perform database access operations on its own. For this, sqlite provides a large set of functions through its API. The problem is: While it's fairly easy to invoke them from C code, it's rather tedious to do in Xojo, because they're accessed through an array of function pointers (note the pApi parameter the sqlite3_extension_init gets passed - that's where all those pointers are located). In C, macros take care of making them convenient to call, but in Xojo it means a bit more work for us.

Basically, to call a function, e.g. sqlite3_result_int, we must first find out which place its function pointer occupies in said array. And that can only be found out by looking at the header file (sqlite3ext.h) and counting through the positions. Turns out that sqlite3_result_int is at index 82, for instance.

I have then located some of the other functions to access passed parameters and to set returned results and make proxy functions for them to make calling them easier. It should also make it clear how to add more API functions if needed.

Download it all

To sum this all up, I'm now presenting to you a ready-made Xojo project along with compiled libs for both extensions and their source code with Xcode project files. The project shows how to load both extensions and then tests the results, proving that both the unicode extension and the added custom functions work. Note, however, that it only works on OS X out of the box. I've tested it with both Real Studio 2012r2.1 and Xojo 2015r2.2.

The proejct uses the older REALSQLDatabase class. If you replace it with the newer SQLiteDatabase class, you need to also add this line each time before you try to install an exension, or you'll get an "Not authorized" error:

db.LoadExtensions = true

I believe the extensions can also be built for Windows and Linux without too much effort if you follow the sqlite instructions under the topic "Compiling A Loadable Extension". If you succeed in this and like to share your results, I'm happy to include them in my downloadable file or link to yours.

As usual, my demo code is free of restrictions. Use it as you like.

Download here.

Alternative by MonkeyBread Software

MonkeyBread Software has recently added ICU support to their SQLite plugin as well.

17 May 2015

Ricoh / Pentax Flucard reverse engineering

I have done some quick reverse engineering on the FluCard (a wireless memory card for Pentax DSLR cameras), revealing some additional commands it offers.

I'd like to do more work on this, such as actually decoding some of its busybox code, but have no time for that currently.

Details are here: http://www.tempel.org/Photo/FLUcard

08 January 2015

OS X Kernel Hacking (fixing the psignal kernel bug)

At the German Macoun conference in Oct 2014 I gave a talk about rootkits and hacking the OS X kernel.

This article is a summary and follow-up on this topic.


Since Yosemite (OS X 10.10), kernel extensions (KEXTs) need to be code signed or they won't get loaded any more. The code signing certificate needs to be optained from Apple, and it's not easy to get it - and even if you have one - once you do something with it that Apple doesn't approve of, they can simply invalidate it, preventing further use of KEXTs signed with that certificate.

As usual for Apple in recent years, we developers (at least in my case who received said certificate from Apple) are not told clearly what would be acceptable by Apple. Clearly, Apple should use this to block malicious software from getting installed (such as rootkits and "jailbreaks" that undermine the security of the user's data and resources). But what about things that benefit users but that Apple simply does not agree with due to their desire to control everything lately? Is patching bugs allowed? Is patching features allowed as long as it's desired by and beneficial to the users, with their express agreement? We don't know. But we'll find out a bit of that, I guess. See below.

It is possible to disable the signing requirement for KEXTs, though. This is done by setting a particular firmware parameter (see this StackExchange answer for instructions) on the computer where you like to load an unsigned KEXT. This setting is meant for developers so that they can test their KEXT without having to sign it each time (besides, performing code signing requires an internect connection, which may not always be available). But users can apply this setting as well, of course. For example, anyone having installed a 3rd-party SSD into their Mac may want to install a patched version of the disk driver in order to have TRIM support enabled (Apple only enables it for their own drives by default).

Patching a little bug in the Kernel

There's this bug in the kernel that affects signal handling in threads. The author of the article, Russ Cox, shows how to patch the kernel file on disk to fix this bug. That is a bit dangerous because, when one makes a mistake and tries to boot this system, the Mac may become inoperational, and fixing this is a bit tricky. Also, it's not everyone's pleasure to patch the kernel the way.

I now like to show how to perform the same patch dynamically, from within a kernel extension. The advantage of this procedure is that it can be enabled and disabled at will, simply by loading and unloading the KEXT. As it doesn't modify the kernel on disk but only updates it in memory, it should work with future kernel updates as well, patching the bug in there as long as it hasn't been fixed by Apple. Should the bug not be found by the KEXT, then the KEXT simply quits and doesn't alter the system further.

To make this work several 3rd party components are used:

1. For locating non-exported kernel symbols in memory: The Flying Circus framework.
2. For locating the to-be-patched code in possibly any kernel version without having to know the exact location or code for each kernel release. This is accomplished by using the diStorm3 disassembler.

The entire project can be downloaded here:

It contains an Xcode project with two targets: One builds the KEXT, the other builds the test program as provided by Russ Cox on this website. It also contains a readily usable KEXT signed with my own certificate.

To install the psignal fix into your OS X system, open Terminal and issue the following commands:

cd /path/to/OSX-psignal-fix-folder
sudo chown -R 0:0 psignal-fix.kext
sudo kextload psignal-fix.kext

That should have installed the patch. If you got an error, open Console.app and look for error messages containing "[psignal-fix]".

If you're happy with the functionality of the fix, you may also simply place it into /Library/Extensions/ and reboot to load it permanently. You have have to issue the chown command again after copying the kext to that folder, though.

To unload the KEXT, issue this command:

sudo kextunload psignal-fix.kext

If you have Xcode installed, you can also run the test to verify that the patch works. Open the project and select the "psignal-test" target from the popup menu in the window title. Run it and see its console output. This is the essential output without the fix:

500 signals received.
500 on main
0 on looper 0
0 on looper 1
0 on looper 2
0 on looper 3

And this is the output with the fix working:

500 signals received.
0 on main
129 on looper 0
123 on looper 1
128 on looper 2

122 on looper 3

The difference is that with the fix intact, the signals will occur nearly evenly distributed on all looper threads and none on the main thread.

If you build the KEXT (target "psignal-fix") yourself, you need to disable the KEXT signing requirement first if you're on OSX 10.10 or later, unless you have your own KEXT signing certificate, of course.


* Disable KEXT signing requirement : http://apple.stackexchange.com/questions/163059/
* Trim Enabler: http://www.cindori.org/software/trimenabler/
* Description of kernel bug: http://research.swtch.com/macpprof
* The Flying Circus: http://www.phrack.org/papers/revisiting-mac-os-x-kernel-rootkits.html

26 November 2014

Making CUPS printer drivers work again on OS X Yosemite (OS X 10.10)

I recently found out that some printer drives using the CUPS printing system stopped working on OSX 10.10 (Yosemite) that worked fine with the previous OSX Mavericks (10.9).

After some googling around I found various suggestions, but none seemed to help. In the end, it turned out that I had to follow THREE different suggestions to make it work. To save you the same odysee, here's my recipe for making printer drivers work again that stopped working on Yosemite:
  1. Remove any troublesome printers from the "Printers & Scanners" System Preferences.
  2. Re-Install the printer driver.
  3. Fix the permissions of the installed cups drivers by starting the "Disk Utility" program, selecting the startup disk and clicking on "Repair Disk Permissions".
  4. Turn off the tighter Sandboxing checks in Yosemite with these two Terminal commands:
    • sudo sh -c 'echo "Sandboxing Relaxed" >> /etc/cups/cups-files.conf'
    • sudo launchctl stop org.cups.cupsd
  5. Connect the printer and re-add the printer in the "Printers & Scanners" settings.
Hope this helps. If it does, please spread the word. If it doesn't, let me know please.

And, if you found this useful, also check out some other work of mine, such as:

  • Find Any File - The program that finds the files that Spotlight doesn't.
  • iClip - Improves the workflow with text snippets and the clipboard in general.

04 November 2014

Find Empty Folders - a free OS X tool

Today someone asked me if I knew a way to find and delete empty folders, even if they contain an invisible .DS_Store file.

I googled and found nothing.

So I quickly whipped something out with Real Studio.

The result can be found here:


18 July 2014

Something adventurous for a change (Nelly Cootalot)

Today I'm just helping promote someone else's work.

With this blog being directed mainly to other (programming) geeks, I think you might even be the right audience:

Did you enjoy playing Monkey Island, Leisure Suit Larry and other funny point-and-click adventures? (Though, the first Larrys I played were running still on DOS, without mouse support.)

And do you enjoy british humour?

And, possibly, are you on Steam?

Then have a look at this hilarious Nelly Cootalot promo!

The game is designed by a british film maker, and it's not his first go at an adventure, however this one is made much more professionally. Support his quest for getting the game onto Steam - don't be a coward!

I support this because long time friends of mine are involved in the development and distribution, and at some time I was almost becoming part of the dev team as well. I'd have loved to but I'm already busy with so many other projects I couldn't take on another one.

The game already got its basic funding through Kickstarter earlier this year, and now this is a step up to get onto Steam, giving it a much wider reach.

Also, it'll run on Macs, which is always good.

07 July 2014

OS X: Moving Files to Trash from your own application, with optional user interaction!

When you're writing a interactive Mac OS X program, it's possible that you need to delete files.

And if they're files the user has access to, then it's often a good idea to move them to the trash instead of erasing them right away, in order to give the user the opportunity to restore them again (saving you the need to provide some kind of Undo command for the destructive operation).

Moving items to the Trash is rather complex, and so it's best to just let the Finder perform that task.

The Finder is, fortunately, still scriptable, and so this task should be easy. (I must say, though, that I don't know if you can get such an app into the App Store - if you know, please comment.)

The challenge is to have the Finder tell the user if there's a problem, such as when the item is locked or when an admin login is needed.

Now, if you're trying to be lazy, you'd just write an AppleScript and invoke that from your application (this is especially easy to do in Real Studio / Xojo, but not much harder in ObjectiveC, either).

The downside of using an AppleScript is that it won't show these dialogs - if the item can't be moved to the trash, the function will return and give you an error message telling you about the problem.

To solve this, you need to invoke the AppleEvent directly, without going through an AppleScript.

The Xojo code would look like this, simplified:

  dim ae as AppleEvent = NewAppleEvent ("core", "delo", "MACS")
  dim list as new AppleEventDescList
  list.AppendFolderItem (theItemToDelete)
  ae.DescListParam("----") = list
  ok = ae.send

However, this won't give us the user interaction either, yet. That's because Xojo's Send command is lacking the options that the OS X API offers.

We have to call the native AESend() function via a declare, instead:

  const kAEWaitReply = &h00000003 ' sender wants a reply and will wait
  const kAEAlwaysInteract = &h00000030 ' interact where appropriate
  const kAECanSwitchLayer = &h00000040 ' interaction may switch layer
  declare function AESend lib "Carbon" (ae as Integer, reply as Ptr, 
    sendMode as Integer, prio as Integer, timeoutTicks as Integer,
    idle as Ptr, filter as Ptr) as Integer
  dim sendMode as Integer
  sendMode = kAEWaitReply + kAEAlwaysInteract + kAECanSwitchLayer
  dim err as Integer = AESend (ae.Ptr, nil, sendMode, 0, 0, nil, nil)
The same can be applied to copying files with user interaction as well - use the "clon" instead of the "delo" value for the AppleEvent.

I've made a REALbasic / Xojo project with both a MoveItemsToTrash() and CopyItems() function for your own use. Download here.

18 May 2014

Using the Cocoa animation API with Xojo

I've just published a Xojo / Real Studio class for doing smooth fade/mode/resize animations with windows and controls.

Find out more about it here: http://www.tempel.org/RB/CocoaAnimation.

26 March 2014

Customizing Xojo's controls on OS X with Cocoa

This article is about some advanced techniques for Xojo users that write OS X Cocoa apps.

If you are using Xojo to build OS X Cocoa applications, you can customize the controls much further than Xojo offers directly.

Let's take the PopupMenu. It's fairly limited in appearance (one look only) and capabilities (no icons in the menu, for instance).

Changing the appearance of such a control is easy because for most of Xojo's controls there is a Cocoa control equivalent, and you can directly alter those Cocoa controls from your code.

First, find out what you like to change in a control. You could look up Apple's documentation for the NSPopupButton, for instance. An easier way is to use Xcode as you can then see immediately the results in Xcode's interface designer.

Get Xcode from developer.apple.com, launch it, and choose New Project... from the menu.
Choose OS X - Application - Cocoa Application. Click Next. Give the project a name and check "Create Document-Based Application". Click Next and save it, e.g. to the Desktop.

In the browser menu on the left, select "Document.xib". That will show you the window editor, similar to Xojo's layout editor.

At the right of the Xcode window, at the bottom, there is a list of controls to choose from:

Find the "Pop Up Button" and drag it into the layout window. The window should then look like this:

Click on the added button to select it, and on right side of the Xcode window, click on the properties button (the one shown in blue here).

You will see the various properties you can alter.
For example, let's change the Style to Inline. Do that and see how the appearance of the popup menu changes:

Now let me show you how to do that in your Xode program.

I assume you know how to add a PopupMenu control to a window in Xojo.

Add an Open event to the PopupMenu in the window, and add the following code:

  Declare Sub setBezelStyle Lib "Cocoa" _
    Selector "setBezelStyle:" (obj as Integer, v as Integer)
  const NSInlineBezelStyle = 15
  setBezelStyle (me.Handle, NSInlineBezelStyle)

If you run your Xojo project, the popup button should now have the changed appearance.

However, it might be that the up+down arrows are centered, not on the right as they should. If you look at the Xcode project and play around with the settings for the Popup control, you'll see that the Arrow property controls this - if it's set to "Center", it's placing the arrows in the center. Let's see if we can fix the Popup menu in Xojo to have its Arrows set to "Bottom" instead. Add these lines:

  Declare Sub setArrowPosition Lib "Cocoa" Selector "setArrowPosition:" (obj as Integer, v as Integer)
  Declare Function cell Lib "Cocoa" Selector "cell" (obj as Integer) as Integer
  const NSPopUpArrowAtBottom = 2
  setArrowPosition (cell (me.Handle), NSPopUpArrowAtBottom)

Now I'll explain what's going on:

setBezelStyle is a function of NSButton, which is the super class of NSPopUpButton. Unfortunately, Xcode does not tell us that its "Style" property is actually called "bezelStyle", so I had to search the documentation for it anyway. me.Handle is the Xojo control's reference to the Cocoa object (i.e. the NSPopUpButton object), and we've then declared a Xojo function with the name setBezelStyle that invokes the "setBezelStyle:" selector, passing it an Integer value. Note that for every Cocoa property you find, there is always a setter function named after this pattern, e.g. "name" -> "setName:".

Getting access to the Arrow property was a little more complicated. Neither NSPopUpButton nor any of its super classes provides a function to set this property. Instead, it's found in the NSPopUpButtonCell class. To get to values of this class, we need to use the cell function (found in the NSControl class).

If you do not like to figure all this out yourself, you can also take a look at MacOSLib or use the MBS plugins. Especially with MBS it's very easy to acess these Cocoa control properties  or add icons to a PopupMenu's items.

But wait, there's more!

This is getting pretty advanced now.

Cocoa is built on Objective C, which, contrary to C++ and Xojo, lets you get right down to the lowest level where you can call functions (called messages in ObjC) by name. From a String.

Which means that you can store the function name you want to call in a String, or even read it from a file, and then invoke it.

Here's an example to show you what I mean. The above code for setting the style could also be written like this:

  Declare Function NSSelectorFromString Lib "Cocoa" _
    (aSelectorName as CFStringRef) As Ptr
  Declare Sub objc_msgSend_int Lib "Cocoa" Alias "objc_msgSend" _
    (obj as Integer, sel as Ptr, v as Integer)
  dim f as String = "setBezelStyle:"
  objc_msgSend_int (me.Handle, NSSelectorFromString(f), NSInlineBezelStyle)

See how "setBezelStyle:" is stored in a string variable here?

This Cocoa control also has an "alignment", so let's try altering that as well. The function name to change the value is thus "setAlignment:" (the ":" at the end is important and specifies that the function gets one value passed to it). So add this code:

  dim f as String = "setAlignment:"
  objc_msgSend_int (me.Handle, NSSelectorFromString(f), 1)

And with that, we've right-aligned the text in the popup menu.

With this procedure, you could now define customizations for your Xojo/Cocoa controls by specifying the properties and values in a file, and generically assign those properties in your code, without the need to write a special case for every property you might want to change.

Demo project!

Click here to download a demo project using above techniques.

18 February 2014

iClip 5.1.2 with Search and "Smart Sets" now in beta

For the past few years I've taken over the development of iClip 5, a Clipboard / Pasteboard tool for OS X after the original author abandoned it to instead get insanely rich with selling iOS apps and organizing Bundle deals.

iClip is now again in beta testing. The new version adds a search feature that also lets you create smart search sets in order to see ony specific clippings, e.g. only those from a particular applications.

If you like to help testing iClip, even if you've never used it before, please sign up with MacDeveloper, then subscribe to iClip there. The MacDeveloper site helps us (i.e. me and the folks at Irradiated Software) manage bug reports and feature requests, and in return you as a tester can collect points in order to get the software for free if you don't own it yet.

14 December 2013

New: OS X Prefs Editor for editing app defaults

Mac OS X 10.8 introduced a caching system for app preferences ("cfprefsd"). While this probably increases performances for apps, it makes it harder for developers to manipulate preference values quickly for testing, because making changes directly to the plist files in the ~/Library/Preferences folder does not work any more with editors such as "Property List Editor.app" and the similar one in Xcode.

Therefore, I've just written a new tool that works like the plist editor in Xcode but uses the CFPreferences functions to modify the values instead of editing the plist files directly.

If you've worked with "Property List Editor" or Xcode, you should have no trouble using my tool. It supports even Copy + Paste the same way. The only things I have not implemented (yet) are Undo and changing the type of existing elements (the reason for the latter is that REALbasic's framework does not offer a way to show a popup menu inside a listbox cell, and so I was too lazy finding a work-around for this to imitate Xcode's UI).

It even has one feature that Xcode's editor doesn't: You can delete all entries at once with Select All, then delete.

Also note that due to the use of the CFPreferences API, any changes you make are immediately committed to the preferences as seen by other apps, just like when you use the "defaults" command. For the same reason, changes made to the prefs by other apps (or with the "defaults" tool) will be seen in this tool after a change, a front switch or by pressing the "Sync" button. Hence, there is no Save command - updates are instant. For that reason Undo would be quite useful. Alas, I can live without it for now.

The tool is free for your use. Get it here:

It includes the REAL Studio (Xojo) source code as well. But I retain the copyright and distribution rights to the program, so please do not distribute the app yourself without getting my permission first!

To learn of new updates, use the RSS feed on this page or watch it on MacUpdate.

(Shameless plug: If you'd like to have a customized version, let me know, I also do work for money ;) )

30 October 2013

iTunes 11 modal alert dialog overflow

Do you tend to show modal dialogs to show issues to the user when processing his data? Well, think twice, or you might end up upsetting your users like Apple manages with iTunes 11.

See here for a screen recording I created when trying to update my iOS apps after about a week from the last update.

Blogger was supposed to show a video box here,
but that doesn't seem to work.

Admittedly I am suffering from iTunes 11 more than most users because I have two iTunes accounts: A German and a U.S. account. And I have downloaded iOS apps from both accounts. This leads to iTunes wanting to update all apps for both accounts at the same time, even if it only shows me the German updates.

However, before iTunes 11, this was still manageable. iTunes 11 made this much worse.

Also, there are other alerts in there that even the usual Apple customers with just one iTunes account might experience, such as the late-appearing "age restriction" notice that pauses downloading, and messages about problems with particular apps without naming those apps.

Overall, iTunes 11 does this all plain terribly. Apple used to excel at good user interfaces. Nowadays, I get the impression they take the kids right from college, let them work on these programs with no supervision from anyone who still has a clue about how to avoid such bad user experiences.

(I've also reported this to Apple via their bugreporter, see http://www.openradar.me/15350892)

11 October 2013

Understanding the Xojo (REALbasic) language

This is a small assortment of tips and tricks to make you understand the Xojo language better.

The differences between identifiers, keywords and reserved words

Even though the IDE paints them all the same color, they're not the same in the way you can use them in code code:
  • Identifiers are words you can use to name your methods, properties, variables etc.
  • Reserved Words (or reserved names) are all those that you can never use as identifiers. This includes: if, then, to, function, end, attribute
  • Keywords are a mix of the above. They are all those words that are pre-defined by the compiler. This not only includes the Reserved Words but also standard types such as String, Integer, Boolean and Color. But it does not include Date, for instance - that's a name from the framework, i.e. it's practically code written in Xojo that you do not see but that the compiler secretly adds to your source code when building your program. You can tell the difference if you use XojoScript (RbScript), which is a bare bones access to the compiler without most of the framework: While you can use Color there, the Date class is not available. (Note, however, that even XojoScript gives you a small framework to work with that includes some strings operations and the Print function, for instance).
Now, here's the interesting part:

Whereas you cannot declare a variable with the name attribute, you can declare one with the name boolean!

Here is an example that actually works:
  dim boolean as integer
  boolean = 2
  MsgBox "The value of the variable 'boolean' is "+Str(boolean)
While the above is quite some nonsense, it can be used in other ways, too, for instance in an enumeration. You can declare an Enum named Types and add the values String, Integer, Boolean to it. Then you can write:
  dim t as Types
  t = Types.Integer
BTW, for that very reason that reserved words (if, then etc.) are not the same as predefined types (String, Integer etc.), I tend to write the former always in lower case while I write types (both predefined and self-declared ones) always in upper case. So, while some may think my mixed case code writing is inconsistent, it actually follows rules based on the understanding of the language.

Boolean logic

I often see beginners not making best use of Boolean types.

For instance:
  if found = True then
could also be written as:
  if (not (found <> true)) <> false then
The second example is clearly unreadable and therefore quite counterproductive. The first one is, to some lesser extent, similar in that nature.

The recommended variant would be:
  if found then
Read it aloud. Doesn't that sound better than the first example?

In the same way,
  if found = False then
should be written as:
  if not found then
Think about this the next time when you choose a name for a Boolean property or variable. Name it so that it reads well in a "if ... then" sentence, avoiding "= false" and "= true" constructs. It clutters your code and makes it often less readable.

Similarly, you may want to avoid constructs like these:
  if me.Value = true then
    button1.Enabled = true
    button1.Enabled = false
Why not simply:
  // enable button1 only if this checkbox is checked
  button1.Enabled = me.Value
Though, some may argue that "me.Value" is not self-explanatory, thus "me.Value = true" is adding some context. Here's an alternative that's even more self-explanatory:
  dim checked as Boolean = me.Value
  if checked then
and, analogously:
  dim checkboxChecked as Boolean = me.Value
  button1.Enabled = checkboxChecked

27 September 2013

Using Xojo or Real Studio? Please help me test Arbed

I am close to releasing the next version of my tool "Arbed" for Xojo and Real Studio project files.

As explained in my previous post, I have added code that performs a self-check to make sure it doesn't miss anything when reading and writing a Xojo project in textual (VCP) format.

If you haven't yet, please download the latest beta from http://files.tempel.org/Arbed/beta/1.7.0/b13/ and launch it, then open your .rbvcp and .xojo_project projects with it. (Either use the "Open" command from the File menu, or drag and drop your projects onto its main "Arbed Drop Pad" window, into the "Project Editor" group box.)

If you get an error or verification failure message, please let me know and I'll fix the issue within a day or two.

01 September 2013

Lessons in working with file formats in flux

As you may know, my tool Arbed for Xojo (formerly known as Real Studio) is capable of reading and modifying Xojo's project files. This ability is used to perform often needed operations that the Xojo IDE doesn't offer (yet), such as comparing projects and single classes, scripted code replacement, code obfuscation, preparing code for localization and more.

I'm going to tell you a little about the challenges I have to deal with.

There are 3 different project formats supported by Xojo:
  1. Binary, aka RBP, aka RbBF, using extensions .rbp and .xojo_binary_project (sheesh!)
  2. XML, using extensions .xml and .xojo_xml_project
  3. Textual, aka VCP, using extensions .rbvcp and .xojo_project

Dealing with RBP and XML

The first two formats are structurally identical: The RBP format is nothing more than a more compact version of the XML format, using a binary representation of the XML by using 4-letter-codes for the tags and 4 byte length fields to indicate the size of the elements that follow. (Xojo engineers informed me that actually RBP was designed first, and XML is a translation of the RBP format.)

Arbed originally only supported these first two formats, and it was pretty easy to handle. The data was well structured and repetitive in a way that gives little room for (problematic) surprises. If you've ever looked at a xml (or html) file, you probably understand what I mean.

When Arbed reads a project in the RBP and XML formats, it modifies only the parts it well understands while leaving all the other data untouched. For instance, when it changes some source code inside a function, it only modifies the affected <SourceLine> elements. That way, it can safely modify the project even if Arbed doesn't understand everything that's inside the project. And believe me, even though XML is apparently self-explanatory, there are a lot of unexplained things in there that just leaves one guessing.

The blame for this can be put with Xojo for not documenting this. I guess they do not even have an internal documentation. As so often in rushed software engineering, the code is the only documentation (for that, it doesn't even deserve the term "engineering"). When I am about to write complex code, I usually start with documenting and specifying it first, so that I (and others that might join the project) can later reder to that. It really helps, and should be obvious. However, most people do not follow this simple rule. Many even never learn. It even starts with documenting your single subroutines. See my little article on coding guidelines.

Converting between RBP and XML formats.

It gets a little more complicated when using Arbed's Convert operations. They's available in the main window (titled Arbed Drop Pad). For instance, to convert a RBP to a XML file, it has to know how the 4-letter-codes in the RBP file translate to the XML tag names.

It happens every once in a while that Xojo adds new features to the project file format that lead to using newly names element tags. For instance, when the Web Edition was added, a new XML tag named WebApp got added. Now, while Arbed doesn't need to know what this value means, it needs to know how to translate it between the XML tag name and the RBP tag code. Therefore, when you're using an Arbed version that doesn't know this translation yet, it'll tell you about it if you ask it to convert a newer project file with yet-unknown tags in it. I will then have to update Arbed with the new code (which is easy to do, I just have to save the same project in both XML and RBP format in the IDE and see look for the code in question in both files).

Overall, working with RBP and XML projects in Arbed is therefore fairly safe and fast.

Arbed only rarely needs updates, e.g. when a new tag code gets added, and then only to make the Convert functions work (which are not even really necessary because you can just use the IDE to save a project in a different format).

Dealing with the VCP format

The VCP is a different and much more scary beast.

In theory, it's just plain RbScript code, which is fairly well specified (even though Xojo has failed to present a syntax/grammar spec for their language for 15 years now, it's fairly well understood by me by now, and my background in compiler design helps there, too).

I even have a fully working RbScript parser that I hope to use soon to implement some great features such as method name obfuscation (i.e. rename all custom method names in your source in order to foil reverse engineering attempts), automatic code reformatting and dead code identification and removal.

But the reality is harsh. The VCP format has a lot of inconsistencies and hard-to-understand behaviors that make parsing it challenging.

Some examples:

Properties of Controls vary in representation

Recent Real Studio versions tended to write some integer values of Control properties as floating point numbers. If you're using version control you may have noticed that a control's Width and Height were sometimes shown as 1.6e2 when you did input 160. Recent Xojo version seem to have finally fixed this.

Odd spacing in method declarations

This is how a normal Method declaration looks like when using qualified type identifiers:
Sub Foo(x as a.b)
And this is how an External Method looks like:
Declare Sub extmethod Lib ""  Foo(x as a . b)
Note the inserted blanks around the period. They're syntactically allowed but I didn't type them - the VCP format inserted them automagically.

Troubles with Attributes

Let's enter 3 attributes with, admittedly, some unusual values:
  • test: a
  • t2: \x22 (which gets automatically quoted into: "\x22")
  • t3: b=," (which gets automatically quoted into: "b=,""")

Now let's see how they look like in the VCP format.

This is how a normal Method looks like:
Attributes( test = a, t2 = "\x22", t3 = "b=,""" )  Sub Foo()
Alright. That's fairly readable. (Though what's with those spaces inside of the parentheses? Also note the double space before "Sub".)

Here's a similar Delegate declaration:
Attributes( test = a, t2 = "\x22", t3 = "b=,""" ) Delegate Sub Foo()
Looks pretty identical, doesn't it? And now we can also understand that double space: It makes room for plugging a "Delegate" word in there :)

But wait. Method declarations are surrounded by #tag markers, providing extra information that the RBP and XML formats usually store in extra elements or attribute fields. Here's the one for a normal Method:
#tag Method, Flags = &h0
Okay. Now the one for a Declare:
#tag DelegateDeclaration, Flags = &h0, Attributes = \"test \x3D a\x2C t2 \x3D "\x22"\x2C t3 \x3D "b\x3D\x2C""""
Uh, what? Not only it includes the Attributes that are already present - and more much readable - in the declaration source code line, but it's only appearing in this special "delegate" method declaration but not in a normal method - even though they have the same syntax and should thus be created the same way.

But it gets even weirder: The above was created in the latest Xojo IDE. When doing the same with Real Studio 2012r2.1, the above line does not include the Attributes in the #tag line. So, someone must have accidentally added this nonsense just recently, and only to Declares, not to normal functions. Or maybe it wasn't an accident. In any case, it's quite a mess.

Encoding horrors

As a final example, let's focus on the odd encoding of attributes in the #tag line. It's obvious that it tries to escape some codes so that they may also be used inside the attribute values. It looks like a homemade algorithm, but appears to work pretty well. As you can see above, I've used characters as values that it also uses for escaping to see if I can break it. But I couldn't. That is, almost. When I set the value of an attribute to something containing line feeds (returns), then the IDE crashes hard or the compiler spills out inexplicable error messages. Oh, and later I found that if you put \x22 into an attribute's value, it'll be converted to a comma next time you open the project. Doesn't happen in RBP format, of course.

Sure, the test with the return in the attribute value was something that's unlikely to happen, but it shows that the whole thing is put together with little understanding how escaping random data should be done. Heck, since the Xojo IDE is programmed in Xojo, they could have have just base64-encoded the string, or used quoted-printable encoding. That's what they're meant for, and both are functions that Xojo provides anyway! But no, a new technique had to be invented: Something unreadable and prone to crashing.

What this all means

Why do I even bother with this painful stuff?

Well, here's the thing: In 2012, I enhanced Arbed to read and write the VCP format.

It was one of the worst decisions I had made in past years, because it took me several months of intensive work to get even close to being usable. And since then I spent many more days working out all the kinks. These kinks are what you see above: Hardly anything that looks similar also behaves similarly. Behaviors change in different RS and Xojo versions, sometimes without making sense. I had to identify and special-case them all, for each single IDE release, so that Arbed generates the exact same output that the IDE does.

I am still not sure that it was worth the effort, because I do not use the VCP format for myself, and until then, all the features I added to Arbed were needed by myself. But when I started selling Arbed, I believed I had to add this feature because many expected it to just work.

Now, every time Xojo changes the output format of the VCP files only slightly, it might mean that my own code could miss it. For instance, some files use undocument flag values, such as &h1000. No idea what it's for. Formerly, I had just generated these flags from the declaration source line. But now, sometimes, the Flag in the #tag line contains more information and I have to preserve it. Special handling galore!

Arbed's VCP output needs to match that of the IDE exactly

The issue is that, contrary to how Arbed can edit just specific parts of a method's source code in the RBP and XML formats, it can't do this with VCP files. Arbed's project modification code was written to work on the RBP/XML format directly, in order to keep anything unmodified that the user doesn't directly alter. For this functionality to work with VCP files, Arbed has to convert it internally into RBP format, so that my existing code can operate on it. Then, when the user saves his changes, Arbed recreate the VCP file from the updated RBP code (Arbed only rewrites the files that were affected by the changes). But that means that I have to recreate the entire VCP file from RBP data, instead of just modifying the source lines or whatever else was changed by the user as I did with RBP/XML files.

All this requires that Arbed understands every detail of a VCP file in order to properly convert it to RBP and back, internally.

Potential for damage

And that's what bit me a few times already in the past. I would miss small changes in the format, such as that color constants were originally written as hex values (&h...), but recently Xojo changed this to using &c... codes instead. My code was not prepared for this and would then write any color back as a 0 (black).

Basically, Arbed could, unknowingly, damage your VCP files if you edited and saved them in Arbed. You'd notice eventually, but Arbed should notice this before it does such damage.

So, how do you make sure that your code that reads and writes an external file format doesn't damage it just because the file format introduces changes that you are not aware of, yet?

The solution is: When reading the input, recreate the would-be output from it right away and then verify that both match. If there's a mismatch, your code is prone to damaging the file when writing it back.

Simple as that.

Arbed 1.7 is made safer by performing a self-test on any VCP file it reads.

Therefore, from Version 1.7b9 on (which I'll release shortly), Arbed will verify its own VCP read/write code to make sure that it entirely understands the project it reads and (optinally) writes.

So that, when you save a project, anything you didn't specifically change, will remain the same in the rewritten VCP file. So that when you use version control (git, mercurial, subversion) with your VCP files, it won't show lots of unrelated changes after Arbed wrote the file back (even if it's still valid to Xojo). Something Arbed may do even better than the Xojo IDE itself (ever seen those TabStops disappear and reappear randomly?)

Arbed accomplishes this self-check by converting the read project into its internal RBP format, then back the VCP representation, then comparing its VCP output with the original file. If it finds a mismatch, it warns the user that the data was not fully understood. It also creates a file on the user's Desktop containing the specifics of the rendering differences, so that the user can tell what's gone wrong (allowing the more experienced user to decide if the changes are benign), and if the user is sending the file to me, I can quickly fix Arbed and release a new version that deals with it.

Of course, if Arbed reports a problem with its VCP conversion, one can always just use the Xojo IDE to make the conversion: Open your project in Xojo, and use the Save As... menu command to save the project in the binary or XML format. Then use that file with Arbed, and if you've made changes in Arbed and saved them back to the project file, open that again in Xojo and use again Save As... to save it in VCP format. It's tedious but that's the way it's been working even before Arbed added support for VCP projects.