Searching multiple Xojo projects

Finding that useful method or class in one of your numerous Xojo projects

If you've used Xojo (Real Studio) for a while, you'll probably have collected more than just one project.

And each of these projects contains unique code, and some of that may even be re-usable for other projects.

How do you keep track of all the code and methods you've written in the past? E.g, you do remember you've once written that nifty function to count the words in a string, but where is it?

I am going to show you several ways how you can find code across multiple projects.

Mac OS X only: Use Spotlight

For Spotlight to be able to search Xojo projects, it needs to be able to understand their content so that it can extract any text from it and add it to its searchable database. By default, textual project formats (.xojo_project, .xojo_xml_project, .xojo_script, .xml, .rbvcp, .rbs) are regarded as text files and thus be automatically scanned by Spotlight. However, it won't scan binary file formats such as .xojo_project and .rbp - to make them usable by Spotlight, a so-called Spotlight Importer is needed.

Real Studio and early Xojo IDEs did include such an importer, but recent versions don't any more. However, as I wrote this importer originally, I've now made the effort and updated it for the latest Xojo file format (as of Xojo 2015r3). So, if you're on a Mac I suggest you install this latest importer for Xojo if you like to use Spotlight:

Spotlight Importer for Xojo

This Spotlight importer will remember all the class and function names of all your projects, along with any other text it they contain.

Let's assume you have a method names "CountWords". If you enter "countwords" into the Spotlight search field, you should see the project file listed in the results.

However, the results may be cluttered with a lot of other types of files not related to Xojo.

So, here's how you tell Spotlight to only show Xojo project files:

• Open the Spotlight Finder window (e.g. by pressing cmd+F in the Finder).
• If you do not see a popup menu under the Search: row, click the [+] button on the right.
• Click on the leftmost popup menu (which probably reads Kind) and choose Other...
• A new sheet dialog appears in which you can select a search attribute.
• Find the "File extension" attribute. Tick its checkbox under the In Menu column.
• While you're at it, also find the "Xojo class" attribute and tick its checkbox as well.
• Then click OK to dismiss the dialog.

Now, whenever you use this Find window, you can choose File extension from the popup menu to limit the found items to file types that your Real Studio projects use: rbp, rbbas, xojo_binary_project etc.

Furthermore, if you are looking for a particular class, you can choose the Xojo class attribute and enter the (partial) class name there, leaving the main search field empty. That way, you'll see all the projects that contain that class.

All platforms: Searching inside VCP and XML projects

If you are used to saving your projects in the textual VCP (.rbvcp or .xojo_project and related class files) or XML (.xml or .xojo_xml_project) format, then you may have success performing simple searches using Spotlight on OS X or Windows Search on Windows.

If you need more control over what files are found, so that you don't get lots of false results from non-RB files, here are a few 3rd party programs you could try:

OS X

• TextWrangler is free and has a Multi-File search in which you can set up filters with the extensions you want to search.
• EasyFind mainly finds files by names, but if you specify to search files ending in .rbbas, .xojo_class etc., then you can also search for their content. (I'd have liked to tell you that my program Find Any File could also be of help here, but it doesn't search file contents - yet.)

Windows

• UltraFileSearch does a good job. You'd use the Wildcards search in the Files and Folders tab, searching for "*. xojo_*", and then enter the name of the function or code under the Containing Text tab.

Unfortunately, most of these methods (apart from TextWrangler), including Spotlight, won't show you the actual code snippets, and the Xojo IDE also won't open ".xojo_window" and similar files alone, so you'll have to find their main project file and double click that, then use the IDE again to search for the function name.

All platforms: Arbed

Arbed is a tool that was developed mainly for dealing with larger projects and the tasks around them. One of its many features lets you search inside project files within a folder and all its sub folders (and while some features require a license purchase, this is a free feature).

Contrary to Spotlight, Arbed runs on Windows and Linux as well, so you can use this even if you do not use a Mac.

The advantage of Arbed over the above methods is that Arbed can show you the results conveniently, so you don't have to re-enter the search in the IDE before you know if the results contain what you were searching for.

Launch Arbed and then either drop a folder onto its Search Multiple Projects box or choose Find on Disk... from the File menu:

You can then enter the search text, even as a Regex formula, and also choose the folder in which you want to search:

Arbed then lists all found projects:

Double clicking any of the found items opens a new window showing that project, along with a window listing all occurances. Clicking on any of the occurances will show you the related project item (class, function etc.):

Right-Clicking on an item in the project list gives you the option to reveal the file on disk or open it in the Xojo / Real Studio IDE.

New: OS X Automator Action plugin for invoking Services

When I started adding Services to my apps (Find Any File, iClip), I realized that this was not enough for users who want to write workflows with Automator or Applescript because there seems to be no way to invoke Services from Automator nor AppleScript.

So I wrote an Automator Action that allows you to run any Service that operates on Text or Files & Folders:

Find instructions and the downloads on github (including Xcode source).

License is unrestricted, i.e. totally free.

Xojo SQLite databases: Enabling ICU for full Unicode support and adding custom functions

Unicode Support (ICU)

Xojo's default Database class REALSQLDatabase (or its replacement SQLiteDatabase), which is based on sqlite, does not support international (non-english) characters. For instance, if you try to use the SQL function upper on a french phrase such as Tête-à-Tête, you will get TêTE-à-TêTE, i.e. only the plain ASCII chars (a-z) will get uppercased but not the special french chars. This missing unicode support also affects sorting and related comparison functions.

The usual solution to this is to compile the sqlite3 library with ICU support and then include this lib in your program. This is not a viable option with Xojo because you'd also have to write your own Database plugin to make the sqlite lib usable in Xojo (while the sqlite lib is open source, Xojo's plugin is not, or this would be less of an issue). Furtunately, though, Xojo's plugin allows dynamic loading of extensions (like plugins) into the sqlite engine.

Still, you need to have a readily built ICU lib for sqlite before you can load it. And unfortunately, there is none publically available as far as I could see, at least not for OS X. Therefore, I'm making my own built version available, along with the source code. Mind you, only for OS X, though, as I do not have the time to make and test versions for Windows and Linux as well. I believe, however, that you can more easily find .dll and .so files of the ICU-sqlite lib on the web and use them with these instructions.

Custom SQL Functions

I recently also started using sqlite's FTS (Fast Text Search) functionality. At the end of the FTS page, examples are shown on how to implement a smart ranking feature when searching for text. To implement it, a custom ranking function needs to be added to the SQL engine. Again, this is usually done by writing C code, and then load that as an extension. But since my code that controls what needs ranking is written in Xojo, I'd need to have this ranking function written in Xojo as well, in order to have access to the other data I've collected in my program relevant to performing the ranking.

Adding a custom functions to sqlite is fairly easy: Invoke the SQL command "SELECT load_extension('libfile_path', 'init_function_name')" is all it takes.

It would be great if we could simply pass the path to own code that's been compiled by Xojo, along with a custom init function we've prepared for this. Unfortunately, that doesn't work because all methods in Xojo are local (non-exported), making it impossible for sqlite to find any functions in our code.

Extension Stub

We need to use a small helper code file that provides the init function we need to implement. Here's the C code (I've removed some of the non-essential parts):

    #include <sqlite3ext.h>
    SQLITE_EXTENSION_INIT1

    typedef __cdecl int (*callback_func) (const void *pApi, void *db);

    static callback_func callback;

    void SetCallback(void *p) {
        callback = p;
    }

    int sqlite3_extension_init (
      sqlite3 *db,
      char **pzErrMsg,
      const sqlite3_api_routines *pApi)
    {
        SQLITE_EXTENSION_INIT2(pApi);
        if (callback) {
            callback (pApi, db);
        }
        return SQLITE_OK;
    }

The first relevant part here is the SetCallback function. This is a function we'll call from Xojo before loading this extension into the sqlite engine. We pass it the address of a Xojo function in which we'll handle the installation of our custom functions. The SetCallback function stores the address of our Xojo function for later use. Then, once we load this extension using the "SELECT load_extension" command, sqlite3_extension_init will get called. Its code then invokes our previously set Xojo function through the stored callback function pointer. That's basically all it does.

(This code should be portable for Windows and Linux as well - the actual source code contains some extra instructions for Windows DLLs).

Once the extension gets loaded and our callback function gets called, we perform the registration of the custom functions. How that's done is documented in the SQLite docs.

Using the sqlite API

There is another challenge, though. Our custom function certainly wants to be able to receive parameters, return values and maybe even perform database access operations on its own. For this, sqlite provides a large set of functions through its API. The problem is: While it's fairly easy to invoke them from C code, it's rather tedious to do in Xojo, because they're accessed through an array of function pointers (note the pApi parameter the sqlite3_extension_init gets passed - that's where all those pointers are located). In C, macros take care of making them convenient to call, but in Xojo it means a bit more work for us.

Basically, to call a function, e.g. sqlite3_result_int, we must first find out which place its function pointer occupies in said array. And that can only be found out by looking at the header file (sqlite3ext.h) and counting through the positions. Turns out that sqlite3_result_int is at index 82, for instance.

I have then located some of the other functions to access passed parameters and to set returned results and make proxy functions for them to make calling them easier. It should also make it clear how to add more API functions if needed.

Download it all

To sum this all up, I'm now presenting to you a ready-made Xojo project along with compiled libs for both extensions and their source code with Xcode project files. The project shows how to load both extensions and then tests the results, proving that both the unicode extension and the added custom functions work. Note, however, that it only works on OS X out of the box. I've tested it with both Real Studio 2012r2.1 and Xojo 2015r2.2.

The proejct uses the older REALSQLDatabase class. If you replace it with the newer SQLiteDatabase class, you need to also add this line each time before you try to install an exension, or you'll get an "Not authorized" error:

db.LoadExtensions = true

I believe the extensions can also be built for Windows and Linux without too much effort if you follow the sqlite instructions under the topic "Compiling A Loadable Extension". If you succeed in this and like to share your results, I'm happy to include them in my downloadable file or link to yours.

As usual, my demo code is free of restrictions. Use it as you like.

Download here.

Alternative by MonkeyBread Software

MonkeyBread Software has recently added ICU support to their SQLite plugin as well.

Ricoh / Pentax Flucard reverse engineering

I have done some quick reverse engineering on the FluCard (a wireless memory card for Pentax DSLR cameras), revealing some additional commands it offers.

I'd like to do more work on this, such as actually decoding some of its busybox code, but have no time for that currently.

Details are here: http://www.tempel.org/Photo/FLUcard

OS X Kernel Hacking (fixing the psignal kernel bug)

At the German Macoun conference in Oct 2014 I gave a talk about rootkits and hacking the OS X kernel.

This article is a summary and follow-up on this topic.

Introduction

Since Yosemite (OS X 10.10), kernel extensions (KEXTs) need to be code signed or they won't get loaded any more. The code signing certificate needs to be optained from Apple, and it's not easy to get it - and even if you have one - once you do something with it that Apple doesn't approve of, they can simply invalidate it, preventing further use of KEXTs signed with that certificate.

As usual for Apple in recent years, we developers (at least in my case who received said certificate from Apple) are not told clearly what would be acceptable by Apple. Clearly, Apple should use this to block malicious software from getting installed (such as rootkits and "jailbreaks" that undermine the security of the user's data and resources). But what about things that benefit users but that Apple simply does not agree with due to their desire to control everything lately? Is patching bugs allowed? Is patching features allowed as long as it's desired by and beneficial to the users, with their express agreement? We don't know. But we'll find out a bit of that, I guess. See below.

It is possible to disable the signing requirement for KEXTs, though. This is done by setting a particular firmware parameter (see this StackExchange answer for instructions) on the computer where you like to load an unsigned KEXT. This setting is meant for developers so that they can test their KEXT without having to sign it each time (besides, performing code signing requires an internect connection, which may not always be available). But users can apply this setting as well, of course. For example, anyone having installed a 3rd-party SSD into their Mac may want to install a patched version of the disk driver in order to have TRIM support enabled (Apple only enables it for their own drives by default).

Patching a little bug in the Kernel

There's this bug in the kernel that affects signal handling in threads. The author of the article, Russ Cox, shows how to patch the kernel file on disk to fix this bug. That is a bit dangerous because, when one makes a mistake and tries to boot this system, the Mac may become inoperational, and fixing this is a bit tricky. Also, it's not everyone's pleasure to patch the kernel the way.

I now like to show how to perform the same patch dynamically, from within a kernel extension. The advantage of this procedure is that it can be enabled and disabled at will, simply by loading and unloading the KEXT. As it doesn't modify the kernel on disk but only updates it in memory, it should work with future kernel updates as well, patching the bug in there as long as it hasn't been fixed by Apple. Should the bug not be found by the KEXT, then the KEXT simply quits and doesn't alter the system further.

To make this work several 3rd party components are used:

1. For locating non-exported kernel symbols in memory: The Flying Circus framework.
2. For locating the to-be-patched code in possibly any kernel version without having to know the exact location or code for each kernel release. This is accomplished by using the diStorm3 disassembler.

The entire project can be downloaded here:
http://files.tempel.org/OSX-psignal-fix/OSX-psignal-fix.zip

It contains an Xcode project with two targets: One builds the KEXT, the other builds the test program as provided by Russ Cox on this website. It also contains a readily usable KEXT signed with my own certificate.

To install the psignal fix into your OS X system, open Terminal and issue the following commands:

cd /path/to/OSX-psignal-fix-folder
sudo chown -R 0:0 psignal-fix.kext
sudo kextload psignal-fix.kext

That should have installed the patch. If you got an error, open Console.app and look for error messages containing "[psignal-fix]".

If you're happy with the functionality of the fix, you may also simply place it into /Library/Extensions/ and reboot to load it permanently. You have have to issue the chown command again after copying the kext to that folder, though.

To unload the KEXT, issue this command:

sudo kextunload psignal-fix.kext

If you have Xcode installed, you can also run the test to verify that the patch works. Open the project and select the "psignal-test" target from the popup menu in the window title. Run it and see its console output. This is the essential output without the fix:

...

500 signals received.

500 on main

0 on looper 0

0 on looper 1

0 on looper 2

0 on looper 3

And this is the output with the fix working:

...

500 signals received.

0 on main

129 on looper 0

123 on looper 1

128 on looper 2

122 on looper 3

The difference is that with the fix intact, the signals will occur nearly evenly distributed on all looper threads and none on the main thread.

If you build the KEXT (target "psignal-fix") yourself, you need to disable the KEXT signing requirement first if you're on OSX 10.10 or later, unless you have your own KEXT signing certificate, of course.

References

* Disable KEXT signing requirement : http://apple.stackexchange.com/questions/163059/
* Trim Enabler: http://www.cindori.org/software/trimenabler/
* Description of kernel bug: http://research.swtch.com/macpprof
* The Flying Circus: http://www.phrack.org/papers/revisiting-mac-os-x-kernel-rootkits.html