19 July 2017

APFS and fast catalog search

This is about FSCatalogSearch / searchfs support in macOS with the APFS file system.

Updated 3 Oct 2017: Find Any File 1.9 will support fast search on APFS on High Sierra (10.13) by using the searchfs function. Version 1.9 is currently in open beta, see the FAF web site.

Updated 25 July 2017: Clarified why FSCatalogSearch doesn't work on APFS, adds issue about hard links and 64 bit CNID resolving.

Some background on FSCatalogSearch in general


Programs like EasyFind and my own Find Any File (FAF) are able to search for file names (as well as file dates, sizes and a few other rarely needed attributes) on disks in a quite fast manner by using a little-known function macOS offers.

This Carbon level function is known as CatSearch or (FS)CatalogSearch and has been around for more than 25 years. There's also a BSD level function called searchfs.

The advantage of this function is that it performs the search for names at the file system driver level, meaning that when you search for files containing ".png" in their name, the file system can look at the entire directory tree much faster, sorting out the matches, and only report those to the program that initiazes the search and then shows the results to the user.

Without this special function, the search program would have to start at the root of the disk, read each folder (directory) recursively, and then sort out the matches itself, which all takes much more computing time.

For example, a search on a disk with millions of files and folders on it would take only a few seconds with FSCatalogSearch, whereas a classic recursive search would take minutes.

Getting even more technical


Apple added the FSCatalogSearch function in Mac OS long ago, after introducing the HFS file system. This was supported by the fact that  HFS did, unlike Window's FAT, arrange the entire directory tree in one large file on the disk, with interlinked nodes that did not match the hierarchical folder structure. FSCatalogSearch would then iterate over the nodes in a most efficient way, not caring about the folder structure, thereby minimizing disk seek times, which was a significant factor in disk access before SSDs. This also meant that FSCatalogSearch would only work on volume formats that used a single (invisible) file for its entire directory tree, meaning that FSCatalogSearch was never available for FAT disks, for instance. It would also be optimal for NTFS volumes, but since Apple never used NTFS other than to support reading from Bootcamp partitions, they never made the effort to add FSCatalogSearch to their NTFS file system driver.

What about APFS?


Now Apple is about to replace HFS(+) with APFS on macOS. And fans of EasyFind and FAF start wondering: Will I still be able to perform fast disk-wide file name searches the way I'm used to?

The good news is: The APFS file system code has support for the lower level searchfs function, and that's been already added in 2016, apparently, for OS X 10.12. Which ultimately means: Yes, FAF and EasyFind can continue to provide fast search on APFS formatted disks, provided extra work is put into updating the apps accordingly.

However, there are still some issues:
  • The high-level FSCatalogSearch does not work on APFS. Both EasyFind and FAF rely on this function and therefore won't find files the fast way on APFS volumes right now. The reason for this is that APFS uses larger values (64 bit) than HFS+ for identifying the files, and the FSCatalogSearch function cannot handle those larger values. (rdar://33454922)
  • As of now (10.12.6, 10.13 beta 3), the searchfs function does search case-sensitive and not case-insensitive as it should. That means that searching for ".png" won't find files using ".PNG". I confirmed this with an Apple engineer - it's a known issue, just one with a low priority right now. So, there's a chance that this will get resolved eventually, and I hope it'll be done before 10.13 is released. This issue may not get fixed for 10.12.x, though. We'll have to see what Apple does in this regard. (rdar://33455597)
  • Hard links can't be identified correctly - if there are multiple hard links to the same file, then searchfs can't currently tell them apart, and the results will all point to the same directory entry. (rdar://33473247)
  • searchfs() returns CNIDs (Catalog Node IDs, 64 bit wide) instead of paths to the found items. This requires resolving these IDs to the paths later. However, there is currently no documented API provided in macOS to do so. There's a hackish way around this, but that's not a proper solution. (rdar://33507188)

What this all means


Current versions of FAF and EasyFind can't fast search on APFS. They need to be rewritten using the searchfs API.

I will be working on a quick-fix version of FAF that'll add fast search on APFS and which I hope to release before 10.13 (High Sierra) is officially released. I have quite a few other improvements for FAF in the works (64 bit app, content search, icon view, server support etc.) which will have to wait so that I can get this APFS issue resolved ASAP.

05 July 2017

Recover lost BootCamp Windows partitions on an Mac

I have installed several Windows 7 and 10 version on several of my Macs using Apple's Boot Camp feature.

Recently, I found that almost all of them have disappeared: I was not able to boot from them any more when I held down the option (alt, ⌥) key at startup - the Windows partitions would either not appear at all or not boot up.

The main reason in my case was that the MBR was reset to a plain GUID entry, and my Windows versions do not like that, because they cannot handle the EFI / GUID partition info that the Mac prefers. Why that even happened? Probably from repartitioning operations I frequently perform on my disks - and Apple's Disk Utility is quite ignorant of the needs to keep Windows bootable in this regard.

To fix that, I had to edit the MBR partition info again, in order to make the Windows NTFS visible again to the Windows systems.

In short, I used iBored to edit the partition layout of each disk that contains a BootCamp partition from something like this:


Into this:


Note that this reduces the size of the first partition (you could as well change its size to the minimum, which is 33), and adds a new partition with the start and site matching what you can inquire using the Partitions window (see Disk menu):

This modification makes the Windows partition available in the MBR, and after that, I can boot again from it. And it won't mess with macOS booting because that uses the GUID partition info which isn't getting modified by this procedure (for more info, read my older article on using BootCamp on a non-startup disk).

Let me know in the comments or by email if you find this interesting and need more instructions, and I'll see if I can improve this article.

08 April 2017

Adding fast external disks to various Mac models, with benchmarks

2nd Update on 9 Apr 17: See end of article
3rd Update on 18 Apr 17: Added more adapters

I wanted a fast external disk for my three Macs, which are now all 5 years or more old:
  • MacBook Pro Mid 2012 ("MacBookPro10,1"), 2.6 GHz Core i7
  • iMac 27" Mid 2011 ("iMac12,1"), 3.4 GHz Core i7
  • Mac Pro Late 2008 ("MacPro3,1"), 8 Core-Xeon 2.8 GHz

The Mac Pro has no Thunderbolt ports but I've installed a USB 3.0 PCI card from Inateck.

The iMac has Thunderbolt but only USB 2 built-in. I've added USB 3.0 support with the Elgato Thunderbolt 2 Dock (230 €).

I was interested to find out whether I should just go with a cheap USB 3 disk enclosure or use a significantly more expensive Thunderbolt enclosure.

The short answer is: If you have USB 3 ports on your Mac(s), you'll be fine with a USB 3 enclosure from a performance standpoint. But you won't get TRIM support, which is especially important if you plan to use the SSD as a faster boot volume - in that case, I recommend Thunderbolt instead.

For testing, I was using a new SanDisk Ultra II SSD 960GB (SATA III). I connected it to these adapters / enclosures:


Note: Even though the ICY BOX supports USB 3.1, which allows up to 10 Gbit/s, it connects to my Macs only at USB 3.0 speeds (5 Gbit/s) as they do not support 3.1. The provided USB cables are compatible with these Macs' "standard" USB ports.

The Thunderbolt enclosure by Delock requires an external power supply, and the AKiTiO comes with a second cable that needs to be plugged into a regular USB port to provide power to the disk. The USB adapters, on the other hand, do not have nor provide an extra power supply - they pull all the needed power for the disk from the USB port they're connected to.

Test Procedures


The testing was done in various ways, but I always only attempted to figure out the maximum possible transfer speed, such as for copying large single and unfragmented files, as that's the main use case for me (I often transfer entire disks, sector by sector, to another disk, e.g. for migration to a newer Mac, data analysis or quick backups).

For testing read performance, the easiest way is to use the dd command in Terminal.app:

sudo dd if=/dev/rdiskN of=/dev/null bs=512000

N has to be replaced by the disk number, which can be learned by using this command:

diskutil list

Note that this command lists the disk names with the leading "r", but when doing the test, the "r" is important for best speed results.

Write performance testing is a bit more difficult, because you need some data to copy from. You could use a file or another disk, but then you get a result that also involves the reading speed from the other disk. If that disk is an internal SSD, this may not affect the overall result too much, but in my case, it would, at least on the Mac Pro, despite having installed a Samsung SSD 850 EVO 1TB on a Velocity Solo x1 PCI card with a SATA III socket.

Copying with dd from /dev/zero may be a valid method for hard disks, but is not entirely honest for SSDs, as that would indeed test the transfer speed over the USB or TB bus, but since SSDs treat empty sectors specially, by not writing them to their flash memory, that would be cheating. I wanted to measure the true effective write speed, and therefore I'd need to write random (non-zero) data.

Copying from /dev/randon is no solution either, because generating the random data is very slow. I ended up getting unrealistic write speeds around 10 MB/s.

Being the author of a versatile disk editor, iBored, I have instead added a feature to it: Random data is generated once in memory, and then written repeatedly to disk, to consecutive sectors. That gives me realistic results that also correlated fairly well to copying from /dev/zero (i.e. it's a bit slower as expected). The version I used has not been officially released yet, but it's the 1.2b5 in the downloads. The Read/Write Speed Test commands are found in the Disk menu. If you try them yourself, be aware that this tool is not fool-proof. If you accidentally choose the wrong disk, and you could erase your precious files! So, be careful and always have an external backup, just in case.

Due to caching done by the SSD, I also had to make sure to write a large amount of data, to avoid reading and writing just through the faster cache.

Performance Test Results



The graphic does not include the Enateck, Teorder and AKiTiO adapters, as I got them later. The Teorder performed similarly to the Delock 42486, and the AKiTiO similarly to the Delock 42510. The Inateck is about 15 MB/s faster than the ICY BOX on the MacBook (didn't test on the other Macs, yet).

I also made a test on a PC with USB 3 ports, using iBored again. There, read speed was 240 MB/s with the Delock 42486 and the other USB3-SATA II adapter, and 320 MB/s with the ICY BOX. Write speed was only getting to 150 MB/s, though. That's odd, but a test with a regular Windows benchmark tool showed similar results.

Observations


  • The USB 3 performance with the Thunderbolt Dock is not as good as with the built-in USB 3 port on the MacBook Pro. That's not a weakness of the Dock, though, as far as I can tell, but a weakness of the iMac. I suspect its memory bus isn't up to it. When I connected the Dock to the MacBook Pro and then connected the ICY BOX to the Dock's USB 3 port instead of the MacBook Pro's port, I read at 330 MB/s and wrote at 300 MB/s, which is much closer to the built-in USB port.
  • The Delock 42486 USB 3 and the Teorder adapters are mislabelled in my opinion. Both supposedly support SATA III (i.e. 6Gbit/s) but perform like a SATA II (3Gbit/s) adapter.
  • My  Delock 42486 has a problem reporting incorrect power needs over USB, often just 24mA, when it needs 10 times or more, and that causes intermittend problems with some Macs but not with others. Delock support has not been helpful resolving this, insisting that it's not their fault, not even offering a replacement. This problem did not affect the performance testing, though.
  • The Thunderbolt adapters perform about as well or better as a good USB 3 adapter.
  • All the USB adapters have trouble reporting disk information such as name and serial number. While most at least report most of the disk's name ("Ultra II 960GB"), the ICY BOX only reports "ICY BOX" for any disk attached to it, the Inateck even just "2115". Only the Thunderbolt adapters are able to report the complete name ("Sandisk Ultra II 960GB") and the SDD's Serial Number to the Mac. These limitations with the USB adapters are minor, though, and will not affect performance or usability.
  • The Macs are also unable to report S.M.A.R.T. status for disks attached via USB - that's a known limitation of OS X / macOS. Again, with Thunderbolt there is no such issue.
  • UASP support may or may not be having a positive effect on performance. Obviously, the Teorder adapter, claiming UASP support, does not perform well at all. OTOH, the Inateck adapter is slightly faster than the ICY BOX, which may be due to UASP.

Conclusions


If you have a Mac that supports USB 3, there's hardly a need for the Delock Thunderbolt adapter.

The disadvantages of the Thunderbolt adapter (much higher price and having to use an extra power supply) have no advantages I can see.

The only exception I can think of is when you use disk in with that requires more power than USB 3 can supply. Though, my MacBook Pro, in System Profiler, under USB, tells me it can supply 1800mA, while the ICY BOX adapter with the SSD installed only requests 224mA.

It remains to be seen if faster disks (i.e. SSDs) are able to provide even more throughput, and if that would give the Thunderbolt adapter an advantage. However, I believe that SATA III, which is used by all three adapters as the interface to the disk, is the bottleneck here, though there is still a bit of room to fill that out (nominally, SATA III can transmit about 500 MB/s, and suppedly the SanDisk SSD can achieve that, but I have currently no way to verify this).

(9 Apr 17) After considering TRIM support, I've changed my mind on this: The lack of TRIM support in the tested USB 3 makes them not well suited for external SSDs as a Thunderbolt adapter that invariably supports TRIM. See below for more details.

If you have a Mac without USB 3 but with Thunderbolt, you still have two choices:

  • Either purchase a AKiTiO Thunderbolt adapter at about 125 € (as of 12 April 2017 at Amazon). That's probably the cheapest option in the short run.
  • Or get the Thunderbolt 2 Dock from Elgato at about twice the price, plus a cheap USB 3 enclosure for your disk. The Dock has the advantage that it equippes your Mac not only with three USB 3 ports but also with an HDMI port to connect a second monitor, an easier-to-access Audio port (which apparently is also higher-quality than that of some Macs) and even a microphone port. It is more bulky and requires a power supply, too (and here it makes actual sense). That leaves you with a slightly worse performance than with the Thunderbolt case, but therefore your external disk can be used even in older Macs that have no Thunderbolt, and even in regular PCs.

1. Update on 9 Apr 17:

@felix_schwarz pointed out to me that there's an advanced protocol for external disks called UAS (aka UASP). UAS can provide better performance USB (and, according to AKiTiO, apparently also over Thunderbolt), and Mac OS X supports UAS since 10.8, according to WP.

Of the adapters I tried, some claim UASP support, others do not. The results are inconclusive, as one adapter with UASP (Teorder) was very slow, whereas the one with it (Inateck) was slightly faster than the one without claiming UASP support (ICY BOX).

I just learned that AKiTiO makes various Thunderbolt products of interest. There is a Thunderbolt Dock with USB 3 ports as well as an External Thunderbolt + USB SSD that supports UASP. AKiTiO claims that this gives over 500 MB/s effective read speed with their 1TB SSD model, though their benchmark makes me wonder why the effective write speed is below 200 MB/s. Maybe they're using an older type of SSD that is not up to it. The Neutrino model seems to be better in this regard, but it does not offer a 1 TB SDD version. Also, the price is quite high compared to the setups I described above, although actual prices seem to be significantly lower that the given MSRP. If you need maximum performance, these SSDs may still be worth considering. (Update 18 Apr 17: I've now received and tested the AKiTiO Thunder SATA Go Adapter, see above).

2. Update on 9 Apr 17:

I had so far overlooked one important property that affects SSDs: Trim support.

It turns out that the tested USB 3 adapters provide no TRIM support. It's unclear why that is - I suspect that it's simply lack of support for it in OS X, similar to the missing S.M.A.R.T. support.

Missing TRIM support can will quickly lead to performance degradation of the installed SSDs.

If you only use hard disks with these adapters, that won't matter, but for SSDs, especially if they're used frequently for adding and deleting files, TRIM support is fairly significant to the performance. However, if the SSD is mainly used as a permanent backup device, e.g. as a Time Machine backup destination, TRIM support is not that important, even if files get replaced by TM once the disk is full.

Only the Thunderbolt adapters support TRIM. I've verified this by first checking in System Profiler whether the disk is even listed as "TRIM Support: Yes", and then by using the technique described in this Ars Technica article.

This realization inverts my earlier recommendation: Of the tested adapters, the ones with Thunderbolt are the best choice if you want lasting optimal performance, especially if you plan to boot from it regularly (which is a scenario for iMacs with only a HD installed, to speed up their system with the permanently connected SSD that'll host the system software). Of course, that still requires that you enable system-wide TRIM support in OS X / macOS, e.g. by using the "trimforce" command as explained in the Ars Technica article.



There is even an awkward work-around for missing TRIM support, in case you decide to go with an SSD and USB 3 regardless: Regularly fill the free space on the disk with a file that just contains zeros, and then delete the file again, as that'll tell the SSD that this space is actually unused, and should make the disk perform better again for a while. This can be done using iBored (menu Tools, Erase Unused Disk Space...) or with this Terminal command:

cp /dev/zero "/Volumes/YourSSDsName"

Once the copy has finished, filling up the entire disk, delete the file again:


rm "/Volumes/YourSSDsName/zero"


(Update 18 Apr 17): BTW, I now learned how to issue the TRIM command (only for supporting adapters, i.e. Thunderbolt) and will soon add this feature to my disk tool iBored.

28 March 2017

Xojo: How to improve performance when using WeakRefs

If you're building tree structures with leaves (children) that shall keep a reference to their branch (ancestor, parent) they're attached to, the simplest (but not smart) solution is to store a direct references to the parent object in the child object.

Imagine this node class:

class Person
  property name as String
  property parent as Person
  property children() as Person
end class

And then you'd have a Constructor like this:

sub Constructor (parentIn as Person, nameIn as String)
  self.parent = parentIn
  self.name = nameIn
end sub

and create new children like this:

sub AddAChildNamed (name as String)
  dim newChild as new Person (self, name)
  self.children.Append newChild
end sub

This is not optimal because you'd get circular references (parent references children who in turn reference back their parent). And since Xojo uses ref-counting for keeping its objects alive, and does not have a separate garbage collection task, removing all references from your main code to the top parent won't free all those Person objects, because they still keep referencing each others, thereby keeping themselves alive.

One solution would be to add a "FreeAllChildren" method that you'd have to call right before are ready to release the entire tree, but that's not very elegant (it's the fatest solution, though, if you get this right).

A more natural and fool-proof way is to use weak references for the child-to-parent connections, like this:

class Person
  property name as String
  property parent as WeakRef // **changed**
  property children() as Person
end class

sub Constructor (parentIn as Person, nameIn as String)
  self.parent = new WeakRef (parentIn)
  self.name = nameIn
end sub

To access the actual parent, you'd use code like this to temporarily re-create a hard reference to the parent object:

  dim myParent as Parent = Parent (self.parent.Value)

With the above changes, you won't have circular references any more.

However, it's not very efficient. If you have many 1000s of parent objects, you'll get as many WeakRef objects. And the downside of WeakRef is that it involves a search operation to locate the actual parent object when you use the WeakRef.Value function. And the more WeakRef objects you have, the more time it takes to look up these values.

Now consider this: Usually, with these parent-child relationships, there are more children to a parent than vice versa. In the above code, every child creates a WeakRef, often for the same Parent. And that's what we can optimize for:

Instead of having the child create its own WeakRef, let us have the parent provide it, as it'll remain constant.

Change the constructor to accept a WeakRef:

sub Constructor (parentRefIn as WeakRef, nameIn as String)
  self.parent = parentRefIn
  self.name = nameIn
end sub

add a new property to the class:

class Person
  property selfRef as WeakRef
  ...

and update the way a child gets added accordingly:

sub AddAChildNamed (name as String)
  if selfRef = nil then selfRef = new WeakRef (self)
  dim newChild as new Person (selfRef, name)
  self.children.Append newChild
end sub

And that's all.

TL;DR

If you use WeakRefs, and if you create many WeakRef for the same object, consider creating the WeakRef only once and cache it, to gain runtime performance.

24 March 2016

Using USB barcode scanners on Macs with Xojo

Barcode scanners translate 1D and 2D bar codes, such as Code128 and QR Code, into readable characters.

Usually, handheld bar code scanners with a USB interface simulate a keyboard by default. With that, they work out of the box: If you scan a code with such a reader, it'll send keyboard strokes to the computer.

This makes them easy to use, but it also causes some issues:
  • If you have a program that expects input from a scanner, you have to make sure the typed characters are properly caught by your program and not go somewhere they don't belong. E.g, if the user switches accidentally to another program and then scans a bar code, not only will your program not be able to get the code from the scanner, but the keystrokes may cause odd effects in the the program that's active then. It's like your cat walking over your keyboard and wonder what'll go wrong.
  • Even worse, it can pose a security threat. Hackers have been able to show scanners customized bar codes that contained key strokes that would lead to remote controlling computers that the user was not supposed to access that way. Imagine a so-called "kiosk" terminal that's supposed to run only a particular application but gets quit by a alt-F4 or cmd-Q keystroke, then opening another program such as a Terminal to mess with the computer. In other words, using a bar code scanner in the "keyboard emulation" mode opens the computer to all kinds of abuse that you may have wanted to prevent by not giving the user a keyboard and mouse.

One way to prevent these issues is to put the scanner into a mode where it does not emulate a keyboard any more. That mode is usually called a "HID POS" or "USB OEM" mode. In such a mode,  the software needs to run special code to communicate with the scanner directly. Another option would be to not use the scanner's USB interface but rather use a serial interface (many scanners can be used with either) and then use a Serial-to-USB adapter to connect the serially operated scanner to the Mac or PC.

In case you want to use the USB HID POS mode with a scanner (e.g. GoDEX models), I've posted some details and a demo project on the Xojo forum a while ago: https://forum.xojo.com/24557

In the meantime, I've also managed to scanners from Datalogic (Gryphon 44xx models) and similar scanners using the sparely documented IBM / Toshiba "OEM" USB protocol. If you are interested in supporting those, contact me directly.

31 October 2015

Searching multiple Xojo projects

Finding that useful method or class in one of your numerous Xojo projects


If you've used Xojo (Real Studio) for a while, you'll probably have collected more than just one project.

And each of these projects contains unique code, and some of that may even be re-usable for other projects.

How do you keep track of all the code and methods you've written in the past? E.g, you do remember you've once written that nifty function to count the words in a string, but where is it?

I am going to show you several ways how you can find code across multiple projects.


Mac OS X only: Use Spotlight


For Spotlight to be able to search Xojo projects, it needs to be able to understand their content so that it can extract any text from it and add it to its searchable database. By default, textual project formats (.xojo_project, .xojo_xml_project, .xojo_script, .xml, .rbvcp, .rbs) are regarded as text files and thus be automatically scanned by Spotlight. However, it won't scan binary file formats such as .xojo_project and .rbp - to make them usable by Spotlight, a so-called Spotlight Importer is needed.

Real Studio and early Xojo IDEs did include such an importer, but recent versions don't any more. However, as I wrote this importer originally, I've now made the effort and updated it for the latest Xojo file format (as of Xojo 2015r3). So, if you're on a Mac I suggest you install this latest importer for Xojo if you like to use Spotlight:


This Spotlight importer will remember all the class and function names of all your projects, along with any other text it they contain.

Let's assume you have a method names "CountWords". If you enter "countwords" into the Spotlight search field, you should see the project file listed in the results.

However, the results may be cluttered with a lot of other types of files not related to Xojo.

So, here's how you tell Spotlight to only show Xojo project files:
  • Open the Spotlight Finder window (e.g. by pressing cmd+F in the Finder).
  • If you do not see a popup menu under the Search: row, click the [+] button on the right.
  • Click on the leftmost popup menu (which probably reads Kind) and choose Other...
  • A new sheet dialog appears in which you can select a search attribute.
  • Find the "File extension" attribute. Tick its checkbox under the In Menu column.
  • While you're at it, also find the "Xojo class" attribute and tick its checkbox as well.
  • Then click OK to dismiss the dialog.
Now, whenever you use this Find window, you can choose File extension from the popup menu to limit the found items to file types that your Real Studio projects use: rbprbbas, xojo_binary_project etc.

Furthermore, if you are looking for a particular class, you can choose the Xojo class attribute and enter the (partial) class name there, leaving the main search field empty. That way, you'll see all the projects that contain that class.


All platforms: Searching inside VCP and XML projects


If you are used to saving your projects in the textual VCP (.rbvcp or .xojo_project and related class files) or XML (.xml or .xojo_xml_project) format, then you may have success performing simple searches using Spotlight on OS X or Windows Search on Windows.

If you need more control over what files are found, so that you don't get lots of false results from non-RB files, here are a few 3rd party programs you could try:

OS X

  • TextWrangler is free and has a Multi-File search in which you can set up filters with the extensions you want to search.
  • EasyFind mainly finds files by names, but if you specify to search files ending in .rbbas, .xojo_class etc., then you can also search for their content. (I'd have liked to tell you that my program Find Any File could also be of help here, but it doesn't search file contents - yet.)

Windows

  • UltraFileSearch does a good job. You'd use the Wildcards search in the Files and Folders tab, searching for "*. xojo_*", and then enter the name of the function or code under the Containing Text tab.
Unfortunately, most of these methods (apart from TextWrangler), including Spotlight, won't show you the actual code snippets, and the Xojo IDE also won't open ".xojo_window" and similar files alone, so you'll have to find their main project file and double click that, then use the IDE again to search for the function name.


All platforms: Arbed


Arbed is a tool that was developed mainly for dealing with larger projects and the tasks around them. One of its many features lets you search inside project files within a folder and all its sub folders (and while some features require a license purchase, this is a free feature).

Contrary to Spotlight, Arbed runs on Windows and Linux as well, so you can use this even if you do not use a Mac.

The advantage of Arbed over the above methods is that Arbed can show you the results conveniently, so you don't have to re-enter the search in the IDE before you know if the results contain what you were searching for.

Launch Arbed and then either drop a folder onto its Search Multiple Projects box or choose Find on Disk... from the File menu:


You can then enter the search text, even as a Regex formula, and also choose the folder in which you want to search:


Arbed then lists all found projects:


Double clicking any of the found items opens a new window showing that project, along with a window listing all occurances. Clicking on any of the occurances will show you the related project item (class, function etc.):


Right-Clicking on an item in the project list gives you the option to reveal the file on disk or open it in the Xojo / Real Studio IDE.

21 September 2015

New: OS X Automator Action plugin for invoking Services

When I started adding Services to my apps (Find Any File, iClip), I realized that this was not enough for users who want to write workflows with Automator or Applescript because there seems to be no way to invoke Services from Automator nor AppleScript.

So I wrote an Automator Action that allows you to run any Service that operates on Text or Files & Folders:



Find instructions and the downloads on github (including Xcode source).

License is unrestricted, i.e. totally free.

17 September 2015

Xojo SQLite databases: Enabling ICU for full Unicode support and adding custom functions

Unicode Support (ICU)


Xojo's default Database class REALSQLDatabase (or its replacement SQLiteDatabase), which is based on sqlite, does not support international (non-english) characters. For instance, if you try to use the SQL function upper on a french phrase such as Tête-à-Tête, you will get TêTE-à-TêTE, i.e. only the plain ASCII chars (a-z) will get uppercased but not the special french chars. This missing unicode support also affects sorting and related comparison functions.

The usual solution to this is to compile the sqlite3 library with ICU support and then include this lib in your program. This is not a viable option with Xojo because you'd also have to write your own Database plugin to make the sqlite lib usable in Xojo (while the sqlite lib is open source, Xojo's plugin is not, or this would be less of an issue). Furtunately, though, Xojo's plugin allows dynamic loading of extensions (like plugins) into the sqlite engine.

Still, you need to have a readily built ICU lib for sqlite before you can load it. And unfortunately, there is none publically available as far as I could see, at least not for OS X. Therefore, I'm making my own built version available, along with the source code. Mind you, only for OS X, though, as I do not have the time to make and test versions for Windows and Linux as well. I believe, however, that you can more easily find .dll and .so files of the ICU-sqlite lib on the web and use them with these instructions.

Custom SQL Functions


I recently also started using sqlite's FTS (Fast Text Search) functionality. At the end of the FTS page, examples are shown on how to implement a smart ranking feature when searching for text. To implement it, a custom ranking function needs to be added to the SQL engine. Again, this is usually done by writing C code, and then load that as an extension. But since my code that controls what needs ranking is written in Xojo, I'd need to have this ranking function written in Xojo as well, in order to have access to the other data I've collected in my program relevant to performing the ranking.

Adding a custom functions to sqlite is fairly easy: Invoke the SQL command "SELECT load_extension('libfile_path', 'init_function_name')" is all it takes.

It would be great if we could simply pass the path to own code that's been compiled by Xojo, along with a custom init function we've prepared for this. Unfortunately, that doesn't work because all methods in Xojo are local (non-exported), making it impossible for sqlite to find any functions in our code.

Extension Stub


We need to use a small helper code file that provides the init function we need to implement. Here's the C code (I've removed some of the non-essential parts):

    #include <sqlite3ext.h>
    SQLITE_EXTENSION_INIT1

    typedef __cdecl int (*callback_func) (const void *pApi, void *db);

    static callback_func callback;

    void SetCallback(void *p) {
        callback = p;
    }

    int sqlite3_extension_init (
      sqlite3 *db,
      char **pzErrMsg,
      const sqlite3_api_routines *pApi)
    {
        SQLITE_EXTENSION_INIT2(pApi);
        if (callback) {
            callback (pApi, db);
        }
        return SQLITE_OK;
    }

The first relevant part here is the SetCallback function. This is a function we'll call from Xojo before loading this extension into the sqlite engine. We pass it the address of a Xojo function in which we'll handle the installation of our custom functions. The SetCallback function stores the address of our Xojo function for later use. Then, once we load this extension using the "SELECT load_extension" command, sqlite3_extension_init will get called. Its code then invokes our previously set Xojo function through the stored callback function pointer. That's basically all it does.

(This code should be portable for Windows and Linux as well - the actual source code contains some extra instructions for Windows DLLs).

Once the extension gets loaded and our callback function gets called, we perform the registration of the custom functions. How that's done is documented in the SQLite docs.

Using the sqlite API


There is another challenge, though. Our custom function certainly wants to be able to receive parameters, return values and maybe even perform database access operations on its own. For this, sqlite provides a large set of functions through its API. The problem is: While it's fairly easy to invoke them from C code, it's rather tedious to do in Xojo, because they're accessed through an array of function pointers (note the pApi parameter the sqlite3_extension_init gets passed - that's where all those pointers are located). In C, macros take care of making them convenient to call, but in Xojo it means a bit more work for us.

Basically, to call a function, e.g. sqlite3_result_int, we must first find out which place its function pointer occupies in said array. And that can only be found out by looking at the header file (sqlite3ext.h) and counting through the positions. Turns out that sqlite3_result_int is at index 82, for instance.

I have then located some of the other functions to access passed parameters and to set returned results and make proxy functions for them to make calling them easier. It should also make it clear how to add more API functions if needed.

Download it all


To sum this all up, I'm now presenting to you a ready-made Xojo project along with compiled libs for both extensions and their source code with Xcode project files. The project shows how to load both extensions and then tests the results, proving that both the unicode extension and the added custom functions work. Note, however, that it only works on OS X out of the box. I've tested it with both Real Studio 2012r2.1 and Xojo 2015r2.2.

The proejct uses the older REALSQLDatabase class. If you replace it with the newer SQLiteDatabase class, you need to also add this line each time before you try to install an exension, or you'll get an "Not authorized" error:

db.LoadExtensions = true

I believe the extensions can also be built for Windows and Linux without too much effort if you follow the sqlite instructions under the topic "Compiling A Loadable Extension". If you succeed in this and like to share your results, I'm happy to include them in my downloadable file or link to yours.

As usual, my demo code is free of restrictions. Use it as you like.

Download here.


Alternative by MonkeyBread Software

MonkeyBread Software has recently added ICU support to their SQLite plugin as well.

17 May 2015

Ricoh / Pentax Flucard reverse engineering

I have done some quick reverse engineering on the FluCard (a wireless memory card for Pentax DSLR cameras), revealing some additional commands it offers.

I'd like to do more work on this, such as actually decoding some of its busybox code, but have no time for that currently.

Details are here: http://www.tempel.org/Photo/FLUcard

08 January 2015

OS X Kernel Hacking (fixing the psignal kernel bug)

At the German Macoun conference in Oct 2014 I gave a talk about rootkits and hacking the OS X kernel.

This article is a summary and follow-up on this topic.

Introduction


Since Yosemite (OS X 10.10), kernel extensions (KEXTs) need to be code signed or they won't get loaded any more. The code signing certificate needs to be optained from Apple, and it's not easy to get it - and even if you have one - once you do something with it that Apple doesn't approve of, they can simply invalidate it, preventing further use of KEXTs signed with that certificate.

As usual for Apple in recent years, we developers (at least in my case who received said certificate from Apple) are not told clearly what would be acceptable by Apple. Clearly, Apple should use this to block malicious software from getting installed (such as rootkits and "jailbreaks" that undermine the security of the user's data and resources). But what about things that benefit users but that Apple simply does not agree with due to their desire to control everything lately? Is patching bugs allowed? Is patching features allowed as long as it's desired by and beneficial to the users, with their express agreement? We don't know. But we'll find out a bit of that, I guess. See below.

It is possible to disable the signing requirement for KEXTs, though. This is done by setting a particular firmware parameter (see this StackExchange answer for instructions) on the computer where you like to load an unsigned KEXT. This setting is meant for developers so that they can test their KEXT without having to sign it each time (besides, performing code signing requires an internect connection, which may not always be available). But users can apply this setting as well, of course. For example, anyone having installed a 3rd-party SSD into their Mac may want to install a patched version of the disk driver in order to have TRIM support enabled (Apple only enables it for their own drives by default).

Patching a little bug in the Kernel


There's this bug in the kernel that affects signal handling in threads. The author of the article, Russ Cox, shows how to patch the kernel file on disk to fix this bug. That is a bit dangerous because, when one makes a mistake and tries to boot this system, the Mac may become inoperational, and fixing this is a bit tricky. Also, it's not everyone's pleasure to patch the kernel the way.

I now like to show how to perform the same patch dynamically, from within a kernel extension. The advantage of this procedure is that it can be enabled and disabled at will, simply by loading and unloading the KEXT. As it doesn't modify the kernel on disk but only updates it in memory, it should work with future kernel updates as well, patching the bug in there as long as it hasn't been fixed by Apple. Should the bug not be found by the KEXT, then the KEXT simply quits and doesn't alter the system further.

To make this work several 3rd party components are used:

1. For locating non-exported kernel symbols in memory: The Flying Circus framework.
2. For locating the to-be-patched code in possibly any kernel version without having to know the exact location or code for each kernel release. This is accomplished by using the diStorm3 disassembler.

The entire project can be downloaded here:
http://files.tempel.org/OSX-psignal-fix/OSX-psignal-fix.zip

It contains an Xcode project with two targets: One builds the KEXT, the other builds the test program as provided by Russ Cox on this website. It also contains a readily usable KEXT signed with my own certificate.

To install the psignal fix into your OS X system, open Terminal and issue the following commands:

cd /path/to/OSX-psignal-fix-folder
sudo chown -R 0:0 psignal-fix.kext
sudo kextload psignal-fix.kext

That should have installed the patch. If you got an error, open Console.app and look for error messages containing "[psignal-fix]".

If you're happy with the functionality of the fix, you may also simply place it into /Library/Extensions/ and reboot to load it permanently. You have have to issue the chown command again after copying the kext to that folder, though.

To unload the KEXT, issue this command:

sudo kextunload psignal-fix.kext

If you have Xcode installed, you can also run the test to verify that the patch works. Open the project and select the "psignal-test" target from the popup menu in the window title. Run it and see its console output. This is the essential output without the fix:

...
500 signals received.
500 on main
0 on looper 0
0 on looper 1
0 on looper 2
0 on looper 3

And this is the output with the fix working:

...
500 signals received.
0 on main
129 on looper 0
123 on looper 1
128 on looper 2

122 on looper 3

The difference is that with the fix intact, the signals will occur nearly evenly distributed on all looper threads and none on the main thread.

If you build the KEXT (target "psignal-fix") yourself, you need to disable the KEXT signing requirement first if you're on OSX 10.10 or later, unless you have your own KEXT signing certificate, of course.

References


* Disable KEXT signing requirement : http://apple.stackexchange.com/questions/163059/
* Trim Enabler: http://www.cindori.org/software/trimenabler/
* Description of kernel bug: http://research.swtch.com/macpprof
* The Flying Circus: http://www.phrack.org/papers/revisiting-mac-os-x-kernel-rootkits.html

26 November 2014

Making CUPS printer drivers work again on OS X Yosemite (OS X 10.10)

I recently found out that some printer drives using the CUPS printing system stopped working on OSX 10.10 (Yosemite) that worked fine with the previous OSX Mavericks (10.9).

After some googling around I found various suggestions, but none seemed to help. In the end, it turned out that I had to follow THREE different suggestions to make it work. To save you the same odysee, here's my recipe for making printer drivers work again that stopped working on Yosemite:
  1. Remove any troublesome printers from the "Printers & Scanners" System Preferences.
  2. Re-Install the printer driver.
  3. Fix the permissions of the installed cups drivers by starting the "Disk Utility" program, selecting the startup disk and clicking on "Repair Disk Permissions".
  4. Turn off the tighter Sandboxing checks in Yosemite with these two Terminal commands:
    • sudo sh -c 'echo "Sandboxing Relaxed" >> /etc/cups/cups-files.conf'
    • sudo launchctl stop org.cups.cupsd
  5. Connect the printer and re-add the printer in the "Printers & Scanners" settings.
Hope this helps. If it does, please spread the word. If it doesn't, let me know please.

And, if you found this useful, also check out some other work of mine, such as:

  • Find Any File - The program that finds the files that Spotlight doesn't.
  • iClip - Improves the workflow with text snippets and the clipboard in general.

04 November 2014

Find Empty Folders - a free OS X tool

Today someone asked me if I knew a way to find and delete empty folders, even if they contain an invisible .DS_Store file.

I googled and found nothing.

So I quickly whipped something out with Real Studio.

The result can be found here:

http://www.tempel.org/FindEmptyFolders

18 July 2014

Something adventurous for a change (Nelly Cootalot)

Today I'm just helping promote someone else's work.

With this blog being directed mainly to other (programming) geeks, I think you might even be the right audience:

Did you enjoy playing Monkey Island, Leisure Suit Larry and other funny point-and-click adventures? (Though, the first Larrys I played were running still on DOS, without mouse support.)

And do you enjoy british humour?

And, possibly, are you on Steam?

Then have a look at this hilarious Nelly Cootalot promo!

The game is designed by a british film maker, and it's not his first go at an adventure, however this one is made much more professionally. Support his quest for getting the game onto Steam - don't be a coward!

I support this because long time friends of mine are involved in the development and distribution, and at some time I was almost becoming part of the dev team as well. I'd have loved to but I'm already busy with so many other projects I couldn't take on another one.

The game already got its basic funding through Kickstarter earlier this year, and now this is a step up to get onto Steam, giving it a much wider reach.

Also, it'll run on Macs, which is always good.

07 July 2014

OS X: Moving Files to Trash from your own application, with optional user interaction!

When you're writing a interactive Mac OS X program, it's possible that you need to delete files.

And if they're files the user has access to, then it's often a good idea to move them to the trash instead of erasing them right away, in order to give the user the opportunity to restore them again (saving you the need to provide some kind of Undo command for the destructive operation).

Moving items to the Trash is rather complex, and so it's best to just let the Finder perform that task.

The Finder is, fortunately, still scriptable, and so this task should be easy. (I must say, though, that I don't know if you can get such an app into the App Store - if you know, please comment.)

The challenge is to have the Finder tell the user if there's a problem, such as when the item is locked or when an admin login is needed.


Now, if you're trying to be lazy, you'd just write an AppleScript and invoke that from your application (this is especially easy to do in Real Studio / Xojo, but not much harder in ObjectiveC, either).

The downside of using an AppleScript is that it won't show these dialogs - if the item can't be moved to the trash, the function will return and give you an error message telling you about the problem.

To solve this, you need to invoke the AppleEvent directly, without going through an AppleScript.

The Xojo code would look like this, simplified:

  dim ae as AppleEvent = NewAppleEvent ("core", "delo", "MACS")
  dim list as new AppleEventDescList
  list.AppendFolderItem (theItemToDelete)
  ae.DescListParam("----") = list
  ok = ae.send

However, this won't give us the user interaction either, yet. That's because Xojo's Send command is lacking the options that the OS X API offers.

We have to call the native AESend() function via a declare, instead:

  const kAEWaitReply = &h00000003 ' sender wants a reply and will wait
  const kAEAlwaysInteract = &h00000030 ' interact where appropriate
  const kAECanSwitchLayer = &h00000040 ' interaction may switch layer
  declare function AESend lib "Carbon" (ae as Integer, reply as Ptr, 
    sendMode as Integer, prio as Integer, timeoutTicks as Integer,
    idle as Ptr, filter as Ptr) as Integer
  dim sendMode as Integer
  sendMode = kAEWaitReply + kAEAlwaysInteract + kAECanSwitchLayer
  dim err as Integer = AESend (ae.Ptr, nil, sendMode, 0, 0, nil, nil)
        
The same can be applied to copying files with user interaction as well - use the "clon" instead of the "delo" value for the AppleEvent.

I've made a REALbasic / Xojo project with both a MoveItemsToTrash() and CopyItems() function for your own use. Download here.

18 May 2014

Using the Cocoa animation API with Xojo

I've just published a Xojo / Real Studio class for doing smooth fade/mode/resize animations with windows and controls.

Find out more about it here: http://www.tempel.org/RB/CocoaAnimation.

26 March 2014

Customizing Xojo's controls on OS X with Cocoa

This article is about some advanced techniques for Xojo users that write OS X Cocoa apps.


If you are using Xojo to build OS X Cocoa applications, you can customize the controls much further than Xojo offers directly.

Let's take the PopupMenu. It's fairly limited in appearance (one look only) and capabilities (no icons in the menu, for instance).

Changing the appearance of such a control is easy because for most of Xojo's controls there is a Cocoa control equivalent, and you can directly alter those Cocoa controls from your code.

First, find out what you like to change in a control. You could look up Apple's documentation for the NSPopupButton, for instance. An easier way is to use Xcode as you can then see immediately the results in Xcode's interface designer.

Get Xcode from developer.apple.com, launch it, and choose New Project... from the menu.
Choose OS X - Application - Cocoa Application. Click Next. Give the project a name and check "Create Document-Based Application". Click Next and save it, e.g. to the Desktop.

In the browser menu on the left, select "Document.xib". That will show you the window editor, similar to Xojo's layout editor.

At the right of the Xcode window, at the bottom, there is a list of controls to choose from:

Find the "Pop Up Button" and drag it into the layout window. The window should then look like this:


Click on the added button to select it, and on right side of the Xcode window, click on the properties button (the one shown in blue here).

You will see the various properties you can alter.
For example, let's change the Style to Inline. Do that and see how the appearance of the popup menu changes:


Now let me show you how to do that in your Xode program.


I assume you know how to add a PopupMenu control to a window in Xojo.

Add an Open event to the PopupMenu in the window, and add the following code:

  Declare Sub setBezelStyle Lib "Cocoa" _
    Selector "setBezelStyle:" (obj as Integer, v as Integer)
  
  const NSInlineBezelStyle = 15
  
  setBezelStyle (me.Handle, NSInlineBezelStyle)

If you run your Xojo project, the popup button should now have the changed appearance.

However, it might be that the up+down arrows are centered, not on the right as they should. If you look at the Xcode project and play around with the settings for the Popup control, you'll see that the Arrow property controls this - if it's set to "Center", it's placing the arrows in the center. Let's see if we can fix the Popup menu in Xojo to have its Arrows set to "Bottom" instead. Add these lines:

  Declare Sub setArrowPosition Lib "Cocoa" Selector "setArrowPosition:" (obj as Integer, v as Integer)
  Declare Function cell Lib "Cocoa" Selector "cell" (obj as Integer) as Integer
  
  const NSPopUpArrowAtBottom = 2
  
  setArrowPosition (cell (me.Handle), NSPopUpArrowAtBottom)

Now I'll explain what's going on:

setBezelStyle is a function of NSButton, which is the super class of NSPopUpButton. Unfortunately, Xcode does not tell us that its "Style" property is actually called "bezelStyle", so I had to search the documentation for it anyway. me.Handle is the Xojo control's reference to the Cocoa object (i.e. the NSPopUpButton object), and we've then declared a Xojo function with the name setBezelStyle that invokes the "setBezelStyle:" selector, passing it an Integer value. Note that for every Cocoa property you find, there is always a setter function named after this pattern, e.g. "name" -> "setName:".

Getting access to the Arrow property was a little more complicated. Neither NSPopUpButton nor any of its super classes provides a function to set this property. Instead, it's found in the NSPopUpButtonCell class. To get to values of this class, we need to use the cell function (found in the NSControl class).

If you do not like to figure all this out yourself, you can also take a look at MacOSLib or use the MBS plugins. Especially with MBS it's very easy to acess these Cocoa control properties  or add icons to a PopupMenu's items.

But wait, there's more!


This is getting pretty advanced now.

Cocoa is built on Objective C, which, contrary to C++ and Xojo, lets you get right down to the lowest level where you can call functions (called messages in ObjC) by name. From a String.

Which means that you can store the function name you want to call in a String, or even read it from a file, and then invoke it.

Here's an example to show you what I mean. The above code for setting the style could also be written like this:

  Declare Function NSSelectorFromString Lib "Cocoa" _
    (aSelectorName as CFStringRef) As Ptr
  Declare Sub objc_msgSend_int Lib "Cocoa" Alias "objc_msgSend" _
    (obj as Integer, sel as Ptr, v as Integer)
  
  dim f as String = "setBezelStyle:"
  objc_msgSend_int (me.Handle, NSSelectorFromString(f), NSInlineBezelStyle)

See how "setBezelStyle:" is stored in a string variable here?

This Cocoa control also has an "alignment", so let's try altering that as well. The function name to change the value is thus "setAlignment:" (the ":" at the end is important and specifies that the function gets one value passed to it). So add this code:

  dim f as String = "setAlignment:"
  objc_msgSend_int (me.Handle, NSSelectorFromString(f), 1)

And with that, we've right-aligned the text in the popup menu.

With this procedure, you could now define customizations for your Xojo/Cocoa controls by specifying the properties and values in a file, and generically assign those properties in your code, without the need to write a special case for every property you might want to change.

Demo project!


Click here to download a demo project using above techniques.

18 February 2014

iClip 5.1.2 with Search and "Smart Sets" now in beta

For the past few years I've taken over the development of iClip 5, a Clipboard / Pasteboard tool for OS X after the original author abandoned it to instead get insanely rich with selling iOS apps and organizing Bundle deals.

iClip is now again in beta testing. The new version adds a search feature that also lets you create smart search sets in order to see ony specific clippings, e.g. only those from a particular applications.

If you like to help testing iClip, even if you've never used it before, please sign up with MacDeveloper, then subscribe to iClip there. The MacDeveloper site helps us (i.e. me and the folks at Irradiated Software) manage bug reports and feature requests, and in return you as a tester can collect points in order to get the software for free if you don't own it yet.

14 December 2013

New: OS X Prefs Editor for editing app defaults

Mac OS X 10.8 introduced a caching system for app preferences ("cfprefsd"). While this probably increases performances for apps, it makes it harder for developers to manipulate preference values quickly for testing, because making changes directly to the plist files in the ~/Library/Preferences folder does not work any more with editors such as "Property List Editor.app" and the similar one in Xcode.

Therefore, I've just written a new tool that works like the plist editor in Xcode but uses the CFPreferences functions to modify the values instead of editing the plist files directly.




If you've worked with "Property List Editor" or Xcode, you should have no trouble using my tool. It supports even Copy + Paste the same way. The only things I have not implemented (yet) are Undo and changing the type of existing elements (the reason for the latter is that REALbasic's framework does not offer a way to show a popup menu inside a listbox cell, and so I was too lazy finding a work-around for this to imitate Xcode's UI).

It even has one feature that Xcode's editor doesn't: You can delete all entries at once with Select All, then delete.

Also note that due to the use of the CFPreferences API, any changes you make are immediately committed to the preferences as seen by other apps, just like when you use the "defaults" command. For the same reason, changes made to the prefs by other apps (or with the "defaults" tool) will be seen in this tool after a change, a front switch or by pressing the "Sync" button. Hence, there is no Save command - updates are instant. For that reason Undo would be quite useful. Alas, I can live without it for now.

The tool is free for your use. Get it here:


It includes the REAL Studio (Xojo) source code as well. But I retain the copyright and distribution rights to the program, so please do not distribute the app yourself without getting my permission first!

To learn of new updates, use the RSS feed on this page or watch it on MacUpdate.

(Shameless plug: If you'd like to have a customized version, let me know, I also do work for money ;) )