Category Archives: Mac OS X

Catalina’s Custom Keyboard Viewers

Long-time Mac users will remember an app called “Key Caps”, which later become “Keyboard Viewer”, a feature of the Mac that is now accessible via the menu bar’s “Input Methods” item. If you’ve never played with this, I encourage you to enable it and check it out. Apple has detailed instructions for configuring the menu and these options.

I don’t use the Keyboard Viewer often, but when I do, it’s a real life-saver. I brought it up recently while I was debugging an issue with keyboard shortcuts in FastScripts, my scripting utility app. The Keyboard Viewer not only reflects every bona fide keystroke you make on a hardware keyboard, but also allows you to simulate keystrokes by tapping on the keys of the on-screen keyboard.

On macOS 10.15 Catalina, Apple has evidently dramatically overhauled the Keyboard Viewer. I don’t see any hint of this on the Apple marketing sheet for the OS, but this is what the Keyboard Viewer looks like on my Mac now:

NewImage

Well, isn’t that spiffy? But what I really want to talk about is that little Gear Button in the upper right corner of the window. Click it, and this what you get:

Popup menu with various options for customizing keyboards

A whole slew of options for tweaking the behavior of the virtual keyboard, and an enticing “Customize…” item at the bottom. When you select it, a dedicated application called “Panel Editor” opens up. It’s essentialy a construction set for building virtual keyboard layouts:

Custom keyboard editor with silly bitsplitting.org keyboard layout

This example is obviously comical, but the point is you can create and layout tappable regions that correspond to whatever keystrokes you desire. The options for configuring these keys even include options to perform multiple keystrokes, open apps, run scripts, etc. It’s a powerhouse of utility superpowers.

How did they possibly find time to add all this great functionality in one OS upgrade? They didn’t. Folks who are familiar with Apple’s Accessibility Keyboard have no-doubt recognized my screenshots as being familiar from past OS releases. I personally had never seen it before, but it’s been hiding in the System Preferences Accessibility tab. What happened in macOS 10.15 Catalina is that Apple has evidently recognized its superiority in all ways to “Keyboard Viewer” and allowed the Accessibility Keyboard to simply take its place.

This is an excellent example of software being designed to assist people with specific needs, yet actually being useful to everybody. That is the heart of accessible software design, and I think we’ll see more and more “accessible” software released from the relative obscurity of the Accessibility tab as we move forward.

Terminal Security Profiles

In macOS Mojave, Apple introduced a number of new security features that impact the day-to-day use of the computer. Activities such as running scripts, or using apps that access private information, are altered now such that users are prompted with one-time permission-granting requests.

One consequence of these changes is that you can no longer access certain parts of your home directory from the Terminal. Don’t believe me? Try opening Applications > Utilities > Terminal, and run the following command:

ls ~/Library/Mail

In all previous macOS releases, this would list the contents of Apple’s internal Mail files. As a privacy enhancement, access to these files is now restricted unless apps have requested or been proactively granted access.

If you really wanted to regain access to these files via the Terminal, you have to grant the app “Full Disk Access.” This is a new section of the Security & Privacy pane in System Preferences.

Well, that’s fine. Now you can “ls” anything in your home folder, but absolutely every other thing you run in Terminal can as well. To grant myself the ability to list files in ~/Library/Mail, am I willing to grant the same access to every single thing I’ll ever run in Terminal?

This isn’t earth-shattering: it’s been the case forever that tools you run in the Terminal have access to “all your files.” But the new restrictions in macOS Mojave shine a light on a problem: the bluntness of security restrictions and relaxations with regard to Terminal.

I’ve run into a variation of this problem in the past. I use the excellent TripMode to limit bandwidth usage when I’m traveling, and tethered to my phone. A consequence of this is that, unless I grant unlimited network access to Terminal, I can’t perform routine tasks such as pushing git changes to a server.

Ideally these permission grants would be applicable at the tool level, rather than at the application level. It would be better if I could say “let ls access my Mail” rather than “let anything I run from Terminal access my Mail.”

I don’t completely understand the limitations there, but I suspect that because commands in the Terminal are running as subprocesses of Terminal, there is some technical challenge to making the permissions apply at such a fine-grained level.

As an alternative, I wonder if Apple could introduce some kind of “Security Profiles” feature for Terminal so that individual windows within the app could be run when different permissions? This could build on Terminal’s existing support for “Profiles” which already support varying Terminal settings dramatically on a per-window basis.

With Security Profiles, a user would be configure an arbitrary number of named profiles, and security privileges acquired by Terminal would be stored separately for the active profile. Each profile would be considered by the system effectively as a different app. For example, given my uses of Terminal, I might set up a few profiles for the types of work I regularly do:

  • Personal: Everyday productivity tasks including running scripts, editing files in my home directory, etc.
  • Administrative: Tasks that pertain to the overall maintenance of my Mac: examining system logs, delving into configuration files, etc.
  • Collaborative: Tasks that involve installing and running third-party tools that I trust, committing to shared source repositories, etc.
  • Experimental: Tasks that involve installing or running third-party tools that I am not familiar with and do not have a high degree of faith in.

These are off the top of my head, and just to give an idea of the kinds of profiles that might make sense here. Switching between these modes would also switch the system’s active list of entitlements for Terminal. If I run a script that accesses my Calendar items from the “Personal” profile, the system would prompt me once to ask my permission, but never prompt me again in that profile. When I switch to “Experimental” and run some unfamiliar third-party tool that tries to access my calendar, it would ask permission again for that profile.

I filed Radar #45042684: “Support a finer-grained permissions model for Terminal”, requesting access for this or something like it.

More on Mojave’s Automation Sandbox

I wrote last month about macOS Mojave’s restrictions on automation, and how users can reset the database that controls them. In that post, I cited Felix Schwarz’s excellent article on the subject.

In recent weeks, Apple has made changes to the behavior of macOS Mojave, and added some API calls to help developers better handle the restrictions of the system. Felix is back with an updated post, describing the changes, and what he thinks can still be improved.

Reauthorizing Automation in Mojave

The macOS Mojave betas include a significant enhancement to user control over which applications can perform automation tasks. When we talk about automation on the Mac, we usually think of AppleScript or Automator, but with a broader view automation can be seen as any communication from one application to another.

One ubiquitous example of such an automation is the prevalence of “Reveal in Finder” type functionality. For example if you right-click a song file in iTunes, an option in the contextual menu allows you to reveal the file in the Finder. This is a very basic automation accomplished by sending an “Apple Event” from iTunes to the Finder.

In the macOS Mojave betas, you’ll notice that invoking such a command in an application will most likely lead to a panel asking permission from the user. The terminology used is along the lines of:

“WhateverApp” would like to control the application “Finder”.

If the user selects “OK”, the application sending the command will be thereafter whitelisted, and allowed to send arbitrary events (not just the one that prompted the alert) to the Finder. If you’re running macOS Mojave you can see a list of applications you’ve already permitted in System Preferences, under “Security and Privacy,” “Privacy,” “Automation”.

These alerts are a bit annoying, but I can get behind the motivation to give users more authority over which applications are allowed to control other applications. Unfortunately, there are a number of usability issues and practical pitfalls that come as side-effects of this change. Felix Schwarz made a great analysis of many of the problems on his blog.

I ran into another usability challenge that Felix didn’t itemize: the problem of denying authorization to an application and then living to regret it. I guess at some point I must have hastily denied permission for Xcode (Apple’s software development app) to control the Finder. This resulted in a seemingly permanent impairment to Xcode’s “Show in Finder” feature. I’m often using this feature to quickly navigate from Xcode’s interface to the Finder’s view on the same files. After denying access once, the feature has the unfortunate behavior of succeeding in activating the Finder (I guess that one is whitelisted), but failing silently when it comes to revealing the file.

OK, that’s fine. I messed up. But how do I undo it? Unfortunately, the list of applications in the Security and Privacy preference pane is only of those that I have clicked “OK” for. There’s no list of the ones that I’ve denied, and no apparent option to drag in or add applications explicitly. For this high level problem, I filed Radar #42081464: “TCC needs user-facing mechanism for allowing previously denied privileges.”

What’s TCC? I’ll be darned, I don’t know what it stands for. But it’s the name of the system Apple uses for managing the system’s so-called “privacy database.” This is where these and other permissions, granted by the user, are saved. For instance, in macOS 10.13 when the system asks whether to grant access to your Address Book or Contacts, the permission is saved, and managed thereafter, by TCC.

Resetting TCC Privileges

I knew from past experience testing Contacts privileges in my own apps, that Apple supports a mechanism for resetting privileges. Unfortunately, it’s pretty crude: if you want to change the authorization setting for an application you’ve previously weighed in on, you have to universally wipe out all the privileges for all apps using a particular service. For Contacts, for example:

tccutil reset AddressBook

This completely removes the list of apps authorized to access Contacts. (The AddressBook naming is a vestige of the app’s former user-facing name.) In fact, if you type “man tccutil” from the Terminal, you’ll find that AddressBook is the only service explicitly documented by the tool. Fixing my Xcode problem is not going to happen by resetting AddressBook privileges. So what do I reset? I tried the most obvious choice, “Automation,” results in an error: “tccutil: Failed to reset database”.

What’s the service called, and does tccutil even support resetting it? After a crude search of the private TCC.framework’s binary, I discovered I was looking for “AppleEvents”:

tccutil reset AppleEvents

After running this, I quit and reopened Xcode (the TCC privileges seem to be cached), and selected “Show in Finder” on a file. Voila! The Finder was activated and I was again asked if I wanted to permit the behavior. This time, I made sure to say “OK.”

You can get a sense for the variety of services tccutil apparently supports resetting by dumping the pertinent strings from the framework:

strings /System/Library/PrivateFrameworks/TCC.framework/TCC | grep kTCCService

The list of matching strings includes names like AppleEvents and AddressBook, as well other names for things I don’t recognize, and a seemingly useful “All,” which can presumably be used to wipe out all authorizations across all services.

Because the tccutil is far more useful than is advertised, and because users are undoubtedly going to end up needing to reset services more than ever in Mojave, I also filed Radar #42081070: “Documentation and command-line help for tccutil should enumerate services.” There are some items in the dumped list that appear likely to be private to Apple, but anything genuinely useful to customers (or more likely, the consultants who fix their Macs) should be listed in the manual.

Lighten Up, Eh?

While I support the technical and user-facing changes suggested by Felix Schwarz in the previously linked blog post, some issues would be avoided by simply giving apps the benefit of the doubt for widely used, innocuous forms of automation.

I mentioned earlier that the Apple Event sent by Xcode to “activate the Finder,” was apparently whitelisted by the system. Evidently Apple saw wisdom in the thinking that simply causing another application to become active is unlikely to be widely abused. I think the same argument holds for asking the Finder to reveal a file. I filed Radar #42081629: “TCC could whitelist certain widely used, innocuous Apple Events.”

I mentioned before that I can support Apple’s effort to put more power into users’ hands with this feature, but one side-effect of requiring the authorization even for innocuous events like “Show in Finder” is that apps that do not otherwise offer automation functionality to users will nonetheless require that users grant that power.

If the merit in the feature is to allow users to limit what kinds of automation apps can perform, then supporting a “Show in Finder” feature for an application should not require me to simultaneous permit it to do whatever kind of Finder automation it chooses to. For example, an application so-authorized is now empowered, presumably, to send automation commands to the Finder that modify or delete arbitrary user files.

These days Apple always seems to be pushing the privacy and security envelope, and in many ways that is great for their users and for their platforms. With a little common-sense and some extra engineering (“It should be easy” — Hah!), we can get the best of the protection these features offer, while suffering the fewest of the downsides.

Unified Swift Playgrounds

Usually I write about “programmer stuff” on Indie Stack, and “other tech stuff” here.

Today I had it in mind to write about my belief that Apple should get rid of Xcode Playgrounds and instead focus on adapting the iOS Swift Playgrounds app to the Mac.

I thought at first I would post it here because it’s not strictly about programming, but I ended up posting it there instead:

Unified Swift Playgrounds

May be of some interest even to folks not strictly interested in programming!

Accessible Resistance

Accessibility in software refers to the noble ambition of ensuring that software is usable by as diverse a user base as possible. To that end, software is made more accessible by adapting to a variety of physical or cognitive impairments that may affect any individual user.

In the United States and other countries, there is an ugly trend towards supporting politicians who don’t believe that people from diverse backgrounds, or with specific impairments, should be accommodated by society as a whole.

Many developers are looking for concrete ways to fight these politicians who don’t value diversity and inclusion. One small thing we can all do to push back, to resist, is to ensure our own apps are as accessible as possible.

During my many years as an indie Mac developer, I have often prioritized accessibility in my apps. I have heard from many MarsEdit users, particularly those with vision difficulties, who tell me its accessibility makes it a better alternative to many other blogging solutions.

I am gratified to hear about the ways I have gotten accessibility right, but I am still not satisfied that I have done enough. There are nuances of MarsEdit’s accessibility that can yet be improved, while some of my other apps, such as Black Ink, are still hardly accessible at all.

If you are a Mac or iOS developer who is committed to improving the accessibility of your app, a great place to start is with the WWDC 2016 What’s New In Accessibility session. Apple is always enhancing the variety of accessible features that are built in to iOS, macOS, tvOS, and yes!, even watchOS.

Spend a half hour watching this video, and start getting up to speed with how you will enhance the accessibility of your app. No matter where you live in the world, you can be a strident voice for inclusion by declaring, through your actions in Xcode, that your software is designed to be used by everyone.

Bitter iCloud Truth

A few days ago, I set about tidying up my completely unruly Contacts database. Over the years I have accumulated 2,953 distinct contact cards. Thousands are duplicates, hundreds were added automatically by Mail or macOS, some are for people whom I no longer know, and a very few are for folks I no longer want to know.

The first thing to do was obviously to take care of those duplicates. Contacts on the Mac features a couple seemingly handy menu items for dealing with this problem: “Look for Duplicates” and “Merge Selected Cards.” I cannot recommend using either of these features.

When I invoked “Look for Duplicates” on my 3000 cards, of which I could tell by visually scanning that at least 1000 were duplicates, Contacts came back at me with a stunningly unhelpful offer to fix 9 of them:

NiceTry

Seeing that I would have to identify the clones myself, I committed to selecting groups of two or three clearly duplicated identities, and selecting Card -> Merge Selected Cards. I generally trust Apple software to do the right thing, but recognizing the heartache that would come with losing any information from these cards, I decided to proactively export my Contacts database so that I would be safe should anything go wrong. Selecting File -> Export… -> Contacts Archive from the menu bar produced a 60MB bundle that seemed certain to contain all of the contact information on my Mac.

I tried “Merge Selected Cards” on a few items, and it seemed to do the trick. Where data from the two cards was identical, it merged them cleanly. Where something was in conflict, for example if a Company Name had changed from “Google” to “Apple” at some point, the conflict was resolved by picking one and appending the other to the merged card’s “Note”.

After painstakingly merging cards in this manner for an hour or so (!), I had stopped paying close attention to whether conflicting data was being persisted well or not. At one point I stumbled upon the realization that I lacked the phone number for a contact whom I had sent an SMS message just within the past week. Other contacts were missing key data, too. An outdated email address here, a missing mailing address there. Whoops! Abort mission! Time to recover from that backup file.

I double-clicked the backup file in the Finder, and agreed when Contacts asked if I was sure I wanted to “replace my Contacts data” with the contents of the archive.

Screenshot 1 3 17 1 14 PM

You’re damned right, I am sure! Give me back my data.

After restoring from Backup, I found that I didn’t have any duplicate entries in Contacts. In fact, I only had around 130 cards. Somehow I had lost the vast majority of cards in this process. What’s going on here?

I think what happened is that exporting from Contacts on my Mac is creating an archive that, when imported, possibly reflects the state of my Contacts database from years ago, before I ever agreed to sync Contacts with iCloud. No amount of futzing with local data on my Mac, including disabling iCloud syncing, and restoring the contents of Contacts’s data folder (~/Library/Application Support/AddressBook) would coerce Contacts into showing me all of my Contacts. I began to panic. Had I actually lost all of my Contacts data? In spite of dutifully backing up my local data and making a proactive archive from Contacts, the app had a very different idea of what my Contacts “truth” was.

I was relieved to discover an iCloud.com feature for recovering Contacts to the state they were in on a previous date. You get to it by navigating to iCloud Settings -> Advanced, and clicking the “Restore Contacts” link.

NewImage

After restoring Contacts through iCloud.com, and syncing to iCloud from my Mac, my Contacts were finally restored to their absurdly broken, but data-intact, state.

The overarching lesson here is that when it comes to iCloud-synced data, you cannot count on local data, and worse: you can’t count on local archives. If you care to archive your data, navigate to iCloud.com and use the pertinent export feature from the web site. For example, from iCloud.com’s Contacts interface, click on a contact, then select “Select All”, and then “Export vCard…”.

ExportVcard

Amusingly, downloading this file will probably cause Contacts on your Mac to immediately attempt to import all of the contacts, which of course you will want to decline to do. Apple has a reference page, with detailed instructions for archiving various kinds of iCloud data.

The other lesson of course is you can’t trust Contacts to do a good or even acceptable job merging Contacts. I’ll be looking for another solution on this front. On Twitter, a f waller suggested Smart Merge Pro for iOS, but I tried that and it unfortunately is also blind to the duplicate nature of my cards! I guess something special is making my cards appear non-duplicated to both Contacts and this third-party app.

Update: I figured out why “Look for Duplicates” behaved so poorly: it wasn’t even considering my iCloud-based cards. So when I invoke it in Contacts on my Mac, it evaluates only the ~140 cards that are considered local to my Mac, and completely ignores the ~3000 cards that are synced via iCloud. Unfortunately there is no “Look for Duplicates” on iCloud.com, so what is a Mac-based user, for whom Smart Merge Pro doesn’t work, to do?

I think the way it works is it only considered “On my Mac” cards, if there are any local cards. If all your cards are iCloud based, then the “Look for Duplicates” command seems to work as expected.

Touch Bar Everywhere

Apple announced on Thursday that many of their new MacBook Pros will ship with a “Touch Bar,” a narrow, high resolution touch screen in place of the Mac keyboard’s traditional Function Key row.

Many people immediately wondered whether we can expect Apple to release an external keyboard with the Touch Bar. This would bring the technology to the much wider audience of Mac users who are not ready to update to the latest MacBooks, or who prefer desktop Macs, or who prefer the flexibility of using their MacBook in a desktop-style configuration.

The question was discussed on the latest Accidental Tech Podcast, in which, if memory serves, John Siracusa and Marco Arment argued different angles of the “no” argument, citing the hardware cost, the extent to which a Touch Bar keyboard would complicate the accessories lineup, and perhaps most significantly, that Tim Cook does not care enough about the Mac to prioritize pushing any such technology.

Casey Liss watched football. Zing! Actually, Casey pushed back against the cynicism, suggesting that Apple’s apparent lack of enthusiasm for the Mac does not reflect a lack of commitment to improving it. I, on the other hand, take exception with the very suggestion that Apple lacks enthusiasm or is not investing heavily in the Mac. Or, at least in this feature.

I think Apple intends to push the Touch Bar as as widely as it possibly can. The current MacBook Pro lineup is the most practical computer to debut the feature, but as it becomes possible to bundle it with external keyboards, and on notebook computers at every price point, they will do so.

Why am I so assured of Apple’s big plans for the Touch Bar? Because while many people assert that Apple is not investing seriously in the Mac, the Touch Bar’s hardware and software support appear to have been a major priority for the company in the year or two leading up to its release.

A massive amount of design work must have gone into the Touch Bar’s physical hardware, structuring the information it represents, and deciding how users will most usefully interact with it. I suspect that the Touch Bar merited an amount of design effort perhaps less than, but not completely incomparable to a standalone product like the Watch.

Thanks to the Touch Bar simulator in Xcode 8.1, we can also already take stock of the sheer amount of engineering effort, across many disparate groups in the company, that went into supporting the Touch Bar from a wide range of different apps and modes in macOS. Leave the simulator running while you go about your work, and prepare to be repeatedly surprised by the variety of novel use cases that have already been identified and implemented.

I find it impossible to believe that Apple would go to all this work, both on the Touch Bar itself, and across the entire range of its own apps and OS features, unless it had a grand vision for the Touch Bar that extends way beyond the internal keyboard of its premium notebook computers.

Instead, I think Apple sees the Touch Bar as a long-term, distinguishing aspect of using a Mac. Users will always be able to get by without one, just as they do for example when a multi-touch trackpad is not available. But macOS, and nearly every app that users run on it will work better with a Touch Bar. One day we’ll expect to always have access to one, and will feel that something is missing if we don’t.

It’s easy to see why Apple couldn’t come charging out of the gate with this vision fully realized. The Touch Bar hardware is no doubt expensive, and there are probably practical considerations with respect to the security of Touch ID, bandwidth between the device and the Mac, and managing its power needs in a user-friendly manner.

I say give Apple time. They’ve made a huge investment in Touch Bar, and all indications are they are prepared to continue prioritizing support for it down the road. We’re only on the brink of entering the early adopter phase, but in years to come I do think the Touch Bar will be everywhere.

Log Littering

Apple has dramatically revamped its standard logging mechanism. Unified Logging, available in macOS 10.12 and iOS 10, replaces various file-based logging approaches with a centralized, database-backed repository for log information of all levels of interest.

The big win, both for developers and for users, is performance. Where the simple act of logging some debugging information used to have enough of a cost to dissuade rampant use, the new system makes it so cheap that developers inside Apple and out are being encouraged to “log like the dickens” in the name of generating more evidence for posthumous debugging. The new system also offer impressive data mining abilities, making it far easier to filter for what you’re looking for and to track logging information across specific activities, even if the activity spans multiple processes.

The two big losses, in my opinion, are that the sheer size, number, and variety of logging messages makes it impractical for users to skim the console for “real problems,” and that the resulting logging archives are so large that it’s impractical to casually include them with bug reports to Apple or 3rd party developers.

The WWDC session on the topic goes into detail on these points, but perhaps most useful for framing my complaints here are Apple’s stated goals for the revamp:

  • One common, efficient logging mechanism for both user and kernel mode
  • Maximize information collected while minimizing observer effect [the phenomenon that enabling or adding logging changes the behavior of the system you are trying to understand]
  • We want as much logging on all the time as possible
  • Design privacy into the system

Two of the stated goals, “maximizing information collection,” and “as much logging on all the time as possible,” exacerbate the problems of there being so much log data that it’s both difficult to skim, and cumbersome to share.

Imagine a society in which all packaging has been transformed into fast-composting, biodegradable materials. Your soda bottles, snack wrappers, cigarette packages, etc., all biodegrade to dirt within 48 hours of use. What a boon for the world: the major, global problem of trash accumulating in our towns and environment would be gone. Poof!

Or would it? When “littering has no cost,” I suspect that we’d face a new problem: far more people would litter. Why bother finding a place to throw out that bottle, when nature will take care of it in within 48 hours? Multiply this attitude out over even a small portion of the world’s billions of people, and we’d be guaranteed to be buried in trash. All trash that will be gone in 48 hours, mind you, but constantly replenished with a fresh supply.

I think this metaphor points to a similar problem with unified logging: the sudden onslaught of “low cost logging” has left our developer society unprepared in some specific ways to deal with the consequences of the new reality.

So, how can we, and Apple fix this? We need specific solutions that make skimming for problems, and sharing pertinent log data easier, yet don’t impact the very positive goals outlined by Apple above. Here is my advice:

  1. Don’t litter the logs. Just because you can log it, and just because doing so is cheap, doesn’t mean there isn’t a cost. Some of the recurring log messages littering my Console arrive at a rate of thousands per minute. There may be good arguments that certain subsystems, in certain states, should log this extensively if the resulting data will justify easier diagnosis of problems, but much of the junk I see are redundant reiterations of relatively useless information:
Not switching as we're not in ~/Library/Keychains...
...
CSSM Exception:...
...
Etc. Etc. Etc.

Some of these seem to be well and truly noise, but some of them, even if they seem highly redundant and noisy to me, could in fact represent useful logging if you happened to, for example, be tracking down a subtle bug in Apple’s security libraries. That’s fine, I’ll accept that some amount of log diarrhea in the name of better software, but only if the following accommodation is made…

  • Annotate high-frequency logs for easier filtering. The logging system offers a variety of tools for annotating log messages, but even internal Apple groups do not seem to use these extensively or appropriately. The system supports the notion of three levels of log message: default, info, and debug. Only the “default” level messages are displayed by default in the Console app, yet all of the above-described garbage is displayed in that default mode. The Console will become significantly more useful when the worst offenders have marked their messages for severity, subsystem, and category so that they can be effectively omitted from the default view, and so that they can be mined later for use by folks who can actually make sense of them.

  • Generate context-specific sysdiagnoses. One of the worst practical outcomes of unified logging is that capturing a “sysdiagnose” on my Mac now generates a compressed archive that is still larger than 400MB. These archives are typically requested by most groups at Apple in response to bug reports, regardless of how severe the bug is or how readily reproducable it appears to be. In the world of Apple bug reports, sysdiagnoses are treated like lightweight bits of information that should be appended to every issue, except they’re now anything but lightweight.

    All the annotation work advised above should really pay off when Apple provides a streamlined mechanism for capturing sysdiagnose information pertinent to a specific subsystem or product. The company already offers product-specific advice for capturing log information, sometimes requiring the installation of custom profiles and other shenanigans. The lightweight unified logging system empowers Apple to both require that internal groups properly annotate their log messages and to facilitate smarter gathering of that data.

    Currently if you want to capture a sysdiagnose at any time a Mac, you simply press Ctrl-Opt-Cmd-Shift-Period. On an iOS device it’s done by holding both volume keys and the power key, but you have to enable logging first with a custom profile. This shortcut will generate one of those mondo 400MB style sysdiagnose archives, which you can upload to attach to your bug reports in your copious spare time.

    I envision a prompt that appears after invoking the existing shortcut, or else a new shortcut for “interactive sysdiagnose” where you could specify the category of bug you are reporting. The prompt would list categories correlating to groups at Apple who had done the work of providing streamlined log filtering data that will effectively strip out all the useless (to that group) noise from your log data.

    In fact, sysdiagnose already takes a “process” parameter when invoked from the command line, and my understanding is that this enables it to capture data that is pertinent to the target process. The unified logging system seems to provide the infrastructure for even smarter capturing along these lines.

    I realize that in this time of flux, where much log data is not properly annotated, there is still an incentive for many groups to capture as much as possible and sort it out on Apple’s end. These groups should know, however, that the larger the sysdiagnose archive you ask users to upload, the lower the chances they will bother actually following through on filing the bug or on providing the pertinent information you have requested.

  • Annotate specifically for user-concerning issues. Power users on the Mac have, for years, counted on being able to skim the console log for information that might explain a slow Mac, persistent crashes, or other inexplicable behavior. Given the huge increase in number of log message and the lackluster annotation of these messages by Apple, the Console app is effectively useless to users for this purpose.

    On top of getting the annotation right in general, I think a new special level of log message should be created, indicating particular concern to end-users. These would identify log messages that developers specifically think users should take notice of and follow up about. I mentioned there are three basic levels of log message: default, info, and debug. On top of that, there are two special levels called “fault” and “error”, which can be filtered on in the Console, and which receive special treatment from the logging system.

    These special levels seem close to what users might find interest in, but they’re not quite right either.

    A “user interest” level of logging message would facilitate the kind of high level skimming that seems impossible in Sierra today. These log messages would convey information that a user might gain some insight from. They would stand in stark contrast to the noise of messages like these, from the filecoordinationd process:

    Claim D32EDEBE-C702-4638-800C-E4BAB9B767F3 granted in server

    Nobody knows what a claim is, or cares whether it was “granted in server” … unless they are actively debugging filecoordinationd. On the other hand, a message indicating the failure to grant a claim “because the disk is full,” could be very useful to users indeed.

    I realize the line could be difficult to draw in some circumstances, but a huge number of messages currently being logged are obviously not of user interest. Perhaps then, the opposite approach could be taken, and log messages could be annotated as being specifically useful for posthumous, internal debugging by developers.

I tell you, things are bleak. We’re swimming in a morass of useless console noise, and there is little we can do about it. There is room for optimism however, as the logging infrastructure is good, and seems to support many thoughtful measures that could attenuate the problems and make the system more useful and less cumbersome to developers, end-users, and well-meaning bug reporters.

The console is littered with trash. Sure, it will biodegrade in 48 hours, but that doesn’t mean we shouldn’t be concerned with cleaning it up. A combination of more thoughtful logging, proper annotation, and appropriate filtering will get us out of this mess.

Bad Preference Gatekeeper

With the release of OS X 10.11.4, developers of standalone preference panes face a new challenge with respect to users installing their software.

Apparently, the validation process that Apple applies to downloaded software, Gatekeeper, fails to validate OS X preference panes, even if they are signed with a legitimate Developer ID code signature.

The upshot of this is when users download a bona fide 3rd party preference pane such as Noodlesoft’s excellent Hazel, instead of having the software install as expected, a scary warning is displayed indicating the purported untrustworthiness of the software.

According to Paul Kim of Noodlesoft, the problem affects every preference pane he’s tested, including a freshly built, completely plain preference pane built with Apple’s latest tools. I put this to the test in Xcode 7.3, running on 10.11.4, by creating a new Preference Pane project from Apple’s template, setting it to sign with my Developer ID, and creating a release build of the project.

Running Apple’s “spctl” tool on a binary is a reasonably approximate way of determining whether Gatekeeper would reject the binary after downloading it from the web. Here’s the result for all the affected preference panes:

% spctl -av ./TestPanel.prefPane
./TestPanel.prefPane: rejected
source=obsolete resource envelope

Ah, that pesky “obsolete resource enveloped” message. Those of us who survived the transition from Version 1 to Version 2 code signing remember it well. But it’s not an accurate assessment in this case:

 % codesign -dv ./TestPanel.prefPane              
Executable=/Volumes/Data/daniel/Desktop/TestPanel 2016-03-31 14-03-51/Products/
[...]
Sealed Resources version=2 rules=12 files=2
Internal requirements count=1 size=220

The “version=2” indicates we are using an appropriate version for the signed resources. It would be hard, perhaps impossible, to do otherwise on a modern system with a modern Xcode toolchain.

The “spctl” tools supports a command line option to ask for more and more verbose results by adding “v”s to the command line. Unfortunately “spctl -avvvvvvv” doesn’t yield anything more informative than the seemingly inaccurate “obsolete resource envelope.”

I wondered if there was some magic flag that preference panes must now exhibit, or some new requirement that internals be signed in a different way than before. Surely, if anybody could get this right, it would be Apple! Their “Network Link Conditioner” is the only downloadable preference pane I could think of, and what do you know, it was updated as part of the Hardware I/O Tools for Xcode 7.3 download package, released on March 20. I downloaded a fresh copy to be sure I had the best that Apple could offer, located the preference pane, and double-clicked it.

Untrusted

You know it’s bad when even Apple’s own downloads are portrayed as untrustworthy.

This is a minor annoyance for folks trying to install an obscure development tool, but it’s a major issue for developers like Noodlesoft whose entire livelihood is built on the distribution of software packaged as a preference pane. The scary wording in the dialog casts doubt on the reputation of the developer, and for the more savvy, on the reputation of Apple’s ability to properly assess the trustworthiness of software that we download.

Let’s hope Apple can address this problem soon. Although it doesn’t pose a security risk, it seems appropriate that they could include this in a security update. After all, it has everything to do with preserving trust between users, developers, and Apple.

Update: According to Paul Kim, it’s not just preference panes that are affected, but any standalone non-app code bundle. So, for example, color palettes, screensaver modules, and, if anybody ever used them anymore, Dashboard widgets, are all affected. Pretty, pretty, pretty, pretty bad.

(Radar #25468728)

Selectively Smart Typography

OS X offers a system-wide feature to “Use smart quotes and dashes” when typing text. The option can be enabled or disabled in the System Preferences “Keyboard” pane, inside the “Text” tab:

Keyboard

This essentially ensures that when you rattle off a bit of prose, any quotation marks will be “smartened” so that they curl inwards towards the quoted phrase, and sequences of double-hyphens will be converted to typographic dashes.

It’s a good thing.

However, the feature can be vexing when typing in contexts where typographic beauty is clearly not the priority. For example the behavior would not make sense in a code editor like Xcode or BBEdit. These apps are more often used to write text that will please a programming language compiler or interpreter, where a string for example must be enclosed by "straight quotes", and “smart quotes” would only lead to a syntax error.

Apple actually makes it possible for developers to control the behavior of this smart typography on a very fine-grained basis. For example, my own app MarsEdit features an HTML text editor in which it makes sense to support smart quotes and dashes, but not within markup tags. Because the app is aware of when the user is typing within a tag or not, it simply disables the functionality as appropriate for the context.

Depending on the apps you use, and what you use them for, you might find yourself vexed by the unwanted conversion of quotes and dashes. You could turn the feature off at the System Preferences level, but what if you want to keep it in all the other apps where it works just as you want it to?

Thanks to the flexibility of Apple’s “user defaults” system for registering preferences on a system-wide and app-specific basis, you can impose override preferences for these options on the specific apps of your choosing. I was inspired to write this article by Jesse Atkinson, who complained on Twitter that the behavior was meddling with his code snippets in Slack:

So, here’s a general recipe for overriding the settings in any app:

  1. Quit the app.
  2. Determine the “bundle identifier” for the app. Slack’s, for example, is “com.tinyspeck.slackmacgap”.
  3. Open the OS X Terminal app.
  4. Type the following commands to impose overriding defaults for the pertinent features:
    defaults write com.tinyspeck.slackmacgap NSAutomaticDashSubstitutionEnabled -bool NO
    defaults write com.tinyspeck.slackmacgap NSAutomaticQuoteSubstitutionEnabled -bool NO
    
  5. Reopen the app.

You should now find that this specific app behaves as though the system-wide setting for these features were turned off.

If you want to undo your handiwork, just delete the overrides you put into place:

  1. Quit the app.
  2. Open the OS X Terminal app.
  3. Type the following commands to delete the overriding defaults:
    defaults delete com.tinyspeck.slackmacgap NSAutomaticDashSubstitutionEnabled
    defaults delete com.tinyspeck.slackmacgap NSAutomaticQuoteSubstitutionEnabled
    
  4. Reopen the app.

If you’re a Slack user who has also been annoyed by the smart quotes and dashes functionality, the above may prove helpful to you. If you’re not, maybe a similar situation will arise where you can use the power of OS X’s default registration system to fine-tune the behavior of your favorite apps.

Update: My old friend Chris DeSalvo, who also happens to work at Slack on the Mac Slack app points out that there is a much simpler solution for Slack in particular:

I just confirmed that this seems to work well with Slack! So was this whole article for naught? In my experience, toggling the Substitutions flags as Chris describes only changes the setting temporarily. This is because those menu items within an app are usually configured to communicate directly with the text field you are focused on. A developer has to go out of her way to intercept and persist such a toggle, to ensure that it “sticks” the next time the app launches.

So! If you have this problem with Slack, just toggle the setting in the Edit -> Substitutions menu. With most other apps? Perhaps the technique outlined above will help.