To support the table of contents button in the OS X Help Book app you need to implement the following code. This isn't documented anywhere but I found it in Apple's own help files and tested it to work. I believe it only works in the more recent versions of OS X, so you if your app supports really old versions you might want to check it.

apple help book example.txt

//disable TOC button
window.HelpViewer.showTOCButton(false);

//enable TOC button
//If you call this on window load it will flash active for a brief second and then disable again.
//Call if after a delay of 250ms and is works fine
//Not sure what the second variable does yet, but passing false works fine
window.HelpViewer.showTOCButton( true, false, function() {
	 //do something to toggle TOC in your webpage
});

//Apple stores the status of the TOC using a window session variable, you can use this to keep the TOC open or closed
//when transitioning between pages.
window.sessionStorage.getItem("toc");

I'm on 10.14 and I've figured out some stuff about this.

The meaning of the parameters appears to be:

window.HelpViewer.showTOCButton(enabled, showCallback, hideCallback)

The reason you have to click the button twice on 10.13+ is that Help Viewer looks at your HTML to see if the TOC is visible, and it only calls your callback if it thinks the visibility state has changed. So, you need to update your HTML to tell Help Viewer whether your TOC is visible.

Specifically, Help Viewer executes the following JavaScript to check if your TOC is visible:

document.querySelector("nav[role=navigation]").getAttribute("aria-hidden")

So, you need to do something like this:

<nav role="navigation" aria-hidden> </nav>

function showTOC() {
    document.querySelector('nav').setAttribute('aria-hidden', false);
    // show your TOC
}

function hideTOC() {{
    document.querySelector('nav').setAttribute('aria-hidden', true);
    // hide your TOC
}}

setTimeout(function() {{
    window.HelpViewer.showTOCButton(true, showTOC, hideTOC);
}}, 250);

Somebody remind me why I develop for this platform...

Also: In order to enable the Share menu - specifically to enable users printing a user guide - the HPDBookRemoteURL must be specified and set to a copy of the Resources folder on your server