Publishing with Silicon

We have built solutions using InDesign Server since it came out, and before that we were building solutions based on InDesign desktop for 5 years. So we know the XMPie space pretty well.

XMPie is a really well-built program, that to me has three main benefits:

1. It lets you easily define a data source for variable content (using uPlan) and reference that data source directly in InDesign (via uCreate)
2. It manages XMPie jobs (via the uProduce server), with functionality exposed as Web Services
3. It optimizes print output, producing VPS (which has been known to work), PPML, “VIPP” (which is known not to work; it is not VIPP but a VIPP wrapper around PostScript), etc.

XMPie is salvation for the designer at a mail house: they can bypass Programming entirely and set up their own “campaign” based on new InDesign/data input from a client.

Yet these days, we never meet such a designer. We meet enterprise clients, who consider themselves very special and do things a very special way. XMPie invariably meets their needs 40-80% of the way, but the other 20-60% can take a supreme effort. So we need to request extensibility from XMPie, and in many case fuse together an XMPie workflow with a very non-XMPie workflow. They may be sick of my requests, but they have given us more and more extensibility over time.

InDesign Server has to its advantage complete flexibility, but if you use InDesign Server alone you have to build several features that are pre-existing with XMPie. It really depends on specific workflows/document types/staff whether XMPie is the right fit.

Adobe has finally defined an XML format for Flash, something those of us in the SVG World have long waited for. Well, we weren’t waiting for Flash, it was an XML server-based description of interactive graphics, but who’s counting?

Now that we have XFL and FLA renaming it, can we get a server? I really look forward to this but it remains completely unannounced.

I was wondering why a dirt-simple InDesign file packaged to 6 Meg… and I discovered that the images were about 1 Meg each. I had created very simple images, colored rectangles in Illustrator CS4 and saved as straight .ai files with default settings.

Amazing, what valuable info is it persisting that makes this the right size for a primitive image? When I export as SVG I get 6 lines of code, which look verbose already, but way tiny by compare…

Does anyone remember the quote attributed to Bill Gates that PCs would never need more than 640k of memory? Poor Bill could only see the lower 2/3rds of this rectangle with the dream computer of his age. The only thing more amazing than the continuous exponential increases in memory, disk space, and bandwidth is the way such improvements are consumed by new applications the moment they are available.

At the Adobe MAX 2009 conference, one of the coolest sessions I attended was “Creating an eBook for Distribution on Sony Reader Digital Book, Amazon Kindle, and Apple iPhone” by Colin Fleming. The recording is online, among the many sessions available through the MAX 2009 Conference Scheduler. Colin’s blog post on the subject includes sample files.

Now this is quite cool, given a manual workflow. Just one problem… our company never wants to do anything manually, we want to automate everything. So on the surface, it appeared that we could probably just do this using InDesign Server, at least the export to Digital Editions. However, no such luck.

It turns out that the feature to export to Digital Editions, which does most of the work, is implemented via JavaScript. This piece of InDesign is in the Scripts > XHTML For Digital Editions folder, and it is compiled JavaScript, the wonderful jsxbin format that lets one distribute scripts without letting users view source or know what parameters the script expects.

Therefore, in order to automate this, one would have to guess the parameters to pass to the scripts. No documentation, and no source for the scripts, is yet available from Adobe. Why did they even put these scripts into the InDesign Server application? It is only usable from the desktop application, and only from the UI.

Olav Kvern was quick to tell me that he will help correct this problem. I look forward to seeing either source for the scripts or at least documentation on how to run them from code. Then we will be half way there in fully automating conversion to Kindle: of course the setup of source InDesign documents is important, too.

The other half is figuring out how to automate the step that Calibre takes care of in the process. At least that is open source.

I am thrilled that Adobe builds parts of InDesign with scripting: we have long maintained that scripting should be a first-class citizen with C++, and this proves our point. But they should not abandon the extensibility and complete exposure to automation that make InDesign and InDesign Server the first-class tools that they are.

I finally posted to the IDML Developers Group that I started on Yahoo. I hope this group takes off as developers begin to use IDML more frequently in InDesign automation.

IDML is wonderful, but it doesn’t on its own do quite everything one has to do to compose and edit dynamic documents. While you can define text and formats in a Story, for example, there is not a pure IDML-based approach to managing in-flow continued headers or copyfitting of text. You can’t use IDML alone to conditionally fill space based on how text flowed: IDML essentially instructs InDesign what to flow into a document, but it is not so different than MIF, XSL-FO, or other document descriptions that often require a two-pass approach to document composition (XSL-FO in some implementations does have extensions that go further than MIF or IDML in this respect): flow the content with IDML then clean things up and reflow based on scripting or plugin-based automation. It would be nice to see the page description itself include such state-based features.

While there are tactics that one can use to include metadata in IDML that round trips with documents, it would also be nice if IDML were more extensible in terms of allowing object-level metadata to persist when the IDML document is round-tripped with InDesign.

I was pleased to see that my old example still runs in CS4. Using VBA was very cool, it just suffered from dependency on Windows. But functionally we did many applications back in the early days of InDesign that worked just great with VBA. I think it was VB and AppleScript prior to JS and the cross-platform scripting? Let me go look that up.

In any case while JS scripting seems far more useful now, or “ExtendScript” as they call it, we still suffer from the sandbox and the inability to shell out to data sources and do work with them then get right back to automating InDesign. Using Java one can do such things with InDesign Server, but the Server-only nature of the Java makes debugging difficult. Its strongly-typed nature is a big headache, as with C# or any strongly typed language: you spend more time figuring out what type object you’re working with than you do manipulating the object.

I want to translate the VBA example to JavaScript with some database such as MySQL. In general the power of InDesign to do data-generated publishing is awesome.

Yes, JavaScript is better. VB is actually buggy… How come this JS works:

var myDocument = app.documents.item(0);
var myRectangle = myDocument.rectangles.item(0);
myHyperlinkURL = myDocument.hyperlinkURLDestinations.add("http://www.publishingsilicon.com");
myHyperlinkSource = myDocument.hyperlinkPageItemSources.add(myRectangle);
myHyperlink=myDocument.hyperlinks.add(myHyperlinkSource,myHyperlinkURL);
myHyperlink.visible=false;

and this equivalent in VB doesn’t:

Dim oInDesign As InDesign.Application
Dim oDocument As InDesign.Document
Dim oPage As InDesign.Page
Dim oImageRect As InDesign.Rectangle
Dim oHyperlinkURL As InDesign.HyperlinkURLDestination
Dim oHyperlinkSource As InDesign.HyperlinkPageItemSource
Dim oHyperlink As InDesign.Hyperlink
Set oInDesign = CreateObject("InDesign.Application.CS4")
Set oDocument = oInDesign.ActiveDocument
Set oImageRect = oDocument.Rectangles.Item(1)
Set oHyperlinkURL = oDocument.HyperlinkURLDestinations.Add("http://www.publishingsilicon.com")
Set oHyperlinkSource = oDocument.HyperlinkPageItemSources.Add(oImageRect)
Set oHyperlink = oDocument.Hyperlinks.Add(oHyperlinkSource, oHyperlinkURL)
oHyperlink.Visible = False

?

The VBA version bombs on the line with oDocument.HyperlinkPageItemSources.Add… and this sort of thing was posted to the forums over a year ago. I guess yet another reason the cross-platform JavaScript form of InDesign scripting is better is that it enjoys support. Can hardly blame Adobe for limiting support, especially as they cut 9% of their workforce when holiday season comes along. You can’t fight every battle.

I was asked how to script hyperlinks in InDesign, based on someone who dug up this old article that I wrote many years back, when I still used VBA for InDesign. Well, nowadays I only use JavaScript when scripting, but hopefully this basic example will help:

var myDocument = app.documents.item(0);
with(myDocument.pages.item(0)){
//Get a reference to the text frame.
var myTextFrame = textFrames.item(0);
var myParagraph = myTextFrame.paragraphs.item(0);
myHyperlinkURL = myDocument.hyperlinkURLDestinations.add("http://www.publishingsilicon.com");
myHyperlinkSource = myDocument.hyperlinkTextSources.add(myParagraph);
myHyperlink = myDocument.hyperlinks.add(myHyperlinkSource,myHyperlinkURL);
myHyperlink.visible=false;
}

This assumes a text frame exists with some text in it, and that when you export the PDF you either check Include\hyperlinks in the PDF Export dialogue, or do the equivalent with code if exporting programmatically.

I need to get around to updating that data-generated example some day. Amazing how many people around the world seem to stumble upon it.