Publishing with Silicon

DITA will take over the world… or maybe more like lay under it, as XML does currently.

From my perspective, DITA (or a good part of DITA – there is also the tech doc focus) is the next step in core SGML/XML. IBM started SGML itself, and later had a fair amount to do with XML: now the same sort of people are working on DITA, making XML safe for the world.

DITA extends SGML constructs such as entities with constructs such as conrefs. Everyone loves the idea of re-use of content, but XML 1.0 is a bit too flexible in this regard. It doesn’t say much about *how* you re-use, associate, and aggregate content, thus tools will do the same thing different ways, or won’t support re-use well at all. DITA fixes this, then immediately (concurrently) applies it to Tech Doc.

DITA is based on the practical experience of some IBM tech doc teams and while their goals and requirements were specific to tech doc, many of the core constructs are not.

Similar to XML itself, which is a meta-language (or language for creating languages), DITA has a powerful specialization methodology, that allows for completely custom document structures, yet a backwards compatibility with the core DITA constructs. If your <eBookPara> tag is read by a DITA rendition tool that only knows the <p> of DITA, you will at least get things rendered, though perhaps not in the special “eBook” way that you prefer. At least the tools don’t break.

It is somewhat confusing that the drivers for DITA remain squarely in the Tech Doc space, yet the solution it provides is often fairly universal. Maybe what DITA needs to do is split into the tech-doc specific DITA and the generic DITA, the way XSL split into XSLT and XSL-FO.

What is the world coming to? Never thought I’d see IE supporting SVG. We lobbied so hard 9 years ago, 8 years ago, and 7 years ago, until it felt like we were getting nowhere.

I remember Microsoft tried to hire me in 2002, having found me on the… SVG developers list. Now that was strange, what on earth were they doing stalking us XML geeks?

In a year or so, it became clear; XAML was highly derived from SVG, and would form the basis of WPF and Silverlight later. Unable to embrace a standard, MS had decided to copy standards activity into their own proprietary technology.

The poor SVG black sheep was even abandoned by Adobe itself when they eyed Macromedia/Flash, and enjoyed almost ZERO serious support over a few years, unless you count intensive emulation with XAML and later FXG, or the tireless efforts of a few diehards in places like the Mozilla project and Opera that kept SVG alive.

Fast forward 7 years, and we find Microsoft in the same boat with Apple, falling further behind Adobe’s Flash on the RIA front, with Silverlight piling up on the junkheap of obscurity along with Quicktime. With both proprietary efforts dead in the water, SVG is suddenly appealing to these would-be monopolies, and we find a bizarre rally behind a 10-year-old standard.

Why did they even bother to throw SVG into the mix with HTML5? Certainly the Canvas functionality can accomplish most or all of the core Flash capability that everyone (other than Adobe) wants. SVG and Canvas seem to have complimentary performance depending on what you’re doing. Still, who wants to learn how to do everything two different ways? Perhaps those railroading HTML5 through “spec” processes realize they won’t catch everything with the canvas approach, but more likely, they realize that this 2010 form of “standard” with Apple/Google pushing their rush “standard” out as Microsoft tails along, can have a better chance of flying with some stapled-on integrity from a bygone era.

It is still great to see, there is something really nice about the simplicity of core SVG, and it is fully ironic that its enemies have ended up having to support it despite their traditional opposition to standards. Apple, Google, Adobe, Microsoft have the same monopolistic agendas, yet are forced to co-exist, and let flowers like SVG grow through the cracks.

Today Steve Jobs called Adobe “Lazy.”

Flash has dramatically improved since Adobe bought Macromedia. Papervision 3D is a 3D engine that runs on Actionscript, this sort of capability was unheard of back when Macromedia ran Flash. 

Flash is spreading all over the place: set top boxes, TVs, phones, everywhere except the iPhone. It is ubiquitous for ads and video on the web. No wonder Steve is jealous. What is the install base of Quicktime?

Flash runs fine on every other phone. Is the iPhone buggy? no, it is intentionally dumbed down in the interest of rabid monopolistic tendencies of one eccentric genius.

Steve should remember that Apple would have died save for its use as a graphics platform running Adobe technology for a long stretch of time.

The iPhone will be a better device when it supports Flash.

As much as he has done for the company and the world, Steve Jobs really has conquered everything that needed conquering; the world needs a little less conquering and fewer dumbed-down, closed-source, “no VM allowed” systems like the iPhone and iPad.

With the iPad we witness the first case in history of computing where the limitations of a small device float upwards into a bigger device, instead of the opposite (remember when Moore’s law was a good thing?): who is lazy?

I was on the phone with a prospective client today, who shall remain nameless and unidentifiable. This could be any company, as they face the essential predicament of anyone trying to get the same content to go to both web and print effectively.

On the one hand, there is so much commonality and re-use of their content across the web and print media, it is absurd to have two entirely different workflows. On the other hand, the tools that lend themselves to a real multi-channel workflow, such as real XML content management, take extreme effort and time to implement and often have expensive associated software. Even after that effort, authors or content sources may not fit in with the required content process at all. Beyond that, moving content over from an unstructured to a structured format can be really difficult.

Inevitably, XML demonstrations make business users underestimate the challenge. “If you show us something, show us with our content!,” he said; evidently they were shown a rosy picture where perfectly marked up XML flowed easily out into web, print, braille, video, whatever. It is true; if you have rich semantic markup the publishing capabilities are amazing.

The challenge is getting that richly marked up content. It is hardly automatic. The extreme best case for authoring such content is the world of technical documentation, where the authors are typically really technical, and highly-evolved schemas/toolchains like DITA give them guidance on how to structure content. But at the other extreme, with writers who are non-technical, it is hard to get them to work with tools that are too constraining, or to get them to follow rigorous guidelines. No pain, no gain:  without the rich markup, publishing becomes more of a channel-by-channel basis.

I believe over time things will get easier, with standards like DITA, greater support for XML authoring in tools, and better example workflows for organizations smaller than the Department of Defense. But the pace of such improvement is slow.

We have built solutions using InDesign Server since it came out, and before that we were building solutions based on InDesign desktop for 5 years. So we know the XMPie space pretty well.

XMPie is a really well-built program, that to me has three main benefits:

1. It lets you easily define a data source for variable content (using uPlan) and reference that data source directly in InDesign (via uCreate)
2. It manages XMPie jobs (via the uProduce server), with functionality exposed as Web Services
3. It optimizes print output, producing VPS (which has been known to work), PPML, “VIPP” (which is known not to work; it is not VIPP but a VIPP wrapper around PostScript), etc.

XMPie is salvation for the designer at a mail house: they can bypass Programming entirely and set up their own “campaign” based on new InDesign/data input from a client.

Yet these days, we never meet such a designer. We meet enterprise clients, who consider themselves very special and do things a very special way. XMPie invariably meets their needs 40-80% of the way, but the other 20-60% can take a supreme effort. So we need to request extensibility from XMPie, and in many case fuse together an XMPie workflow with a very non-XMPie workflow. They may be sick of my requests, but they have given us more and more extensibility over time.

InDesign Server has to its advantage complete flexibility, but if you use InDesign Server alone you have to build several features that are pre-existing with XMPie. It really depends on specific workflows/document types/staff whether XMPie is the right fit.

We worked with Adobe a bit on their Scene7 product, and I have to say that it is some of the most promising technology out there for Web to Print. There are two big gotchas that I hope are overcome soon:

1. The text that is possible with Flash 10 is not fully functional: this stands to improve once FXG 2.0 is available, the hope is that FXG 2.0 will be fully supported. As of now the text is more like FXG 1+, it isn’t quite robust enough for our typical clients.
2. The pricing model is crazy. I think they priced it so high that they would make sure not to get slammed with too many initial implementations. $50K/year as a base price with multiple forms of transaction/bandwidth costs on top of that is hardly a SaaS model. You either pay as you go or you pay up front, they can’t ask for both…

Also, it appears in their early concepts of how the app would be used, they imagined one would hit the server for the Flash renditions! I think the whole beauty of sharing the XML model between PDF and Flash is leveraging what the client can do…

Demonstration of the Scene7 web-to-print solution

Anyway, in all my years of working with great programmers, including many at Adobe, I have never seen a group as great as those working on Scene7 web to print. I am very optimistic about its future.

The fundamental beauty of the Scene7 model for web to print is that it uses the same XML to describe the web document and the print document. It also extends the XML used in Flash (FXG) to support requirements of print such as CMYK color. Tricky, as this would ideally not be done in a separate namespace, but would be part of the core FXG spec itself. In general it is awkward how the different groups at Adobe work together: they are all focused on their own short-term deliverables and can’t often reconcile or coordinate the overlapping parts of their efforts.

Which brings us to… InDesign Server. One might have guessed that Scene7 would use InDesign Server rather than build their own form of PDF generation totally independent, with a different text engine (common with Illustrator/Flash, not InDesign) and different XML model (FXG vs. IDML). Sadly, the InDesign project does represent the ultimate in text engines, the ultimate in document feature sets for long documents, etc., but there has not been a desire to use it from the Scene7 group. They didn’t find it much of a true server product, apparently, which is quite understandable. The “server” dimension of IDS is minimalist, it is essentially the rendition half of the desktop product with a few hooks and enough “build it yourself” aspects that solution providers like us have a fairly endless stream of opportunity.

So Scene7 will hopefully become a big part of our work next year, assuming the 2 issues above are handled, yet InDesign Server will remain, especially for longer documents and those cases where extending a desktop InDesign workflow to the server is easier when avoiding issues around reconciling text and layout engines. We don’t really mind two systems, but some day I’m sure we’ll hit a hybrid case where we use both, and in some long-term road map (CS7?) they should actually get reconciled.

Adobe product managers have managed to calm down my early complaints about non-reconciliation of these two engines. One pointed out the incredible backwards compatibility responsibilities of IDS: they can’t just start over… one tiny bug in one tiny dot release can screw up a million documents for a client, they are not as agile as a SaaS shop.

In terms of SaaS; as of now, Scene7 is almost only SaaS and IDS is almost only self-hosted. It is likely that both products will cross over the other direction. We can host IDS in an EC2 environment just fine, with great scalability, yet the licensing is not SaaS friendly. In similar fashion, Scene7 can install just fine as a self-hosted software, yet they only allow this in “special” situations and tend to push for SaaS at all costs.

I just set up a basic WordPress blog for Paris Tompkins. This took only a few minutes’ spare time, including the hosting, DNS, customization. She chose the theme herself; I just made the side look OK and added the vital “Add to Any” plugin.

I still barely know WordPress, but I have had almost no trouble with any aspect of it recently: I played around with various blogging software about 8 years ago, and it was nowhere near this easy. I have only had to do something with PHP within a WordPress site once, and probably because of the nature of the template.

I am an impatient person and I will never be content with the pace of publishing technology, but I have to say that when you look back to where the web was in 1996 and where it is today, it has generally gone the right direction. Well, that is after starting out with a fairly complete misinterpretation.

Proposal for the World Wide Web

The original concept was of course brilliant, and you should check out the proposal for the web if you haven’t already. Basically, the intent of the web was to facilitate two-way communication, but of course the mindset of a Television-soaked population at first thought of it more like a single-direction, one-to-many, broadcast medium. We only understand things in relation to what we’ve seen before, at least at first.

So the Web started out with much knee-jerk reproduction of the Television model, and only with the gradual evolution of blogging and social networking (and concurrent evolution of tools for this) has it become really easy for a person like Paris to get out there and express herself. Now even the large corporations are hyping Facebook and Twitter, with their own YouTube channels and real estate in SecondLife. I think Paris will fare better than most corporations, because the playing field is leveled.

Adobe has finally defined an XML format for Flash, something those of us in the SVG World have long waited for. Well, we weren’t waiting for Flash, it was an XML server-based description of interactive graphics, but who’s counting?

Now that we have XFL and FLA renaming it, can we get a server? I really look forward to this but it remains completely unannounced.

I was wondering why a dirt-simple InDesign file packaged to 6 Meg… and I discovered that the images were about 1 Meg each. I had created very simple images, colored rectangles in Illustrator CS4 and saved as straight .ai files with default settings.

Amazing, what valuable info is it persisting that makes this the right size for a primitive image? When I export as SVG I get 6 lines of code, which look verbose already, but way tiny by compare…

Does anyone remember the quote attributed to Bill Gates that PCs would never need more than 640k of memory? Poor Bill could only see the lower 2/3rds of this rectangle with the dream computer of his age. The only thing more amazing than the continuous exponential increases in memory, disk space, and bandwidth is the way such improvements are consumed by new applications the moment they are available.

At the Adobe MAX 2009 conference, one of the coolest sessions I attended was “Creating an eBook for Distribution on Sony Reader Digital Book, Amazon Kindle, and Apple iPhone” by Colin Fleming. The recording is online, among the many sessions available through the MAX 2009 Conference Scheduler. Colin’s blog post on the subject includes sample files.

Now this is quite cool, given a manual workflow. Just one problem… our company never wants to do anything manually, we want to automate everything. So on the surface, it appeared that we could probably just do this using InDesign Server, at least the export to Digital Editions. However, no such luck.

It turns out that the feature to export to Digital Editions, which does most of the work, is implemented via JavaScript. This piece of InDesign is in the Scripts > XHTML For Digital Editions folder, and it is compiled JavaScript, the wonderful jsxbin format that lets one distribute scripts without letting users view source or know what parameters the script expects.

Therefore, in order to automate this, one would have to guess the parameters to pass to the scripts. No documentation, and no source for the scripts, is yet available from Adobe. Why did they even put these scripts into the InDesign Server application? It is only usable from the desktop application, and only from the UI.

Olav Kvern was quick to tell me that he will help correct this problem. I look forward to seeing either source for the scripts or at least documentation on how to run them from code. Then we will be half way there in fully automating conversion to Kindle: of course the setup of source InDesign documents is important, too.

The other half is figuring out how to automate the step that Calibre takes care of in the process. At least that is open source.

I am thrilled that Adobe builds parts of InDesign with scripting: we have long maintained that scripting should be a first-class citizen with C++, and this proves our point. But they should not abandon the extensibility and complete exposure to automation that make InDesign and InDesign Server the first-class tools that they are.