File-based links in InDesign

When I first met InDesign, in 2000, I (like many users then and since) thought maybe URL-based assets would be a feature that was already part of this then-new product: wouldn’t it be cool to work with web-based assets? But no, it wasn’t part of the product, and such a feature wasn’t necessarily essential to getting work done. With InDesign and InDesign Server, you just have to make sure things are in the right place on the file system. WebDAV offers a sort of linking to Web-based assets, but it is typically clunky and for the most part we put effort into moving files around and validating that they are in the right place when we automate InDesign Server.

In February, 2009 I went to cold New York City to attend the “Adobe InDesign and InDesign CS4 SDK Post-launch Developer Briefing.” I was tired and almost slept, but I woke up when they said how the linking architecture had been fundamentally rewritten underneath, to support URI-based linking, two-way linking, and generally what looked like anything one would want to do with linking. I asked, immediately, “can this be scripted?” and was quite disappointed to find out that it not only required a plug-in to leverage any of this power, but there was quite a bit left to developers to create what one would imagine as a basic use of URI-based linking.

Still, I never forgot this, and recently we finally had a chance to build our own plug-in to exploit this underlying capability. After taking some time to figure out how to do this right at a low level, thanks to some truly expert developers, it works!

HTTP-based Links in InDesign

With what we are calling “Silicon Connector” we can interface InDesign with DAMs (so far Day CQ5 and MediaBeacon, but looking at others), Rendition Systems (Scene7), and generally any remote asset. We are working on specific implementations that already validate improvement in authoring workflows, but there are a wide range of possible use cases:

• In OPI workflows, users can swap FPO assets with high-res assets more easily, from a selection for the entire document
• Documents can connect to multiple DAMs or repositories when different sorts of content are in different places: the “gathering” process can be completely controlled from InDesign
• Authentication of the rights to an asset can occur from within InDesign: a photo library could let users work with watermarked images, then calculate the document total for purchasing appropriate-sized images, and complete the transaction from within InDesign
• Bi-directional linking has all sorts of possibilities, probably most are more cool than practical, yet some are probably quite useful

This capability compliments and enhances the CS Extensions that came out with InDesign CS5. We now have what we naively expected out of InDesign 1.0. Better late than never!

Seeing the range of possibilities and finding different visions and functional requirements of different DAM vendors and developers/users, I see now why Adobe left this capability so far under the hood. There is not a single generic approach that would work for everyone.

I’ve made shoes for everyone, even you, while I still go barefoot
Bob Dylan

I am still pinching myself to see if I am awake as I look at the whitepaper on our web site. I have worked as the tireless scribe of other companies for so long, working on 10+ whitepapers for Adobe and others in the early 2000s; it is refreshing to get to the point where we can have a whitepaper of our own. By John Parsons, no less.

Pleased as I am with John’s work, I am more happy still about the topic of the whitepaper, and the state of the technology it describes. FXG is a joy to work with, now that we are past the struggles with the beta forms of the associated products, and on to real implementations.

Web to Print Workflow

The Scene7 platform was built really well, and almost anything once can imagine in Web to Print can be accomplished between Scene7 and Flash. It will no doubt evolve, but ever since FXG 2.0 we have a very practical strong foundation.

I am not a patient person at all. It was a good year and a half of naively thinking that FXG 2.0 was right around the corner: I am the sucker that believes the projected delivery dates of software products, just as I naively believed in XML rendition standards in the late 90s and early 2000s. But better late than never, and of course there’s an explanation: it seems that Flex 4 was more ambitious than anticipated, and it got late enough that CS5 integration made sense, and meanwhile S7 was waiting for the standard to be complete for the sake of interoperability.

Adobe Scene7 FXG Web to Print Workflow

We are working on several Silicon Designer implementations on top of Scene7 now; I think it has just eclipsed InDesign Server in popularity, but the two are not mutually exclusive and our latest solution uses both. XMPie and Scene7 can interact as well: the beauty of more recent technology is that with Web Services and the various techniques of translating the different forms of XML (semantic, transformation, and rendition) interoperability has gotten much less opaque than it once was in this industry.

Silicon Designer

I like John’s observation that WTP was born in the dot com era: it has suffered from the same brittle, rush-job poor foundations of such an era until now, when Adobe finally comes to their senses and acquires a company like Scene7 with true vision, and has the guts to empower them to build something of this caliber.

When Adobe bought Scene7 in May of 2007, Scene7 was a leading provider of imaging for retail web sites, known for feeding up images on demand so users could zoom in on a purse, rotate a watch with pseudo-3D technology, or see a furnished room as it would look with various selections of colors and materials in a photo-realistic way. The scalable, hosted application would serve up images in different raster formats at different resolutions based on the parameters of a URL. While it was possible to produce raster images of sufficient quality for print, they did not produce vector-based output, nor did they have a strong text engine, so Scene7 at the time was unsuitable as a complete solution for web-to-print. Rather, it would typically serve up images that would be used by other tools (server software with robust PDF output such as InDesign Server or Quark DDS) to the extent that Scene7 images became part of print output.

The Restoration Hardware configurator - an application of traditional pre-Adobe Scene7 technology

Meanwhile, InDesign Server was a very different sort of application, available as software that users would put on their own servers (not a hosted solution: initially the license even prohibited use within a Software as a Service or SaaS model). There was very little “server” in InDesign Server, instead the core rendition part of the desktop application was split out from the UI and handed off to the developer community with a simple SOAP interface and no job queuing: it was up to us developers to figure out how to spawn instances, queue jobs and make it produce output as fast as was possible given the slow nature of the rendition engine. In the five years of desktop product evolution, emphasis had been on attaining ultimate rendition capability more than speed of composition. InDesign Server went exponentially slower than earlier server-based composition engines, because it offered very robust features that were computation-intensive. Using the InDesign Paragraph Composer to lay out text, for example, involves analyzing the entire paragraph before deciding where to place the second character. The complexity of rendition, coupled with a lack of initial high-throughput focus, means that InDesign Server offers the ultimate in quality of output, at the expense of throughput.

A Web-to-print solution built on InDesign Server

Soon after Adobe bought Scene7, they embarked on adding web-to-print functionality to the product. There was some interaction with the InDesign Server group, even some involvement in Scene7 Professional Services in InDesign Server deployments, but fundamentally Scene7 went down the path of building their own web-to-print by pulling Adobe core technology into their server-based application infrastructure. They opted to base web-to-print on a new XML format that was brewing at Adobe since the Macromedia acquisition: FXG. FXG was a graphic format being built into Flash and Flex, which could be exported and imported from Illustrator; the primary motivation/use case for FXG was Flash Catalyst, which uses this model to enable powerful designer/developer workflows. The XML nature of the graphics and the integration of this XML into Flex MXML as a “native” format provide a good foundation for defining relationships between graphic objects and their behaviors in Flash applications. Notably, Flash Catalyst didn’t really require print-related functionality, so FXG did not (and still does not) include much that supports print: the color model is exclusively RGB, for example.

The first Scene7 Web-to-print release,  in September of 2009, used FXG 1.0. In order to provide capabilities essential to print output that were not part of the FXG 1.0 specification, FXG was extended by a Scene7 namespace, thus “Scene7 FXG” is the core XML used to describe the high-quality print output from Scene7 Web-to-print. FXG 1.0 was put out concurrent with a wonderful new text markup language, the Text Layout Framework (TLF). Unfortunately, at the time of FXG 1.0, the TLF spec was not yet final. The text possible with the first Scene7 Web-to-print, even with S7-namespace enhancements, offered only rather crude text features. This made the first Web-to-print offering from Scene7 less than stunning for text: it was clear, however, that the core rendition functionality was solid, and there were some exciting aspects of the system from a Web-to-print standpoint:

• For the first time in history, the same graphic XML used in the client could power robust print output on the server
• PDF export was blazingly fast
• The possibilities for working with variable data and graphics were very powerful, with tactics such as nested FXG, DOM manipulation, and some built-in methods for defining and using variables
• The Web-to-print functionalities could leverage traditional imaging and image-management techniques available with the general Scene7 platform
• A “solution accelerator” built initially by Scene7 Professional Services, offered a jump start towards a Web-to-print application built in Flex

With the first, FXG 1.0-based release, there was a core infrastructure for Web-to-print, but the application was clearly brand new, and the text capabilities were sorely lacking. Knowing that TLF 1.0, an extremely robust typographic capability, would be part of FXG 2.0, most of us in the development/partner community decided to wait for the release based on FXG 2.0. This wait turned out to be longer than expected, because FXG 2.0 itself suffered from delays in Flex 4 and what became a desire/policy of synchronizing Flex 4 with the CS5 release. Once FXG 2.0 was final, though, Scene7 was quick to make it work with the system: in parallel, what had been the Solution Accelerator was divided into an SDK and a sample application, and improvements were made to document setup and other aspects of Web-to-print based on experience with 1.0 and beta 2.0 implementations. In October 2010, the Scene7 5.0 / FXG 2.0 Web-to-print went live on all Scene7 production servers.

The Basic Configurator sample application that comes with Scene7 Web-to-print

Now that there is a truly robust Web-to-print offering from the Scene7 group, the comparison with InDesign Server, which is at this point a well-proven rendition back end to Web-to-print, can be made. And it should be made… already in our work at Silicon Publishing, we have been in the middle of decisions over which technology to use several times. At a very high level, there is a high degree of overlapping capability, and the buzzword popular at Adobe the past couple years, “disruptive” is well-suited. Adobe is definitely competing with itself, and it is not as trivial as the PageMaker/InDesign “disruption” when you know one technology is slated for extinction: it is likely that both Scene7 and InDesign Server will continue as Adobe offerings indefinitely, and both products will continue to grow and mature. In the overall industry, these two are the leaders from a functional standpoint, as Adobe has such a huge advantage over their competition in this space. So let’s look at the differences, first in a side-by-side comparison:

We can also look at the areas where Adobe Scene7 is ideal, or where InDesign Server is ideal. Constraints such as availability to host yourself vs. availability in a SaaS model may eliminate one or the other off the bat.

• Scene7 Web-to-print is typically the best option when:
  • You want a scalable SaaS solution
  • Speed of server-generated preview is essential to a Web application
  • You want to leverage the single FXG XML standard for Web and print documents (and have developers/applications available to do this with)
  • Your application would benefit from integration with the other features of the Scene7 platform

  assuming that your document types are relatively small and straightforward, i.e. business cards, stationary, signage, brochures, multi-page documents in the low page-count range or where the design is not flow-intensive.

• InDesign Server is typically the best option when:
  • You want to host the solution yourself
  • Speed of server-generated preview is not essential to a Web application, i.e. you can work around limits with Flash-based editing offering most or all of proof rendition
  • You are using the most advanced features of InDesign or InDesign automation and need these fully editable in a Web UI
  • Your workflow requires direct work with the source documents subsequent to the online editing session

Those are the strong indicators that would push it one way or another, although there are requirements driven by workflow/UI and or document characteristics that can push it one way or another.

I will try to keep this updated, please provide feedback based on your experience, or questions that might help make this more clear. In general, both tools are very powerful, and there is something like a 70% overlap, whereby in 70% of the Web-to-print solutions we’ve encountered, either one would work, so it is likely we will work with both for the foreseeable future. Our Silicon Designer product is built to run with either Scene7 Web-to-print or InDesign Server (optionally with XMPie on top of it), and we are planning to continue to support this.

Silicon Designer works with both InDesign Server and Scene7 Web-to-print

One thing to contemplate that also keeps coming up in our work: could they be combined? Yes, they definitely could… this will inevitably happen for some large organizations, but a formal connection is not currently on Adobe’s roadmap as far as we can tell. We would like to see InDesign Server offered as a SaaS component of Scene7, but have no indication this is in the cards. There is nothing like an obvious combination, you would probably see one or the other at the center of such a combined solution: Scene7 just for previews from FXG, or InDesign Server just for long documents or specific document features, that sort of thing. It really depends on what you want to accomplish. Combining the two will probably be very rare unless/until Adobe offers some appropriate bundled pricing.

Keynote – Kevin Lynch started in with discussion technology trends, then moved into presentation of the Digital Publishing features. Wired magazine was presented, and really didn’t show much different than what we’ve seen already. They seem to have reacted to some of the initial bashing. They showed dynamic text wrapping in HTML, identical to SVG text wrapping demos from 10 years ago. Not exactly thrilling, though they do say they are contributing to WebKit. He presented the Adobe Digital Publishing Suite, starting from InDesign, collaboratively produced and distributed, and of course they want to use Omniture to analyze the results. It is quite a stretch to think they will do the full-cycle at all as well as they have built the low-level tools.

Kevin spoke of video… quite an awesome amount of video going out in Flash, but unlike last year they are not showing the trend towards ubiquity, but rather “most video on the web is still shown in Flash.” Flash 10.1 was very quickly adopted, apparently, yet we know where it did not show up. Internet television keeps advancing, and Flash does have quite a head start there: the streaming is really impressive. AIR for TV is out; first launch partner is Samsung. Marc Goldberg of Epix spoke of multi-screen, saying their subscribers watch on a number of different screens: the demo broke, however.

Flash Media Server is working on on-the-fly encoding. Wowza seems to have made some real inroads, based on discussion with some attendees. P2P-assisted video looks kind of cool.

Kevin showed a number of small apps built targeting the tablet computing environment… it would appear they are rewriting things to be more lightweight. He demo’d a table device / PC wireless app, where the iPad served as a palette for a user in Photoshop.

Adobe MAX 2010 - Tablet as artist palette connected to desktop via wireless

He spoke of the enterprise… the impact of multi-screen, as well as LiveCycle. He mentioned the Day Software acquisition, which we have been watching with eager anticipation at Silicon Publishing, and brought their CTO David Nüscheler to present.

David Nüscheler from Day spoke of the need for discrete control of how things work across devices: he explained that the communications are truly different, it is not as trivial as spewing forth variants of the same thing entirely, but there is a true need to have some separate content specifically targeted. David demo’d CQ5, showing editing of HTML and mobile content. He did not dwell on the DAM, which has been our primary interest in Day. We are in the midst of a very exciting integration with the Day DAM, and so far we have been extremely impressed with the DAM aspects of Day. Adobe has done several acquisitions really well, and even though it’s the LiveCycle group that they are subsumed into, we are optimistic they will take over appropriate leadership roles. Wouldn’t it be cool if we can connect the rendition power of Adobe to structured information in LiveCycle? We have witnessed a long and pathetic history of the Acrobat monopoly being squandered on silo-based random junk (Central Output Pro? Graphics Server? Document Server? Forms with dirt-crude formatting?…) for rendition, yet there is good security and DRM in LiveCycle, hopefully the Day team will gut it and connect appropriately to the rendition side of Adobe (Flash, After Effects, InDesign desktop and Server, and above all… Scene7).

Day does introduce a real philosophical paradox. After the keynote, they presented content that had probably mainly been presented before, starting with a 45-minute argument for Open Source. LiveCycle has traditionally been the precise opposite of Open Source, and just the day before at the leadership summit we had heard that the “open source” Flex project had approximately 0 contributions from anybody but Adobe: will Day change this? It will be interesting. It was sort of strange to hear conway’s law cited and consider what that would mean as the organization is not the Apache Foundation any more, it is Adobe. We will see some interesting software!

Kevin demo’d medical imaging technology. Flex really does create powerful front ends for enterprise apps. Flex 4.5 beta is out, I am not sure after the year of Flex 4 beta lifestyle how deeply we will want to dive in. Back then we really needed the TLF badly, so it was probably worth the rush, I think with future Flex betas we can build a bit less production code on top of it.

Mike Lazaridis, CEO of RIM, came on and showed the Blackberry Playbook. Adobe and RIM have been working together with Flash/AIR; fairly obvious that Apple motivates them. “Not trying to dumb down the internet for a mobile device” – NICE stab at Apple. That was, as the 2 of you who read this blog know, my big reaction to the iPad.

Kevin showed some cool games, rendition gets faster over time with more use of the GPU, and AIR is evolving. AIR for Android looks good, Apple has some good competition, hopefully. Deploying from Flash to iPhone OS is once again possible, too.

Social gaming has always been one of Kevin’s interests: he showed the work idol worship, a nice-looking virtual reality game that uses old school animation techniques coupled with slightly newer technologies. He showed some very cool GPU-accelerated 3D capabilities of other games. Flash is really getting nice: the virtual reality driving demo was compelling. With the next Flash 3D, code named Molehill, the immersive 3D graphics should be very good for games and general 3D imaging.

Chrysty Wyatt from Motorola came on to speak of Android and the request of Motorola to put Flash on mobile devices – “anyone who fails to put Flash on a mobile device is not giving you the Internet.” And they are not above bribing us – all attendees at MAX are getting a Droid 2.

I saw the feature yesterday, but it seems to have been turned off today. Maybe this has caught on? Today, I uploaded a 1936 × 2592 image, and as usual Facebook reduced it to 538 × 720.

Max picture with resolution reduced by Facebook upload

Hardly conducive to high quality print, though I have been impressed that even with resolution along those lines, the cards from fidipidi have looked pretty great.

As soon as Facebook turns this feature back on, I will make a high resolution card using fidipidi and report the results.

From my perspective, DITA (or a good part of DITA – there is also the tech doc focus) is the next step in core SGML/XML. IBM started SGML itself, and later had a fair amount to do with XML: now the same sort of people are working on DITA, making XML safe for the world.

DITA extends SGML constructs such as entities with constructs such as conrefs. Everyone loves the idea of re-use of content, but XML 1.0 is a bit too flexible in this regard. It doesn’t say much about *how* you re-use, associate, and aggregate content, thus tools will do the same thing different ways, or won’t support re-use well at all. DITA fixes this, then immediately (concurrently) applies it to Tech Doc.

DITA is based on the practical experience of some IBM tech doc teams and while their goals and requirements were specific to tech doc, many of the core constructs are not.

Similar to XML itself, which is a meta-language (or language for creating languages), DITA has a powerful specialization methodology, that allows for completely custom document structures, yet a backwards compatibility with the core DITA constructs. If your <eBookPara> tag is read by a DITA rendition tool that only knows the <p> of DITA, you will at least get things rendered, though perhaps not in the special “eBook” way that you prefer. At least the tools don’t break.

It is somewhat confusing that the drivers for DITA remain squarely in the Tech Doc space, yet the solution it provides is often fairly universal. Maybe what DITA needs to do is split into the tech-doc specific DITA and the generic DITA, the way XSL split into XSLT and XSL-FO.

I remember Microsoft tried to hire me in 2002, having found me on the… SVG developers list. Now that was strange, what on earth were they doing stalking us XML geeks?

In a year or so, it became clear; XAML was highly derived from SVG, and would form the basis of WPF and Silverlight later. Unable to embrace a standard, MS had decided to copy standards activity into their own proprietary technology.

The poor SVG black sheep was even abandoned by Adobe itself when they eyed Macromedia/Flash, and enjoyed almost ZERO serious support over a few years, unless you count intensive emulation with XAML and later FXG, or the tireless efforts of a few diehards in places like the Mozilla project and Opera that kept SVG alive.

Fast forward 7 years, and we find Microsoft in the same boat with Apple, falling further behind Adobe’s Flash on the RIA front, with Silverlight piling up on the junkheap of obscurity along with Quicktime. With both proprietary efforts dead in the water, SVG is suddenly appealing to these would-be monopolies, and we find a bizarre rally behind a 10-year-old standard.

Why did they even bother to throw SVG into the mix with HTML5? Certainly the Canvas functionality can accomplish most or all of the core Flash capability that everyone (other than Adobe) wants. SVG and Canvas seem to have complimentary performance depending on what you’re doing. Still, who wants to learn how to do everything two different ways? Perhaps those railroading HTML5 through “spec” processes realize they won’t catch everything with the canvas approach, but more likely, they realize that this 2010 form of “standard” with Apple/Google pushing their rush “standard” out as Microsoft tails along, can have a better chance of flying with some stapled-on integrity from a bygone era.

It is still great to see, there is something really nice about the simplicity of core SVG, and it is fully ironic that its enemies have ended up having to support it despite their traditional opposition to standards. Apple, Google, Adobe, Microsoft have the same monopolistic agendas, yet are forced to co-exist, and let flowers like SVG grow through the cracks.

Flash has dramatically improved since Adobe bought Macromedia. Papervision 3D is a 3D engine that runs on Actionscript, this sort of capability was unheard of back when Macromedia ran Flash. 

Flash is spreading all over the place: set top boxes, TVs, phones, everywhere except the iPhone. It is ubiquitous for ads and video on the web. No wonder Steve is jealous. What is the install base of Quicktime?

Flash runs fine on every other phone. Is the iPhone buggy? no, it is intentionally dumbed down in the interest of rabid monopolistic tendencies of one eccentric genius.

Steve should remember that Apple would have died save for its use as a graphics platform running Adobe technology for a long stretch of time.

The iPhone will be a better device when it supports Flash.

As much as he has done for the company and the world, Steve Jobs really has conquered everything that needed conquering; the world needs a little less conquering and fewer dumbed-down, closed-source, “no VM allowed” systems like the iPhone and iPad.

With the iPad we witness the first case in history of computing where the limitations of a small device float upwards into a bigger device, instead of the opposite (remember when Moore’s law was a good thing?): who is lazy?

On the one hand, there is so much commonality and re-use of their content across the web and print media, it is absurd to have two entirely different workflows. On the other hand, the tools that lend themselves to a real multi-channel workflow, such as real XML content management, take extreme effort and time to implement and often have expensive associated software. Even after that effort, authors or content sources may not fit in with the required content process at all. Beyond that, moving content over from an unstructured to a structured format can be really difficult.

Inevitably, XML demonstrations make business users underestimate the challenge. “If you show us something, show us with our content!,” he said; evidently they were shown a rosy picture where perfectly marked up XML flowed easily out into web, print, braille, video, whatever. It is true; if you have rich semantic markup the publishing capabilities are amazing.

The challenge is getting that richly marked up content. It is hardly automatic. The extreme best case for authoring such content is the world of technical documentation, where the authors are typically really technical, and highly-evolved schemas/toolchains like DITA give them guidance on how to structure content. But at the other extreme, with writers who are non-technical, it is hard to get them to work with tools that are too constraining, or to get them to follow rigorous guidelines. No pain, no gain:  without the rich markup, publishing becomes more of a channel-by-channel basis.

I believe over time things will get easier, with standards like DITA, greater support for XML authoring in tools, and better example workflows for organizations smaller than the Department of Defense. But the pace of such improvement is slow.

XMPie is a really well-built program, that to me has three main benefits:

1. It lets you easily define a data source for variable content (using uPlan) and reference that data source directly in InDesign (via uCreate)
2. It manages XMPie jobs (via the uProduce server), with functionality exposed as Web Services
3. It optimizes print output, producing VPS (which has been known to work), PPML, “VIPP” (which is known not to work; it is not VIPP but a VIPP wrapper around PostScript), etc.

XMPie is salvation for the designer at a mail house: they can bypass Programming entirely and set up their own “campaign” based on new InDesign/data input from a client.

Yet these days, we never meet such a designer. We meet enterprise clients, who consider themselves very special and do things a very special way. XMPie invariably meets their needs 40-80% of the way, but the other 20-60% can take a supreme effort. So we need to request extensibility from XMPie, and in many case fuse together an XMPie workflow with a very non-XMPie workflow. XMPie may be sick of my requests, but they have given us more and more extensibility over time.

InDesign Server has to its advantage complete flexibility, but if you use InDesign Server alone you have to build several features that are pre-existing with XMPie. It really depends on specific workflows/document types/staff whether XMPie is the right fit.