Publishing with Silicon

I’ve made shoes for everyone, even you, while I still go barefoot
Bob Dylan

I am still pinching myself to see if I am awake as I look at the whitepaper on our web site. I have worked as the tireless scribe of other companies for so long, working on 10+ whitepapers for Adobe and others in the early 2000s; it is refreshing to get to the point where we can have a whitepaper of our own. By John Parsons, no less.

Pleased as I am with John’s work, I am more happy still about the topic of the whitepaper, and the state of the technology it describes. FXG is a joy to work with, now that we are past the struggles with the beta forms of the associated products, and on to real implementations.

Web to Print Workflow

The Scene7 platform was built really well, and almost anything once can imagine in Web to Print can be accomplished between Scene7 and Flash. It will no doubt evolve, but ever since FXG 2.0 we have a very practical strong foundation.

I am not a patient person at all. It was a good year and a half of naively thinking that FXG 2.0 was right around the corner: I am the sucker that believes the projected delivery dates of software products, just as I naively believed in XML rendition standards in the late 90s and early 2000s. But better late than never, and of course there’s an explanation: it seems that Flex 4 was more ambitious than anticipated, and it got late enough that CS5 integration made sense, and meanwhile S7 was waiting for the standard to be complete for the sake of interoperability.

Adobe Scene7 FXG Web to Print Workflow

We are working on several Silicon Designer implementations on top of Scene7 now; I think it has just eclipsed InDesign Server in popularity, but the two are not mutually exclusive and our latest solution uses both. XMPie and Scene7 can interact as well: the beauty of more recent technology is that with Web Services and the various techniques of translating the different forms of XML (semantic, transformation, and rendition) interoperability has gotten much less opaque than it once was in this industry.

Silicon Designer

I like John’s observation that WTP was born in the dot com era: it has suffered from the same brittle, rush-job poor foundations of such an era until now, when Adobe finally comes to their senses and acquires a company like Scene7 with true vision, and has the guts to empower them to build something of this caliber.

With the release of Scene7 5.0 and its Web-to-print features, there are now two extremely powerful server-side rendition technologies from Adobe. Unless you are intimately familiar with these two technologies and their corresponding branches of the Adobe organization, you might assume that Adobe would develop these technologies with some concept of integration, but in truth they really are two distinct and disconnected offerings that just happen to come from the same parent company.

When Adobe bought Scene7 in May of 2007, Scene7 was a leading provider of imaging for retail web sites, known for feeding up images on demand so users could zoom in on a purse, rotate a watch with pseudo-3D technology, or see a furnished room as it would look with various selections of colors and materials in a photo-realistic way. The scalable, hosted application would serve up images in different raster formats at different resolutions based on the parameters of a URL. While it was possible to produce raster images of sufficient quality for print, they did not produce vector-based output, nor did they have a strong text engine, so Scene7 at the time was unsuitable as a complete solution for web-to-print. Rather, it would typically serve up images that would be used by other tools (server software with robust PDF output such as InDesign Server or Quark DDS) to the extent that Scene7 images became part of print output.

The Restoration Hardware configurator - an application of traditional pre-Adobe Scene7 technology

Meanwhile, InDesign Server was a very different sort of application, available as software that users would put on their own servers (not a hosted solution: initially the license even prohibited use within a Software as a Service or SaaS model). There was very little “server” in InDesign Server, instead the core rendition part of the desktop application was split out from the UI and handed off to the developer community with a simple SOAP interface and no job queuing: it was up to us developers to figure out how to spawn instances, queue jobs and make it produce output as fast as was possible given the slow nature of the rendition engine. In the five years of desktop product evolution, emphasis had been on attaining ultimate rendition capability more than speed of composition. InDesign Server went exponentially slower than earlier server-based composition engines, because it offered very robust features that were computation-intensive. Using the InDesign Paragraph Composer to lay out text, for example, involves analyzing the entire paragraph before deciding where to place the second character. The complexity of rendition, coupled with a lack of initial high-throughput focus, means that InDesign Server offers the ultimate in quality of output, at the expense of throughput.

A Web-to-print solution built on InDesign Server

Soon after Adobe bought Scene7, they embarked on adding web-to-print functionality to the product. There was some interaction with the InDesign Server group, even some involvement in Scene7 Professional Services in InDesign Server deployments, but fundamentally Scene7 went down the path of building their own web-to-print by pulling Adobe core technology into their server-based application infrastructure. They opted to base web-to-print on a new XML format that was brewing at Adobe since the Macromedia acquisition: FXG. FXG was a graphic format being built into Flash and Flex, which could be exported and imported from Illustrator; the primary motivation/use case for FXG was Flash Catalyst, which uses this model to enable powerful designer/developer workflows. The XML nature of the graphics and the integration of this XML into Flex MXML as a “native” format provide a good foundation for defining relationships between graphic objects and their behaviors in Flash applications. Notably, Flash Catalyst didn’t really require print-related functionality, so FXG did not (and still does not) include much that supports print: the color model is exclusively RGB, for example.

The first Scene7 Web-to-print release,  in September of 2009, used FXG 1.0. In order to provide capabilities essential to print output that were not part of the FXG 1.0 specification, FXG was extended by a Scene7 namespace, thus “Scene7 FXG” is the core XML used to describe the high-quality print output from Scene7 Web-to-print. FXG 1.0 was put out concurrent with a wonderful new text markup language, the Text Layout Framework (TLF). Unfortunately, at the time of FXG 1.0, the TLF spec was not yet final. The text possible with the first Scene7 Web-to-print, even with S7-namespace enhancements, offered only rather crude text features. This made the first Web-to-print offering from Scene7 less than stunning for text: it was clear, however, that the core rendition functionality was solid, and there were some exciting aspects of the system from a Web-to-print standpoint:

• For the first time in history, the same graphic XML used in the client could power robust print output on the server
• PDF export was blazingly fast
• The possibilities for working with variable data and graphics were very powerful, with tactics such as nested FXG, DOM manipulation, and some built-in methods for defining and using variables
• The Web-to-print functionalities could leverage traditional imaging and image-management techniques available with the general Scene7 platform
• A “solution accelerator” built initially by Scene7 Professional Services, offered a jump start towards a Web-to-print application built in Flex

With the first, FXG 1.0-based release, there was a core infrastructure for Web-to-print, but the application was clearly brand new, and the text capabilities were sorely lacking. Knowing that TLF 1.0, an extremely robust typographic capability, would be part of FXG 2.0, most of us in the development/partner community decided to wait for the release based on FXG 2.0. This wait turned out to be longer than expected, because FXG 2.0 itself suffered from delays in Flex 4 and what became a desire/policy of synchronizing Flex 4 with the CS5 release. Once FXG 2.0 was final, though, Scene7 was quick to make it work with the system: in parallel, what had been the Solution Accelerator was divided into an SDK and a sample application, and improvements were made to document setup and other aspects of Web-to-print based on experience with 1.0 and beta 2.0 implementations. In October 2010, the Scene7 5.0 / FXG 2.0 Web-to-print went live on all Scene7 production servers.

The Basic Configurator sample application that comes with Scene7 Web-to-print

Now that there is a truly robust Web-to-print offering from the Scene7 group, the comparison with InDesign Server, which is at this point a well-proven rendition back end to Web-to-print, can be made. And it should be made… already in our work at Silicon Publishing, we have been in the middle of decisions over which technology to use several times. At a very high level, there is a high degree of overlapping capability, and the buzzword popular at Adobe the past couple years, “disruptive” is well-suited. Adobe is definitely competing with itself, and it is not as trivial as the PageMaker/InDesign “disruption” when you know one technology is slated for extinction: it is likely that both Scene7 and InDesign Server will continue as Adobe offerings indefinitely, and both products will continue to grow and mature. In the overall industry, these two are the leaders from a functional standpoint, as Adobe has such a huge advantage over their competition in this space. So let’s look at the differences, first in a side-by-side comparison:

We can also look at the areas where Adobe Scene7 is ideal, or where InDesign Server is ideal. Constraints such as availability to host yourself vs. availability in a SaaS model may eliminate one or the other off the bat.

• Scene7 Web-to-print is typically the best option when:
  • You want a scalable SaaS solution
  • Speed of server-generated preview is essential to a Web application
  • You want to leverage the single FXG XML standard for Web and print documents (and have developers/applications available to do this with)
  • Your application would benefit from integration with the other features of the Scene7 platform

  assuming that your document types are relatively small and straightforward, i.e. business cards, stationary, signage, brochures, multi-page documents in the low page-count range or where the design is not flow-intensive.

• InDesign Server is typically the best option when:
  • You want to host the solution yourself
  • Speed of server-generated preview is not essential to a Web application, i.e. you can work around limits with Flash-based editing offering most or all of proof rendition
  • You are using the most advanced features of InDesign or InDesign automation and need these fully editable in a Web UI
  • Your workflow requires direct work with the source documents subsequent to the online editing session

Those are the strong indicators that would push it one way or another, although there are requirements driven by workflow/UI and or document characteristics that can push it one way or another.

I will try to keep this updated, please provide feedback based on your experience, or questions that might help make this more clear. In general, both tools are very powerful, and there is something like a 70% overlap, whereby in 70% of the Web-to-print solutions we’ve encountered, either one would work, so it is likely we will work with both for the foreseeable future. Our Silicon Designer product is built to run with either Scene7 Web-to-print or InDesign Server (optionally with XMPie on top of it), and we are planning to continue to support this.

Silicon Designer works with both InDesign Server and Scene7 Web-to-print

One thing to contemplate that also keeps coming up in our work: could they be combined? Yes, they definitely could… this will inevitably happen for some large organizations, but a formal connection is not currently on Adobe’s roadmap as far as we can tell. We would like to see InDesign Server offered as a SaaS component of Scene7, but have no indication this is in the cards. There is nothing like an obvious combination, you would probably see one or the other at the center of such a combined solution: Scene7 just for previews from FXG, or InDesign Server just for long documents or specific document features, that sort of thing. It really depends on what you want to accomplish. Combining the two will probably be very rare unless/until Adobe offers some appropriate bundled pricing.

We worked with Adobe a bit on their Scene7 product, and I have to say that it is some of the most promising technology out there for Web to Print. There are two big gotchas that I hope are overcome soon:

1. The text that is possible with Flash 10 is not fully functional: this stands to improve once FXG 2.0 is available, the hope is that FXG 2.0 will be fully supported. As of now the text is more like FXG 1+, it isn’t quite robust enough for our typical clients.
2. The pricing model is crazy. I think they priced it so high that they would make sure not to get slammed with too many initial implementations. $50K/year as a base price with multiple forms of transaction/bandwidth costs on top of that is hardly a SaaS model. You either pay as you go or you pay up front, they can’t ask for both…

Also, it appears in their early concepts of how the app would be used, they imagined one would hit the server for the Flash renditions! I think the whole beauty of sharing the XML model between PDF and Flash is leveraging what the client can do…

Demonstration of the Scene7 web-to-print solution

Anyway, in all my years of working with great programmers, including many at Adobe, I have never seen a group as great as those working on Scene7 web to print. I am very optimistic about its future.

The fundamental beauty of the Scene7 model for web to print is that it uses the same XML to describe the web document and the print document. It also extends the XML used in Flash (FXG) to support requirements of print such as CMYK color. Tricky, as this would ideally not be done in a separate namespace, but would be part of the core FXG spec itself. In general it is awkward how the different groups at Adobe work together: they are all focused on their own short-term deliverables and can’t often reconcile or coordinate the overlapping parts of their efforts.

Which brings us to… InDesign Server. One might have guessed that Scene7 would use InDesign Server rather than build their own form of PDF generation totally independent, with a different text engine (common with Illustrator/Flash, not InDesign) and different XML model (FXG vs. IDML). Sadly, the InDesign project does represent the ultimate in text engines, the ultimate in document feature sets for long documents, etc., but there has not been a desire to use it from the Scene7 group. They didn’t find it much of a true server product, apparently, which is quite understandable. The “server” dimension of IDS is minimalist, it is essentially the rendition half of the desktop product with a few hooks and enough “build it yourself” aspects that solution providers like us have a fairly endless stream of opportunity.

So Scene7 will hopefully become a big part of our work next year, assuming the 2 issues above are handled, yet InDesign Server will remain, especially for longer documents and those cases where extending a desktop InDesign workflow to the server is easier when avoiding issues around reconciling text and layout engines. We don’t really mind two systems, but some day I’m sure we’ll hit a hybrid case where we use both, and in some long-term road map (CS7?) they should actually get reconciled.

Adobe product managers have managed to calm down my early complaints about non-reconciliation of these two engines. One pointed out the incredible backwards compatibility responsibilities of IDS: they can’t just start over… one tiny bug in one tiny dot release can screw up a million documents for a client, they are not as agile as a SaaS shop.

In terms of SaaS; as of now, Scene7 is almost only SaaS and IDS is almost only self-hosted. It is likely that both products will cross over the other direction. We can host IDS in an EC2 environment just fine, with great scalability, yet the licensing is not SaaS friendly. In similar fashion, Scene7 can install just fine as a self-hosted software, yet they only allow this in “special” situations and tend to push for SaaS at all costs.