The Return of the Blue Q

When last we saw QuickTime for Java (QTJ), the media API had been unceremoniously and unapologetically broken by Apple's Java 1.4.1 implementation for Mac OS X. As I wrote in a weblog entry at the time, the workaround was to force apps to run with Java 1.3.1, which all Mac OS X users could still use. However, this was an intolerable situation to Mac- and Windows-based Java developers who needed a high-performance media API that supported modern formats and features -- who wants to depend on an API with no apparent future?

With the release of QuickTime 6.4 for Windows and Mac OS X, QTJ has a future again. However, the API has changed greatly in 2003, so much so that nearly any existing QTJ application will remain broken unless it is rewritten for the new QTJ 6.1.

This article will describe the new QTJ by relating the history of why it was broken in the first place, how it was fixed, how to use the new version, and what we might expect to see from QTJ going forward.

From Biscotti to Broken

QuickTime for Java is one of the oldest third-party Java APIs. Originally developed under the codename "Biscotti," and formally known as QuickTime for Java, QTJ was announced in 1998. Starting with QuickTime 3, QTJ development continued alongside the rest of the QuickTime API. As the underlying native library gained features, QTJ would grow more powerful. Features that didn't require new method calls, such as support for a new media format or codec, were available immediately to QTJ. In other cases, a feature might wait one version -- for example, QuickTime 5's API for broadcasting became available to Java developers in QTJ 6.

Related Reading Mac OS X for Java Geeks By Will Iverson

It helps to understand the nature of QTJ's architecture. QTJ's goal is to present a native, straight-C media API to Java developers in a way that makes sense in Java (because it is mostly composed of calls to a native library, it can be thought of as "QuickTime from Java"). It creates an object-oriented view of QuickTime by crafting together C structs and the functions that use them into sensible objects. For example, the Movie struct and several dozen functions from Movies.h are represented as a class called quicktime.std.movies.Movie. This collection of classes is then farmed out into two sets of packages:

1. A wrapper to the underlying QuickTime API, directly translating Java calls into native equivalents. This includes all packages in the quicktime.std hierarchy, as well as a few others.
2. A bridge into Java, providing points of contact between QuickTime and Java, most obviously in providing a means of rendering QuickTime content into AWT or Swing components. These classes are in the quicktime.app hierarchy.

Apple's J2SE 1.4.1 implementation featured a rewrite of the GUI layer in Cocoa, whose object-orientation and threading models were a better fit for Java. QTJ continued to work with Java 1.4 on Windows, but the Mac version of QTJ was still Carbon-based, and calls from Cocoa's world of pre-emptive threading to Carbon's cooperative threading were a dicey proposition. There was also a too-tight integration with the older QuickDraw graphics API. For whatever reason, Apple chose not to address these problems before the launch of their 1.4.1, instead letting QTJ apps just fail with a NoClassDefFoundError when they attempted to access Carbon. Mac developers were up in arms that Apple's QTJ worked on Windows but not on their own platform.

After their Java 1.4.1 release, Apple posted information about the incompatibility on QTJ's home page, saying that the future of QTJ was under consideration, and soliciting feedback as to what features of QTJ were actually being used by developers and users. The quicktime.app was very generous in offering a large collection of display classes: effects and rendering conveniences like an off-screen Compositor, 2D "sprites," AWT and Swing components with a wide range of resizing and display options, etc. Fixing all of that code would require a lot of rewriting. With limited resources, Apple was asking the community to help identify the pieces that would be fixed.

Downsizing QTJ

The new release of QTJ, version 6.1, solves this problem by radically reducing the size and scope of the Java bridge. This new version is delivered as part of QuickTime 6.4 for Mac OS X 10.3 ("Panther") and Windows. Note that on Windows, you need to do a custom install or upgrade to get QTJ, as shown in Figure 1, since QTJ is not part of the default installation.

On Mac OS X 10.2 ("Jaguar"), the situation is a little confusing as of this writing. Updating to QuickTime 6.4 wipes out QuickTime for Java, but a separate update for QTJ 6.1 is available via the Software Update utility. Upgrading to QuickTime 6.4 but failing to get the QuickTime Java update will totally break all QTJ apps, so it's important to update both. It's not unreasonable to expect that some future version of the QT 6.4 updater would install QTJ 6.1 automatically, so this warning may not be necessary in the future.

Downloading an SDK to write QTJ apps is not strictly necessary, since all you need is to put the QTJava.zip file in your classpath. The installer puts this file in /System/Library/Java/Extensions on Mac OS X, or the lib\ext directories of any JRE folders it finds on a Windows machine. However, the QTJ SDK has traditionally included a generous collection of sample code, as well as Javadocs for the QTJ API. A Windows version of this SDK is available from Apple's SDK page. The Mac version currently exists as separate pieces, with the Javadocs in one place and the demos available in another. Mac developers will want to consult the QTJ section of What's New in QuickTime 6.4, which indicates which of these demos do and don't work with QTJ 6.1 and Java 1.4.1.

Looking at the new QTJ, the solution to the complex legacies of the old Carbon QTJ code is apparently to start over with a radically simplified quicktime.app hierarchy, one that only provides a basic ability to get QuickTime content, such as movies and images, into the Java display space. Presumably, this smaller set of functionality represents what developers said they needed, and omits the extra goodies.

Specifically, all of these packages from earlier versions of QTJ are now off-limits when developing for Java 1.4.1 or higher on Mac OS X:

• quicktime.app.actions
• quicktime.app.anim
• quicktime.app.audio
• quicktime.app.display
• quicktime.app.event
• quicktime.app.image
• quicktime.app.players
• quicktime.app.sg
• quicktime.app.spaces
• quicktime.app.ui

These packages, however, are still available, and represent the core of QuickTime functionality:

• quicktime
• quicktime.io
• quicktime.jdirect
• quicktime.qd
• quicktime.qd.text
• quicktime.sound
• quicktime.std
• quicktime.std.anim
• quicktime.std.clocks
• quicktime.std.comp
• quicktime.std.image
• quicktime.std.movies
• quicktime.std.movies.media
• quicktime.std.music
• quicktime.std.qtcomponents
• quicktime.std.sg
• quicktime.streaming
• quicktime.util
• quicktime.vr

Note: several qd3d packages have been omitted from this list because while they still exist, they do nothing on Mac OS X, which does not support QuickDraw 3D.

The only surviving part of the bridge is the quicktime.app.time package, which provides time-based callback functions, such as the TaskAllMovies class. It can be used to give movies time to redraw themselves, though this is rarely used by application developers, since adding a movie to a GUI sets up tasking callbacks for you.

Of course, if you look at the Javadocs, or peer into the .zip file with a jar tf QTJava.zip, you'll see all of the old packages and classes are still present. This is apparently to give developers the opportunity to keep using those classes with Java 1.3.1 on Mac OS X, or to use them with Windows. However, the affected classes are clearly marked as deprecated, and attempting to use one may bring up a runtime exception. For example, if you attempt to run an old QTJ app in Java 1.4.1, trying to set up the GUI's QTCanvas will kick up this exception:

quicktime.QTRuntimeException[QTJava:6.1.0g1]
Unsupported on Mac OS X and Java 1.4 and higher.,QT.vers:6408000

Replacing the old QTJ GUI classes is a small new package called quicktime.app.view, not to be confused with the deprecated quicktime.app.display. This class defines interfaces for components that can be used in AWT and Swing: QTComponent and QTJComponent. In the latter case, the "J" should be understood in the sense of a Swing JComponent, not the last letter of "QTJ."

Using the New Classes

The means of getting and using these components has changed radically. In the old QTJ, you would wire up a Movie to a Java GUI like this:

1. Create a QTCanvas or JQTCanvas component.
2. Add it to your GUI. Optionally, set resizing behavior for the component.
3. Get a Drawable, such as MoviePlayer or QTPlayer, from a Movie, GraphicImporter, or other visual QuickTime source.
4. Call QTCanvas.setClient with the Drawable to wire up the component to the movie, by way of the Drawable.

In the new QTJ, it works like this:

1. Call QTFactory.makeQTComponent() or QTFactory.makeQTJComponent() with your Movie to get a QTComponent or QTJComponent, respectively. Another signature of makeQTComponent() takes a MovieController argument, which causes the returned widget to include a typical QuickTime movie control bar.
2. Call QTComponent.asComponent() or QTJComponent.asJComponent() to get an AWT or Swing component that can be added to your GUI.

One thing to notice about this is that while the classes returned by QTFactory may call themselves "components," suggesting membership in the AWT Component hierarchy, these interfaces really only provide methods to get and set their source movies or images. For compile-time type safety, you need to make the extra asComponent() call to ensure that you have a real AWT Component.

You might also have noticed that the new workflow says nothing about resizing behavior. While the old QTCanvas described specific behaviors for resizing, such as maintaining a movie's aspect ratio or only resizing to even multiples of its original size, the new components have no such controls. Instead, they make the fairly natural default choice of reporting their content's size in getPreferredSize(), while scaling into any size that might be imposed by the AWT layout. Practically speaking, this means if you add a QTComponent as the only member of a java.awt.Frame and then pack() it, the resulting window will be just the right size for the movie. On the other hand, if you want to force a movie to always be, say, 360 by 240, you could force this with something like a GridBagLayout.

Other Changes

A few other changes are worth noting. The QTFactory class that provides these components used to be in quicktime.app, and provided methods for getting Drawables from media sources like files or URLs. The new QTFactory is in quicktime.app.view. Since both versions exist inside of QTJava.zip, it's theoretically possible for the names to collide, but only if you import quicktime.app.*, whose classes have all been deprecated.

One important feature that is now missing is capture. The underlying classes for creating a SequenceGrabber exist in quicktime.std.sg, but there's no clear way to get the captured images onscreen. It may be possible to capture to an offscreen GWorld and then devise a way to get those bits into a component, but I'm not aware that anyone has gotten this working yet. We may just have to wait for the next QTJ.

Rewriting the Examples

I've rewritten all of the example code from previous ONJava.com articles for QTJ 6.1 and included it with this article. For instance, Figure 2 shows the "Simple QTJ Editor" from a few articles back running on Mac OS X 10.3 in Java 1.4.1 with QTJ 6.1.

Here's a summary of the changes for each of the examples:

• "A Gentle Re-Introduction to QuickTime Java, Part 1": The simple player demonstrated in this article simply replaces the QTCanvas with a QTComponent.

• "A Gentle Re-Introduction to QuickTime for Java, Part 2": The editor in this article is a little tricky, since I wanted to force the sizes of the components to be 320 by 240 for the target movie, and 160 by 120 for the source movie. I forced this by putting the QTComponent into the center of a BorderLayout, and then using Swing's Box to create horizontal and vertical struts of exactly the desired width or height, putting those struts in the south and east of the layout, thereby forcing a suitable size for the center.

• "Making Media From Scratch": This two-part article stays entirely within the quicktime.std packages, making rewriting unnecessary. There are a couple of quicktime.app imports, but these appear to have been left over from debugging and aren't used at runtime.

I haven't included an update to "Java Media Development With QuickTime for Java," since I haven't been able to work out some problems with its code. That article offers a bridge from the Java Media Framework to QTJ. More accurately, it uses QTJ as a Player for JMF. One of JMF's basic assumptions is that a movie's "visual component" and its "control panel component" are two separate widgets. In QuickTime, it's more typical to include the default control as part of the visual presentation of the movie. In the old QTJ, it was possible to create a "detached controller," which allowed you to create two QTCanvases or JQTCanvases, one for showing the movie and the other for the control bar. This technique, as described in an Apple tutorial, crashes in QTJ 6.1. An alternative would be to create a controller-less QTComponent with QTFactory.makeQTComponent(Movie), and then make a custom AWT component, for which you'd have to handle mouse-clicks and make appropriate calls to Movie.start(), Movie.setTime(), etc.

Also crashing is the MovieExporter code used in the article "Parsing and Writing QuickTime Files with Java." This is somewhat more troublesome, since the MovieExporters are entirely within the quicktime.std hierarchy, which seemingly didn't change for QTJ 6.1.

I've filed bug reports for these, and Apple developers have repeatedly stressed on mailing lists that they want bug reports and feature requests to be properly filed, which you can do by going to Apple's Bug Reporter. One trait of QuickTime for Java development is that new functions in the native API often don't show up in QTJ until someone asks for them. I complained in an earlier article about the lack of QTJ constants for MPEG-4 and Sorenson 3 video codecs; filing a bug on these took five minutes, and kMPEG4VisualCodecType and kSorenson3CodecType appeared in QTJ 6.1 a short time later.

The Future of QTJ

Is the shrunken quicktime.app going to hold back QTJ in the future? I don't think so. It's likely that many developers use a media API just to play media, meaning a simplified bridge could actually be better, since it is now easier to use -- it was arguably confusing to have to get a Displayable from a Movie and then use setClient() to connect it to a QTCanvas, all just to show some video. Also, QTJ still picks up new features every time the underlying library gains support for new media types or codecs. That brought MPEG-4 playback to Java developers with QuickTime 6, and means that the professional-quality "Pixlet" codec introduced in Mac OS X 10.3 should work in QTJ with no code changes required.

Moreover, the core QuickTime functions in quicktime.std provide APIs for movie editing, transcoding between formats, metadata, low-level sample access, and other tasks. This is the heart and soul of QuickTime, and it's not going anywhere. For creating and working with media, the QuickTime API remains unmatched.

• qtj61-add-timecode-track.tar.gz
• qtj61-editor.tar.gz
• qtj61-make-csequence.tar.gz
• qtj61-make-text-track.tar.gz
• qtj61-player.tar.gz

Chris Adamson is an author, editor, and developer specializing in iPhone and Mac.

Return to ONJava.com.

Comments on this article

Showing messages 1 through 14 of 14.

• Applets and QTJ 6.1 on OSX
  2004-10-05 16:56:24  angorwatt [View]
  
  
  Hi Chris. I am coding on OSX 10.3 jdk 1.4.1. I can get your example QTSimpleplayer and its swing version to work as applications. If I try to appletize them, it doesnt work. I cannot get apple's sample code such as QTStreamingApplet to work in the applet viewer. ( for neither of QTSS streaming server on another machine and a local movie) I dont get errors and it does see the movie, and seems to create the controller, but doesnt show anything up on the canvas. any ideas what I could do. sorry, never coded with QT before.
  thanks!

• Having trouble with frame grabbing (code from another article)
  2004-07-21 09:02:10  johnmangum [View]
  
  
  
  I was trying out the code grabbing a frame from a movie in your article here.
  
  http://www.onjava.com/pub/a/onjava/2002/12/23/jmf.html?page=2
  
  But I'm having trouble getting a good jpeg file from a movie. Here's the code:
  
  

  
  
  			QTSession.open();
  
  			QTFile qtFile = new QTFile("/Users/josh/kelley20.mp4");
  
  
  
  			OpenMovieFile movieFile = OpenMovieFile.asRead(qtFile);
  
  			Movie movie = Movie.fromFile(movieFile);
  
  			int j = 0;
  
  			int step = (movie.getDuration() - movie.getTime())/5;
  
  			for (int i = movie.getTime(); i < movie.getDuration(); i += step) {
  
  				j++;
  
  				Pict pict = movie.getPict(i); // frame number
  
  				File f = new File("/Users/josh/quicktime/test" + j +".jpeg");
  
  				
  
  				pict.writeToFile(f);
  
  			}
  
  			QTSession.close();
  
  			System.exit(0);
  
  

  
  
  It almost works. There is a file saved and I can open it in safari and it's the right picture. Unfortunately it won't open in camino ("this jpeg contains errors") or preview. It's not really a jpeg though. Graphics Converter says its file format is "QuickTime Importer (QuickDraw)".
  
  Any idea how i can convert this into a jpeg? I've tried using GraphicsImporter,on the file like this:
  
  

  
  
  			GraphicsImporter gi = new GraphicsImporter(qtFile);  // get a cantFindHandler error here
  
  			GraphicsImporterDrawer myDrawer = new GraphicsImporterDrawer 
  
  				(gi); 
  
  			QTImageProducer qtProducer = new QTImageProducer (myDrawer, 
  
  					size); 
  
  			Image image = Toolkit.getDefaultToolkit().createImage(qtProducer); 
  
  

  
  
  but I get the "Unsupported on Mac OS X and Java 1.4 and higher" error on the GraphicsImporterDrawer line. I'm using 10.2, could that be a problem? (I think i've updated quicktime java properly).
  
  Could i use ibm's toolkit for this?
  
  Thanks for your help and great articles, J.
  
• Having trouble with frame grabbing (code from another article)
  2004-07-22 10:27:30  Chris Adamson | [View]
  
  

  This is coming in a soon-to-be-announced QTJ book from O'Reilly (pause for effect of thundering applause), which I'm just now able to talk about. I'm trying to get the example code posted on an open-source site (probably java.net) ASAP.

  Anyways, I did the QuickDraw chapter a few weeks ago. What you've done is shockingly close to something in the book.

  Two catches:

  • you don't need to write the Picts to disk. Copy a Pict's bytes to a buffer that's 512 bytes larger, so you have a 512-byte header pad. Point the GraphicsImporter to that memory with setDataReference().
  • there are two GraphicsImporterDrawers. You seem to be picking up the deprecated one from quicktime.app.image (all of which is gone). Check your imports to make sure you're getting the new one from quicktime.app.view.

  
  

  --Chris (invalidname)

  
  
• Having trouble with frame grabbing (code from another article)
  2004-09-13 19:16:07  ktown [View]
  
  
  Has this code now been made available somewhere? Is the new QTJ book announced?
• Having trouble with frame grabbing (code from another article)
  2004-09-25 11:03:30  Chris Adamson | [View]
  
  
  The book is coming in December - working through tech review now. The grab-a-movie-frame trick with the GraphicsImporter and the 512-byte pseudo-header is spelled out with code (from the book) in a message I posted to the quicktime-java mailing list:
  
  http://lists.apple.com/archives/quicktime-java/2004/Jul/msg00032.html
  
  -Chris

• Can't you just use the getMovie() Method of Sequence Grabber?
  2004-06-15 14:31:57  CyrusMobasheri [View]
  
  
  Can't you just use the getMovie() Method of Sequence Grabber, and use this Movie Object in the following line?
  
  Component component = QTFactory.makeQTComponent (movie);
• Re: Can't you just use the getMovie() Method of Sequence Grabber?
  2004-06-16 05:19:18  Chris Adamson | [View]
  
  
  Have you tried it? It will compile, but it seems like the native structure wrapped by the Movie dies as soon as you create it.
  
  Here's the critical section of some test code I wrote:
  
  

  
  
  QTSession.open();
  
  grabber = new SequenceGrabber();
  
  SGVideoChannel vChannel = new SGVideoChannel(grabber);
  
  System.out.println ("got video channel, sourceVideoBounds = " +
  
                      vChannel.getSrcVideoBounds() +
  
                      ", videoRect = " + vChannel.getVideoRect());
  
  
  
  /* this doesn't work - QT native object is non-existent
  
  */
  
  Movie movie = Movie.fromSequenceGrabber(grabber);
  
  System.out.println (movie);
  
  QTComponent qtc = QTFactory.makeQTComponent (movie);
  
  Component c = qtc.asComponent();
  
  

  
  
  Run this with a camera attached (I'm using an iSight) and here's the output (line breaks inserted for clarity):
  
  

  
  
  got video channel,
  
  sourceVideoBounds = quicktime.qd.QDRect[x=0.0,y=0.0,width=1600.0,height=1200.0],
  
  videoRect = quicktime.qd.QDRect[x=0.0,y=0.0,width=1600.0,height=1200.0]
  
  quicktime.QTNullPointerException: The QT native object
  
  represented by this QTJava object is no longer valid:
  
  quicktime.std.movies.Movie
  
          at quicktime.QTObject._ID(QTObject.java:77)
  
          at quicktime.std.movies.Movie.getBounds(Movie.java:845)
  
          at quicktime.std.movies.Movie.toString(Movie.java:2252)
  
          at java.lang.String.valueOf(String.java:2131)
  
          at java.io.PrintStream.print(PrintStream.java:462)
  
          at java.io.PrintStream.println(PrintStream.java:599)
  
          at SGExperiment1.<init>(SGExperiment1.java:33)
  
          at SGExperiment1.main(SGExperiment1.java:18)
  
  

  
  
  So, we have some low-level access to the camera pixel data via the video channel, but trying to use high-level concepts like the Movie bomb out: we can't even println it! And taking out the println doesn't help - you'll just get more QTNullPointerExceptions later on when you try to idle the SG.
  
  I hope to be back soon with a piece on what you can and can't do with QTJ capture. In the meantime, file a bug on QTJ capture being broken. Apple says volume of bug reports is a key determinant of where they apply development effort.
  
  --Chris
• you are right and its even worth
  2004-06-20 07:12:54  CyrusMobasheri [View]
  
  
  I tried to use the grabPict() methode of the sequencegrabber, but all i get is an empty Pict file.

• QTJ image Processing
  2004-04-13 00:48:24  wmalgadey [View]
  
  
  Is somebody out there who is able to describe how to do some image processing with QTJ?
  I need to get the pixel values (RGB) from an Image on the HDD? The only way I see today is to do the decompressing and getting trough the ByteArray of EncodedImage.
  The method .getCPixel(int, int) is not ablicable, because I cannot get a usefull QDGraphic from my encodedImage (or its ImageDescription).
  
  greets wmalgadey

• Capture available yet...?
  2004-01-14 12:42:00  anonymous2 [View]
  
  
  I have seen code that gets the bits from a SequenceGrabber via a QDGraphics (off-screen GWorld) and uses a PixMap to get the pixels for display outside of QuickTime.
  
  Unfortunately, this seems as though it degrades performance and I am trying to figure-out a better way. Anyone know of a way to get a GWorld into a Movie, that is create a Movie out of a GWorld?
  
  vanevery@yahoo.com
• Capture available yet...?
  2004-01-14 13:13:26  Chris Adamson | [View]
  
  
  Interesting. I have been experimenting with something similar for an upcoming article.
  
  The stuff that's in quicktime.std.sg (sg == "SequenceGrabber") substantially works, in that you can get a SequenceGrabber, iterate over devices, look at the audio or video channels, get plausible dimensions for the GWorld, etc.
  
  But on the other hand, creating a Movie with Movie.fromSequenceGrabber(sg) gives you a movie that instantly dies (try to use it, even for a println, and you get a "the native QT object is no longer valid" error).
  
  It's interesting that you can get the pixels to an offscreen GWorld (I'm trying to dump them as a Pict to disk and it's not working yet), and it makes sense that you then have to use them "outside of QuickTime", since you don't have a valid Movie or GraphicsImporter from which to fashion a QTComponent. So what do you figure this code is doing... maybe taking the PixMap and making a BufferedImage out of it (since the PixMap gives you a RawEncodedImage for the Raster and a ColorTable)? You mention performance is bad and that makes sense - from the QT offscreen buffer to a Java offscreen buffer and then blitted to a (J)Component. Eep.
  
  If you have a link to this code, please post it as a follow-up here - it'd be interesting.
  
  That said, even if we can do this, we've still lost a lot of why we'd want capture. QTJ should be able to save the captured movie to memory or disk (as a real QT Movie, so we can edit it, export it, etc.), get us a MediaHandler so we can look at audio levels on the mic ("Karaoke Revolution" anyone?), look for diffs between frames (turn your G5 into a motion detector), stream it if enough of the Presentation API still works (if it ever did), etc. We still need capture to get fixed for real.
  
  Apple says they listen to bug reports, and duplicates tell them what's important, so go ahead and file a "we want QTJ capture back" bug. I have. http://bugreport.apple.com/
  
  --Chris (invalidname)

• Now a member, still can't find source
  2003-11-04 08:49:00  jdhardin [View]
  
  
  Hi folks-
  
  i just registered as an O'Reilly member, and still don't see where the source files are for this nice article.
  
  Thanks!
  Jeff
• Now a member, still can't find source
  2003-11-04 10:16:03  sarahkim [View]
  
  
  Hello,
  
  My apologizes. The example code is now available.
  
  Sarah
  ONJava.com Producer

• Where are the source files?
  2003-11-04 08:41:14  anonymous2 [View]
  
  
  CHris said that he had rewritten the QTJava examples for QTJ 6.1/Java 1.4, but I don't see where these are from the article.
  
  Thanks in advance,
  
  Jeff Hardin
  Professor
  Dept. of Zoology
  Univ. of Wisconsin
  jdhardin@wisc.edu