First let me start with some disclaimers to try to make sure this post is not misinterpreted: I am not arguing that the mouse and keyboard are really dead or that the lack of a mouse on tablets is a bad thing. I am not arguing that the ipad or similar devices are awful for education or content creation. I am just thinking about how to make them even better in these areas, and conceptual and technological roadblocks in the way. Some of the criticisms of the ipad as a content consumption device (here, here), have been addressed with the ipad 2 and other new android tablets with their inclusion of cameras and input ports, and some tablets are even coming out with a stylus, like the HTC Flyer.
But imagine any creative person – creating a 3d character for a game, drawing a picture, composing a music score, creating a graphically rich document or presentation, etc. They likely have something in their hand, or their hands are busy doing something. When that involves interacting with a computer they are likely clicking the mouse to drag something around or edit text, for example, or using a stylus on a digitizer surface (like a wacom) for drawing, or typing away on a keyboard.
These input devices are all essentially gone on new tablets and smartphones. You can still type (slower) on virtual keyboards, and you can click like a mouse with your finger (tap).
An example of the impact of this is rich text editing, like with a word or openoffice or google docs document. Many browser-based wysiwyg editing tools, which are used virtually everywhere, such as in moodle (which uses the TinyMCE editor) or drupal, no longer work when you access them from an ipad or iphone or android device (or other mobile platforms like blackberry or palm webos). Even the newest “HTML5” editors, such as Aloha Editor, pop up an error message if you try to access them from a mobile device. Other browser-based editing and drawing tools also no longer work on these new platforms, or you have to draw with your fingers. Most of Google’s and others’ tools like Google Presentation do not work on mobile platforms. Really, just imagine most any software people use on a desktop to create stuff – like office, or the flash ide, or gimp/photoshop, blender 3d, etc. Even when programming, which really is just typing in plain text, we usually prefer to use IDEs that popup suggestions and corrections to help us out. For many of our desktop apps its hard to even imagine them working on a tablet or phone.
The rich text editing tools in browsers like TinyMCE or CKEditor primarily rely on the contenteditable HTML attribute to support editing. Add that attribute to an HTML element, and the contents of that element become editable inside the web browser. It works in all browsers, including old Internet Explorer versions. It doesn’t really work well or even at all on mobile browsers though (see here, here, here, here).
Newer versions of android, webkit, and mobile firefox have been slowly improving their support for contenteditable, and maybe they will eventually “fix” the issue, but I’m not sure that this will be fixed through engineering alone. Some code editors like codemirror 2 and the ace editor are trying out workarounds like using a hidden textfield that captures key presses. Codemirror 2 works on an ipad somewhat, the ace editor does not. It remains to be seen if a similar trick might work for a rich text editor (it’s tricky enough just to do it for plain text). And like I said, the HTC Flyer and other tablets (esp. those being designed for medical and other professionals) are starting to include a stylus, and it remains to be seen if that will catch on (it didn’t before with older tablets). Others are coming out with dual screen tablets, where the second touchscreen can work like a touchpad on a laptop or nintendo ds, but that also may not catch on.
Another more general alternative strategy to this issue of tablets having no mouse or other input devices other than the touchscreen and the camera (which can be used for gestural or other input), might be to conceptually rethink how to support multimedia creation on these mobile platforms. Perhaps we should drop the notion of “documents” or “pages”. After all, you don’t think of a flash widget as a page or document. You don’t think of a game as a set of pages or documents. And Apple and other developers have already created apps for some specialized types of content creation and creativity, such as musical instrument simulators and so forth.
So, this may be a pre-paradigmatic moment where we’ll see what catches on: will we try to perfectly “emulate” the mouse and stylus and its supported interactions via other means such as gestures, or will new and unique types of interactions continue to catch on (like multitouch stuff). Probably a combination of both, but so far the camera is hardly being used at all for input, other than recording videos or taking pictures. And some may dismiss the idea of a stylus ever catching on again, but Apple has occasionally made “mistakes” before (the first mac didn’t have a floppy drive, for example), and others have been successful in incorporating a stylus, like the Nintendo DS, which my little boy continually loses 🙂