Apple’s disappointingly low-interaction iBooks (pending custom widgets, activity logging)

Apple held a much-anticipated education event today in NYC, and I’m mostly disappointed though not without hope. If I were really cynical, I’d say they’ve made a deal with the big publishers that will benefit both of their businesses without really benefitting the end users. But I’m not that cynical. My gripe is with the potential for interactive engagement (buzzword alert) that might be lost on this new etextbook platform depending on the mysterious side door that is the custom widget. What interactive engagement means in the context of ebooks is perhaps unclear, so this is where I will start.

The Meaning of Interactive

To compare whether Apple’s new platform for electronic textbooks is really revolutionary on the interactivity front, we need to remember that the copy of Physics by Halliday & Resnick (3rd Ed.) on my bookshelf is not exactly a hunk of basalt. It’s at least interactive in the following way: it has cover art, lots of different pages, and I can freely flip through them in any order I want to. And it has pictures, or at least diagrams, which convey things to me that are hard to do with text alone. Some of the text is bold or highlighted in some way, and I own the book so I’m free to write notes in it if I want to.

My iPad is way cooler than my copy of Halliday & Resnick; the color is nice and bright and I can have thousands of textbooks stored in it. But if all those textbooks do is flip pages, show pictures and take my notes, I would not say the interactivity has been substantially upgraded.

The first level of interaction is being able to see content that is richly dynamic and to change that content, albeit in a limited way. Watching videos and animations, zooming in and out of or rotating an image, turning on and off parts of a diagram or switching between multiple display modes–all of these are interactivity level one. It’s nice that these are part of the iBook experience, but it’s really the least you can do.

Interactivity 2.0 means being able to change things in a meaningful way, like moving a bunch of objects around and seeing what happens to their state as a result or writing in your own equation and seeing it graphed as a mathematical curve. Interactive web graphics, apps and games have multiple knobs and gesture-based controls with exciting outcomes that can keep you busy for several minutes or hours, not ten seconds flipping through mode 1 through 5.

Here’s an example taken from life not from the page: Imagine you moved into my house and I showed you to your new room and said, look you can do whatever you want in here, ahem, you can view your room from multiple angles, you can lie down in that bed there, and you can turn the lights on and off. That’s interactivity level 1 and not really all that generous. It’s a far cry from saying you can replace the carpet, move the furniture in, out and around, play music, cover the walls with faux fur, install your own disco lighting and change the window treatments. That’s Interactivity.

Of course the two levels I just sketched are arbitrary demarcations, because interactivity is really measured on a continuum that is a function of how rich a feedback experience you get from the interaction. And it seems to me that Apple’s current offering is rather limited.

I took a peek at the free sample of the only physics textbook in the new store. It’s a digital version of Glencoe Physics: Principles and Problems by Zitzewitz et al., which is actually a pretty good textbook. I even ordered it for my ninth grade class when I was teaching full time. The free sample chapter is Vibrations and Waves, and it is full of pseudo-interactive figures like one on pendulum motion, which is really a three-stage figure with some transitional frames to “animate” it. Can you pull the pendulum and release it? No. Can you change the mass or length of rope or constant of gravity to see what effect each has on the period of oscillation? No. In fact this figure, like all of the interactive figures, is more frustrating than informative because your natural tendency is to try to make things happen that you just can’t. (If after reading this, you’re dying to play with a virtual pendulum, you can use the truly interactive PhET simulation).

I don’t believe this is just a shoddy job on McGraw-Hill’s part. The iBooks Author application imports interactivity through the following set of widgets: Gallery (multiple images you can flip through), Media (video), Review (multiple-choice-type questions), Keynote (for graphs and data, presumably), 3D (image is rotatable with the touch interface) and Interactive Image. Interactive Image means you can select parts of the image which, when clicked on, will zoom the view and maybe show a bubble of text.

You’re not going to get past level one interactivity with them tools. But here’s the rub. Supposedly you can author your own widgets with javascript. I will need to see more, but this gives me hope for the next level interactivity, because I’ve seen wonderfully interactive web graphics using AJAX/javascript/jQuery. If iBooks Author can really incorporate that functionality, I may become a believer yet.

Not interactivity, but the next big thing in education

One feature that I doubt will be possible with iBooks2 is activity logging, and that’s a shame. Activity logging is one way that educational web services learn what students really do given the opportunity, how much time they spend with different features, and—in the best possible integration of learning resources with formative assessment—how that interaction has affected their learning. This information is used to keep improving the resources in a feedback loop that makes things better for everyone and faster than the typical revision cycle. And the data can also be useful to teachers, who can subsequently supplement the offerings in ways the textbook cannot. Some people get privacy concerns about activity logging, but it can be done safely and anonymously. If you think of those bug reports your computer is shooting off to Apple or Microsoft (assuming you give your permission), you can see one simple case in point.

In sum, Apple’s small step falls way short of the potential for interactive online educational tools, but it safeguards a certain proprietary model for publishing, which makes it palatable to educational behemoths. That’s a sound business move. If custom widgets open the door to real interactivity then it will also be a boon for students and independent learners. But unless it grows to include activity logging and meaningful assessment, this new platform will soon be supplanted by higher quality web-based services which will be just as accessible from mobile devices and even more useful for students and teachers.

The Holodeck is the Ideal Classroom. The wallodeck is closer to reality.

I had never heard of this guy before, but he beat me to the point I want to make today, which is that the holodeck is the platonic ideal of the classroom. In the holodeck, anything is possible*, you can experience virtual reality with all of your senses, it is completely safe to experiment, you can pause, slow down or rewind your experience, and there is a nearly omniscient computer available to answer any question you might have (but it will only do so if you ask or get really badly stuck). You can be in the holodeck alone or with real or virtual companions. The holodeck is so amazing that I can see why it would coexist with a war-less future.

* although the interaction is rule-based, the rules can be different from the usual rules.

Is Microsoft building one? Maybe. Is the CAVE sort of like it?

I’m not sure the 3D-realistic-haptic experience is that close on the horizon. Short of the real deal, here is how I envision a physics education wallodeck (2D and not haptic) within the next 5-10 years with the aid of artificial intelligence. It would take place on a surface (just don’t say SMART board) and enable you to:

  • draw objects or write text, and it would recognize objects, text labels, and equations. Examples of research on this are the now-defunct Microsoft center at Brown U.  and pen-based interactions at ISUE by Joe LaViola (a former graduate of the Brown center). Try this fun handwriting to LaTeX converter, if that means anything to you. I am deliberately glossing over the issue of representing 3D objects in 2D for now.
  • assign rules to the objects you create such as: rigid-body, interacts with gravity, objects that collide with it lose 80% of their energy, has no surface friction, slightly magnetic, etc… thus you can “turn on time” and animate anything. Commercial solid modelers like SolidWorks and Autodesk Inventor already do this. Arriving HD for the iPad uses this kind of thing in a game environment. And PheT simulations do it for objects that are predetermined and physics-topical. NetLogo is an amazing tool for writing code to model very simple dynamical systems.
  • describe the system with equation models and the computer will correct you if you make a mistake (assuming you want this feature on). Solve for unknowns as physics practice. (cf. Andes tutor and other Intelligent Tutoring Systems or ITS)
  • create problems, share them, or solve problems created by anyone (cf. LONCAPA) or created on the fly by the wallodeck program, obviating the need for withholding publication of problems to prevent copying graded assignments.
  • know your skill level, what the holes in your knowledge are, how you stack up, and what are the best next moves to help you improve. This is where more machine learning, educational data mining, cognitive science, applied psychological measurement and ITS research fits in. The problem of assessment still looms large.

If you are an animator and are interested in animating this wallodeck concept collaboratively, get in touch with me!

Two rules of data mining

Why no posts? I’m actually working on a paper called Item Response Theory in the Style of Collaborative Filtering. I wanted to write all about it here, but then I thought that might cause trouble if I haven’t finished at least a draft of the paper. Soon. In the meantime, I offer you this:

The Two Rules of Data Mining:
1) Less is more.

Student information by the numbers

I alluded to the increase in information available about student learning from monitoring online interactions. I want to make this explicit and discuss the numbers a little bit. In a typical college course grade book, an instructor will record a dozen maybe fifteen homework grades, half a dozen quiz scores and, say, three exams. If each grade is on a scale of 0-100, we should allocate 7 bits for each grade (homework grades might not even be this detailed in practice). With 24 grades in the final gradebook, that makes 168 bits of information on the basis of which the student is finally evaluated.

In online interactions, students are almost always allowed multiple attempts, and sometimes they are given optional hints or can choose to do a practice problem first. The system can theoretically track every click, but practically speaking, we will at least keep track of the following information: number of attempts (3 bits), hints or intermediate resources used (3-6 bits), time-to-response (3-6 bits), problem-specific parameters quantifying the difficulty and discrimination parameters (10 bits). For each problem solved online, we thus record 20 bits of information. At 250 problems in a typical semester-long course, that makes 5000 bits of relevant information. Note this is more modest than the factor of 100 I have previously suggested, but I am being conservative in both the high and low estimates.

What is the difference between 168 bits and 5000 bits? I hope Daniel Kahneman won’t mind my repurposing an image he uses in his new book Thinking, Fast and Slow. Kahneman introduces the image to make a point about processes that occur very quickly in the brain, such as reading someone’s facial expression. Here is what happens if you have to compromise the eye’s extraordinary resolution for the limited information available in the two cases we are considering:


The image on the left contains about 168 pixels and the image on the right about 5000. I’ve cheated in that the pixel depth is actually 8 bits, i.e. grayscale not just black & white, so it would be more accurate to call them 168 byte and 5000 byte images. Interesting that at 168 pixels, you can tell that you’re looking at a face, whereas at 5000 you can tell that face is about to yell at you.

Some of what is out there

Tom Mitchell of CMU has a fascinating slide talk available online about Human and Machine Learning. Mitchell’s expertise is definitely weighted to the computer science side, but he is also concerned with the cognitive neuroscience component of human learning. Which I just want to point out early on is somewhat orthogonal to anything I am trying to do–I will have very little to say about dopamine response, though there is a natural connection between dopamine response and reinforcement learning as it pertains to machines (scholarpedia on reinforcement learning from both algorithmic and neuronal perspectives).

This work helps me bring into relief the parts of machine learning and human learning that I am personally focused on, at least for now, which subsumes formal education (or school learning, K-16) and educational measurement on the human learning side and data mining and pattern recognition on the machine learning side. Formal education helps to distinguish between the kind of learning humans do when they acquire the skill to recognize a chair for what it is (pre school, presumably) vs when they learn to answer word problems which might involve algebraic solution.

Traditional education research involves a lot of theory of cognition and a lot of case studies, trials, interviews and assessments. But today a whole lot of education is happening in online or computer-based environments, and with that the sheer quantity of hard data increases by at least a factor of 100 (easily more). Educational measurement has gotten more sophisticated as well, employing a growing statistical arsenal. It is irresistible to bring some of these techniques to the data, which is what I actually do.  But I’m trying to resist it a little bit at the moment, because I think there is a learning moment for me to use machine learning techniques to better mine these data, to recognize new patterns and to develop new educational metrics. I want to write a machine learning algorithm which will come up with IRT by itself…and then come up with something even better. The long-term objective of all of this learning about learning is to improve (formal) education.

First training example

Three months ago I started a post-doc in physics education research at MIT in a group called RELATE, which stands for Research in Learning and Tutoring Effectively. [Before that I was a teacher, a furniture-maker/sculptor and a particle physics grad student, in reverse chronological order.] The work at RELATE, in the smallest possible nutshell, is focused on quantitative analysis in physics education research, especially for online learning environments. There is a lot of data mining here, informed by concept maps and cognitive models. Publications on the website tell the story better.

When I arrived at MIT, I inherited a project that had been worked on by at least two former postdocs (R. Warnakulasooriya and later C. Cardamone): applying Item Response Theory (IRT) as an assessment tool in our research. IRT is a psychometric method that allows you to measure parameters about assessment items (i.e. difficulty and discrimination of questions) simultaneously with the ability/skill of the person answering the questions. IRT extracts more information than classical test theory (CTT) and allows you to measure students on the same scale using different questions. It also helps you evaluate the quality of your items. There will be a lot more written about IRT in the future of this blog. One of the facets about IRT that is particularly relevant to mention is that it is not really designed to be used in a dynamic, noisy, online learning environment. IRT was developed for use in high-stakes testing, which is highly controlled. So we have a whole host of other interesting problems.

Because we are interested in the different ways that online learning takes place, I was paying attention when Stanford opened up three courses this Fall (Artificial Intelligence, Machine Learning, and Databases) to a wide on-line audience. Over 100,000 people enrolled in some of these. I started out curious about course format, but I quickly became even more interested in the content, especially Andrew Ng’s Machine Learning course. Much of this work is quite related to the IRT analysis I have been doing, and now my mind is spinning. Can we use machine learning algorithms to develop even better assessment tools than IRT, to…analyze student learning?

In order to keep track of some of these thoughts, I decided to start a blog (my first). The topic is the interplay of human learning and machine learning. On the simplest level, RELATE is interested in studying (and improving) human learning using machines, machines which help us analyze data that are obtained from the interaction of human students with other machines.  On a deeper level, Machine Learning can be used to learn about human learning from the same data.  This is less circular than it sounds. All three pairwise links between the nodes human, learning, and machine contain synergistic ideas.  In fact here is a fascinating workshop description from the NIPS 2008 conference titled Machine learning meets human learning.

That is the starting point.