Making a Difference

You may have already heard about the NeverSeconds blog or the 9-year-old who created it to rate her school's cafeteria food. How after just a few blog posts, the photos and descriptions got national and then world attention. She got the food situation improved (with help from dad), and has created a way for other kids at other schools around the globe to become involved with school food. It's an amazing story. Here's one of her food reviews (source):

Today's meal was on the menu as Cheeseburger and ice cream/biscuit but as you can see I got an ice lolly. I prefer ice cream. I wish they had stuck to the menu. I did get 2 croquettes though only 3 pieces of cucumber when I said no thanks to the peas.  

Food-o-meter-7/10
Mouthfuls- eating and counting and chatting to friends is hard!
Courses- main/dessert
Health Rating- 2/10
Price- £2
Pieces of hair- 0!

This story has made the rounds framed as a human-interest bit, which of course it is. But for me there's a larger story.

This couldn't have happened without the Internet and cheap consumer technology. Imagine just a couple of decades ago what it would have required for a 9-year-old to photograph her lunch every day and share her musings with hundreds of thousands of other people. And now we don't even notice the technological miracle--that's not the story at all. The story is not that this schoolgirl is a prodigy either. It's that she cared about something enough to do something about it, and it made a difference.

This winds back to my point about the medieval mindset that still permeates much of our educational systems: the "prepare and certify" model (see this post for more). In treating students like they are machines on an assembly line, we overlook the fact that they are already capable of doing very cool things, as this 9-year-old demonstrates. Yes, we have to prepare them by helping them learn about the world in ways they wouldn't do accidentally. But we can also lead students to an even more important realization: they can already begin to change the world with the preparation they already have.

The young nutritionist created her project because of intrinsic motivation, not because of a school assignment. Now it stands on its own for anyone to look at and assess as an accomplishment. This is very different from a classroom assignment with purely extrinsic motivation, which results ultimately in a letter grade assessment that gets averaged into others, losing most of the evidence that the event ever happened. As with most 'preparation' activities, it would be entirely transient--a momentary hurdle to be overcome on the way to a very distant graduation from university and ultimate 'certification' in the form of a diploma. I hope the contrast between these is sufficiently stark that you may wonder about how we might do things differently. At the very least, it ought to generate some doubt that it really requires more than twenty years to prepare a human being to be useful in society.

A Social Media Metric Arrives

In "The End of Preparation" I wrote that:

The method of assessing a portfolio is deferred to the final observer. You may be interested in someone else's opinion or you may not be. It's simply there to inspect. Once this is established, third parties will undoubtedly create a business out of rating portfolios for suitability for your business if you're too busy to do it yourself.

It turns out that third parties aren't waiting on portfolios. If you read the I-ACT article, you may recall that participation in professional social networks like mathoverflow.net are part of what goes in my ideal portfolio. These professional networks are not developed yet for every discipline, but there is a company using lowest-common-denominator social networks to rate your overall social impact. The site is Klout.com. Here's what they say:

People have always had the power to influence others, and that power is being democratized with new social media tools. Klout's mission is to provide insights into everyone's influence. We measure your influence based on your ability to drive action in social networks. We process this data on a daily basis to give you an updated Klout Score each morning. Here are a few of the actions we use to measure influence:

I didn't want to give them my Twitter or Facebook password so they could calculate my score. In any event, it can't be very large on their log scale from 1-100:

The average Klout Score is actually 20, not 50. As your Score increases, it becomes exponentially harder to increase your Klout. That's why you see so many 20s and not as many 90s!

I learned about Klout from Reddit (see comments there), which points to a Wired article "What Your Klout Score Really Means" by Seth Stevenson. The article describe the experience of a candidate for a VP position at a marketing agency:

The interviewer pulled up the web page for Klout.com—a service that purports to measure users’ online influence on a scale from 1 to 100—and angled the monitor so that [the candidate] could see the humbling result for himself: His score was 34. “He cut the interview short pretty soon after that,” [the candidate] says. Later he learned that he’d been eliminated as a candidate specifically because his Klout score was too low. “They hired a guy whose score was 67.”

At present, using such a crude instrument probably only damages the hiring process, and is awfully shortsighted. But this is just the beginning. For mathematics, you can already browse mathoverflow.net to see the reputation on this social site devoted to research-level mathematics. The user with the highest assigned reputation is Joel David Hamkins. Take a look at his page there and see how rich it is with information about his professional life.

Means to What End?

The title for this article comes from a 4/22 Commentary in the The Chronicle entitled "Stop Telling Students to Study for Exams" by David Jaffee. Here's a bit of it:

If there is one student attitude that most all faculty bemoan, it is instrumentalism. This is the view that you go to college to get a degree to get a job to make money to be happy. Similarly, you take this course to meet this requirement, and you do coursework and read the material to pass the course to graduate to get the degree. Everything is a means to an end. Nothing is an end in itself. There is no higher purpose.

I put in bold the headline quote so you couldn't miss it. Instrumentalism is the idea that predicting cause and effect is more important than "understanding reality," and I'm not sure it's exactly the right concept for this argument. But the argument is still valid, and summed up in this ubiquitous practice:

On the one hand, we tell students to value learning for learning's sake; on the other, we tell students they'd better know this or that, or they'd better take notes, or they'd better read the book, because it will be on the next exam; if they don't do these things, they will pay a price in academic failure. This communicates to students that the process of intellectual inquiry, academic exploration, and acquiring knowledge is a purely instrumental activity—designed to ensure success on the next assessment.

This is the "prepare and certify" model that I dissected in "The End of Preparation." In theory, the preparation (the cause) enables students to be functional in graduate school, employment, entrepreneurship, performance, public service, or some other worthy human endeavor after graduation. The reason I don't think our prepare/certify model is instrumentalism is because it's rare for anyone to check this connection between the preparation and its ultimate impact. That's mainly because it's so hard to do. Yes, we get studies about how much an undergraduate degree is worth in terms of life wages, but that doesn't say anything causal about the education itself (correlation is not causation).

The whole article is worth a read. Maybe I'm saying that because it reaches the same conclusions I have:

Authentic assessments involve giving students opportunities to demonstrate their abilities in a real-world context. Ideally, student performance is assessed not on the ability to memorize or recite terms and definitions but the ability to use the repertoire of disciplinary tools—be they theories, concepts, or principles—to analyze and solve a realistic problem that they might face as practitioners in the field.

I have gone further and tried to show how we can do that. See "I-ACT: An Alternative to Prepare-and-Certify."

If you have some time to read it, there's a provocative article on the philosophy of science that is related to instrumentalism (my assessment), and does have a connection to the subject matter, albeit from the perspective of natural selection: "The Interface Theory of Perception: Natural Selection Drives True Perception To Swift Extinction" by Donald D. Hoffman. Here's the abstract:

A goal of perception is to estimate true properties of the world. A goal of categorization is to classify its structure. Aeons of evolution have shaped our senses to this end. These three assumptions motivate much work on human perception. I here argue, on evolutionary grounds, that all three are false. Instead, our perceptions constitute a species-speci c user interface that guides behavior in a niche. Just as the icons of a PC's interface hide the complexity of the computer, so our perceptions usefully hide the complexity of the world, and guide adaptive behavior. This interface theory of perception o ers a framework, motivated by evolution, to guide research in object categorization. This framework informs a new class of evolutionary games, called interface games, in which pithy perceptions often drive true perceptions to extinction.

I have added emphasis to the point I think connects to the current context: the way we think of the world, how this forms cause-effect models and the language we construct to process it, collectively form an "interface" that guides behavior in a niche, as the author puts it. If the 'niche' is negotiation of short-term hurdles using short-term memory and becoming skillful at doing minimal work to earn grades, that's a very different thing from being productive in the grandest way humans are capable of: through art and rhetoric, leadership and service--the effects we are actually hoping for when betassled students walk over the stage.

Memory Games

A while back I wrote "Memory as SLO," about the possibility of developing short term memory as an intentional learning outcome. I came across an article that has a particular program for doing that. It seems like this would be a relatively easy bit of research: randomly select some first year students and have them go through a program based on the methods describes.

Read it for yourself at NYT in "Can You Make Yourself Smarter?" Here's a quote:

In a 2008 study, Susanne Jaeggi and Martin Buschkuehl, now of the University of Maryland, found that young adults who practiced [this method] also showed improvement in a fundamental cognitive ability known as “fluid” intelligence: the capacity to solve novel problems, to learn, to reason, to see connections and to get to the bottom of things. The implication was that playing the game literally makes people smarter.

Teaching and Assessment (Video)

This is a long video, but worth the time.

Pattern Matching

A friend gave me a desk calendar with daily puzzles. Usually the numerical ones are easy, but last Friday the one above came up, and I couldn't immediately solve it. If you want to try to solve it yourself, you may not want to read any further until you're done with it.

I assumed the 'game' in the puzzle was to use the four numbers in the corners to mathematically derive the one in the middle, although there was the possibility that there was a non-numerical 'trick' answer that relied on the way numerals are spelled or something. I stuck with it for a while, trying simple calculations, but couldn't find a suitable pattern that worked for all three examples. Then I remembered reading about Eureqa, from Cornell Creative Machines Lab--a genetic pattern finder that should be able to chew this problem up in no time. It's also free.

I labeled the input values x1, x2, x3, and x4 for top left, top right, bottom left, and bottom right, respectively, and the center (output) number became y. It's easy to enter this in the program like a spreadsheet:

The y column was originally 4, 5, 3, which are the numbers in the puzzle. The 8, 9, 3 that appear in the image are explained below.

I skipped the "Prepare Data" tab because I didn't need to smooth data or fill in missing values, etc. Things started to get interesting with the "Prepare Data" tab. The program guessed my target expression correctly. And here I made a mistake. You get to decide what sorts of operators the solution is allowed to use. This is a meta-problem, where you have to think like the test designer. Is it likely that the solution uses the hyperbolic tangent function? Probably not. So I picked the operations one learns about in grade school: addition, subtraction, multiplication, and division. Part of the screen is shown below.

Notice that I left "Constant" and "Integer Constant" unchecked, and hence unavailable for the program to use in a solution. I reasoned that the puzzle designer would not have included an arbitrary number, as in y=(x1+x2-x3-x4+15). It seemed inelegant and made the problem space much bigger to include even integer constants. But this was a case of premature optimization, as we will see. I chose the "Minimize the absolute error" metric of success and started the search.

This was my first time using the interface, but it only took a moment to figure out what I was looking at. The "fitness" is the error function, which is optimized at zero, meaning no errors in predicting y. The formula that it found immediately was y=x4. If you look at the puzzle again, you'll notice that the lower right number is the same as the center one. This is the simplest pattern that matches, and the automatic searcher has a bias for low-complexity (i.e. small) formulas. I was sure the problem designer intended all four input numbers to be used, so this couldn't be it.

So I tried obfuscation to trick the solver into using all four inputs. I did this by transforming the problem by creating a function z, and asking the solver to find a formula f so that f(x1,x2,x3,x4) = y + z(x1,x2,x3,x4). The idea was that this would rule out the simple y = x4 solution, and force the solver to look for answers that  used all four inputs. Then I could subtract off the z part and have my solution. For example, one function I tried was z = 2x1 - x2 - x3*x4.The input screenshot above shows one of these attempts, with the y column transformed in this way.

This yielded results. After using two different z functions, I got two new solutions. Here they are:

f(x1,x2,x3,x4) = x2+x3-x3*x3
f(x1,x2,x3,x4) = -2x1+x2+2x3x4-x3*x3+x4

These both work, and you can check them by plugging in the puzzle inputs to see that the calculations equal the number in the middle. But the first one above doesn't use x4, and although the second one does, it seems too complex for a casual puzzle.

At this point I looked at the "official" answer. Here it is:

f(x1,x2,x3,x4) = (10*x1+x2)/(10*x3+x4)

As you can see, there are integer constants in the solution, which I had ruled out when I set up the problem. Although the formula looks opaque as I have written it, for a human it's natural to read two digits across as a single number, so that the 9 and 6 on the top of the first example becomes 96, which is formulized as 9*10 + 6. The bottom is 24. I had tried that trick myself, but not noticed that 96/24 = 4.

I was naturally annoyed at myself for having ruled out the possibility of finding the official solution by unchecking "Integer Constant," so tried again with constants enabled. This time I got

f(x1,x2,x3,x4) = 24-2*x1-x2-x4-x3*x4

This was after 6931 generations. It still isn't what I'm looking for. My transformation was making sure all the inputs were being used, but it was a crude hack that ended up making the solution more complex than it needed to be.

What I really needed was a way to modify either the target expression or the error metric in order so that an optimal solution has exactly one occurrence of each of x1 through x4. I couldn't figure out how to do that with the interface, and it may not be possible with this version of the software. So I tried another approach.

The puzzle is flawed in that it allows the y = x4 solution, so I created some more examples for the solver to chew on: 22/11 = 2 and 26/12 = 3. With the original puzzle values plus these two extras, the solver began to converge to solutions with smaller and smaller errors:

Unfortunately, integer-based problems like this just aren't amenable to gradient-based approaches. Unlike horseshoes and hand grenades, close isn't good enough. After seven minutes, the formulas were all variations on high powers of the inputs, overfitting the limited data to find a solution.

I tried 'seeding' the search with y=x1/x3 This is the same as y = (10*x1)/(10*x3), which is pretty close to the solution. After more than 20,000 generations that resulted in a perfect solution, but not the one I wanted:

y = x4 + -1/x4 + (x2 + x3 - x3*x3)/(x4*x4)

It did recognize the 'correct' solution when I fed that directly in. This is pictured below (third one down).

The 'fit' is the error, which is zero for the right answer. It immediately lost the integer constants, however, and went off looking for lower-complexity solutions, which do exist (see the ones listed earlier).

To be fair, Eureqa isn't really designed for solving this sort of problem. Finding patterns with real world data usually means accounting for noise and missing values, and looking for simple approximate relationships that might tell you something about underlying relationships. In this case, y = x4 is still a pretty good approximation for the examples I put in. It occurred to me that if this were real data, the solution I'd be looking for is y = x1/x3, which would normally be a reasonable approximation to the perfect solution. When I added a few more rows of example data, the solver found this immediately. So it works as intended, which is not necessarily for the purpose of solving arbitrary puzzles humans design for each other.

Update: There are a few comments about the above on /r/machinelearning, which you can find here.

Education News Dot Org

I discovered Education News this week. It seems like a comprehensive, smart source for ed news, and has a clean design to boot. Check it out at EducationNews.org. For bloggers, one of the nice things is that they allow linked backtracks in comments. This isn't true at all sites, and I suppose there is a downside to it from the publisher's perspective (it invites spam links, which have to be weeded out).

Note that the site isn't new. In fact it's as old as my daughter, which means that the editor should keep a shotgun by the door and hide the car keys.

Here's a direct link to the section on higher education.