A STUDIO VISIT WITH JER THORP AND BEN RUBIN OF THE OFFICE FOR CREATIVE RESEARCH
Big Data scares people. It begets ominous headlines and underpins global controversies. Faceless entities automate the collection of facts that make up a human life, without consent or transparency. Businesses invade privacy and profile people from afar. The intersection of the wholly personal and the merely mechanical is inherently unsettling. When we envision Big Data, we think not only of search histories, Twitter feeds, and online bank records, but also of this information collected together and its potential uses, either benevolent or nefarious. Who has access to it and who doesn’t. If Big Data is a tool, we wonder: what is its purpose. If it is a hammer, where is the nail, we think, reflexively rubbing our heads.
Yet envisioning the term as exclusively beholden to technology is a miscalculation. Big Data according to Google (arguably a foremost expert on the subject), is defined as “data sets that are too large and complex to manipulate or interrogate with standard methods or tools.” Clearly, that definition holds true for more than just your Facebook account or email history.
In fact, if you believe MIT’s Seth Lloyd, “Merely by existing, all physical systems register information. And by evolving dynamically in time, they transform and process that information. The laws of physics determine the amount of information that a physical system can register (number of bits) and the number of elementary logic operations that a system can perform (number of ops). The universe is a physical system.” Lloyd goes on to posit that “The universe can have performed no more than 10120 ops on 1090 bits.”
In other words, it’s Big Data everywhere. Your breakfast is Big Data. My sweater is Big data. Jack Black is Big Data. The Cretaceous Period is Big Data. Waking up in the morning is Big Data. We’re surrounded by information and always have been. The difference today is that we have increasingly powerful tools to engage it. So, it’s not the information itself that scares us, but the one-two punch of how it’s collected and ultimately used. Which should make us wonder: are there other ways to gather information, and are there other ways to use it? Ways that are less scary? Can Big Data amount to more than buying and selling? More than targeted ads? Something other than the NSA or identity theft? Is data always moving slowly and inevitably toward a PowerPoint presentation?
The work of Jer Thorp and Ben Rubin answers these questions with resounding clarity. Their organization, The Office for Creative Research (OCR), “is a multidisciplinary research group exploring new modes of engagement with data, through unique practices that borrow from both the arts and sciences.” Their work is not parading ultimately toward a PowerPoint slide, but often to an art installation or theater performance. Their work doesn’t manipulate data to better target ads; rather, they’ve created a tool to help you track and counter when that targeting happens.
Recently, we sat down with Jer Thorp and Ben Rubin to talk about the The Office for Creative Research, their working process, and the future of art / science collaborations.
SIENA ORISTAGLIO: [Points.] The iBook, that’s quite a relic.
JER THORP: When we started, there were two of us. Now, there are ten of us. So we performed an archeological dig to make space, and this is all the ephemera that’s come out of that. There are police scanners and Webby Awards from ten years ago, and old telephones, and every model of Mac computer. We recycled a lot of them. We just kept the ones that we thought were important to keep.
SIENA ORISTAGLIO: So, we’ve all done research into your work. We’re all really curious as to how The Office for Creative Research got started.
JER THORP: The genesis of this studio happened in two steps. Ben Rubin and Mark Hansen met in 1999. Mark was a scientist at Bell Labs. I like to say it all started at Bell Labs, because it all started at Bell Labs. So Ben and Mark met because — this is right up your alley — they were doing a project that paired scientists with artists. Mark was the scientist and Ben was the artist. They worked together to build a project called Listening Post (2002) which is one of my favorite works of art ever. It’s completely amazing.
SIENA ORISTAGLIO: Can you describe Listening Post?
JER THORP: It’s an array of screens in a half a circle that show the transcripts of chat rooms. There are two of them: one is in San Jose and there’s another one in the Science Museum in London. So, then I met Mark. I was the data artist-in-residence at The New York Times for two and a half years. Mark was there as well, doing a sabbatical. Ben and Mark did a piece in The New York Times lobby called Moveable Type (2007), if you’ve ever been in the building. If you haven’t, you should go just to see the piece. It’s incredible.
SIENA ORISTAGLIO: I haven’t been. Where is The New York Times building?
JER THORP: Not surprisingly, Times Square. It’s on 8th Ave between 40th and 41st. It’s the third tallest building in the city. Who knew? Well, the people who work in the building. Since then, the three of us have collaborated on many projects together. Then in January of last year, we decided to create The Office for Creative Research and bring some more people in. We have this set of strategies and techniques that we apply to art projects that could also be applied to other hard problems that aren’t necessarily in the art space. We do a lot of what we call “R&D” for companies. Microsoft is one of our big clients and we’re doing a bunch of work with a company called Acumen. We’re also working on some museum and gallery pieces. We just started creating a big piece for the Boston Public Library and we’re doing a yearlong residency at MoMA. But the reality is, we don’t really treat any of these projects much differently from one another. The end product might be slightly different if it’s going to be in a gallery or a museum than it would be otherwise, but the tactics that we use and the thinking — that’s all kind of shared. We have a research agenda. We follow that research agenda through these various projects.
SIENA ORISTAGLIO: How do you conceptualize your agenda?
JER THORP: It’s somewhat nebulous, but it centers around this border between data and culture — where data is becoming culture and culture is defining and producing data — and this region where data and humans are interacting in various ways. We’re really just taking the first step; there is so much room to expand. Most of the industry is very narrow. We’re really interested in getting out of that. We’re trying to push things and trying to think of new ways to engage with data.
SIENA ORISTAGLIO: You say the industry is narrow. How so?
JER THORP: I think it’s just the way business works. There are brief periods of innovation followed by long periods of refinement. With this Big Data thing, three years ago there were a lot of new things happening, and now we’re stuck in one of those periods of refinement. Everyone is doing exactly the same thing. Eventually, we’ll come out of that. But I think even just the focus on data visualization shows that. Data visualization was built on methods that were made for print 70 years ago. It’s an extraordinarily conservative field, both in its technique and in its politics. We sort of exist as a rogue element. We do things that are unusual and sometimes they don’t work, but that’s the way we position ourselves. Lots of people can solve your data problems in really conservative, safe ways but that’s not what we’re interested in.
SIENA ORISTAGLIO: That makes sense. In data visualization, do you find that there aren’t that many people that consider themselves artists?
JER THORP: Well, you’re comparing an occupation with a technique, which is hard. I’m pretty sure I coined the terms “data art” and “data artist” back in 2008 or 2009. Now, they get used a lot by contemporary artists working in this world, as well as by people who just want to make their job sound cooler. It’s kind of the “sandwich artist effect.” They’re not making sandwiches — they’re making cool things — but they like calling themselves artists. There is a lot of “art equals pretty” in the data visualization world, which I find pretty hard to take, but maybe I will get around that a little bit. Although that is sort of the usual thing, “It’s pretty so I’ll call it art.”
KARINA VAHITOVA: I completely agree with you. I’m a communication design major at Parsons right now and I spent the entire last semester arguing with my professor about art and data, how I can be both an artist and designer at the same time, and about how to bridge the two.
JER THORP: I was having this discussion with Paola Antonelli three weeks ago about exactly this — design versus art. Both of us agreed that we should make the work and let other people decide. We’re going to put a work in a design show — so we’re going to be thinking of this object in the context of design — but we don’t really care whether it is or isn’t design or is or isn’t art. We’re just going to think about it in that context. Because if I put it in a box, as soon as we take it out of the box, it can be used in another context. It’s not actually about the object but rather the frame in which you place the object. That’s the way I think about the work that we do — we make it and other people sort it.
SIENA ORISTAGLIO: So, the intention for it to be “art” isn’t necessarily there.
Lots of people can solve your data problems in really conservative, safe ways but that’s not what we’re interested in.
— Jer Thorp
JER THORP: Well, I think most of the work we do exists in what I consider to be a continuous art practice. But some of it is more obviously such and some of it isn’t. I don’t really care if people see the stuff on the far edge and look at it and say, “That’s not art.” The way that I’m thinking about it is as a piece of a bigger puzzle. This thing on its own may not be, but this thing combined with this list might be. Sometimes, we overthink to such a level that it can prevent us from doing really good work.
EVALINA PATIÑO: There is an art historian that said that art doesn’t exist, only artists exist. It’s a compelling idea.
JER THORP: Yeah, we think within a very similar argument. You’re not making art — it’s about the framework. I’ve always thought that it’s in the eye of the beholder. That object over there, that Mac, was my first computer. Not that individual one, but that model was my first computer. That object carries with it this incredible emotional weight and story to me, and to other people it’s just a weird old thing. That’s okay — that’s how the world works.
DOLAN MORGAN: And also, there are utilitarian objects in museums. You go to the Met or the Smithsonian, for example.
JER THORP: MoMA has a huge collection of all of that in the Architecture and Design Collection.
KARINA VAHITOVA: It’s funny to think that a thousand years from now, they’ll be looking at this SLR and saying, “Wow, that’s what they used to photograph things?”
JER THORP: That’s another really interesting discussion. How do you choose? I’ve had pieces of that discussion with Paola. It’s like, how do you determine not only which SLR, but which collection? Which model? From what year? Is it the first one off the production line? Is it something a famous photographer used? Is it a random one that’s still in its box? The curatorial process is so much weirder when it involves everyday objects.
SIENA ORISTAGLIO: “What is the archetype for a particular utilitarian object?”
JER THORP: Now they’re collecting software and typefaces, the problem becomes even harder. What do you have left? … We can see if we should give up on Ben — he’s probably lost in SoHo. Let me just make sure he hasn’t already come in. [He leaves and returns.] At what point should I call the search party? [Laughs.]
SIENA ORISTAGLIO: I think what you said about context is really interesting. We talk a lot about that in terms of performance. Something that Marina Abramovic often says is that if a baker bakes in a bakery, he’s a baker, but if he goes into a gallery and bakes bread, then the work is an artwork. The context and the intention are two factors we consider in answering the question, "What is performance art?”
JER THORP: You should meet my girlfriend. My girlfriend was in art school and a sculptor, and started making these gigantic cakes. And then she realized she really liked making cakes and so then she left art school and became a baker. Now she’s a pastry chef. She liked the making of these things that people consume and it was so intriguing to her that she left her art career and became a baker.
It’s not actually about the object, but rather the frame in which you place the object.
— Jer Thorp
SIENA ORISTAGLIO: That's great. So, we’re also really interested in scientists who focus on duration and time perception — like Michel Siffre’s biological clock experiments in 1962 and 1972. We have begun to see overlaps in the characteristics of performers who create long durational works and scientists who experiment with duration — specifically, in the rigor and patience that it requires. In this vein, we research long durational natural phenomena, which is why we'll be collaborating with Rachel Sussman, who put us in contact with you. We’re interested in archiving these phenomena and also partnering with conservation efforts. [Shows an image.] Crystal formation, for example, is an endless natural process.
JER THORP: I think I know that guy. [Points to a man in the image.] He’s a photographer for National Geographic.
SIENA ORISTAGLIO: Yes, you worked for National Geographic, right?
JER THORP: I’m a National Geographic Emerging Explorer.
SIENA ORISTAGLIO: Explorer! What does that mean exactly?
JER THORP: They give this award out to like ten people every year and then we get support from National Geographic and we get to go there each year and hang out with other people who do much more exciting things. I say, “What did you do this year?” They say, “Oh, I discovered four new species and lost an arm on the top of a mountain. What have you been doing lately?” And I’m like, “I did some programming. It was good though, it was hard.” [Laughs.]
SIENA ORISTAGLIO: Speaking of difficult projects, with many artists that we approach, we say, “Okay, we love your work. What would you do in a six-hour format?” From an audience perspective, Jeanette Winterson talks about “art at a trot.” People walk through galleries and don’t pay attention to what’s there. They take a photo and Instagram it and then leave. The idea is that with long durational works of performance, you have to be there live and you have to commit the time for that.
KARINA VAHITOVA: There’s a study done by the Metropolitan Museum of Art that said that the average amount of time someone spends in front of a painting is seventeen seconds.
SIENA ORISTAGLIO: Yes, there have been a few of studies like that. I’d actually be curious about the data there.
JER THORP: I don’t pay much attention to that study, to be frank, because then we would expect a museum-goer to stand in front of every painting for a long period of time? What I want to see is the maximum time that a visitor spent. I don’t even care if they barely stand in front of most of the paintings, but if they stand in front of one for nine minutes, win. The problem is that the people who created the studies say, “Okay, well they walked by like seven hundred artworks. They stood in front of that one for like nine minutes, so that divided by seven hundred..." They're going to get this small number. We’ve been doing all this work for MoMA so that’s why that figure keeps coming up. It’s such a bad example. I don’t think what you want is a museum experience where everyone parks themselves in front of every work for a prescribed amount of time and then moves on. The way that it's always been is that you walk into a room and you’re like, “Eh. Eh. Eh. WOW!” And your “Wow!” is maybe not somebody else’s “Wow!” and that’s great. That study is kind of a misuse of statistics. The average is not what we want. The maximum would be far more interesting.
SIENA ORISTAGLIO: That makes sense, yes. We just featured a long durational work by performance artist Jonathan Van Dyke, who stood in front of Jackson Pollock’s Convergence (1952) for 40 hours. He stared at the painting for five days, eight hours a day, taking only one 20-minute bathroom break daily. So, we’d be looking at a maximum, there.
JER THORP: Yes. He would have skewed the dataset.
SIENA ORISTAGLIO: The other thing is that we’re dealing with performance, so it’s different. It’s not like a gallery where you’re selecting from various objects. With performance, there’s typically one work in a space that lasts a certain length of time. There are many ways that artists who create these works engage their audiences. For instance, with the Christian Marclay piece The Clock (2010), the audience can come and go. For some pieces, the public really has to stay to experience a narrative structure. We recently collaborated with video game designer, Pippin Barr, who we found because he created this 8-bit video game version of “The Artist Is Present” (2010). It’s very literal — you can’t play the game on Tuesdays because the MoMA’s closed, you have to advance your character forward or you get kicked out of line, and it can take five hours to sit with a tiny Marina.
JER THORP: This reminds me of a durational video game that people are obsessed with, called Truck Driver. You have to drive for as long as you can and if you take your attention off of it, you drift off the road.
DOLAN MORGAN: You get one point if you get all the way there, right?
JER THORP: Yeah. People are obsessed with it. That’s all it is. It’s not entertaining in any way. It’s nighttime and you’re on an open road and you just have to hold the wheel. If you cross the line at all at any point, you have to start over. And do you know the guy who is trying to get to the edge of Minecraft? This is where I get excited about all of these things that don’t have art as an intention and are amazing. In theory, Minecraft is an infinite terrain, but the way that they use the algorithm, it falls apart near the edges. This guy heard about this and decided that he would start walking towards the edges. When he started, he made a calculation and thought it would take two months but as it turns out, it’s going to take him like eight years. So he has this YouTube channel where he records himself walking and he just talks. He has something like five million followers and all he’s doing is walking through the game and talking. He does it every night for like five hours. It’s going to take him a decade.
DOLAN MORGAN: Minecraft is complicated enough that you can’t just leave a book on the keyboard.
JER THORP: Yeah, and you can’t pause the game. What you have to do is build yourself a structure, put yourself in there, and close the door so that the zombies don’t eat you overnight. Then you can wake yourself up, undo it, and keep on going. But the game never stops.
KARINA VAHITOVA: What happens when he gets to the edge?
JER THORP: Well, it still never stops, but in theory, it should get weirder and weirder. Like, mountains in the sky and upside-down trees. In theory. Every child between the age of 10-15 in the world plays this game. We have no idea. There’s another guy in Minecraft who’s been making these clocks. There are logical circuits you can build and he’s made a clock that will not finish for a million years. He builds them all by hand. In the game, you basically have to place all of these blocks to make all of these things. They work in orders of ten, so the first clock will work in an hour, and the next will be ten hours, and then a hundred hours, and then a thousand hours. So he couples them.
SIENA ORISTAGLIO: So, Pippin created a digital version of the Institute. He also did these games that are kind of like that, where there are three million grains of rice and sesame and you have to separate them one by one. At the end, it just says “exercise complete.” You don’t win anything.
JER THORP: Except nobody will ever finish it.
SIENA ORISTAGLIO: Correct. So, this is one way that we engage collaborators. We also like to pair people who we think might work well together, and we’re really interested in promoting and featuring works of people whose works are long durational.
JER THORP: I think that there are a lot of links to projects we’ve worked on. But the most obvious one is a collaboration we did with Elevator Repair Service (ERS). Do you know of them?
SIENA ORISTAGLIO: Yes, the theater and performance company.
JER THORP: We work a lot with them. “Gatz” (2006) is one of their most well-known works.
SIENA ORISTAGLIO: Yes, the eight-hour interpretation of The Great Gatsby. We featured that work and we’re going to be in touch with them, actually, because we’re building an archive.
JER THORP: Part of what we’re doing at MoMA is a performance. It’s not, in the scope that it sits right now, a long durational work because we’ve just been experimenting. One of the things we’re trying to build is a conceptual API [Application Programming Interface] that will allow people to interact with the archive in interesting and unexpected ways.
[Ben arrives. Introductions.]
JER THORP: I just got the low-down on the Institute from them, which is really amazing. The short summary is that they're focusing on two things, long durational artwork — meaning anything more than six hours long — and collaborations between artists and scientists, so right in our power alley.
Sometimes the performers couldn't even hear each other, but a person who was in between might hear everything. What we were working with were dynamics of order and disorder...
— Ben Rubin
SIENA ORISTAGLIO: “Power alley.” That's a good term.
JER THORP: I was just talking about the MoMA project and the series of performances we're going to do in the gallery. We don't really know how they're going to appear, but the core is a conceptual API where people can interact with the collection of 120,000 objects in ways that are unexpected. One of the sketches that we made and proposed to them originally was a kind of interface where you would have to look at an artwork for longer than a certain amount of time before you could find out information about it that isn't usually available to the public. So, here's the artwork and here is the little info card. [Gestures.] If you watch it for long enough, it flips open and there's some more information. If you watch it for even longer, then something else comes down. So you have access to things you wouldn't otherwise have access to, but only through this dedication to the art. Which is kind of the intended experience, right?
KARINA VAHITOVA: The more you look, the more you see.
JER THORP: Yes. Ben has a long history of building works that are long durational to an extent. Some of them obviously so, and some of them less obviously — like the piece San Jose Semaphore (2006 - Ongoing) that Ben installed in San Jose, which is like a performance, almost.
BEN RUBIN: The way I understand long durational work is that it is time-based. It unspools over time.
BEN RUBIN: Well, in our work Elevator Repair Service, certainly — and in the pieces that we do that have no time bounds.
JER THORP: Yes, where there's not a start and an end.
BEN RUBIN: The model for what we're planning to do at MoMA is based a little bit on this piece called “Shuffle” (2011) that we worked on with ERS.
SIENA ORISTAGLIO: In the New York Public Library?
BEN RUBIN: Yeah, exactly. It's infinite, in a way. It’s somewhere between an installation and a performance. The audience can circulate as the performance goes on.
SIENA ORISTAGLIO: Can you describe that piece?
BEN RUBIN: So ERS had done “Gatz,” they had done “The Sound and the Fury,” and they had done “The Select,” which is based off The Sun also Rises. We decided to take those three pieces and treat them as a database of text and as a performance. And since all the performers we were working with had worked on at least two — if not all three — of those pieces, we found that we could cut those pieces up into small fragments of language and reorganize them without necessarily a regard for which source it came from. We mixed them up based on rhythm or sound or other — not necessarily semantic — but other aspects of language, and then came up with these rhythmic or poetic or associative sequences of lines. We would generate these scripts live, so the performers would be fed these scripts live on some sort of device.
Some actors had teleprompters and others had small screens in their hand and their script would be scrolling by. Each performer had an individual script and it directed them where to go and what to say when. The scripts were all synchronized to each other, so they could create this beautiful choreography where people would be in a group in one part of the library and then sort of break up, and another group would reassemble over here [gestures], and then there were these echoes going back and forth in the space. Sometimes the performers couldn't even hear each other, but a person who was in between might hear everything. What we were working with were dynamics of order and disorder and lots of people talking at once, and then quieting down to a single voice. A lot of motion or stillness.
SIENA ORISTAGLIO: And for how long did the performance take place?
BEN RUBIN: We did it in twenty, twenty-five minute cycles. They would all start in exactly the same way and then in about twenty seconds it would start to diverge. We'd do it for four hours and then take a break and then do it for another four hours.
A lot of our work is, by its nature, durational. In many ways, there is no end.
— Jer Thorp
JER THORP: Last year, we did a piece at the Vancouver Art Gallery called Grand Hotel (2013). It was about the changing cultural roles of hotels over the past 150 years. We built a piece that used a massive database of hotels — 6,000 hotels from all around the world. We also built a system that could get relatively up-to-date ratings and other information like room availability. This is a data set that is so vast that it's hard to think about how to engage with it in any interesting way. All of our basic tools of data visualization kind of fall apart. We can draw a map or we can draw a chart but that feels like you're only getting a tiny little slice of something that is gigantic and human. So we built a system that took characters from famous novels and plotted out their trips.
EVALINA PATIÑO: I saw the Lolita piece.
JER THORP: Right. And then we put those characters in the world, in real-time, and they have to stop at hotels. So the characters from Lolita were traveling on the Eastern seaboard and they had to find a hotel, and they had no money. So, depending on the availability and what day of the week it was, they might find one hotel or another. We'd show all the pictures, and there are cockroaches, and there are complaints. And then we might also have another character — we'd have temporally appropriate ones, but also not — like Ulysses, where he'd have lots of money, in the Greek islands, staying in these big palaces. The piece started at the opening of the exhibition and then ran for four months. I think that's what Ben was touching on. A lot of our work is, by its nature, durational. In many ways, there is no end.
The Shakespeare Machine (2012) at the Public Theatre is another example. It's a sculpture that's living in the theatre, but also it's a sculpture that's exploring, trying to find patterns, and will never repeat, but will play on. It will probably be there for 50 years. In some ways, it's a permanent piece but you don't understand it as such because you read it as sculpture. And even though sculpture is by default durational, we don't think of it as performative, whereas I think these works that Ben and I make are performative in their nature. They're doing something — they're literally performing. They're exercising something, whether it's an algorithm or a pattern or something likething.
BEN RUBIN: Before we really started work on The New York Times piece for the lobby, I got to meet with Renzo Piano and talked to him about what he imagined that piece could be. His vision for it — he didn't have any visual, specific shape or anything — but he said he wanted something that could live in the lobby, like an organism that could inhabit the lobby and wake up in the morning and have certain behaviors and be like a lobby creature. He drew this bug — I still have the drawing somewhere. That's a nice way of thinking about these pieces. Almost like organisms that have certain metabolisms.
SIENA ORISTAGLIO: I think an insect is a good metaphor because there is a “consciousness” but also not really a consciousness there.
KARINA VAHITOVA: There's only so many things that they do, but they do them constantly.
BEN RUBIN: To circle back to the “Shuffle” performance with ERS, we're imagining for the MoMA piece at the end of the year to retake that performance strategy but with the MoMA database for the underlying text — bringing it up into the galleries. Meaning that we can work with the original artifacts to which the text refers, the titles of the works, in the performance in some way and occupy the space acoustically and physically. It's really exciting to imagine those games of choreography.
JER THORP: But also, rehearsing in the galleries is what we want to do, putting that into a public space so the piece lives and is formed there and is performed there.
BEN RUBIN: To some degree. Obviously not all the rehearsals, but some portion of rehearsing during museum hours with the public and then the ultimate performances as well.
That's a nice way of thinking about these pieces. Almost like organisms that have certain metabolisms.
— Ben Rubin
SIENA ORISTAGLIO: A transparency of process or creating opportunities for engagement in the early stages of the work? That's interesting.
BEN RUBIN: That's another thing we did at The New York Times, which sort of naturally emerged. We got access to the building shortly before they brought people in to occupy it. We got in there and there weren't floors but we put tables in the lobby and we just stayed there. They put the floors in and they opened the doors and they started to move people in floor by floor. Mark Hansen and I were at this little table for five months from seven in the morning until seven at night, programming and testing, so we got this firsthand sense of the space we were making the piece for.
SIENA ORISTAGLIO: You became the organism that lives in the lobby. You were the lobby creature! When you're talking about the MoMA database, are you talking about the current pieces on display or past works?
JER THORP: The collection.
SIENA ORISTAGLIO: You were speaking about having an interaction with the objects currently on display, but this is a different way of engaging with the archives?
JER THORP: This is the bottom of the iceberg. You see this little piece of it [gestures to top portion of a triangle] when you're in a museum and I think you need to be careful not to think it's about the object but about the record of the objects, which is not the same thing. It's different, in a good way. Every time a work has been in an exhibition, every time it's been repaired, when it's been loaned — if that history exists at all, we never see it in a museum. We interact with that history in a very specific way when we look at a painting but there is a subtler piece of the painting that is there that viewers never see. This is our way of engaging with that. Our goal is to bring this idea to MoMA, where they have a whole other substrate for artists to interact with their collection, not only as physical objects but with this data structure. It's been really interesting to see how they react to this, especially with respect to the performance. When they said, "We're going to bring in these data artists,” I'm sure they didn't expect us to do a performance. But that was one of the main reasons we chose to do it.
DOLAN MORGAN: I like the “record” aspect too because it almost reveals the long durational nature of things that we normally look at as static objects — but they have these huge tails, these lines behind them.
BEN RUBIN: There's a social network among the artworks in a way that you can imagine. You can take a work that's hanging in the gallery not necessary touch on any of the data directly associated with that work, but rather, “What are all the works that have hung in the same exhibition as this work? What works have hung next to it in the past? Does it share any lineage of ownership, was it collected at the same time as certain other things?” to sort of pull together and create this world that surrounds any particular artwork.
SIENA ORISTAGLIO: Not to get too theoretical, but Walter Benjamin calls the history that is attached to an art object the “aura” of the work. There must be a way to bring that to life and display it.
JER THORP: In engagements with the API we're going to build, we have ideas on how to address that. For instance, queries against the database are almost always done by one person but what it would it mean if they were required to query with two people? Every query would have to be cooperative. One person would ask for one thing and another person would have to ask for another and then those results would be returned. You could never selfishly query the database — you could only do it in pairing with somebody else and, as a result, at any given time, you'd get what you asked for but you'd also get some other stuff. Forced serendipity is really important. You get things that you might not have asked for before. Those are the types of mechanisms we're really interested in. One of the nice things about the idea of building an API is that we're inserting an artwork into the functional workings of the museum. That's a way to allow a piece to be longer-lived. I have this dream that they'll kind of just forget about it — that it'll be there, but then they’ll be like, "Wait… Oh yeah." And it will keep on running and people will still query it, and it'll be this thing that will keep on going in a way that a piece on a wall can't because people would be like, "What's that thing on the wall? It's been there for a long time."
BEN RUBIN: Getting into the bloodstream of the institution.
JER THORP: And then the tissue forms around it and suddenly it's, "Was that always there?"
DOLAN MORGAN: I really like the idea of getting the viewer to be engaged. There's a real sense of play in what you do with bodies of information by turning them into visual representations but there isn't necessarily a sense of play in how I experience the thing that you have an open-ended excitement about making. Inviting people to that level of interaction, to be able to poke around to see what does what — that’s exciting. Do you have any other projects like that that you've been working on or are moving toward?
JER THORP: For people who work with computers, all three of us are not typically interactive people. That's not to say there aren't interactions with the work that we do. I always feel super uncomfortable with interactive artwork because it's really facile. I don't know if there are many examples of art that you interact with in the visual realm where that interaction is really meaningful, other than, "Look, when I wave my arms it does something." We're kind of over that. Maybe in the beginning we were like, "OH MY GOD THE COMPUTER CAN SEE ME!" But now it's like, “Oh, whatever.”
SIENA ORISTAGLIO: Well, I don't know, the Rain Room (2012) at the MoMA was very popular.
JER THORP: That was a very well-made example of what I'm talking about. On the very edges, those pieces can really work, but I think that type of interaction can be very restrictive. Whereas the work that we tend to do together, and also separately, is really open-ended. I don't think we like to tell people how they should experience the work, so the interactivity comes from the unique experience you get from the piece. This is due to the way these projects are constructed algorithmically but also because of the narrative in our work. It's a very postmodern narrative. You don't get a "beginning-middle-end" — it's a bunch of pieces that people can pull and take and make a story from, which, to me, is far more rewarding. This is why the whole API idea excited me. It allows for a more direct “interaction” while still leaving the doors open to a gigantic range of experience. One of most exciting things to me is that these projects are designed to be built upon. Artworks built upon artworks. In some ways, it’s a work that is meant to be consumed by machines. The output could be centered on humans or it could never reach humans. It could be recycled amongst machines until some alien output comes at the end.
SIENA ORISTAGLIO: Do you feel that working with performers changes your process?
JER THORP: It definitely does.
BEN RUBIN: Oh yeah.
JER THORP: The way that I think about it is that, say there’s a “thing” in the world and that we have data that is a record of that “thing.” Then I produce a kind of representation of that “thing” using the data. Two phase shifts have happened. Performance is really interesting because it introduces a third. We produce these scripts and then the performers interpret them. Even though, on paper, it kind of brings you further from the original “thing,” there’s something magic that happens where I think you can actually reach back and touch it. You know, because it’s humans and because it’s spoken. These are forms that are so much more closely connected to our brain that then watching something on screen.
It's a work that is meant to be consumed by machines. The output could be centered on humans or it could never reach humans. It could be recycled amongst machines until some alien output comes at the end.
— Jer Thorp
BEN RUBIN: We generate these sorts of abstractions in language. The performers really have to work hard to find a place to hold on to. That’s what’s amazing — they’ll just come up with some kind of a situation or a game, or a trick, or a dynamic, or something that makes it work for them, and suddenly they are invested in it in a way that it would have been impossible to ever be invested in just a sort of visual output.
SIENA ORISTAGLIO: Do you give the performers a crash course in data visualization or do you just hand them the script and then let them connect with it in their own way?
JER THORP: The reason why ERS work so well is because that’s what they do. They work with source material that is not traditionally considered to be performative. We would have had a really hard time doing that MoMA piece with people who are not ERS. They could have done it, but it would be a lot of what you’re describing, whereas ERS just say, "Eh! Okay! Here we go! We’re just performing this thing!”
BEN RUBIN: They don’t know if it’s gonna work or not.
SIENA ORISTAGLIO: That attitude, that mentality is exciting in a collaboration.
BEN RUBIN: For me, you know, I moved to New York in 1993 and it wasn’t what I thought I would be doing. I went to grad school at MIT in film but after grad school, I was working with Steve Reich on a video for a performance that he and Beryl Korot were doing, and that’s what kind of brought me to New York and brought me into the performance realm. I did a lot of that in the 90’s and then with these public artworks, it kind of got onto a different track — out of performance, to some extent. With ERS, it’s really fantastic.
JER THORP: I come from the other side of the whole art and science thing. I studied biology. I had a huge amount of interaction with it. In five weeks, I am going on a 15-day expedition in the Okavango Delta. It’s in the middle of Botswana with an ornithologist who does all these transects of the delta and goes to all these places that nobody wants to go to because it’s too crazy and dangerous and there are too many crocodiles and hippopotamuses and it’s hard to navigate and it’s really hard to get to. The only way you can get there is with these flat bottom canoes.
SIENA ORISTAGLIO: Wow. What are you doing there?
JER THORP: We’re using the data from that expedition. First of all, we will broadcast it live so that anybody can access it and make artwork with it or do whatever they want with it. Also, during the expedition, I will be making these little things. But it’s mostly an exercise in making the data more human.
SIENA ORISTAGLIO: What is the data like?
JER THORP: It’s species counts and then the heart rates of all the expedition members, temperature, environment. We’re also deploying a network of 30-50 remote sensors this time, so they will be reporting back data. We have a guy on the expedition who is a satellite engineer, so he’s involved with remote UAVs [Unmanned Aerial Vehicles]. We’re gonna use drones. Data is the product of science — it's what science does. As an artist, I'm really interested in what happens when we let that out. Usually, scientists keep it close to themselves because they want to publish a paper on it and then, when it's published, no one cares anymore. In this case, as we make these species counts, they're live. We're going to be working with school kids who can help us do the science but also create artworks based on the photographs of the sand that we're recording. That's a big project that's close to my heart — trying to understand where the boundary between science and art becomes less obvious. That's what I'm really interested in.
Last month, I was the Artist in Residence on an oceanographic expedition with a scientist named Cindy Lee Van Dover. She's brings artists along on her contracts. She writes it into her NSF grant — it’s so amazing. We went to the bottom of the ocean in a submarine. She's very interested in art and science collaborations. I would suggest there would be a great opportunity to collaborate because the deep ocean is very much about long durations. These are the oldest ecosystems in the world, where life began.
KARINA VAHITOVA: You are definitely an explorer. You sold yourself very short earlier.
SIENA ORISTAGLIO: Some research we've done says a lot of the oldest grasses are underwater.
JER THORP: And that stuff is nothing compared to the really deep stuff. We don't know anything about it.
KARINA VAHITOVA: Yes. Most of it isn't explored because it's impossible to get to certain depths.
JER THORP: We've actually been talking to Cindy over the last 12 days and we were having this conversation about doing an artists' cruise. She's done scientists’ cruises with artists but we want to do an artists’ cruise with scientists. There are usually 30 people on board, and it's usually 28 scientists and two artists. What would it be like if it were 24 artists and six scientists? Enough scientists to make the dialogue relevant but really be there to have a chance to go and make art. I would be really interested in — this could take a lot of work and budget — just leaving something there. Scientists leave things there all the time, probes, more and more, for data collection. They leave them and then they pick them up later and they are these lonely objects that sort of do their thing at the bottom. They wake themselves up and then close themselves down and then wake themselves up. To make an artwork that would do that as well — in a place that is without audience — would be so beautiful. Cindy's whole message is that in order for the deep ocean to become relevant to us, we need to make it relevant to us. It's part of the largest biomass on the planet, where life began. It's also an incredibly rich source of minerals and petrochemicals so there's no question that it's the next frontier for resource extraction. We've got to get there beforehand so that people give a shit about it in our lifetime.
That's a big project that's close to my heart — trying to understand where the boundary between science and art becomes less obvious.
— Jer Thorp
SIENA ORISTAGLIO: That's the beauty of what art can do in the realm of science. Many of the scientists we've talked to are interested in collaborating with the arts because they feel that it has that element to it — especially if you find an artist that is really good at communicating science. Another thing that we talk a lot about is long durational collaboration. This is the idea that if you just throw an artist and a scientist into the same room for an hour, you likely end up with very little content or meaning from that meeting. But, as in this case, if you put them on a boat...
JER THORP: For four weeks in the middle of the ocean! They’re usually four-week cruises. And you cannot get away from each other. I was on the boat for seven days and it was great. You can have great conversations that last for four weeks. I've been thinking about what you've been doing because you eat three meals a day with each other in this thing, you're in a lab together all day, you have nothing to do on the boat at night other than working or watching a movie together, and then you're in a seven-foot titanium sphere going to the bottom of the ocean. That's a durational experience.
DOLAN MORGAN: I have a question I’d like to ask you about empathy. You’d mentioned the idea of making data human, making it something we can access and feel something about. I’m wondering which set of data you’ve worked with that you feel you most successfully squeezed some empathy out of. I’m thinking along the lines of something where you wouldn’t naturally think empathy would occur. Can you think of a time when you’ve been given a certain set of data and you felt like you really managed to get people to feel things about it?
BEN RUBIN: I mean, for me, it was the original piece I worked with Mark on, Listening Post. That started as a project to do experiments in data sonification. That’s what we undertook, that’s what we got a grant to do. Initially, this was in 1999, we were looking for, “Well, what kind of data can we care about enough?” We had weather data and stock data and web traffic data and it was like, “Oh this is interesting, we can make cool sounds and whatever but who cares? We don’t care about this data.” Then it was Mark who had the idea that chat — the original social media before there was social media — might be data that we could care about. It turned out, in fact, to be. It’s all these people reaching out across space. I was a CB radio kid, in my bedroom late at night just listening to the truckers pass by. It’s the same — you know, this basic human impulse to be like, “Hi, is there anyone out there?” Whether that was in sports chats or sex chats or day-trader chats or whatever, you still got that feeling. So to me, that’s for sure the most directly human data. In a way, it wasn’t even about transforming the data as much as it was about figuring out ways of distilling it and choreographing it to make it tell some kind of story.
SIENA ORISTAGLIO: Storytelling is an interesting part of this because that’s where the human element really comes in.
JER THORP: I had this discussion with someone who was interviewing me for a book he’s writing. He was very rigorous and he has a very strict idea of what a story can be. My idea of a story is completely different. Story doesn’t exist. Story’s not a thing. It’s a phenomenon that occurs in our brains. You’re never making a story — you’re making something that somebody could read a story from. For instance, Mark and I did a piece around the eBay database about two years ago.
BEN RUBIN: Right, that’s a great example.
JER THORP: The eBay database is the most unbelievably human, beautiful, ridiculously amazing dataset. You have all these people selling these items that are a piece of their lives. Mark and I would just sit there and be like, “Oh my god.” Every little corner in the database was an epic treasure trove of beauty and tragedy and poetics. When I said to people that we were working on the eBay database, they’d be like, [cringes] and I would say, “WHAT?” If you can’t think of something to do with this astounding collection of human experience and exchange and these objects, then something is broken. This was one of the first times that I realized that our societal understanding of what data is is really broken. When we were doing that project, we would be interviewed by journalists and they would say, “How do you go about making art from something as dry as the eBay database?” I would say, “WHAT? WHAT?” Maybe we have made artwork from some data that is more dry but I think that we always work with things that are human and beautiful. It’s so shocking that people would not even spend ten seconds to just think about it and say, “Oh okay, there’s something there.” So, the challenge is not that at all! It’s, “How do you deal with this unbelievable tornado of human experience and story and humanity and narrative and beauty and horror and tragedy and make it into something that you can think about and touch?” Our job was the opposite of what people thought it was. People thought it was to manufacture humanity from something that wasn’t human but in fact, it was to take this unbelievably human, ridiculously-scaled thing and bring it into something that people could even think about.
BEN RUBIN: Yeah, it’s a scale issue more than a humanizing one. Bringing it into human scale so that it gets its humanity back.
My idea of a story is completely different. Story doesn’t exist. Story’s not a thing. It’s a phenomenon that occurs in our brains. You’re never making a story — you’re making something that somebody could read a story from.
— Jer Thorp
JER THORP: The way I think about it is that these databases are these gigantic surfaces and we just always look at it from there. [Holds up a piece of paper and looks at it from the thin side.] That doesn’t look that interesting but if you flip it [flips the paper to the wide side], it’s like, “WHAT?” The scale of this piece of paper that we turn over, in the case of eBay, is so gigantic. What do you do then? It’s overwhelming and crazy.
BEN RUBIN: We look at it like this. [Holds a piece of paper closely up to his face.] We’re too close to see. So the key is to get it back to here. [Holds the paper at mid-arms length from his face.]
SIENA ORISTAGLIO: Why do you think the perception of data is that it’s dry? Is it because of how conservative the field is typically? Are there other people who see it the way that you guys do?
BEN RUBIN: What it means to see data usually is to see it either aggregated or graphed or in some way summarized and reduced to a moving average or a line or a sequence of numbers — a characterization of the mass in which we completely lose sight of what the individual elements are. We try to always make sure that the individual elements that make up the data are visible and perceptible or audible or given a good representation. In normal mass media practice and academic practice, it's very rare that that happens. It’s always aggregated and summarized.
JER THORP: It’s a cultural phenomenon of data being the product of computers and computers being a kind of fringe element. It’s only now — maybe over the last few years — that generations have existed for whom computers are just an integral part of life. I also think that we don’t even understand the impact that the computer age is going to have on humanity yet. The thing I always think about is radio. If you ask people about the impact of radio, even now, people will be like, “Marconi radio, radio stations, music, blah, blah, blah.” Really, it could be argued that this [holds up his cell phone] is by far the most dramatic impact of radio. This is a radio device and it’s changing the world monumentally and yet, for the first hundred years of radio, we didn’t know that yet. Our idea of radio was this broadcast technology that allowed people to listen. Then, suddenly, someone invented the cell phone, which took twenty years to get going, and now it’s changed. When the story is written a hundred years from now about the impact that radio had on culture, it will probably talk about the cell phone more than it will talk about Marconi and radio stations. We’re likely in the same place with computers.
SIENA ORISTAGLIO: Building the foundation.
JER THORP: Yeah, and then suddenly something else is going to happen and we’re going to go, “Oh.” Data is the same way. We’re in that phase with data right now. We’ve had data for forever and computerized data for 60 years, but for a long time it was just this thing. I think we’re at this point where that’s changing and becoming bigger and more culturally impactful. That’s what happens when technology starts to bleed into culture. That’s when real change happens. Like when the internet went from something that nerds used to bleeding out into culture and when when radio became something we carried in our pockets. That’s where we’re getting to with data. For me, the real crux of what we do is to try to be a voice in that transition — to speak about things that people aren’t speaking about and to try to find possibility where maybe other people aren’t seeing it.
How do you deal with this unbelievable tornado of human experience and story and humanity and narrative and beauty and horror and tragedy and make it into something that you can think about and touch?
— Jer Thorp
BEN RUBIN: I think in this studio, we take it for granted that data means huge collections of human information of all kinds, with all kinds of semantic depth and many modes of representation. I think a more mainstream definition of data that people carry around with them is that it is numerical, that it’s just something you can put in a spreadsheet and make formulas for. I think, generally, the data we work with would be hard to work with in a spreadsheet.
SIENA ORISTAGLIO: Do you feel that there are "nefarious" forces in data that you you’re resisting, in a sense, by using data in positive humanitarian ways?
JER THORP: I would be careful about using the word nefarious because, unlike in comic books, villains never think they’re villains, right? But I think there are bad actors and there are varying definitions of those bad actors. There’s the NSA. which we can all agree have, in various forms, been very bad actors. There are large companies who are applying these technologies in ways that I think are not being considerate of the human systems that they’re impacting, and there are people and organizations whose focus is on profit. Data is the new power. The more data that you have, the more powerful you are and currently, the largest holders of data are either large governments or these incredibly large corporations who are under almost no oversight.
BEN RUBIN: It’s always been a little hard to put a finger on a great example of really bad data practice.
JER THORP: Until the NSA came in!
BEN RUBIN: Well, the NSA. But in a way, the NSA just collects data and what they do with it we don’t know... Also, Natasha Singer wrote a piece for the Times about Big Data and healthcare. As she was researching the article, she found companies who are bragging about their abilities when it comes to identifying high-profit patients and about being able to help hospitals and insurance companies steer them in certain ways to the most profitable channels of treatment. When you think about the implications of that — it’s so bad. If you really want an example of bad practice, wow.
JER THORP: One of the most highly-valued companies in Silicon Valley right now is this company called Palantir that basically does that. It’s basically what they do.
BEN RUBIN: It’s marrying all the things that we sort of know are going on in the advertising realm with tracking and whatnot — which we’re learning about with our ad-tracking project, Floodwatch — directly to what options you’re offered as a patient in terms of your healthcare. This means that “profitable” patients who are likely to be induced by their insurance companies to pay for lots of treatments are steered in one direction and “unprofitable” patients who don’t have the resources are going to go a different direction and not have the same options and care. And that is the definition of really bad data practice, but even that, you wouldn’t say, is nefarious.
JER THORP: Redlining is the example that always gets used. Because not all the people who were doing that were like, “MUAHAHAHAH, cackle, cackle, cackle.” But the exercise was really an evil thing. It was a time when there was this kind of growth in the belief that you could engineer cities through urban planning and so, in a way, they were exercising some of these newfound powers in the same way that we now have powers that are tied to data. You would hope that we would have enough of an ethical balance now to say, “This is not a good idea,” but within the boardrooms of these health insurance companies, they’re already five or six steps past the ethical boundaries…
BEN RUBIN: But they don’t see it as evil. They’re even touting it on the front page of their website. “Helping healthcare providers to maximize the options available to patients.” That’s how it’s sort of framed but if you look at what the actual result is in society, it’s [shakes head.]
SIENA ORISTAGLIO: I’d like to see that article.
BEN RUBIN: I’ll send it to you.
JER THORP: Well we should probably get back to work, but I love the fact that our meeting was so durational. If it was short, then I would have been like, [sighs loudly].
SIENA ORISTAGLIO: Well, we’re here to interview, but we’re also here to open up the doors to collaboration of some kind and we’d love to continue the conversation.
JER THORP: Yes, we’d love to keep talking.
Over the last decade, Ben Rubin, Jer Thorp, and Mark Hansen the three principals, have each had active careers exploring the expressive possibilities of algorithms, code and data, with work realized in print and site-specific installations. Through OCR, we continue our individual work and also expand our collective practice, learning from each other as well as from new client-focused research projects.