Podcast of Julie Zhuo’s talk on How Facebook Uses Data

About Julie Zhuo

“Wow!” is the word that comes to mind when all of us here at ZURB think back on the hopping ZURBsoapbox event last Friday. We had by far the largest attendance yet. So many people showed up that we had to move the talk into support side of the office with people sitting on floor, desks, and chairs huddled around Julie Zhuo (Facebook’s Design Manager) giving us the low down on how her team uses data to inform decisions. Julie was also kind enough to hang out after the talk to answer questions and catch up with the attendees. As usual feel free to listen to the podcast below as you skim through the summary of takeaways. Enjoy!

Being data informed and not data driven

Julie mentioned that her product design team uses data to understand how people are using different features. She used photo uploading as an example. The team found that out of everyone who wanted to upload a photo:

• 87% reached the first screen with the type in your album name prompt
• 57% opened the file selector to select the photos they wanted to upload
• 52% clicked the upload button
• 48% actually waited through the progress bar to finish the upload.

Problem: Less than half of Facebook users were able to successfully upload photos.

Solution: To boost uploads Facebook switched from java/flash Facebook file selector to the browser native file selector which increased uploads by 11%. They have also streamlined the upload process by taking out the Album Creation before the uploads.

Problem: The team found that of 85% who uploaded a photo would only upload one photo.

Solution: The team thought perhaps people don’t know how to hold down shift and select multiple photos to upload. The added a tip before the upload process begin on how to select multiple photos. The metric dropped from 85% to 40% just with that tip!

Using data for a sanity check

Julie used the Status Updates Composer as the example. The team wanted to improve content sharing and turned to the composer for an answer.

They tried a number of things to improve sharing activity:

• It’s super easy to type your status update into the composer, but not very easy to share a picture or a link through it. The team added more features on the composer to make it easy to perform those actions. What was result? Not much changed.


• The team thought maybe people will update their status more if they saw their last status when they signed in. Added that change in. What was result? Not much changed.


• The team then thought perhaps it takes too much to have to click composer to update your status so they made the composer active by default. What was result? Not much changed.

Using data to evaluate project success

Facebook had a really boring deactivation page when you close your account which said – “We’re sorry you’re leaving. Tell us why Facebook was not useful.”. So one of the designers thought closing your account should be more like leaving summer camp (you know a place which has all your friends, and you don’t want to leave.)

So he created this page above for deactivation which has all your friends waving good-bye to you as you deactivate. Give you that final tug of the heart before you leave. This reduced the deactivation rate by 7%.

We could go on and on summarizing all the amazing insights (such as using data to engage users or when using data does not make sense) which followed after but we want to save some surprises for you as you listen to the podcast. We’d like to thank Julie Zhuo once again for such a PHENOMENAL soapbox!

Transcript

Julie: Some companies have a segmentation of like a visual designer, interface designer, design strategy. And for us it's really just one role. What this role is, you know what, we should just do what we think is most important and for us when we go newsfeed, we really want to make sure that the content that you see of your friends is as high as possible. And he thought that the act of deactivating should be more like leaving summer camp.

Moderator: Super excited to have Julie here. It's going to be an awesome, awesome time. You know, Julie, she's spoken in the part. She has written in the New York Times. She's designed for Facebook, I think it was 2006. Since they had 8 million people all the way up until they had 500 million people. It's crazy. So, all the features you guys have seen, all the little decision they have been making, she's been involved in it. So super excited to have her talk about how they made these decisions, how they use data at Facebook. With that, let's welcome Julie to ZURBsoapbox.

Julie: Thank you, guys. Is this thing working? Can you guys all hear me? OK. Really happy to be here. Thanks to ZURB for hosting these soapboxes. So just a quick around the room first, how many of your guys are designers in the audience? Ooh, awesome. Okay. How about developers, engineers? Okay. Couple of those. And then other tech people I guess. Everyone else. Awesome.

So, definitely I saved some time for questions at the end, but today what I wanted to talk about was how we make decisions around design at Facebook and how we try to use data to inform our designs, but not necessarily drive our designs. A little introduction about Facebook first. We believe in small teams, so we have at this point in time a team for search, a team for newsfeed, a team for the profile, a time for ads. Generally, those teams are pretty tiny. Like we have, generally, at one P.M., one designer who is responsible for the whole feature or even a vertical in some instances. We have a handful of engineers.

As much as we can we like to have everyone work together but keep sort of a tight-knit kind of community so that each team can sort of feel like it's one small company in and of itself. I am a designer. I actually manage half of the product design team and, right now, the product design team is about 18 people and the way that we think of product design at Facebook is some companies have like a segmentation of, you know, a visual designer, interface designer, design strategy and for us it's really just one role.

Traditionally, we've also tried to hire really technical designers and people who can go into the code base and, you know, write up the front end. Or at least have some familiarity with the front end area so they don't have to go an always ask an engineer to like tweak something by five pixels. So the design team is, like I said, 18 people.

We also have a separate design team for communication design and they focus on marketing and on brand and whenever we host an event like our Developer Conference at F8, they're the ones who sort of figure out what the space layout should look like and all of the messaging that we want to convey to developers, advertisers and users.

Focus of the talk is obviously data for us is really important. We use data to inform certain positions and I want to talk a little bit about what we mean by being data informed and necessarily data driven. So the first thing that we think data is hugely important for just an understanding of how users are using our products and features. And what that means is, you know, we put out something like photos, or we put out something like newsfeed and we really want to know like, okay, how are people using it? What are the paying points that they're honoring? What can we do to figure out what's wrong with this flow that we can improve?

I'll give an example. So, photo uploading, now this is, we came out with photos, I think, in 2006 and even then we sort of knew that we weren't going to have the best photo product on the market. We didn't support high resolution photos. We didn't support like all sorts of awesome things that a lot of other photo sites did, but what we really focused on was social photos and we wanted people to just whenever they went to a party or had an event to just post their photos, tag their friends and have those experiences and those memories distributed across their social graph.

So we put it out there and we basically didn't touch the photos product for about three years and it sort of grew and it grew and it grew organically. But, then last year we decided, OK you know what, this is actually hugely important. This is what a lot of people come to Facebook to do is to look at photos, upload photos and we need to invest in what's wrong. So one of the first things we did was to just try to look at the data, try to understand how people are using photos and how their uploading photos. What we found was people were having a lot of trouble uploading photos.

This is the photos dashboard. You would go click to the top corner, upload photo. Then we would give you a screen that says hey, tell us about your photo, give your album a name, give it a description. And then you fill that out and then we would say, hey, select some photos to upload. At that point, we had tried a couple of different photo browsers. The first one was, I don't know if you guys remember, a really, really janky Java blanket. Then we decided to have our own and we created our own flash plug-in that we thought was really awesome because it would open up a custom file browser, let you browse your photos, you can select a bunch at a time. And we thought that was awesome. Unfortunately, our users didn't think so. And people just had a lot of trouble using this.

So we said, OK, what is the simplest way that we can get users to upload photos and we decided to ditch our custom interface and go back to a native browser upload. Now, right now, we use what's native to Mac or Windows and when you get the file browser and you select the photo. And you can select multiple photos at a time and upload those and then once you're done you click upload photo. We show you a progress bar. And eventually, then after all of those steps, then you're done.

What we found was of all the users who tried to upload photos in our session, 87% reached the first screen, which is like type in your album information. 57% actually opened the selector to start selecting photos. Of those, 52% clicked the upload button. And then 48% of them would actually wait through the progress bar and wait for their photos to finish and so basically half, less than half, of our users are able to successfully upload their photo and that was kind of disheartening, right. And even though when we switched from our old custom flash plug-in to using the more sort of native dialogues for selecting, that sort of increased the photo upload rate from 34% to 45%, which is pretty good but still not where we wanted to be.

We found that of the 85% of users who uploaded a photo, any photo, they would actually only upload one. And that's because it's not obvious at all that you can go and select multiple photos, right. You have to remember to hold down the shift key, you have to know to select like five things they all have to be in the same folder for you to easily do so. And a lot of people just like, do what they normally do when they upload a file, just click one and they say go. And they might do this over and over again to get like seven photos on the site. It's kind of a bad experience.

What we did to address that was we thought maybe this is an education problem. Maybe if we told people that they can upload a bunch of things at a time and teach them they can use hold down select and select three things, we can help them solve this problem of like going back and forth and uploading one photo at a time. And this little change, just showing an uploading tip as you're about to upload your photo made it so that it went from 85% to 40% in terms of the users that just wanted to upload one photo, which tells us that a lot of people do want to just upload one photo at a time onto their wall. But a lot of people wanted to upload entire albums and they just didn't know. And so by understanding and really analyzing the data we were able to provide them with just a simple tip and that helped them solve their problem.

We saw photos per upload increase from 3 to around 8 or 9. So, this is sort of the graph of the numbers of uploads per day. Anybody have a guess for why there are all these peaks? Yeah, Sundays. Sundays are when people, they finished the weekend, had all sorts of parties and then they go Sunday nights and upload all their photos.

So another thing that we use data for is just to sanity check our product decisions. One example was our composer. So, our composer is a thing that you see at the top of the newsfeed that you can use to update your status or upload a photo and this is what our composer used to be. Just a little blank field and it said, what's on your mind? And we wanted a user to go ahead and click it, focus and then type something and then share that with their friends. Here's what that looks like.

One thing that we tested was that this makes it very easy for people to go and type a status message, but what if they wanted to share some other type of content? What if they wanted to share a link on the internet? Or what if they wanted to share a photo album? This makes it very difficult. In order to upload a photo album, you have to go to the photos dashboard and click that upload button. And so we thought by introducing it at the top, giving people more options, we might be able to see an increase in the different types of media that they shared.

So this is what we, we tried it. We tried asking people to upload their status, their question, add photos. Instead of just having that text box … anyone have a guess in what the reaction to this change was? A huge drop in what? Status updates decreased by 1% and photo uploads increased by 1.5%. Some changes, not huge, nothing too big. The other thing that we tried was, OK, OK well, what if we tried to get you to update more status by telling you, like showing your last status so you can see, you know, if it started to get stale that you wanted to like update it and tell us what new stuff is happening. So any guesses as to how this performed? Basically, it wasn't statistically, like it didn't do anything is what we found out.

So then we thought well what if we, we don't want to make it like you have to click something so what if we showed you the text field right there and prompted you to type it in. Any guesses as to whether, what the magnitude of this change was? Yeah, that's true, not very much. I think we saw a 2% increase in status updates and that was about it.

So what we learned was that all of these things that we're optimizing, none of them were actually significantly changing the amount of things that people were sharing. What this told us was, you know what, we should just do what we think is most important. For us, when we go to newsfeed we really want to make sure that the content that you see of your friends is the highest possible. Even though having the text field open by default did improve a little bit, we didn't think it was worth it. And all of this testing showed that like this isn't really the way to like move the needle on sharing.

So a third thing that we use data to do is to retroactively evaluate a project's success. This is actually a fun little project that one of our designer just decided to do in his spare time. We have a deactivation page and sometimes users don't want to use Facebook anymore and they go there and they say, I want to deactivate my account. And this is what we had before and we said, "OK, you know, sorry to you see you go but like, hey, please give us some information for why Facebook wasn't useful and then click the button and deactivate."

One of our designers decided to take this page and he thought that the act of deactivating should be more like leaving summer camp. Like it shouldn't be so strict and like, hey, tell us why and go. It should be more like, oh, you know this place is your family and your friends and you've had all these social good times, are you sure you want to leave? So this is what, this is a proposal that he had that was, hey, let's show you some pictures, photos you were tagged in and just bring some faces to this experience. What are the guesses for what this did? Yeah. It actually reduced deactivation by 7%. Which is pretty cool. I think we ended up shifting this version, but, I think, there's a lot more work we could do with the deactivation page. Question?

Audience Question : When you say you ended up shifting this version, was there a subset of people that you rolled the prototype out to?

Julie: Yeah, we usually A/B test a lot of these design decisions. We don't A/B test everything, but for something like this where there's like, here's what the current state is and here's a new proposal, we want to make sure that the new thing actually works.

Audience Question : And what percentage to you roll out to?

Julie: Usually, it's very small. I think for something like this, a very personal page, like it's not something that a lot of your friends are going to see, we will keep that at like less than 1% of our user base. There are instances where we launch a feature that tends to be like, you sort of need the network effects and chat's one of them. Right? Like you can't just launch chat willy-nilly to like one out of ten people. You have to launch it to networks at a time. And so when we were testing chat, we picked a small country like Chile and so we launched it to the population of Chile because we assume everyone there, most of your friends when you live in Chile are probably also going to live in Chile. And then we can sort of see the effect of how people are using chat.

One thing that we are really weary of is being overly data driven. For a number of reasons, but the first is it's really difficult for any metric. When you're looking at data, you're going to pick a couple of metrics that you think are important, but it's not really going to paint the full picture of what is actually important, you know, in terms of user experience and for what you want as a company.

Also quantitative data and a lot of the A/B testing and the metrics that we look at is really only one small piece of the equation, right? I mean, you have qualitative data, like a lot of times the numbers aren't going to tell you how people feel about something. You've got the strategic goals of the company, you've got what experience you really want to provide your users, you've got competition, you've got business interests. And like all of those together is actually what you're trying to design for and the problems that you're trying to solve when you tackle any design problem. Keeping in mind that quantitative is only a small piece of the puzzle is really important for us and something that we try to tell ourselves every single day.

A few months ago, we created a team focused on engagement and this is sort of, this was sort of a tricky team for us because, on the one hand, we wanted to sort of basically have a way to measure what was successful about the level of engagement that our users are having, but our first attempt was to basically say, okay, we want to look at reads and writes. We're going to define reads as content views, so the number of photos you looked at or status posts or videos or linked items. How much is a user consuming. Writes we defined as how much content is this user publishing.

One of the ideas that came out of this was: if we want to increase reads and writes, we should definitely come out with this feature, comment liking. Like that's sort of a huge win. So we launched this and then we made it so that if someone posts a particularly witty comment the other people can like that. Here's an example of what that looks like. But what we found was that yes, this definitely improved the metric a lot, we got a lot more writes into the system, but overall what we found was actually that like 85% of the reads and writes are generated by 20% of the users.

So what we're optimizing for by allowing people to like comments is yeah, we're giving these 20% of users more ways to engage and express themselves, which is really great, but, and even though that really boosted the metric, we're leaving out like this long tail of 80% of people who aren't really doing anything on Facebook and we're not meeting their needs. We had to basically go and really think about, okay, is this really the right metric for what we want to define engagement as and the team right now is still working on trying to figure out what could be a better metric because this isn't representing the full picture of our users and our engagement.

Audience Question : [inaudible @18:13] all that engagement information into a score for the user but based on the user's rights?

Julie: Yeah, we basically just looked at number per user I think. Number of reads, number of writes per user. The other reason why we tend to be pretty weary of data is that sometimes you get so focused on a metric that you just micro-optimize and that is not going to actually move the needle in a really big way. I think a really good example of this is the application menu.

If you guys remember, in 2007, this is what Facebook looked like. We had this left nav and that's where all your applications were and, if you had third party games or photos, this is the way that you would access it. Users were very comfortable with that paradigm. Here's what that looked like.

Then in 2008, we decided to redesign the site and we wanted to move a lot of the navigation to just a really minimal top bar and so we thought, okay, well what to do with this application menu and we thought, hey, great idea, we'll just put it at the bottom and this can sort of be, like your top level navigation … at the top and that's your user profile and your message and applications, which are secondary. Like if you play games, you want to access your photos, dashboard, let's put them at the bottom. It's still visible, it's docked at the bottom of every page and users will find it.

Well, obviously users didn't find it and we just we had a lot of, the stats show that basically nobody really clicked on it. And so we thought, geez, users aren't seeing this. It's too subtle. It's this gray bar. They're more focused on the content. How can we make users see this application click and so we had a bunch of ideas. We thought, OK, we just need to make it more noticeable. We're going to make it big and blue and we're going to highlight that a little bit more. We ran this A/B test and, yeah, this made users see it more.

I think this probably tripled the number of clicks to the application bar and the number of people who started accessing their applications. But I think what we realized is that once we actually took a step back and looked at this and saw how ugly and how basically it just doesn't fit our aesthetic, it sort of felt like we were optimizing for a local maximum. You can tweak and tweak and you can get the stats up, but really in order to make a significant impact that works with what you want to do, it's not about micro-optimizing in these small ways.

It's about actually rethinking the whole problem. And what we think of the problem is, okay, this navigation is fundamentally flawed. We tried to be more minimal by having the bar at the top and that would give more space for the rest of the content, but actually it just made it so that you have navigation here, navigation here and it's just not as cohesive.

So, in our most recent redesign, we returned back to the idea of a left nav. The one change that's different from this and the 2007 version is that left nav is actually only accessible on the home page. So, we do still have the rest of the page so other pages, like the profile, can still access the full width of the page and that was one of the reasons we wanted to move away from 2007. But this navigation structure makes a lot more sense. Users come here, nav is always on the left.

The top important stuff you can still access from anywhere, globally on the top, but this is ultimately the best solution for people not seeing the application menu. It isn't just a button. We always have this example that if you want users to click on something you make it bright red or bright green and make it blink. Yes, you're going to get the number that you're looking for, it's going to impact the metrics, but is that really what you want? Is this really the best optimization or is this the best change or is it just a small optimization?

Finally the last reason that we are so weary of data is that we think that real innovation is going to involve disruption. It isn't going to be just about tweaking what you have better and making step-wise functions. It's really about doing something, like thinking back to some really big ideas and taking some risks on those really big ideas.

I think newsfeed is a really good example. Back in 2006, this is what the Facebook homepage looked like when you logged in and what we launched was just a way for you to see all the news that was happening around you with your friends. I don't know if you guys were on Facebook then or remember the backlash, but I think 10% of our users joined the "I hate the newsfeed" group and that is one out of ten users was, I hate the this thing, it's like the worst thing ever.

Even though we were a small company at that point, we only had, I think, 10 million users. This was such a big deal that all these news outlets who otherwise never wrote any articles about us were like, hey guys, all of your users hate you. But ultimately this was a big risk and we weathered that and it ultimately turned out to be the best thing for the company. I think people are very comfortable with the idea of a newsfeed, of going in and seeing the aggregated activity from your friends.

Another example of a big risk we took was …

Audience Question : [inaudible @23:46] when it first launched you couldn't control what was on this [inaudible @23:51]. Is that something you search for in the data or you just [inaudible @23:55]

Julie: We did actually, like immediately, when the backlash started, we started looking at how we, the decision tree, was either we turn this thing off … it's a failure or we try to figure out what people didn't like about it and maybe assume that they just didn't like it because it was a change and they would get used to it. But try and address some of those user needs and some of those concerns were really about, I wish I had more control.

So we gave people, within three days we launched a preference pane where you could select to opt out of certain pieces of information being shared about you through your friends' newsfeeds and I think that contributed a lot to just helping people feel more comfortable with the newsfeed. There was just, control is very important. Beacon is another big risk we took where we thought, hey, wouldn't it be great if you did something on another site it would automatically share that to Facebook?

And again, okay, automatically share, like that's kind of risky, right? Another huge backlash. A lot of users hated it. And here we realized, okay, this is a fundamental problem, users want control, we're just going to shut the whole thing off. This is a, yeah, this is what it would look like basically. We started giving users, things wouldn't be published in the newsfeed until they explicitly said yes, I want this to be published in the newsfeed.

Finally, just all of the homepage redesigns that we've done. This is something that you're going to be able to see via data because the way that users initially use something isn't going to be how they use it over time and how they, like them and their entire social web will start to use something. We've definitely redesigned the site a lot, but each time we've been trying do a better job of at least anticipating what the biggest user concerns are and addressing that before we launch but users are always ... when you ask them do you want this change or not?

Generally, they're going to say no just because they're familiar with the old thing that you had. I think the thing that we're really trying to focus on is how to tell a better story of how this change is good for users and that's sort of forcing us to think about why did we make this change. What's the narrative around the changes? How can we really demonstrate value and if we can do a better job of that then I think that users will be more open to a lot of the changes that we're making.

This is sort of a funny group that was formed around the time of our latest newsfeed, just to demonstrate like, hey, change is hard. Every time you change you have something fundamental. You have to be really weary of how users perceive it. But ultimately we feel like the greatest risk is taking no risk and I don't know where that last word went. And that's all I have for you guys. Thank you very much for being a great audience. I'm happy to answer questions that you have.

Moderator: Let's do some questions and answers. I'd like to do just a couple from. I'll just do a couple from the web that came in and then we'll just open it up because we have tons of time for it. So Chip Connelly, who is the founder and CEO of one of the largest hotel chains in the Bay area, he says that the status update that you talked about gives a unique window into people's emotional well-being and that Facebook actually already tracks the happiness index. So he wants to know what does Facebook use to create this happiness metric? Do they see happiness as something more of a practice? Something you have to work at and then a pursuit? What means could they use to track practice versus pursuit?

Julie: To be honest, I'm not super, like it's sort of a secret sauce in the data science area, like how they come up with a lot of their analysis. My understanding of how the happiness metric is calculated is it just looks at positive sentiment and negative sentiment based on just a category of words. These words tend to reflect negative sentiment, these words tend to reflect positive sentiment so we're going to basically just run that through millions of millions of status updates and try to get a sense of where that sentiment is at.

It's definitely not, I think, a science per se. I think it's something that we find very interesting just because we want to sort of, I think a big goal of what we're trying to do is really try to understand how users want to best share information with the people that they care about by sort of understanding the sentiment and the emotions that people have around their lives or what they want to share.

That's sort of what we were more interested in. I think, in general, even though our goal is to provide the best platform and tool for allowing people to share, we don't want to prescriptive about what people should say or how people should feel. It's more trying to give them that platform to be expressive in the way that they want to be.

Moderator: And just one more from Josh Ellman, he's a Facebook platform guy, former Facebook platform guy and current project manager at Twitter. And he says at Facebook the approach is to try and replicate real world social norms by emphasizing human qualities in conversation. People's faces, real names, great biographies and places that they visit. So how different are the behaviors on mobile versus web in terms of adding comments, photos, liking. Facebook is one of the rare services that has a strong mobile web presence.

Julie: Yeah, we're starting to definitely get into mobile a lot more because we recognize that as more and more people get phones and like interact with their friends while they're on the go. It's sort of a very different environment than when they are sitting in front of their computers just like trying to read up on the newsfeed. I think that right now, the way everything about it is like, okay, Facebook itself is a mechanism for entertainment, for discovery.

Like generally you're sitting in front of the computer, sitting there for a long stretch of time or at least more than like one or two minutes and so you might be interested in reading articles. You might just want to sort of get a summary of like the day's news and that's why the first few we show you is top news. When you're on the go, we're trying to understand what people want to do and a lot of it goes back to they want to know what's happening right now. They want to know what their friends are doing right now. They want to know what's being said. They want to know what parties are going on right now, like the activities around them. And so we are trying to focus the mobile experience more around the happening now.

One change that we have is that when you go to the newsfeed it's not the same as the newsfeed on Facebook, it's not just the summary of here are all the things that happened in the last 24 hours that are interesting. It's more like here is what was said a minute ago, or five minutes ago. I think we can definitely move more towards the idea of trying to understand what people want and delivering the whole like, this is what's going on in our world around me kind of experience.

Audience Question : [inaudible @31:00]. My question is about process. How do you integrate the data with actual contact with users through either focus groups? Is there a user engagement [inaudible @31:22] going on here?

Julie: Yeah, absolutely. We have a user research team. It's actually a relatively newer team. I think we only got it up and running about two years ago and it's still a very small team. I think we have like four or five researchers. So this is a big area that we're trying to expand in. Traditionally, in the very early stages we were using research more to validate flows and to make sure that users understand how to get from point A to point B and what are some of the different pain points that they're seeing and hopefully fix before the thing goes out.

I think today we're trying to shift the focus of the research. I think that's still important. I think the bigger opportunity lies in a lot of the broad area strategic research, like right now for example, we're trying to do a lot of ethnographic studies around different demographics. I think today a bunch of people are going out and interviewing a lot of new mothers and the experience of having a new baby and how you want to share that with your family and friends is really important.

A lot of times people use Facebook to communicate weddings, engagements, the birth of family and so we really wanted to understand what do people want to do, how are they feeling, like what are the things that they want to share and express? So the research team is diving a lot more deeply into trying to understand these market segments so that we can use that to affect the strategic decisions that we're making around and about how do we do photos, or how do we do newsfeed, or how do we do profile.

Audience Question : How do you take that qualitative data that the user research team is gathering whether it's [inaudible @33:05] and marry that with the quantitative data to tell a more thorough story about what's going on? Do you guys have any kind of approaches that you can take to that?

Julie: I think at this point we probably, because we have just a lot more data, I feel like we generally have more pieces of quantitative data and what we'd like to do is just build out a research team so we can . . . we don't have to, like every time we go in and make some decisions about photos it's not just by looking at like the numbers, it's a lot by looking at the way that people are using things. I think that where we're leaning more heavily now is we definitely have more quantitative stuff and we want to get more of the qualitative aspects so that …

I think the qualitative aspects can be used a lot more for, like I mentioned, the strategic planning and the overall product directions and the quantitative data can be more just to sort of make sure that like the current flow that you have, that you understand the way that people are using it and the problems in order to make the next version even better.

I guess one is more like towards, if you look at product development from like here to here, a lot of the qualitative work will be like in the beginning stages and definitely some on just making sure the flow is right. The quantitative stage, you're going to have, the thing has to be built and it's either looking at what's already existing and the current experiences that people are having on the site today and also just to validate like at the tail end whenever you launch something new to do the A/B test and to make sure that all of the hypotheses that you went into are actually validated by the data.

Audience Question : Kind of an add-on to that regional cultural differences. How do you [inaudible @34:50].

Julie: To be frank, I don't think we're doing a great job of that right now. I think the philosophy that we had going into building Facebook is that we wanted this to be really universal, you know, we didn't want to target just a particular demographic and we thought that like photos or status, that a lot of those things are things that people do in the real world with their real friends so we thought that those would be fairly universal and that's sort of how we've been trying to tackle the problems is to just approach it in that sort of manner.

But I think we are starting to realize definitely more and more that different countries, different regions, even different demographics are, we have to study that and understand those differences. I mean, for example, about a year ago we built an engineering office in Japan and it's because we realize that the way that Japanese users approach social networking is completely different, right? We were all about identity and your real names and your real pictures but if you look at Japan that just doesn't happen.

When you go online, their culture has a lot of advantages, different facets, like here I am hanging out with my co-workers and my co-workers don't actually know who my family is, who my girlfriend is. Like, those walls are sort of kept very clear and as a result, nobody wants to go on the internet and display all this information about everything that's happening and use your real names. All of the social networks that have been very successful have been anonymous in Japan.

This is when we realized, OK, we really actually have to have a team in Japan to really understand those cultural differences and try to build up a product that's going to work in Japan. If, over time, that cultural attitude shifts than that's great, but we really have to understand the way that people want to share and communicate with their friends in these different regions and I think China and a lot of the Asian countries are definitely huge and very different from the way that the Western, America or Europe or the UK, approach sharing.

In terms of other demographics, we're starting to look more at like older users. Our site has traditionally been 11 pixels and that wasn't a huge problem when everyone using it was under the age of 30. Yeah, we're doing studies now to understand whether or not the font size that we're using is going to cause eye strain or accessibility is another big one, right? We want to make sure that our site is accessible, that the contrast is high enough, that it's easy to browse and peruse for deaf readers. So, as we grow we're trying to put more resources into understanding different facets of our audience.

Audience Question: [inaudible @37:38].

Julie: That was a little bit of a . . . that's a good question. OK. Just from a consistency perspective, we weren't happy with the treatment of type on the site. We had, very traditionally everything was 11 pixel, all content was 11 pixel. Then about two years ago we thought, OK, we want this newsfeed to be about voice and we really want to emphasis voice and the hierarchy would be made easier if we upped what people were saying to 12 pixels and then kept everything else at 11. But it's not a very consistent treatment.

Like, if you went to a profile or you went to photos, everything was still 11 pixels. I think as a design team we just never felt that good about, it just sort of seemed like an arbitrary decision so we decided to approach it. If we wanted to make everything consistent and feel like it's designed with voice, how are we going to do that?

As a result, it's not just that the type changed, we tried to restructure the way that the stories were and so now we have the name on one line and then what they said on the next line and all of that was sort of meant to try and make it easier because we knew that by decreasing the font size that things were going to get a little bit harder to read and parse.

We also increased the margins so that the width of the line text would be shorter and as other, so a lot of decisions went into just trying to establish a very consistent visual language and I think what we realized was that afterwards, when a lot of people did start complaining, was OK, well does this even make sense or should we explore a consistent treatment for larger type across the site. This is what we're trying to study right now.

Moderator : [inaudible @39:35] Let's thank her for an awesome, awesome time.

Julie: Thank you, guys.