EDIT (23/01/2015): Changed some text to make clear that you can still run Netvizz by grabbing the source code, registering a new app, and running it in developer mode.
EDIT (25/01/2015): I have written a small install guide for the source code on github. I’m unfortunately unable to help with individual problems, if you’re unfamiliar with server administration, your department’s tech support team should be able to help.
EDIT (28/01/2015): Since Facebook has changed the way apps are created, you can apparently no longer run just scripts requiring extended permissions in newly created apps, even in developer mode (making my source code useless for you). I have therefore whipped up a version of Netvizz that can only do pages and groups without requiring extended permissions. Since this does not have to go through review, you can use the app directly here.
EDIT (29/01/2015): Facebook’s policy review has accepted the new version of Netvizz (with personal network functions removed) and the app is again accessible here. API v1.0 is still going to be retired in April and this may pose problems, but this is something for another day.
EDIT (02/05/2015): API v1.0 has now been retired, but a new version of Netvizz (v1.2) has survived the changes and should continue functioning in the foreseeable future. Personal and group friendship networks are gone for good.
Original Post:
Today Netvizz, an app that allows researchers to download data from the Facebook platform, was suspended by the company and I received a mail explaining why:
Your app is violating the following Platform Policies:
Platform Policy Section 1: Build a quality product.
Platform Policy 1.1: Build an app that is stable and easily navigable.Platform Policy 3.3: Only use friend data (including friends list) in the person’s experience in your app.
To clarify, your app should be stable and easy to use and shouldn’t stall escessively. Additionally, you should not allow friend data export, even if that data is anonymized. You can access the full list of our Platform Policies here: https://developers.facebook.com/policy/.
Since Facebook has recently been very preoccupied with app privacy – for very good reasons actually – this does not come as a surprise. I have been anticipating API changes and the retirement of version 1.0 that comes with some very sensible changes in how data is delivered to platform apps for a while. Apps are clearly one of the biggest problems when it comes to Facebook’s privacy puzzle and most changes make a lot of sense. As Bernie Hogan wrote here, friendship connections are one of the casualties, as they will no longer be available to apps at all (v2.2 no longer makes them available). I was hoping to stall a little by moving to API v2.0, which still runs until April 2016, but this seems no longer viable after this morning’s news. As much as I agree with the general changes Facebook is making, I think it is a real shame that the analytical possibilities apps like Netvizz afford will no longer be available to researchers.
Over its roughly five year life span, what started as an inquiry into Facebook’s API, ultimately had over 60K unique users and analyzing their friendship network has been the start into graph analysis for many people. GetNet, a modified version of Netvizz, was used by Lada Adamic in her highly successful Coursera MOOC, allowing students to look at a network they are intimately familiar with, making network visualization much more tangible. GetNet actually still works, but will probably break in April 2015, if not shut down earlier.
For me personally, Netvizz has been a ambivalent project. On the one side, I enjoyed the tinkering with the API, but on the other, maintaining a complex tool in my spare time has often been a challenge. As anybody who offers software online for free will tell you, the mass of not always friendly emails can be daunting. I’m also not a computer scientist and I work in a humanities department, where technical work does not really count in performance reviews.
But the real problem with the current situation has little to do with me and much more with the many courses and research projects that have been relying on Netvizz. They are left out in the cold. So here are some elements that will hopefully help them deal with the situation:
- Despite my hesitation to make software public that can be used very easily to download large amounts of non-anonymized data, there is so much code already in the wild that another set of scripts is not going to make much of a difference. I’m therefore making Netvizz’ source code publicly available.
This should allow research projects relying on Netvizz to take the source code, register their own app at developers.facebook.com and run it in developer mode (just to make this clear, since I am the developer, I can actually still run the app, but it is no longer publicly available), which should work until April 30, 2015, the day v1.0 of the API retires.I apologize for the crappy code quality, this is one of those projects that grow and grow and never get a real redesign. - I will try to enter into further communication with Facebook to see what can be done, but I don’t expect much from that.
- If that does not work, I will submit a version of Netvizz version for review that excludes personal network features and focuses on pages and groups. It’s still going to “stall excessively”, though, since it gets a lot of data.
I have no idea how long any of this make take. In the meantime, check out this list for alternatives, most of which hopefully still work. But make no mistake: this may well be the beginning of the end for external Facebook research with digital methods.
50 Comments
Leave a Reply to marta severo Cancel reply
Tech support questions will not be answered. Please refer to the FAQ of the tool.
Andreas Birkbak says:
January 23, 2015 at 1:38 pm /
This is very sad news. Thanks for writing such a considerate and thorough post to relieve some of the shock, Bernard. And thanks for the hard work on Netvizz that so many of us benefited from!
Personally, I don’t think it would be much of a sacrifice to lose the opportunity to export friend data. Pages and groups were always the most interesting data for me, but maybe others feel differently about this. I would love to see a new Netvizz that only focused on pages and groups, even if one would have to ungeek the interface and make some data extraction restrictions in order to pass under Facebook’s radar.
Bernhard says:
January 25, 2015 at 9:38 am /
Hi, yeah, I’m currently trying that. But in the end, if there’s not more clarity and accountability in these areas, I don’t think that I’d want to invest much more work into something that can disappear any time. And providing publicly accessible research infrastructure for free without dedicated funding for tech support is in any case a model I have strong doubts about…
best,
Bernhard
Paul Girard says:
January 23, 2015 at 2:42 pm /
API world…
The privacy argument on facebook applications is definitely relevant.
A different way to generate friends networks is to use a scrapper.
More specifically a client-side scapper, i.e. an application used directly by one facebook users which will navigate through facebook web pages as if the user would do it manually.
Privacy issues can be handled by:
– Open Source code letting the world check what this scrapper software does with the data
– client-side only data treatment making sure no data are aggregated in any central place
Still, using a scrapper most likely breaks Facebook usage rules and building such a scrapper isn’t the easiest thing on earth. Thus the risk is delegated to the user only not to the application.
I would use artoo.js to do this embed in a browser extension…
Finally, thank you Bernhard for Nettviz !
Was indeed a great tool to engage people with social networks…
Bernhard says:
January 25, 2015 at 9:32 am /
Hey Paul, sure, scraping is always possible, but also necessarily hacky (and not viable for very large pages) – and it’s always going to limit who can do this kind of research. I’d really hope that research would receive specific legal provisions (with added obligations) to access these data silos. There are exceptions for academic research in other areas (e.g. copyright), and we really need something similar. I hope to write something about this in the near future…
best,
Bernhard
Alexis Brailas says:
January 24, 2015 at 2:26 pm /
Bernhard thank you so much for Netvizz application. It was a wonderful opportunity for me to have a research insight into Facebook’s social networks. I am sorry this is becoming harder or impossible now and although I agree privacy issues should be handled and respected I think this is a step back as regarding Internet democratization and research freedom.We know that Facebook experimented with human subjects without their consent (Experimental evidence of massive-scale emotional contagion through social networks by Kramer et al)so I think there is a hypocrisy here.
Bernhard says:
January 25, 2015 at 9:28 am /
Yes, it’s definitely not super coherent, but I think that the only solution can come through legislation, where research objectives are explicitly recognized as public interest and treated differently from “normal” apps. But the public interest doesn’t have a lobby…
Pingback: Just some reads, links and clips #2 | Geert Verhoeff
marta severo says:
January 24, 2015 at 6:13 pm /
Bernhard, thank for Netvizz! It’s just worrying that they directly stopped the application without leaving you the chance to modify it.
Bernhard says:
January 25, 2015 at 9:24 am /
Yeah, that’s how the procedure works. I’ve submitted a version without personal network features for review, we’ll see whether this gets through…
best,
Bernhard
Tobias Bornakke says:
January 24, 2015 at 8:21 pm /
So sad news. My interest have also always been the page and groups – i’m however unsure how they will work with the upcoming API changes.
Bernhard says:
January 25, 2015 at 9:22 am /
Hi, well, v2.0 is still running until April 2016 and still has FQL, so, technically, one could upgrade tools to that version. But since there is going to be a review process for many apps, this may not work.
best,
Bernhard
taerg says:
January 25, 2015 at 5:35 am /
I want to use netvizz , when i create a website and use your code,there are some problems.
This application has features that require extended permission to access data.It won’t work without them.
I’m a fresher,don’t know how to configure it,can you tell me how to solve that problem?
Thanks a lot.
Bernhard says:
January 25, 2015 at 9:20 am /
Hi, I wrote an install guide on github, hope that helps.
best,
Bernhard
Nina says:
January 25, 2015 at 11:26 am /
Thank you very much for opportunity to work with your appplication!!
I even do not know how to work without it!
Give us hope that will be able to solve this problem!!
I need to use the real data for teaching students of the main mathematical approaches of social network analysis
Thanks for advance)
Marcela Canavarro says:
January 25, 2015 at 3:51 pm /
Hi Benhard, first of all congrats and many thanks for Netvizz!
I’m a researcher from Brazil, studying in Portugal and I was actually closing my doctors thesis proposal, which includes Netvizz as one of main tools for data extraction, when I found out it was not running anymore. I searched on Google and found this post. That’s very sad news.
But I totally agree with previous comment by Andreas Birkbak. Page and groups data can still do a lot for researchers. I’m in the social area research and all data I expect to extract from FB is out of pages, groups AND events (something I was anxiously expecting netvizz to provide, before the latest bad news).
Thanks again and I hope there’s a way to keep it going!
muka says:
January 26, 2015 at 4:53 am /
I am so sad to know this news :(…in first, I have worked on fb page data through using netvizz then also have planned to finish my research. But unfortunately this has happened to me now. Bernhard thank you for hard work on it and I wait your submmited version results. I hope it will work.
Mirkó Petényi says:
January 26, 2015 at 10:46 am /
Hi Bernard,
First of all thank you for your work and effort what you invested to this app. It helped a lot for me, I built my dissertation on the datas what I downloaded with netvizz. Now, I’m a little bit despaired, because without this tool I can’t imagine how can I finish the dissertaiton. So, I hope somehow this project can continue.
Thank you that you shared the app on Girhub, now I try to install it.
I think most of researchers use the group and page datas that are publicly available, I can’t understand why does it offend the Facebook’s privacy rules.
Thank you again!
Best wishes,
Mirkó
Janna Joceli says:
January 27, 2015 at 5:44 pm /
This is soo sad and I have no words to describe my disappointment :/
I´ve been working with Netvizz for more than a year. It is really an amazing and useful tool!!
I hope FB welcome back the Netvizz version without personal network features.
Thanks for sharing an alternative list of tools, Bernhard 🙂
bw.
Miguel Suarez Xavier Penteado says:
January 27, 2015 at 8:09 pm /
Hi Professor
I’m sad for Netvizz too. Anyway congrats for all Work.
I hope facebook reconsider your application.
Thank you !
Sam says:
January 27, 2015 at 10:09 pm /
Hi Bernard,
thank you for the code. I have registered an app and installed netvizz on a server.
When I start I get this: “Submit for Login Review
Some of the permissions below have not been approved for use by Facebook.
Submit for review now or learn more.”
I didn’t submit the app for review an hoped that other features as the friends stuff would work. Unfortunately it gets in a reload loop an changes the access token every second.
I actually don’t want to submit my copy of netvizz for a review, since you are somehow in a conversation with the the facebook people.
Would it be possible to make a branch with only groups and pages functions? Particularly I am mainly interested in the page page like function and was using that to identify page-page cliques using Clique Percolation Method.
Can you help me out please?
Thanks in advance and best wishes,
Sam
Bernhard says:
January 28, 2015 at 7:45 am /
Thanks for the heads up, I didn’t know that the review process changed. I now wrote a version that can do groups and pages (linked above). Hope that works!
Sam says:
January 28, 2015 at 11:10 pm /
Thank you for the new version. It is still helpful. 🙂
Benaz Batrawi says:
January 31, 2015 at 8:44 pm /
Hi Dear Dr. Bernard,
Thanks for your efforts in this, it is really a great job.
I was thinking to use Netvizz to study page data for user activity around posts on pages as part of my thesis but I do not know after reading this discussion if that will work as if I want to do so it will be in March or April will that still work at that time you think? I did not start doing any work regarding this so do you advice me to go on in this or leave it for another alternative? I want to study three pages or four at max to see which issues in these pages are most liked and commented on not more than that so If this module will be still valid in March and April what exactly should I follow from this point, should I download any extra thing I have windows computer, and is is necessary to have a researcher account or I can use my personal one?
Thanks so much for the feedback
Benaz
NMDC program
UVA
Bernhard says:
February 1, 2015 at 9:19 am /
Hi Benaz, it’s difficult to say something definite, since the upcoming closing of v1.0 of the API in April will affect all tools and how Facebook will handle specific cases is hard to predict. But the part concerning pages is certainly the safest, in the sense that no extra permissions are required for page analysis and Facebook considers this the “public” part of its service. I am pretty confident that this will be fine.
best,
Bernhard
Benaz Batrawi says:
February 1, 2015 at 10:05 am /
Thanks a lot Dr. Bernard,
Appreciate your response, can you please reply to this part as well “should I download any extra thing as I have windows computer, and is it necessary to have a researcher account or I can use my personal one?
Moreover can you please ad the video link that I can watch for page data module? I have watched the introduction but I need a specific one if there is any if you please.
Thanks so much for the feedback
Bernhard says:
February 1, 2015 at 10:11 am /
Hi, for the first part, please discuss this matter with your thesis advisor. For the second, no, there is no video…
Benaz Batrawi says:
February 1, 2015 at 10:29 am /
Thanks a lot Dr. Bernard, will do for sure.
Benaz
Anders Kristian Munk says:
February 1, 2015 at 2:10 pm /
Hi Bernhard,
Just wanted to give you a big thanks for your efforts with this. Netvizz has been an amazing tool for introducing our students to network visualization and social media research. Indeed, it has been very useful for research purposes as well. So thanks for your (voluntary and no doubt labour intesive) efforts! I completely agree on the necessity of negotiating a more stable framework for public interest research in areas like this. Let us know what we can do.
Walter says:
February 1, 2015 at 8:25 pm /
Thanks Bernhard for this amazing tool, i have used it during the Social Network Analysis Course and after that. I can in one hand do not believe what this FB people are doing, like they have no codes in the community; in the other hand, that may have been a political decision from somebody that has no idea about the meaning of a community. I know that is not your problem, it is just a way to express my thoughts. Thanks for the hard work and this interesting website! Walter from Argentina (Buenos Aires).
Aphrodite says:
February 10, 2015 at 9:17 am /
Hello,
First of all, I would like to thank you for this amazing tool and your efford. It’s very helpful and useful. Too bad I’ve discovered it so late. I’m doing a research recently and what I’m intrested in is the group data. I noticed that the field for the group id fits up to 15 characters, when my group has a 16-character id. So I get an error message to check the id. I tried with other group ids it works just fine. I would really appreciate any help. Thanks in advance and best wishes,
Aphrodite
Bernhard says:
February 10, 2015 at 10:09 am /
Oops! Fixed it, thanks a lot for the notice!
Aphrodite says:
February 11, 2015 at 9:18 am /
Oh! Great, thank you!!
Pingback: The “rules” of social data mining | Geert Verhoeff
Pingback: Just some reads, links and clips #4 | Geert Verhoeff
Antoinet says:
February 12, 2015 at 11:11 am /
Hello Bernhard,
Well, first, thanks for the great work! Here in Barcelona, we’ve been using Netvizz in our hands-on sessions every single year since 2011 and it was just great!
Working for free is tough because, on one hand, it’s hard to keep-up and, on the other hand one may feel guilty for not doing more or better (true but also irrelevant! Perfect is the enemy of good!).
Now, let me cite Henry David Thoreau: “If you have built castles in the air, your work need not be lost; that is where they should be. Now put foundations under them.”
Best! (from a Python Lover!)
Bernhard says:
February 12, 2015 at 12:33 pm /
Thanks a lot!
Ulises says:
February 19, 2015 at 11:23 am /
Thank you very much for keeping this working as long as you did, Bernhard. I used netvizz for years to teach my students network visualization, and I think they got a lot out of it. Now I need to go change my syllabus! But there’s an important lesson for students in this as well: do you really own your data?
Diego says:
February 22, 2015 at 4:41 pm /
Dear Bernhard,
at first, thank you so much for all your efforts in order to “save” Netvizz!
While using the new version of Netvizz today I have noticed that if a page has published more than 600 hundreds posts in a year: i) you can download no more than 600 hundreds posts by starting from the last post of the year; ii) you can’t download the other posts (e.g., if a page has published 700 posts in 2014, you can download the last 600 posts, but you can’t download the other 100 posts even if you ask Netvizz to download only the post of January 2014). With the former version of Netvizz you did not have this kind of problem.
Have you noticed it?
Thank you again for everything, you are our hero!
Bernhard says:
February 23, 2015 at 5:55 am /
Hi, thanks for the report. May I ask you two things? Could you have a look at the FAQ for how to report a bug? I’ll need much more information to try and reproduce the problem. And could you send that to tools@polsys.net? Thanks a lot!
Jack G says:
March 26, 2015 at 1:09 am /
Hi Bernhard,
Thanks so much for Netvizz! It’s key data collection tool for me to teach Gephi and SNA at Bowdoin.
I found a weird bug recently. When I download a group’s data, it just lists the edges and not the nodes. It’s also missing the nodedef and edgedef headers. We manually added them–which was great for students to figure out how a .gdf file is made!–but others may not figure that out.
Thanks again!
Bernhard says:
March 26, 2015 at 4:03 am /
Hi, I can’t replicate this error, please check the Netvizz FAQ for how to submit a bug report, thanks!
Pingback: Moje facebooková síť – získávání dat (Datamining pro humanitní vědy) | Žbluňkův brloh
Pingback: Review of “Analysing Social Media Data and Web Networks” by Cantijoch, Gibson & Ward (eds.) « Anders [Olof] Larsson
Pingback: How to visualize your Facebook network | Dinesh Ram Kali.
Pingback: Facebook. Визуализация социальных связей |
Pingback: How to visualize your Facebook network - Linkurious
Pingback: Annotation – Content Marketing Through Data Mining on Facebook Social Network – Adventures in Information Science
Pingback: An Overabundance of Research, Data, and Tools – Adventures in Information Science
Pingback: Facebook’s app review and how independent research just got a lot harder
Shabana Chandrasekaran says:
September 7, 2019 at 5:24 am /
“But the real problem … much more with the many courses and research projects that have been relying on Netvizz”. This is absolutely TRUE in our case. We have published a couple of our research with Netvizz and had to drop a promising future research project now. It’s serious handicap for most of the social science researchers. You were our savior and hero Mr Reider. Thank you very much.