Identifying cliques and clique comembers in R using the SNA package

April 10, 2017April 10, 2017 / willchernoff / Leave a comment

The R package SNA provides a number of tools for analyzing social network data. This post reviews the function clique.census from the SNA package, and shows how it can be used to better understand the group structure among a list of network members.

Let’s start by creating a toy network. Say Henry, Sarah, Rick, Joe and Annie are all colleagues in a criminology department. Now imagine they’ve been asked to identify the most important people they collaborate with. The results of this fictitious effort are shown in the following edgelist.

edgelist <- matrix(c("Henry", "Sarah", "Henry", "Joe", "Sarah", "Joe", "Henry", "Sarah", "Henry", "Rick", "Sarah", "Rick", "Sarah", "Henry", "Sarah", "Rick", "Henry", "Rick", "Sarah", "Henry", "Sarah", "Joe", "Henry", "Joe", "Sarah", "Rick", "Sarah", "Annie", "Rick", "Annie"), ncol = 2)

Recall, a clique is a group where everyone in it “likes” everyone else. To identify cliques among our network of criminology researchers, we first transform it into a network object and then apply the SNA function clique.census.

library("network")

net tabulate.by.vertex=FALSE, enumerate=TRUE, clique.comembership="bysize") # Identify cliques
net_cc$clique.count

After applying the function clique.census, we see there were two cliques among our respondents, each involving three researchers.

To identify the comembers of these cliques, we inspect the contents of the variable net_cc$clique.comemb[3, , ].

To paraphrase the SNA documentation, the variable net_cc$clique.comemb is a three dimensional matrix of size max clique size x n x n. In this example we observed cliques involving only three network members each. As such, the 3-clique comembership information is stored in the variable net_cc$clique.comemb[3, , ]. (Note: if we observed one or more cliques with more than three members, say, a 4-clique, we could examine their comembership using the variable net_cc$clique.comemb[4, , ]).

Notably, the format by which net_cc$clique.comemb[3, , ] organizes clique comembership takes some getting used to. In fact, the main point of this post is to explain this organizational scheme in a more everyday kind of way.

Given our criminologist cliques, here’s how we find out who was in them. Recall, the largest clique we observed contained three individuals. Further recall that our network only contained five respondents. As such, the matrix net_cc$clique.comemb[3, , ] is of size 5 x 5. This matrix mimics the network structure itself. That is, network members are listed along the rows and, in the exact same order, listed again across the columns. The values within the matrix then identify researchers who were in cliques together.

Let’s go through a couple columns together to better understand what this matrix is telling us exactly.

The values in the column Joe show who Joe was in a clique with. Zero values indicate who Joe was not in a clique with, while values greater than zero indicate who he was in a clique with. More than just showing who Joe was in a clique with, these values identify the different cliques he was a part of. Reading the column from top to bottom, Joe and Annie were in zero cliques together. Next we see that Joe was in a clique with Henry. Notably, Joe was in one clique with himself (i.e., there was only one clique that involved Joe). The rest of the column values show that Joe was in zero cliques with Rick and one clique with Sarah. In total, this column tell us there was one 3-clique that involved Joe (see Joe’s value for Joe) and that this clique involved the researchers Henry and Sarah.

Let’s look at a trickier column. The values in the column for Henry ranged from 0 to 2. As with Joe, the values show who Henry was in a clique with and how many times they were in the same clique. Going down the column, we see a zero value for Henry and Annie. That is, Henry and Annie were not part of the same clique. Notably, Henry is in two cliques with himself. That is, there were two cliques, each of which involved Henry. The rest of the values show that Henry was in one clique with Joe, another clique with Rick and two cliques with Sarah.

Combined, these column values tell us there were two 3-cliques: one clique involving Henry, Sarah and Joe and another involving Henry, Sarah and Rick.

Plot Network Data in R with iGraph

December 15, 2015 / willchernoff / Leave a comment

I recently had a conversation on Twitter about a plot I made a while back. Recall, the plot showed my Twitter network, my friends and my friend’s friends.

Here’s the Twitter thread:

@willchernoff Hey! I saw your awesome graph there: https://t.co/dnFZQdYpxW Did you use https://t.co/NTKRXB4nB1?

— Antoine Dusséaux 柳华 (@ADssx) November 24, 2015

@ADssx glad you liked the graph! I made it using #iGraph in #R. @januverma makes great #python and #d3 stuff.

— Yung Spielbergo (@willchernoff) November 24, 2015

@ADssx @januverma I can share code next Monday. I'm traveling all week.

— Yung Spielbergo (@willchernoff) November 24, 2015

@willchernoff OK. Thanks!

— Antoine Dusséaux 柳华 (@ADssx) November 24, 2015

And here’s the R code:

#### Load R libraries
library("iGraph")

#### Load edgelist
r <- read.csv(file="edgelist_friends.csv-03-25.csv",header=TRUE,stringsAsFactors=FALSE)[,-1]

#### Convert to graph object
gr <- graph.data.frame(r,directed=TRUE)

#### gr
# Describe graph
summary(gr)
ecount(gr) # Edge count
vcount(gr) # Node count
diameter(gr) # Network diameter
farthest.nodes(gr) # Nodes furthest apart
V(gr)$indegree = degree(gr,mode="in") # Calculate indegree

#### Plot graph
E(gr)$color = "gray"
E(gr)$width = .5
E(gr)$arrow.width = .25
V(gr)$label.color = "black"
V(gr)$color = "dodgerblue"
V(gr)$size = 4

set.seed(40134541)
l <- layout.fruchterman.reingold(gr)

pdf("network_friends_plot.pdf")
plot(gr,layout=l,rescale=TRUE,axes=FALSE,ylim=c(-1,1),asp=0,vertex.label=NA)
dev.off()

Create a dictionary of authors and author attributes and values for a journal article using the Scopus API and Python

August 13, 2015September 20, 2015 / willchernoff / 13 Comments

As an exercise to brush up my Python skills, I decided to tinker around with the Scopus API. Scopus is an online database maintained by Elsevier that records and provides access to information about peer reviewed publications. Not only does Scopus let users search for journal articles based on key words and various other criteria, but the web services also allows users to explore these articles as networks of articles, authors, institutions, and so forth. If you’re interested in risk factors that lead to scholarly publications, publication citations, or impact factors, this is a place to start.

The following code yields a dictionary of author information by requesting content through the abstract retrieval API. This request is made using the Python package requests and parsed using the package BeautifulSoup. Enjoy!

#### Import python packages
import requests
from bs4 import BeautifulSoup


#### Set API key
my_api_key = 'YoUr_ApI_kEy'


#### Abstract retrieval API
# API documentation at http://api.elsevier.com/documentation/AbstractRetrievalAPI.wadl
# Get article info using unique article ID
eid = '2-s2.0-84899659621'
url = 'http://api.elsevier.com/content/abstract/eid/' + eid

header = {'Accept' : 'application/xml',
          'X-ELS-APIKey' : my_api_key}

resp = requests.get(url, headers=header)

print 'API Response code:', resp.status_code # resp.status_code != 200 i.e. API response error

# Write response to file
#with open(eid, 'w') as f:
#    f.write(resp.text.encode('utf-8'))

soup = BeautifulSoup(resp.content.decode('utf-8','ignore'), 'lxml')

soup_author_groups = soup.find_all('author-group')

print 'Number author groups:', len(soup_author_groups)

author_dict = {}

# Traverse author groups
for i in soup_author_groups:

    # Traverse authors within author groups
    for j in i.find_all('author'):

        author_dict.update({j.attrs['auid']:j.attrs}) # Return dictionary of attributes
      
        j.contents.pop(-1) # Pop dicitonary of attributes
 
        # Traverse author contents within author
        for k in j.contents:

            author_dict[j.attrs['auid']].update({k.name : k.contents[0]})
            
print author_list

Plot and Highlight All Clique Triads in VISONE

August 27, 2014September 20, 2015 / willchernoff / Leave a comment

Description

This post describes how to identify group structures among a network of respondents in VISONE. For a network of selections we identify any cliques involving three or more members. A clique is defined as a group containing three or more members where everyone has chosen everyone else.

Identify all triads

A tried is a network structure containing exactly three members. There are many types of triads. A group of three members where everyone chooses everyone else is a triad (i.e., a clique). A group of three members where two people choose each other and nobody choose the third member is another type of triad. There are 16 unique ways three people can select each other.

Identify all triads. Click the ‘analysis’ tab. Next to ‘task’, select ‘grouping’ from the drop down list of available options. Select ‘cohesiveness’ from the drop down list next to ‘class’. Select the option ‘triad census’ next to ‘measure’. Click ‘analyze’.

Highlight all cliques

Highlight all cliques. Click an empty part of the graph. Press the keys ‘Ctrl’ and ‘a’. Open the attribute manager. Click the ‘link’ button. Click the ‘filter’ button. Select ‘default value’ from the first drop down list. Select ‘triadType300’ from the second drop down list. Select ‘has individual value’ from the third drop down list. Click the radial button ‘replace’. Click ‘select’. Click ‘close’.

From the main VISONE drop down bar, select ‘links’. Click ‘properties’. Click the given color next to ‘color:’. Select ‘rgb’ tab. Set the ‘red’, ‘green’, and ‘blue’ values to 0. Set the ‘alpha’ value to 255. Set ‘opacity’ to 50%. Click the ‘close’ button. Set the ‘width:’ value to 5.0. From the ‘edge properties’ dialogue box, click the ‘apply’ button. Click ‘close.

Reduce visibility of all non-clique selections. Select all nodes and links. Click an empty part of the graph. Press the keys ‘Ctrl’ and ‘a’. Open the attribute manager. Click the ‘link’ button. Click the ‘filter’ button. Select ‘default value’ from the first drop down list. Select ‘triadType300’ from the second drop down list. Select ‘has individual value’ from the third drop down list. Click the radial button ‘remove’. Click ‘select’. Click ‘close’.

From the main VISONE drop down bar, select ‘links’. Click ‘properties’. Click the given color next to ‘color:’. Set ‘opacity’ to 20%. Set the ‘width:’ value to 2.0. From the ‘edge properties’ dialogue box, click the ‘apply’ button. Click ‘close.

Visibility and Privacy on Twitter: A Metadata Post

September 1, 2013September 20, 2015 / willchernoff / Leave a comment

The Twitter API teaches me that at best anyone on Twitter can follow anyone else on Twitter without anyone followed knowing anything about it. The Twitter API reveals this through the resource ‘GET statuses/user_timeline‘, a resource which allows anyone with a little programming know-how to anonymously download 3,200 of the most recent tweets made by any public Twitter user. While following Twitter users in this way is readily available to those with a coding background, it is itself not particular to people with such knowledge. Any Twitter user can do it. It is as simple as typing a phrase into the search field (or the search field of the search page) and hitting enter on the keyboard. All those tweets that barrel down the Twitter stream arrive in this way, literally without warning. Advanced search options also exit, but you get the idea. The ability to search and recall tweets anonymously on Twitter is not a defect, it’s a feature.

While Twitter allows users to get some content from other users without contacting them about it first, it, as a problem, is not entirely out of control. Twitter does offer privacy settings and, to be fair to Twitter, these settings provide some aggressive protections. In setting tweet privacy to ‘Protect my Tweets’, Twitter states that a person, in this way, can prevent those without unauthorized permission from viewing their tweets. More specifically, Twitter safeguards include:

Your Tweets will only be visible to users you’ve approved.
Other users will not be able to retweet your Tweets.
Protected Tweets will not appear in Twitter search or Google search.
@Replies you send to people who aren’t following you will not be seen by those users (because you have not given them permission to see your Tweets).
You cannot share permanent links to your Tweets with anyone other than your approved followers.

In this way, Twitter makes their service available to a number of people who don’t want everything they have to say on Twitter made publicly available. But even with privacy settings in place, some pieces of user information aren’t protected.

Even with security settings in place, Twitter makes certain pieces of non-tweet, user information publicly available. Some of these include:

Tweets directed at private users

Any Twitter user can see some of what other users are saying about any other user on Twitter.

Profile summary

Twitter users share with other Twitter users their basic profile information, information which includes tweets, following, and followers counts. At most this can include a location, say a city or state, and a personal webpage.

Detailed profile summary (available through the API)

A Twitter profile can be seen in greater detail through the API, so that any Twitter user can know when any other Twitter account was created.

Social network connections

Who follows a Twitter user and who else these followers follow is not entirely hidden, connections which can help reveal social networks.

Not everything a user does on Twitter is hidden from sight, even when privacy settings are in place. This issue is problematic in that it leads to a false sense of data security, where in which what Twitter and Twitter user’s think is hidden do not necessarily correspond with one another. I have no reason to suspect the social media organization Twitter of actively threatening the security and privacy of its users, but they could make more of an effort to identify and present, in an accessible way, the metrics about their users which are publicly knowable. They know these things. It’s not necessarily outrageous to think they could let us know these things too. Twitter probably knows as well as anyone else that they doesn’t know what’s best for their users; and that only their users actually know what’s best for themselves, but users can’t adequately weigh the pros and cons of their use when they themselves don’t know what’s not hidden. By taking the time to do this, Twitter can better inform their users (and potential users) about their social media presence and better support us all in our use of social media.

Hypothesis to test

On average, when setting their security options, users implicitly assume that what they can allow or deny others to see, as provided by any social media service, is an exhaustive list of all accessible information.

Download Twitter Data using JSON in R

May 25, 2013September 20, 2015 / willchernoff / 6 Comments

Here we consider the task of downloading Twitter data using the R software package RJSONIO.

Before we can download Twitter data, we’ll need to prove to Twitter that we are in fact authorized to do so. I refer the interested reader to the post Twitter OAuth FAQ for instructions on how to setup an application with dev.twitter.com. Once we’ve setup an application with Twitter we can write some R code to communicate with Twitter about our application and get the data we want. Code from the post Authorize a Twitter Data request in R, specifically the keyValues() function, will be used in this post to handle our authentication needs when requesting data from Twitter.

## Install R packages
install.packages('bitops')
install.packages('digest')
install.packages('RCurl')
install.packages('ROAuth')
install.packages('RJSONIO')


## Load R packages
library('bitops')
library('digest')
library('RCurl')
library('ROAuth')
library('RJSONIO')
library('plyr')


## Set decimal precision
options(digits=22)


## OAuth application values
oauth <- data.frame(consumerKey='YoUrCoNsUmErKeY',consumerSecret='YoUrCoNsUmErSeCrEt',accessToken='YoUrAcCeSsToKeN',accessTokenSecret='YoUrAcCeSsToKeNsEcReT')

keyValues <- function(httpmethod,baseurl,par1a,par1b){  	
# Generate a random string of letters and numbers
string <- paste(sample(c(letters[1:26],0:9),size=32,replace=T),collapse='') # Generate random string of alphanumeric characters
string2 <- base64(string,encode=TRUE,mode='character') # Convert string to base64
nonce <- gsub('[^a-zA-Z0-9]','',string2,perl=TRUE) # Remove non-alphanumeric characters
 
# Get the current GMT system time in seconds 
timestamp <- as.character(floor(as.numeric(as.POSIXct(Sys.time(),tz='GMT'))))
 
# Percent encode parameters 1
#par1 <- '&resources=statuses'
par2a <- gsub(',','%2C',par1a,perl=TRUE) # Percent encode par
par2b <- gsub(',','%2C',par1b,perl=TRUE) # Percent encode par
 
# Percent ecode parameters 2
# Order the key/value pairs by the first letter of each key
ps <- paste(par2a,'oauth_consumer_key=',oauth$consumerKey,'&oauth_nonce=',nonce[1],'&oauth_signature_method=HMAC-SHA1&oauth_timestamp=',timestamp,'&oauth_token=',oauth$accessToken,'&oauth_version=1.0',par2b,sep='')
ps2 <- gsub('%','%25',ps,perl=TRUE) 
ps3 <- gsub('&','%26',ps2,perl=TRUE)
ps4 <- gsub('=','%3D',ps3,perl=TRUE)
 
# Percent encode parameters 3
url1 <- baseurl
url2 <- gsub(':','%3A',url1,perl=TRUE) 
url3 <- gsub('/','%2F',url2,perl=TRUE) 
 
# Create signature base string
signBaseString <- paste(httpmethod,'&',url3,'&',ps4,sep='') 
 
# Create signing key
signKey <- paste(oauth$consumerSecret,'&',oauth$accessTokenSecret,sep='')
 
# oauth_signature
osign <- hmac(key=signKey,object=signBaseString,algo='sha1',serialize=FALSE,raw=TRUE)
osign641 <- base64(osign,encode=TRUE,mode='character')
osign642 <- gsub('/','%2F',osign641,perl=TRUE)
osign643 <- gsub('=','%3D',osign642,perl=TRUE)
osign644 <- gsub('[+]','%2B',osign643,perl=TRUE)
 
return(data.frame(hm=httpmethod,bu=baseurl,p=paste(par1a,par1b,sep=''),nonce=nonce[1],timestamp=timestamp,osign=osign644[1]))
}

Next, we need to figure out what kind of Twitter data we want to download. The Twitter REST API v1.1 Resources site provides a useful outline of what kind of data we can get from Twitter. Just read what is written under the Description sections. As an example, let’s download some user tweets. To do this, we find and consult the specific Resource on the REST API v1.1 page that corresponds with the action we want, here GET statuses/user_timeline. The resource page lists and describes the download options available to the task of getting tweets from a specific user, the thing we want to do, so it’s worth it to the reader to check it out.

Here we download the 100 most recent tweets (and re-tweets) made by the user ‘Reuters’.

## Download user tweets
# Limited to latest 200 tweets
# Specify user name
user <- 'Reuters'
 
kv <- keyValues(httpmethod='GET',baseurl='https://api.twitter.com/1.1/statuses/user_timeline.json',par1a='count=100&include_rts=1&',par1b=paste('&screen_name=',user,sep=''))
 
theData1 <- fromJSON(getURL(paste(kv$bu,'?','oauth_consumer_key=',oauth$consumerKey,'&oauth_nonce=',kv$nonce,'&oauth_signature=',kv$osign,'&oauth_signature_method=HMAC-SHA1&oauth_timestamp=',kv$timestamp,'&oauth_token=',oauth$accessToken,'&oauth_version=1.0','&',kv$p,sep='')))

At this point in the post you should have the 100 most recent tweets made by the user ‘Reuters’ as well as values on several variables recorded by Twitter on each tweet. These are stored in a list data structure. You are now free to do list things to these data to explore what it is you have.

For instance, let’s see the tweets.

theData2 <- unlist(theData1)
names(theData2)
tweets <- theData2[names(theData2)=='text']

Plotting Twitter Networks in R

March 25, 2013September 20, 2015 / willchernoff / 1 Comment

My friends and my friends’ friends on Twitter. Can you guess which cluster is associated with the National Science Foundation (@NSF)?