Interactive Link Visualizer Application
- Introductory Forum Post for GSoC
- Discussions held mostly on XWiki Matrix Channel
Description
Note: This design page covers the technical aspects and key decisions behind the application.
The project revolves around creating a link visualization application for XWiki. The main aim of this app is to help users visualize and interact with a network graph that represents the pages in their XWiki instance. It's designed to show the relationships between different pages in a clear and interactive way. Users can explore, analyze, and interact with link connections within XWiki pages using dynamic network graphs, where the nodes represent documents/pages and the edges represent various links between the pages (such as backlinks and included page references). This application provides valuable insights into the structure and connections within users' wikis.
Use Cases
Given the application's focus on data and analysis, its primary utility lies in examining the documents within a wiki. The use cases are centered around this objective, outlined below:
Use Case 1: Content Creator
As a content creator or editor, I can employ the link visualization application to scrutinize the relationships and interlinking patterns among various wiki pages. This functionality aids in identifying gaps in links or content, pinpointing redundant information, and highlighting opportunities for generating additional material. This, in turn, contributes to enhancing the overall coherence and structure of the wiki's content.
Use Case 2: Wiki Administrator
For wiki administrators or community managers, the link visualization application serves as a powerful tool to elevate user engagement and facilitate navigation within the wiki. By presenting users with a visual representation of link connections, administrators can effortlessly explore related content, navigate fluidly between pages, and delve into specific topics of interest.
Use Case 3: Information Researcher
As a knowledge seeker or researcher, I can harness the link visualization application to unearth fresh information and delve into correlated subjects within the wiki. Through visual exploration of the network graph, I can uncover less apparent links, follow intriguing paths, and cultivate a more comprehensive understanding of the wiki's content.
For instance, consider a researcher utilizing the link visualization application within XWiki to delve into sustainable energy. While navigating the network graph visually, they uncover interconnected pages on solar energy, wind power, hydropower, and bioenergy. Additionally, they stumble upon a cluster of nodes dedicated to energy storage technologies, leading them to the exploration of pages discussing battery technologies, energy grid integration, and smart grid systems. By traversing the network graph visually, the researcher uncovers concealed connections, embarks on captivating pathways, and broadens their insights, ultimately unearthing new discoveries within the wiki.
Data Model
- Page: Represents a page or document within the wiki. Each page has a unique identifier and contains content and metadata.
- Link: Represents a link between two pages. It includes the source page and the target page, establishing the relationship between them.
- Network Graph: Represents the overall network graph structure that visualizes the link relationships between pages. It consists of nodes (representing pages) and edges (representing links).
- Node: The spherical points in the graph; represents a node in the network graph, corresponding to a page. It includes information such as the page's unique identifier, title, document reference, wiki, spaces and name.
- Edge: The arrow-lines in the graph; represents an edge in the network graph, corresponding to a link between two pages. It includes information such as the source and target page identifiers.
- Linked Pages Panel: It shows the visualization only around the currently opened document i.e., shows pages/documents that are linked to the currently opened page, as well as any backlinks associated with the currently opened page.
- Solr Facets Filteration: It facilitates targeted exploration of the graph based on specific criteria, enhancing the user's ability to tailor their interaction with the visualization.
- Search bars: We have 2 search bars - the first one is the main search query bar where we can even put Solr queries; it is responsible for tinkering with the graphData. The other one is the Search in nodes... bar where we can find for pages within the graph.
- Filter input: By default, the first 1000 pages will be visualized. However, a number input field is given to override the number and visualize however many documents a user wish.
- Interactive Buttons: Many interactive buttons like zoom & full-screen controls, iteration increase etc. are given for better a11y.
When the data model is well defined, we will decide the most appropriate way for the implementation of the data source (this is something that will fetch us the information about the links present in a wiki). The most suitable way is to use the existing XWiki Solr Search API, where we'll use JavaScript to help us return the link information. Once we have the link information, we'll pass the data to a visualization library to generate the graph.
Visualization library
We will use graphology for implementing the data structures required to represent various graph types in memory: directed, undirected, mixed, multi, etc., and sigmajs for rendering and interacting with network graphs in the browser. Since XWiki can contain thousands of documents, Sigmajs seems to be perfect for it performance-wise.
Why not D3.js?
Sigma.js renders graphs using WebGL. It allows drawing larger graphs faster than with Canvas or SVG based solutions.
Graph layout
Since we cannot put the x and y co-ordinates in every node, we have to use a layout for placing the nodes in the graph container. But we cannot initialize a layout in Graphology without having some initial x and y co-ordinates, so I fixed the issue by initializing some important graph attributes as:
graph.forEachNode((node) => {
graph.setNodeAttribute(node, "x", i++);
graph.setNodeAttribute(node, "y", i);
i++;
});
graph.forEachEdge((edge) => {
graph.setEdgeAttribute(edge, "size", 5);
});
Now we can use any layout we wish to. We have circular layout, random layout as described here and we have some force dependent layouts like force layout and force-atlas-2. To avoid overlapping of graph nodes, we also have a choice of noverlap-layout.
Now, we are interested in the ForceAtlas2 layout's webworker variant (with the #.inferSettings to be precise) because that's the one with the optimal settings for tuning the layout. We didn't use noverlap layout for example, because it doesn't utilize the complete space of the graph container.
const sensibleSettings = forceAtlas2.inferSettings(graph);
const positions = forceAtlas2(graph, {
iterations: 50,
settings: sensibleSettings
});
const sensibleSettings = forceAtlas2.inferSettings(500);
Graph custom settings
We need to have some settings specific to our needs in XWiki. Node color and size are some of them. Before the size of the nodes, we are concerned about the theme compatibility inside XWiki as XWiki has a lot of pre-defined themes. So for example if we are on the Darkly theme (or dark-mode) then we'd want to have lighter color for the nodes label. We solve issues of this kind through implementing the XWiki theme variables inside our application.
var themeColors = $jsontool.serialize({
'nodeColor': $theme.linkColor,
'labelColor': $theme.textColor,
'fadeColor': $theme.highlightColor,
'labelContainerColor': $theme.pageContentBackgroundColor
});
Now we can access these theme variables through our API we created 
Arrow-head size for labels
We don't have much options for customization in Sigmajs. The arrow-head size is hardcoded for example. To change it's size, we have to tinker with the root class of EdgeArrowHeadProgram like this:
process(
sourceData: NodeDisplayData,
targetData: NodeDisplayData,
data: EdgeDisplayData,
hidden: boolean,
offset: number
) {
data.size *= 4 || 1; // Increase the arrow-head size 4 times its original value
super.process(sourceData, targetData, data, hidden, offset);
}
}
const EdgeArrowProgram = createEdgeCompoundProgram([
EdgeClampedProgram,
customEdgeArrowHeadProgram
]);
// Then we need to put the settings in the renderer settings, to make sure that we are over-riding the existing method
const rendererSettings = {
...
edgeProgramClasses: { arrow: EdgeArrowProgram },
...
};
This way we can use whatever arrow-head size we want. This is really helpful for the panel as the arrows were barely visible in the default Sigmajs settings.
Changing the label's container color
There is a bug is Sigmajs (I have already reported it) which is that the label's container has hardcoded white color - which is a problem because there is a setting for labelColor and if someone sets it to a lighter color, then the text won't be visible at all due to label container color being white as well. To fix the issue, we need to override the drawHover method like this:
function customDrawHover(
context: CanvasRenderingContext2D,
data: PartialButFor<NodeDisplayData, "x" | "y" | "size" | "label" | "color">,
settings: Settings
): void {
const size = settings.labelSize,
font = settings.labelFont,
weight = "bold";
context.font = `${weight} ${size}px ${font}`;
// Then we draw the label background
context.fillStyle = "red"; // YOUR FAVORITE COLOR HERE FOR THE LABEL CONTAINER ;)
context.shadowOffsetX = 0;
context.shadowOffsetY = 0;
context.shadowBlur = 6;
context.shadowColor = "pink"; // Whatever you wish
const PADDING = 3;
if (typeof data.label === "string") {
const textWidth = context.measureText(data.label).width,
boxWidth = Math.round(textWidth + 5),
boxHeight = Math.round(size + 2 * PADDING),
radius = Math.max(data.size, size / 2) + PADDING;
const angleRadian = Math.asin(boxHeight / 2 / radius);
const xDeltaCoord = Math.sqrt(
Math.abs(Math.pow(radius, 2) - Math.pow(boxHeight / 2, 2))
);
context.beginPath();
context.moveTo(data.x + xDeltaCoord, data.y + boxHeight / 2);
context.lineTo(data.x + radius + boxWidth, data.y + boxHeight / 2);
context.lineTo(data.x + radius + boxWidth, data.y - boxHeight / 2);
context.lineTo(data.x + xDeltaCoord, data.y - boxHeight / 2);
context.arc(data.x, data.y, radius, angleRadian, -angleRadian);
context.closePath();
context.fill();
} else {
context.beginPath();
context.arc(data.x, data.y, data.size + PADDING, 0, Math.PI * 2);
context.closePath();
context.fill();
}
context.shadowOffsetX = 0;
context.shadowOffsetY = 0;
context.shadowBlur = 0;
// And finally we draw the label
drawLabel(context, data, settings);
}
// Then put this in the renderer settings
const rendererSettings = {
labelColor: { color: "blue" }, // YOUR SECOND FAV COLOR FOR THE LABEL
zIndex: true,
hoverRenderer: customDrawHover // IMPORTANT
};
const renderer = new Sigma(graph, container, rendererSettings);
Click & drag events and cursor style change
Since we implemented the functionality that allows us to open a page's corresponding URL by clicking on it, we also need to change the cursor style to indicate that node is basically a link:
renderer.on("clickNode", ({ node }) => {
if (!graph.getNodeAttribute(node, "hidden") && allowClick) {
window.open(graph.getNodeAttribute(node, "pageURL"), "_self");
}
});
renderer.on("enterNode", () => {
container.style.cursor = "pointer";
});
renderer.on("leaveNode", () => {
container.style.cursor = "default";
});
We also faced some trouble in distinguishing a click and an drag (since we allow to drag nodes here and there in the graph), so we fixed it in this issue.
Generating graph data
We have XWiki's Solr service to fetch us the information. But how did we convert the information that Sigmajs and Graphology understands? We created the following logic for generating the graph data. It also considers important things like some documents cannot have links.
tempData = response;
const nodes = tempData.map(function(obj) {
return {
key: obj.reference,
attributes: {
label: obj.title_,
color: themeColors.nodeColor,
size: 8,
pageURL: new XWiki.Document(XWiki.Model.resolve(obj.reference)).getURL()
}
};
});
const edges = [];
tempData.forEach(function(obj) {
if (obj.links && obj.links.length > 0) {
obj.links.forEach(function(link) {
var target = link.replace(/^entity:/, '');
var isValid = tempData.some(function(item) {
return item.reference === target;
});
if (isValid) {
edges.push({
key: edgeKeyCounter.toString(),
target: target,
source: obj.reference
});
edgeKeyCounter++;
}
});
}
});
const graphData = {
nodes: nodes,
edges: edges
};
});
This way the finally constructed graph data is a JSON object containing the nodes and link information in attributes.
Solr queries for the panel
How did we collect the currently opened document's data? The basic approach was to query the currently opened document first (first solr query) and then query the links after also querying the previous query. You can have a look at this code to get the idea:
const solrServiceURL = new XWiki.Document('SuggestSolrService', 'XWiki').getURL('get');
$.post(solrServiceURL, {
outputSyntax: 'plain',
nb: 1000,
media: 'json',
query: [
'q=reference:' + documentQuery,
'fq=type:DOCUMENT',
'fl=title_, reference, links, wiki, spaces, name'
].join('\n'),
input: " "
}, function(firstResult) {
if (firstResult.length == 1) {
let extraDocuments = "";
if (firstResult[0].links) {
extraDocuments = ' OR ' + firstResult[0].links.map(link => 'reference:' + escapeQueryChars(link.replace(/^entity:/, ''))).join(' OR ');
}
$.post(solrServiceURL, {
outputSyntax: 'plain',
nb: 1000,
media: 'json',
query: [
'q=reference:' + documentQuery + ' OR links:' + linkQuery + extraDocuments,
'fq=type:DOCUMENT',
'fl=title_, reference, links, wiki, spaces, name'
].join('\n'),
input: " "
...
}
});
});
Integration of Solr Facets
One of the most complicated tasks of the project was to integrate solr facets inside it. Solr facets allows extra filteration of the graph data. For this, wee put the entire HTML code for the visualization inside the #macro (displaySearchResults) macro like this:
{{html}}
<div id="top-bar">
<div id="search">
<i class="fa fa-search"></i>
<input
type="search"
id="search-input"
list="suggestions"
placeholder="Find pages in graph…"
title="Find pages from the visualized graph">
</input>
<datalist id="suggestions"></datalist>
</div>
<div title="Displays information about graph" id="graph-info">
<span id="node-count"></span>
<span id="edge-count"></span>
</div>
</div>
<div id="sigma-container" data-results="$escapetool.xml($jsontool.serialize($searchResponse.results))">
<div class="buttonwrapper" id="graph-buttons">
<button class="icon-button" title="Zoom In" id="zoom-in">$services.icon.renderHTML('search-plus')</button>
<button class="icon-button" title="Zoom Out" id="zoom-out">$services.icon.renderHTML('search-minus')</button>
<button class="icon-button" title="Increase Graph Iterations: If graph is not layouted properly & need more iterations" id="iteration-button">$services.icon.renderHTML('refresh')</button>
<button class="icon-button" title="Default Zoom" id="zoom-reset">$services.icon.renderHTML('world')</button>
<button class="icon-button" title="Fullscreen" id="view-fullscreen">$services.icon.renderHTML('arrows')</button>
<button class="icon-button hidden" title="Kill Graph" id="kill-graph-button">$services.icon.renderHTML('delete')</button>
</div>
</div>
{{/html}}
#end
To get only the relevant fields for our visualization we override the queries like:
#set ($discard = $query.bindValue('fl', 'title_, reference, links, wiki, name, spaces'))
#end
To override the default number of results (rows) in the visualization and to have the custom number of rows field input, we can follow this approach (note that an empty search still returns/shows the default 100-node visualization. That is totally intentional and the above 3 lines are responsible for that!)
## Override default no. of rows
#set ($rows = $numbertool.toNumber($request.rows).intValue())
#if ("$!rows" == '')
#set ($rows = 1000)
#end
##
#set($void = $services.progress.startStep('#displaySearchForm'))
{{html clean="false"}}
<form class="search-form row" action="$doc.getURL()" role="search">
<div class="hidden">
<input type="hidden" name="sort" value="$!escapetool.xml($sort)"/>
<input type="hidden" name="sortOrder" value="$!escapetool.xml($sortOrder)"/>
<input type="hidden" name="highlight" value="$highlightEnabled"/>
<input type="hidden" name="facet" value="$facetEnabled"/>
## The parameter used to determine if the request has been redirected with default search filters.
<input type="hidden" name="r" value="$!escapetool.xml($request.r)"/>
#if ("$!request.debug" != '')
<input type="hidden" name="debug" value="$escapetool.xml($request.debug)"/>
#end
## Preserve the current facet values when submitting a new search query.
#foreach ($entry in $request.parameterMap.entrySet())
#if ($entry.key.startsWith('f_') || $entry.key.startsWith('l_'))
#foreach ($value in $entry.value)
<input type="hidden" name="$escapetool.xml($entry.key)" value="$escapetool.xml($value)"/>
#end
#end
#end
</div>
<div class="col-xs-12 col-sm-6">
<div class="input-group">
<input type="search" name="text" class="form-control withTip useTitleAsTip"
title="$services.localization.render('search.page.bar.query.title')" value="$escapetool.xml($text)"/>
<span class="input-group-btn">
<button type="submit" class="btn btn-primary">
$services.icon.renderHTML('search')
<span class="sr-only">$services.localization.render('search.page.bar.submit')</span>
</button>
</span>
</div>
<div>
<label for="rows" style="margin-right: 2%;">No. of pages to visualize:</label>
<input id="rows" type="number" name="rows" title="Number of documents to display in graph"
placeholder="1000" value="$!escapetool.xml($request.rows)"/>
<span>
<button type="submit" id="refresh-button" class="btn btn-primary">
$services.icon.renderHTML('refresh')
</button>
</span>
</div>
</div>
</form>
{{/html}}
#if ($text == '')
#set ($text = "*")
#end
#end
To disable the search highlighting option (because we don't need it in the visualization 
) we can proceed as follows:
#set ($defaultSortOrder = $solrConfig.sortFields.get($type))
#if (!$defaultSortOrder)
#set ($defaultSortOrder = {'score': 'desc'})
#end
#set ($sortOrderSymbol = {
'asc': $services.icon.render('caret-up'),
'desc': $services.icon.render('caret-down')
})
(% class="search-options" %)
* {{translation key="solr.options"/}}
#if($facetEnabled)#extendQueryString($url {'facet': [false]})#else#extendQueryString($url {'facet': [true]})#end
* [[{{translation key="solr.options.facet"/}}>>path:${url}||class="options-item#if($facetEnabled) active#end" title="$services.localization.render('solr.options.facet.title')"]]
(% class="search-results-sort" %)
* {{translation key="solr.sortBy"/}}
#foreach ($entry in $defaultSortOrder.entrySet())
#set ($class = 'sort-item')
#set ($sortOrderIndicator = $NULL)
#set ($targetSortOrder = $entry.value)
#if ($sort == $entry.key)
#set ($class = "$class active")
#set ($sortOrderHint = $services.localization.render("solr.sortOrder.$sortOrder"))
#set ($sortOrderIndicator = "(% class=""sort-item-order"" title=""$sortOrderHint"" %)$sortOrderSymbol.get($sortOrder)(%%)")
#set ($targetSortOrder = "#if ($sortOrder == 'asc')desc#{else}asc#end")
#end
#extendQueryString($url {'sort': [$entry.key], 'sortOrder': [$targetSortOrder]})
* [[{{translation key="solr.sortBy.$entry.key"/}}$!sortOrderIndicator>>path:${url}||class="$class"]]
#end
#end
And at the end, we just have to call #handleSolrSearchRequest and that will take care of the updated data that comes through the solr facets
