Interactive Link Visualizer Application (Proposal.InteractiveLinkVisualizerApplication)

Note: This design page covers the technical aspects and key decisions behind the application.

The project revolves around creating a link visualization application for XWiki. The main aim of this app is to help users visualize and interact with a network graph that represents the pages in their XWiki instance. It's designed to show the relationships between different pages in a clear and interactive way. Users can explore, analyze, and interact with link connections within XWiki pages using dynamic network graphs, where the nodes represent documents/pages and the edges represent various links between the pages (such as backlinks and included page references). This application provides valuable insights into the structure and connections within users' wikis.

8

== Use Cases ==

Given the application's focus on data and analysis, its primary utility lies in examining the documents within a wiki. The use cases are centered around this objective, outlined below:

13

14

==== Use Case 1: Content Creator ====

15

16

As a content creator or editor, I can employ the link visualization application to scrutinize the relationships and interlinking patterns among various wiki pages. This functionality aids in identifying gaps in links or content, pinpointing redundant information, and highlighting opportunities for generating additional material. This, in turn, contributes to enhancing the overall coherence and structure of the wiki's content.

17

18

==== Use Case 2: Wiki Administrator ====

19

20

For wiki administrators or community managers, the link visualization application serves as a powerful tool to elevate user engagement and facilitate navigation within the wiki. By presenting users with a visual representation of link connections, administrators can effortlessly explore related content, navigate fluidly between pages, and delve into specific topics of interest.

21

22

==== Use Case 3: Information Researcher ====

23

24

As a knowledge seeker or researcher, I can harness the link visualization application to unearth fresh information and delve into correlated subjects within the wiki. Through visual exploration of the network graph, I can uncover less apparent links, follow intriguing paths, and cultivate a more comprehensive understanding of the wiki's content.

25

26

For instance, consider a researcher utilizing the link visualization application within XWiki to delve into sustainable energy. While navigating the network graph visually, they uncover interconnected pages on solar energy, wind power, hydropower, and bioenergy. Additionally, they stumble upon a cluster of nodes dedicated to energy storage technologies, leading them to the exploration of pages discussing battery technologies, energy grid integration, and smart grid systems. By traversing the network graph visually, the researcher uncovers concealed connections, embarks on captivating pathways, and broadens their insights, ultimately unearthing new discoveries within the wiki.

== Data Model ==

* Page: Represents a page or document within the wiki. Each page has a unique identifier and contains content and metadata.

31

* Link: Represents a link between two pages. It includes the source page and the target page, establishing the relationship between them.

32

* Network Graph: Represents the overall network graph structure that visualizes the link relationships between pages. It consists of nodes (representing pages) and edges (representing links).

33

* Node: The spherical points in the graph; represents a node in the network graph, corresponding to a page. It includes information such as the page's unique identifier, title, document reference, wiki, spaces and name.

34

* Edge: The arrow-lines in the graph; represents an edge in the network graph, corresponding to a link between two pages. It includes information such as the source and target page identifiers.

35

* Linked Pages Panel: It shows the visualization only around the currently opened document i.e., shows pages/documents that are linked to the currently opened page, as well as any backlinks associated with the currently opened page.

36

* Solr Facets Filteration: It facilitates targeted exploration of the graph based on specific criteria, enhancing the user's ability to tailor their interaction with the visualization.

37

* Search bars: We have 2 search bars - the first one is the main search ##query## bar where we can even put Solr queries; it is responsible for tinkering with the ##graphData##. The other one is the ##Search in nodes...## bar where we can find for pages within the graph.

38

* Filter input: By default, the first ##1000## pages will be visualized. However, a number input field is given to override the number and visualize however many documents a user wish.

39

* Interactive Buttons: Many interactive buttons like zoom & full-screen controls, iteration increase etc. are given for better a11y.

40

41

When the data model is well defined, we will decide the most appropriate way for the implementation of the data source (this is something that will fetch us the information about the links present in a wiki). The most suitable way is to use the existing XWiki Solr Search API, where we'll use JavaScript to help us return the link information. Once we have the link information, we'll pass the data to a visualization library to generate the graph.

42

43

== Visualization library ==

44

45

We will use [[graphology>>https://graphology.github.io/]] for implementing the data structures required to represent various graph types in memory: directed, undirected, mixed, multi, etc., and [[sigmajs>>https://sigmajs.org]] for rendering and interacting with network graphs in the browser. Since XWiki can contain thousands of documents, Sigmajs seems to be perfect for it performance-wise.

46

47

==== Why not D3.js? ====

48

49

Sigma.js renders graphs using WebGL. It allows drawing larger graphs faster than with Canvas or SVG based solutions.

=== Graph layout ===

Since we cannot put the ##x## and ##y## co-ordinates in every node, we have to use a layout for placing the nodes in the graph container. But we cannot initialize a layout in Graphology without having some initial ##x## and ##y## co-ordinates, so I fixed the issue by initializing some important graph attributes as:

let i = 0;

graph.forEachNode((node) => {

58

graph.setNodeAttribute(node, "x", i++);

59

graph.setNodeAttribute(node, "y", i);

60

i++;

61

});

62

graph.forEachEdge((edge) => {

63

graph.setEdgeAttribute(edge, "size", 5);

});

Now we can use any layout we wish to. We have circular layout, random layout as described [[here>>https://graphology.github.io/standard-library/layout.html]] and we have some force dependent layouts like [[force layout>>https://graphology.github.io/standard-library/layout-force.html]] and [[force-atlas-2>>https://graphology.github.io/standard-library/layout-forceatlas2.html]]. To avoid overlapping of graph nodes, we also have a choice of [[noverlap-layout>>https://graphology.github.io/standard-library/layout-noverlap.html]].

68

Now, we are interested in the ForceAtlas2 layout's [[webworker variant>>https://graphology.github.io/standard-library/layout-forceatlas2.html#webworker]] (with the ###.inferSettings## to be precise) because that's the one with the optimal settings for tuning the layout. We didn't use ##noverlap## layout for example, because it doesn't utilize the complete space of the graph container.

69

70

71

import forceAtlas2 from 'graphology-layout-forceatlas2';

72

const sensibleSettings = forceAtlas2.inferSettings(graph);

73

const positions = forceAtlas2(graph, {

74

iterations: 50,

75

settings: sensibleSettings

76

});

77

const sensibleSettings = forceAtlas2.inferSettings(500);

78

79

80

== Graph custom settings ==

81

82

We need to have some settings specific to our needs in XWiki. Node color and size are some of them. Before the size of the nodes, we are concerned about the theme compatibility inside XWiki as XWiki has a lot of [[pre-defined themes>>https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/Skins]]. So for example if we are on the Darkly theme (or dark-mode) then we'd want to have lighter color for the nodes label. We solve issues of this kind through implementing the XWiki theme variables inside our application.

83

84

85

#template('colorThemeInit.vm')

86

var themeColors = $jsontool.serialize({

87

'nodeColor': $theme.linkColor,

88

'labelColor': $theme.textColor,

89

'fadeColor': $theme.highlightColor,

90

'labelContainerColor': $theme.pageContentBackgroundColor

});

Now we can access these theme variables through our API we created ;)

95

96

==== Arrow-head size for labels ====

97

98

We don't have much options for customization in Sigmajs. The arrow-head size is hardcoded for example. To change it's size, we have to tinker with the root class of ##EdgeArrowHeadProgram## like this:

99

100

101

class customEdgeArrowHeadProgram extends EdgeArrowHeadProgram {

102

process(

103

sourceData: NodeDisplayData,

104

targetData: NodeDisplayData,

105

data: EdgeDisplayData,

hidden: boolean,

offset: number

) {

data.size *= 4 || 1; // Increase the arrow-head size 4 times its original value

110

super.process(sourceData, targetData, data, hidden, offset);

}

}

const EdgeArrowProgram = createEdgeCompoundProgram([

115

EdgeClampedProgram,

116

customEdgeArrowHeadProgram

117

]);

118

119

// Then we need to put the settings in the renderer settings, to make sure that we are over-riding the existing method

120

121

const rendererSettings = {

122

...

123

edgeProgramClasses: { arrow: EdgeArrowProgram },

...

};

This way we can use whatever arrow-head size we want. This is really helpful for the panel as the arrows were barely visible in the default Sigmajs settings.

129

130

==== Changing the label's container color ====

131

132

There is a bug is Sigmajs (I have already [[reported it>>https://github.com/jacomyal/sigma.js/issues/1368]]) which is that the label's container has hardcoded white color - which is a problem because there is a setting for ##labelColor## and if someone sets it to a lighter color, then the text won't be visible at all due to label container color being white as well. To fix the issue, we need to override the ##drawHover## method like this:

133

134

135

import drawLabel from "sigma/rendering/canvas/label";

136

137

function customDrawHover(

138

context: CanvasRenderingContext2D,

139

data: PartialButFor<NodeDisplayData, "x" | "y" | "size" | "label" | "color">,

140

settings: Settings

141

): void {

142

const size = settings.labelSize,

143

font = settings.labelFont,

144

weight = "bold";

145

146

context.font = `${weight} ${size}px ${font}`;

147

148

// Then we draw the label background

149

context.fillStyle = "red"; // YOUR FAVORITE COLOR HERE FOR THE LABEL CONTAINER ;)

150

context.shadowOffsetX = 0;

151

context.shadowOffsetY = 0;

152

context.shadowBlur = 6;

153

context.shadowColor = "pink"; // Whatever you wish

const PADDING = 3;

if (typeof data.label === "string") {

158

const textWidth = context.measureText(data.label).width,

159

boxWidth = Math.round(textWidth + 5),

160

boxHeight = Math.round(size + 2 * PADDING),

161

radius = Math.max(data.size, size / 2) + PADDING;

162

163

const angleRadian = Math.asin(boxHeight / 2 / radius);

164

const xDeltaCoord = Math.sqrt(

165

Math.abs(Math.pow(radius, 2) - Math.pow(boxHeight / 2, 2))

);

context.beginPath();

context.moveTo(data.x + xDeltaCoord, data.y + boxHeight / 2);

170

context.lineTo(data.x + radius + boxWidth, data.y + boxHeight / 2);

171

context.lineTo(data.x + radius + boxWidth, data.y - boxHeight / 2);

172

context.lineTo(data.x + xDeltaCoord, data.y - boxHeight / 2);

173

context.arc(data.x, data.y, radius, angleRadian, -angleRadian);

context.closePath();

context.fill();

} else {

context.beginPath();

context.arc(data.x, data.y, data.size + PADDING, 0, Math.PI * 2);

context.closePath();

context.fill();

}

context.shadowOffsetX = 0;

184

context.shadowOffsetY = 0;

185

context.shadowBlur = 0;

186

187

// And finally we draw the label

188

drawLabel(context, data, settings);

189

}

190

191

// Then put this in the renderer settings

192

193

const rendererSettings = {

194

labelColor: { color: "blue" }, // YOUR SECOND FAV COLOR FOR THE LABEL

195

zIndex: true,

196

hoverRenderer: customDrawHover // IMPORTANT

197

};

198

199

const renderer = new Sigma(graph, container, rendererSettings);

200

201

202

==== Click & drag events and cursor style change ====

203

204

Since we implemented the functionality that allows us to open a page's corresponding URL by clicking on it, we also need to change the cursor style to indicate that node is basically a link:

205

206

207

// On click, we open the corresponding page URL

208

renderer.on("clickNode", ({ node }) => {

209

if (!graph.getNodeAttribute(node, "hidden") && allowClick) {

210

window.open(graph.getNodeAttribute(node, "pageURL"), "_self");

}

});

renderer.on("enterNode", () => {

215

container.style.cursor = "pointer";

216

});

217

renderer.on("leaveNode", () => {

218

container.style.cursor = "default";

});

We also faced some trouble in distinguishing a click and an drag (since we allow to drag nodes here and there in the graph), so we fixed it [[in this issue>>https://github.com/jacomyal/sigma.js/issues/1373]].

223

224

==== Generating graph data ====

225

226

We have XWiki's Solr service to fetch us the information. But how did we convert the information that Sigmajs and Graphology understands? We created the following logic for generating the graph data. It also considers important things like some documents cannot have links.

function(response) {

tempData = response;

const nodes = tempData.map(function(obj) {

return {

key: obj.reference,

attributes: {

label: obj.title_,

color: themeColors.nodeColor,

237

size: 8,

238

pageURL: new XWiki.Document(XWiki.Model.resolve(obj.reference)).getURL()

}

};

});

const edges = [];

tempData.forEach(function(obj) {

244

if (obj.links && obj.links.length > 0) {

245

obj.links.forEach(function(link) {

246

var target = link.replace(/^entity:/, '');

247

var isValid = tempData.some(function(item) {

248

return item.reference === target;

});

if (isValid) {

edges.push({

key: edgeKeyCounter.toString(),

253

target: target,

254

source: obj.reference

});

edgeKeyCounter++;

}

});

}

});

const graphData = {

nodes: nodes,

edges: edges

};

});

This way the finally constructed graph data is a JSON object containing the nodes and link information in attributes.

269

270

==== Solr queries for the panel ====

271

272

How did we collect the currently opened document's data? The basic approach was to query the currently opened document first (first solr query) and then query the links after also querying the previous query. You can have a look at this code to get the idea:

273

274

275

require(['jquery', 'visualisationAPI', 'xwiki-meta'], function($, visualisationAPI, xm) {

276

const solrServiceURL = new XWiki.Document('SuggestSolrService', 'XWiki').getURL('get');

277

$.post(solrServiceURL, {

278

outputSyntax: 'plain',

nb: 1000,

media: 'json',

query: [

'q=reference:' + documentQuery,

283

'fq=type:DOCUMENT',

284

'fl=title_, reference, links, wiki, spaces, name'

285

].join('\n'),

286

input: " "

287

}, function(firstResult) {

288

if (firstResult.length == 1) {

289

let extraDocuments = "";

290

if (firstResult[0].links) {

291

extraDocuments = ' OR ' + firstResult[0].links.map(link => 'reference:' + escapeQueryChars(link.replace(/^entity:/, ''))).join(' OR ');

292

}

293

$.post(solrServiceURL, {

294

outputSyntax: 'plain',

nb: 1000,

media: 'json',

query: [

'q=reference:' + documentQuery + ' OR links:' + linkQuery + extraDocuments,

299

'fq=type:DOCUMENT',

300

'fl=title_, reference, links, wiki, spaces, name'

].join('\n'),

input: " "

...

}

});

});

==== Integration of Solr Facets ====

310

311

One of the most complicated tasks of the project was to integrate solr facets inside it. Solr facets allows extra filteration of the graph data. For this, wee put the entire HTML code for the visualization inside the ###macro (displaySearchResults)## macro like this:

312

313

314

#macro (displaySearchResults)

<input

type="search"

id="search-input"

list="suggestions"

placeholder="Find pages in graph…"

324

title="Find pages from the visualized graph">

</input>

</div>

</div>

</div>

<button class="icon-button" title="Zoom In" id="zoom-in">$services.icon.renderHTML('search-plus')</button>

336

<button class="icon-button" title="Zoom Out" id="zoom-out">$services.icon.renderHTML('search-minus')</button>

337

<button class="icon-button" title="Increase Graph Iterations: If graph is not layouted properly & need more iterations" id="iteration-button">$services.icon.renderHTML('refresh')</button>

338

<button class="icon-button" title="Default Zoom" id="zoom-reset">$services.icon.renderHTML('world')</button>

339

<button class="icon-button" title="Fullscreen" id="view-fullscreen">$services.icon.renderHTML('arrows')</button>

340

<button class="icon-button hidden" title="Kill Graph" id="kill-graph-button">$services.icon.renderHTML('delete')</button>

</div>

</div>

#end

To get only the relevant fields for our visualization we override the queries like:

348

349

350

#macro(setHighlightQuery $query)

351

#set ($discard = $query.bindValue('fl', 'title_, reference, links, wiki, name, spaces'))

#end

To override the default number of results (rows) in the visualization and to have the custom number of rows field input, we can follow this approach (note that an empty search still returns/shows the default 100-node visualization. That is totally intentional and the above 3 lines are responsible for that!)

356

357

358

#macro (displaySearchForm)

359

## Override default no. of rows

360

#set ($rows = $numbertool.toNumber($request.rows).intValue())

#if ("$!rows" == '')

#set ($rows = 1000)

#end

##

#set($void = $services.progress.startStep('#displaySearchForm'))

## The parameter used to determine if the request has been redirected with default search filters.

374

375

#if ("$!request.debug" != '')

376

377

#end

378

## Preserve the current facet values when submitting a new search query.

379

#foreach ($entry in $request.parameterMap.entrySet())

380

#if ($entry.key.startsWith('f_') || $entry.key.startsWith('l_'))

381

#foreach ($value in $entry.value)

#end

#end

#end

</div>

<input type="search" name="text" class="form-control withTip useTitleAsTip"

390

title="$services.localization.render('search.page.bar.query.title')" value="$escapetool.xml($text)"/>

391

392

393

$services.icon.renderHTML('search')

394

<span class="sr-only">$services.localization.render('search.page.bar.submit')</span>

</button>

</span>

</div>

<div>

<label for="rows" style="margin-right: 2%;">No. of pages to visualize:</label>

400

<input id="rows" type="number" name="rows" title="Number of documents to display in graph"

401

placeholder="1000" value="$!escapetool.xml($request.rows)"/>

402

<span>

403

404

$services.icon.renderHTML('refresh')

</button>

</span>

</div>

</div>

</form>

#if ($text == '')

#set ($text = "*")

#end

#end

To disable the search highlighting option (because we don't need it in the visualization :)) we can proceed as follows:

418

419

420

#macro (displaySearchResultsSort)

421

#set ($defaultSortOrder = $solrConfig.sortFields.get($type))

422

#if (!$defaultSortOrder)

423

#set ($defaultSortOrder = {'score': 'desc'})

424

#end

425

#set ($sortOrderSymbol = {

426

'asc': $services.icon.render('caret-up'),

427

'desc': $services.icon.render('caret-down')

428

})

429

(% class="search-options" %)

430

* {{translation key="solr.options"/}}

431

#if($facetEnabled)#extendQueryString($url {'facet': [false]})#else#extendQueryString($url {'facet': [true]})#end

432

* [[{{translation key="solr.options.facet"/}}>>path:${url}||class="options-item#if($facetEnabled) active#end" title="$services.localization.render('solr.options.facet.title')"]]

433

434

(% class="search-results-sort" %)

435

* {{translation key="solr.sortBy"/}}

436

#foreach ($entry in $defaultSortOrder.entrySet())

437

#set ($class = 'sort-item')

438

#set ($sortOrderIndicator = $NULL)

439