Mercurial > hg > drupalISMI

annotate importFromOpenMind/importer/ismi2neo4j.py @ 28:a9bfd49355f8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
updated config for ismi-dev.
author casties
date Wed, 18 Nov 2015 15:22:05 +0100
parents ca1e02a2a9c4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
1 import urllib.request
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
2 import json
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
3 from neo4jrestclient.client import GraphDatabase, Node
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
4
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
5 ## configure behaviour
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
6
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
7 # add inverse relations as "<relation"
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
8 add_inverse_relations = True
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
9
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
10 # add relations to these objects as attributes with the relations name
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
11 contract_relations_into_attributes = ['PLACE', 'ALIAS']
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
12
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
13 # label added to all nodes
28
a9bfd49355f8 updated config for ismi-dev.
casties
parents: 19
diff changeset
14 project_label = '_ismi'
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
15
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
16 # OpenMind base URL
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
17 baseURL="http://localhost:18080/ismi-richfaces/jsonInterface?"
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
18
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
19 # neo4j base URL
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
20 neo4jBaseURL = "http://localhost:7474/db/data/"
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
21
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
22
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
23 entsURL=baseURL+"method=get_ents&oc=%s"
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
24
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
25 entsByIdURL = baseURL+"method=get_ents&include_content=True&ids=%s"
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
26
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
27 entURL=baseURL+"method=get_ent&id=%s&include_content=True"
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
28
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
29
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
30 def readJSON(url):
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
31 #print("JSON loading %s"%url)
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
32 wsh=urllib.request.urlopen(url)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
33 txt = wsh.read()
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
34 return json.loads(txt.decode("utf-8"))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
35
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
36 defs_json = readJSON(baseURL+"method=get_defs")
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
37
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
38 # current list of all definitions
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
39 ismi_defs = [atts['ov'] for atts in defs_json['defs']]
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
40
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
41 #ismi_types=["PERSON","WITNESS","CODEX","PLACE","COLLECTION","REPOSITORY"]
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
42
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
43
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
44 gdb = GraphDatabase(neo4jBaseURL, username="neo4j", password="neo5j")
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
45
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
46 n4j_nodes = {}
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
47 ismi_relations = {}
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
48 n4j_relations = {}
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
49
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
50 ent_exclude_attrs = [
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
51 'lw',
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
52 'node_type',
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
53 'nov'
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
54 ]
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
55
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
56 def fixName(name, is_src_rel=False, is_tar_rel=False, att_from_rel=False):
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
57 # these are too embarrassing...
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
58 if 'FLORUIT' in name:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
59 name = name.replace('FLORUIT', 'FLOURISH')
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
60
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
61 elif 'floruit' in name:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
62 name = name.replace('floruit', 'flourish')
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
63
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
64 if is_src_rel:
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
65 #name = name + '>'
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
66 pass
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
67
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
68 if is_tar_rel:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
69 name = '<' + name
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
70
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
71 if att_from_rel:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
72 # clean up relations as attribute names
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
73 name = name.replace('is_', '')
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
74 name = name.replace('has_', '')
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
75 name = name.replace('was_', '')
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
76 name = name.replace('_of', '')
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
77
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
78 return name
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
79
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
80
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
81 def getNode(ismi_id=None):
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
82 if ismi_id is not None:
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
83 res = gdb.query("match (n {ismi_id: %s}) return n"%ismi_id, returns=(Node))
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
84 if len(res) > 0:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
85 return res[0]
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
86
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
87 return None
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
88
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
89
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
90 def nodeFromEnt(ent, etype):
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
91 """Create a Neo4J node from the given JSON entity.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
92
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
93 Creates the node in gdb and returns the node.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
94 """
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
95 attrs = {}
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
96 # go through all attributes
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
97 for att in ent['atts']:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
98 ct = att.get('content_type', None)
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
99 if ct is None or ct.lower() in ['text', 'arabic', 'bool', 'boolean', 'url', 'language']:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
100 # normal text attribute (assume no content_type is text too...)
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
101 key = att['name']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
102 val = att['ov']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
103
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
104 if key in ent_exclude_attrs:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
105 # exclude attribute
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
106 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
107
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
108 # keep attribute
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
109 attrs[key] = val
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
110
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
111 elif ct == 'num':
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
112 # number attribute
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
113 key = att['name']
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
114 val = att['ov']
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
115
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
116 if key in ent_exclude_attrs:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
117 # exclude attribute
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
118 continue
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
119
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
120 # keep attribute, assume num is int
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
121 attrs[key] = int(val)
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
122
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
123 elif ct == 'date':
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
124 # date attribute
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
125 key = att['name']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
126 val = att['ov']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
127 #print("don't know what to do with date: %s=%s"%(key,val))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
128
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
129 elif ct == 'old':
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
130 # ignore attribute
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
131 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
132
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
133 else:
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
134 print("WARN: attribute with unknown content_type: %s"%repr(att))
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
135 # ignore other content types
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
136 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
137
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
138 # process base attributes
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
139 oc = ent['oc']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
140 if oc != etype:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
141 print("ERROR: entity type doesn't match!")
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
142 return null
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
143
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
144 attrs['type'] = fixName(oc)
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
145
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
146 ismi_id = ent['id']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
147 # rename id to ismi_id
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
148 attrs['ismi_id'] = ismi_id
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
149
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
150 ov = ent.get('ov', None)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
151 if ov is not None:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
152 # save ov as label
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
153 attrs['label'] = ov
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
154
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
155 # create node with attributes
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
156 node = gdb.nodes.create(**attrs)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
157 # add labels
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
158 node.labels.add([project_label, fixName(etype)])
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
159 return node
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
160
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
161
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
162 def relsFromEnt(ent, relations):
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
163 """Extract all relations from JSON entity.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
164
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
165 Adds JSON to dict relations under relation's id.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
166 """
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
167 # go through src_rels and tar_rels
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
168 rels = ent.get('src_rels', []) + ent.get('tar_rels', [])
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
169 for rel in rels:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
170 rel_id = rel['id']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
171 if rel_id in relations:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
172 old_rel = relations[rel_id]
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
173 if rel != old_rel:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
174 print("ERROR: relation is different: %s != %s"%(repr(rel), repr(old_rel)))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
175 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
176
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
177 relations[rel_id] = rel
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
178
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
179 return relations
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
180
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
181
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
182 def n4jrelationsFromRels(rels, nodes):
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
183 """Create relations in Neo4J.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
184
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
185 Args:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
186 rels: dict of JSON relations
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
187 nodes: dict of existing Neo4J nodes
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
188 Returns:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
189 dict of Neo4J relations
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
190 """
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
191 # go through all rels
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
192 print("importing %s relations"%len(rels))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
193 cnt = 0
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
194 for rel in rels.values():
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
195 cnt += 1
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
196 if cnt % 100 == 0:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
197 print(" %s relations"%cnt)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
198
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
199 rel_id = rel['id']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
200 rel_name = rel['name']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
201 src_id = rel['src_id']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
202 tar_id = rel['tar_id']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
203 src = nodes.get(src_id, None)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
204 if src is None:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
205 print("ERROR: relation %s src node %s missing!"%(rel_id,src_id))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
206 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
207
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
208 tar = nodes.get(tar_id, None)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
209 if tar is None:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
210 print("ERROR: relation %s tar node %s missing!"%(rel_id,tar_id))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
211 continue
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
212
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
213 if contract_relations_into_attributes:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
214 # contract source relations
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
215 tar_type = rel['tar_oc']
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
216 if tar_type in contract_relations_into_attributes:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
217 att_name = fixName(rel_name, att_from_rel=True)
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
218 # TODO: clean up attribute names
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
219 while src.get(att_name, None) is not None:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
220 # attribute exists
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
221 if att_name[-1].isnumeric():
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
222 # increment last digit
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
223 att_name = att_name[:-1] + str(int(att_name[-1]) + 1)
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
224 else:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
225 att_name += '2'
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
226
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
227 # add target node's label as attribute
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
228 #print("contracting tar to attribute %s on id=%s"%(att_name, src_id))
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
229 src.set(att_name, tar.get('label'))
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
230
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
231 # contract target relations
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
232 src_type = rel['src_oc']
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
233 if src_type in contract_relations_into_attributes:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
234 att_name = fixName(rel_name, att_from_rel=True)
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
235 # TODO: clean up attribute names
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
236 while tar.get(att_name, None) is not None:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
237 # attribute exists
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
238 if att_name[-1].isnumeric():
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
239 # increment last digit
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
240 att_name = att_name[:-1] + str(int(att_name[-1]) + 1)
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
241 else:
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
242 att_name += '2'
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
243
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
244 # add target node's label as attribute
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
245 #print("contracting src to attribute %s on id=%s"%(att_name, tar_id))
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
246 tar.set(att_name, src.get('label'))
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
247
17
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
248 if add_inverse_relations:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
249 n4j_rel = [gdb.relationships.create(src, fixName(rel_name, is_src_rel=True), tar),
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
250 gdb.relationships.create(tar, fixName(rel_name, is_tar_rel=True), src)]
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
251
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
252 else:
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
253 n4j_rel = gdb.relationships.create(src, fixName(rel_name), tar)
4dfd832e9cd9 added automatic creation of inverse relations.
casties
parents: 16
diff changeset
254
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
255 n4j_relations[rel_id] = n4j_rel
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
256
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
257 return n4j_relations
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
258
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
259
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
260 def importEnts(etype):
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
261 """Import all entities of the given type.
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
262 """
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
263 # read json for all entities of given type
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
264 json = readJSON(entsURL%etype)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
265 ents = json['ents']
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
266 print("importing %s %ss"%(len(ents),etype))
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
267 size = 100
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
268 batches = [ents[pos:pos + size] for pos in range(0, len(ents), size)]
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
269 cnt = 0
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
270 for batch in batches:
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
271 cnt += size
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
272 if cnt % 100 == 0:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
273 print(" %s %ss"%(cnt, etype))
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
274
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
275 # extract list of ismi ids
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
276 ismi_ids = [str(ent['id']) for ent in batch]
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
277
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
278 # fetch full data for list of entities
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
279 ent_json = readJSON(entsByIdURL%','.join(ismi_ids))
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
280 ents_data = ent_json['ents']
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
281
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
282 # iterate through results batch
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
283 for ent_data in ents_data:
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
284 ismi_id = ent_data['id']
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
285 if ismi_id in n4j_nodes:
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
286 print("ERROR: entity with id=%s exists!"%ismi_id)
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
287 return
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
288
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
289 # create neo4j node
18
0827156df210 added contraction of relations into attributes.
casties
parents: 17
diff changeset
290 node = nodeFromEnt(ent_data, etype)
19
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
291
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
292 # save node reference
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
293 n4j_nodes[ismi_id] = node
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
294
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
295 # extract relations
ca1e02a2a9c4 unfilteredIsmi: openmind to json exporter like filterISMI.
casties
parents: 18
diff changeset
296 relsFromEnt(ent_data, ismi_relations)
16
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
297
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
298 #if cnt >= 100:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
299 # return
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
300
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
301
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
302 # In[119]:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
303
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
304 def importAllEnts(etypes):
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
305
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
306 for etype in etypes:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
307 importEnts(etype)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
308
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
309 n4jrelationsFromRels(ismi_relations, n4j_nodes)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
310
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
311
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
312 # In[120]:
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
313
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
314 #importAllEnts(ismi_types)
de0a06eef13b new neo4j importer for network visualisation frontend.
casties
parents:
diff changeset
315 importAllEnts(ismi_defs)