Methods to determine and visualise clusters in data graphs
On this put up we’ll determine and visualise completely different clusters of most cancers sorts by analysing illness ontology as a data graph. Particularly we’ll arrange neo4j in a docker container, import the ontology, generate graph clusters and embeddings, earlier than utilizing dimension discount to plot these clusters and derive some insights. Though we’re utilizing `disease_ontology` for example, the identical steps can be utilized to discover any ontology or graph database.
In a graph database, fairly than storing information as rows (like a spreadsheet or relational database) information is saved as nodes and relationships between nodes. For instance within the determine under we see that melanoma and carcinoma are SubCategories Of cell sort most cancers tumour (proven by the SCO relationship). With this type of information we will clearly see that melanoma and carcinoma are associated regardless that this isn’t explicitly said within the information.
Ontologies are a formalised set of ideas and relationships between these ideas. They’re much simpler for computer systems to parse than free textual content and subsequently simpler to extract which means from. Ontologies are extensively utilized in organic sciences and you might discover an ontology you’re all in favour of at https://obofoundry.org/. Right here we’re specializing in the illness ontology which reveals how several types of illnesses relate to one another.
Neo4j is a device for managing, querying and analysing graph databases. To make it simpler to arrange we’ll use a docker container.
docker run
-it - rm
- publish=7474:7474 - publish=7687:7687
- env NEO4J_AUTH=neo4j/123456789
- env NEO4J_PLUGINS='["graph-data-science","apoc","n10s"]'
neo4j:5.17.0
Within the above command the `-publish` flags set ports to let python question the database immediately and allow us to entry it by way of a browser. The `NEO4J_PLUGINS` argument specifies which plugins to put in. Sadly, the home windows docker picture doesn’t appear to have the ability to deal with the set up, so to observe alongside you’ll want to put in neo4j desktop manually. Don’t fear although, the opposite steps ought to all nonetheless be just right for you.
Whereas neo4j is operating you may entry your database by going to http://localhost:7474/ in your browser, or you should use the python driver to attach as under. Be aware that we’re utilizing the port we printed with our docker command above and we’re authenticating with the username and password we additionally outlined above.
URI = "bolt://localhost:7687"
AUTH = ("neo4j", "123456789")
driver = GraphDatabase.driver(URI, auth=AUTH)
driver.verify_connectivity()
Upon getting your neo4j database arrange, it’s time to get some information. The neo4j plug-in n10s is constructed to import and deal with ontologies; you should use it to embed your information into an present ontology or to discover the ontology itself. With the cypher instructions under we first set some configs to make the outcomes cleaner, then we arrange a uniqueness constraint, lastly we really import illness ontology.
CALL n10s.graphconfig.init({ handleVocabUris: "IGNORE" });
CREATE CONSTRAINT n10s_unique_uri FOR (r:Useful resource) REQUIRE r.uri IS UNIQUE;
CALL n10s.onto.import.fetch(http://purl.obolibrary.org/obo/doid.owl, RDF/XML);
To see how this may be completed with the python driver, take a look at the complete code right here https://github.com/DAWells/do_onto/blob/important/import_ontology.py
Now that we’ve imported the ontology you may discover it by opening http://localhost:7474/ in your internet browser. This allows you to discover a bit of of your ontology manually, however we’re within the larger image so lets do some evaluation. Particularly we’ll do Louvain clustering and generate quick random projection embeddings.
Louvain clustering is a clustering algorithm for networks like this. In brief, it identifies units of nodes which can be extra linked to one another than they’re to the broader set of nodes; this set is then outlined as a cluster. When utilized to an ontology it’s a quick solution to determine a set of associated ideas. Quick random projection alternatively produces an embedding for every node, i.e. a numeric vector the place extra comparable nodes have extra comparable vectors. With these instruments we will determine which illnesses are comparable and quantify that similarity.
To generate embeddings and clusters we now have to “challenge” the components of our graph that we’re all in favour of. As a result of ontologies are sometimes very giant, this subsetting is an easy solution to pace up computation and keep away from reminiscence errors. On this instance we’re solely all in favour of cancers and never some other sort of illness. We do that with the cypher question under; we match the node with the label “most cancers” and any node that’s associated to this by a number of SCO or SCO_RESTRICTION relationships. As a result of we wish to embody the relationships between most cancers sorts we now have a second MATCH question that returns the linked most cancers nodes and their relationships.
MATCH (most cancers:Class {label:"most cancers"})<-[:SCO|SCO_RESTRICTION *1..]-(n:Class)
WITH n
MATCH (n)-[:SCO|SCO_RESTRICTION]->(m:Class)
WITH gds.graph.challenge(
"proj", n, m, {}, {undirectedRelationshipTypes: ['*']}
) AS g
RETURN g.graphName AS graph, g.nodeCount AS nodes, g.relationshipCount AS rels
As soon as we now have the projection (which we now have known as “proj”) we will calculate the clusters and embeddings and write them again to the unique graph. Lastly by querying the graph we will get the brand new embeddings and clusters for every most cancers sort which we will export to a csv file.
CALL gds.fastRP.write(
'proj',
{embeddingDimension: 128, randomSeed: 42, writeProperty: 'embedding'}
) YIELD nodePropertiesWrittenCALL gds.louvain.write(
"proj",
{writeProperty: "louvain"}
) YIELD communityCount
MATCH (most cancers:Class {label:"most cancers"})<-[:SCO|SCO_RESTRICTION *0..]-(n)
RETURN DISTINCT
n.label as label,
n.embedding as embedding,
n.louvain as louvain
Let’s take a look at a few of these clusters to see which kind of cancers are grouped collectively. After we’ve loaded the exported information right into a pandas dataframe in python we will examine particular person clusters.
Cluster 2168 is a set of pancreatic cancers.
nodes[nodes.louvain == 2168]["label"].tolist()
#array(['"islet cell tumor"',
# '"non-functioning pancreatic endocrine tumor"',
# '"pancreatic ACTH hormone producing tumor"',
# '"pancreatic somatostatinoma"',
# '"pancreatic vasoactive intestinal peptide producing tumor"',
# '"pancreatic gastrinoma"', '"pancreatic delta cell neoplasm"',
# '"pancreatic endocrine carcinoma"',
# '"pancreatic non-functioning delta cell tumor"'], dtype=object)
Cluster 174 is a bigger group of cancers however principally carcinomas.
nodes[nodes.louvain == 174]["label"]
#array(['"head and neck cancer"', '"glottis carcinoma"',
# '"head and neck carcinoma"', '"squamous cell carcinoma"',
#...
# '"pancreatic squamous cell carcinoma"',
# '"pancreatic adenosquamous carcinoma"',
#...
# '"mixed epithelial/mesenchymal metaplastic breast carcinoma"',
# '"breast mucoepidermoid carcinoma"'], dtype=object)p
These are wise groupings, based mostly on both organ or most cancers sort, and shall be helpful for visualization. The embeddings alternatively are nonetheless too excessive dimensional to be visualised meaningfully. Thankfully, TSNE is a really helpful methodology for dimension discount. Right here, we use TSNE to scale back the embedding from 128 dimensions right down to 2, whereas nonetheless retaining carefully associated nodes shut collectively. We will confirm that this has labored by plotting these two dimensions as a scatter plot and colouring by the Louvain clusters. If these two strategies agree we should always see nodes clustering by color.
from sklearn.manifold import TSNEnodes = pd.read_csv("export.csv")
nodes['louvain'] = pd.Categorical(nodes.louvain)
embedding = nodes.embedding.apply(lambda x: ast.literal_eval(x))
embedding = embedding.tolist()
embedding = pd.DataFrame(embedding)
tsne = TSNE()
X = tsne.fit_transform(embedding)
fig, axes = plt.subplots()
axes.scatter(
X[:,0],
X[:,1],
c = cm.tab20(Normalize()(nodes['louvain'].cat.codes))
)
plt.present()
Which is strictly what we see, comparable varieties of most cancers are grouped collectively and visual as clusters of a single color. Be aware that some nodes of a single color are very far aside, it is because we’re having to reuse some colors as there are 29 clusters and solely 20 colors. This provides us a fantastic overview of the construction of our data graph, however we will additionally add our personal information.
Beneath we plot the frequency of most cancers sort as node dimension and the mortality charge because the opacity (Bray et al 2024). I solely had entry to this information for just a few of the most cancers sorts so I’ve solely plotted these nodes. Beneath we will see that liver most cancers doesn’t have an particularly excessive incidence over all. Nonetheless, incidence charges of liver most cancers are a lot increased than different cancers inside its cluster (proven in purple) like oropharynx, larynx, and nasopharynx.
Right here we now have used the illness ontology to group completely different cancers into clusters which provides us the context to match these illnesses. Hopefully this little challenge has proven you visually discover an ontology and add that info to your personal information.
You may take a look at the complete code for this challenge at https://github.com/DAWells/do_onto.
Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R. L., Soerjomataram, I., & Jemal, A. (2024). World most cancers statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 international locations. CA: a most cancers journal for clinicians, 74(3), 229–263.