ConceptMiner -Creativity Support System, Integrating qualitative and quantitative data to create a foundation for collaboration between humans and AI

Mindware Research Institute is currently developing a SaaS called ConceptMiner. This system obtains embeddings from text chunks describing entities collected according to arbitrary topics, models them using its own fuzzy growth batch neural gas + MST (minimum spanning tree), and places the entities in a conceptual space.

The basic idea behind this system is that there are certain rules for compiling qualitative research reports, and we noticed that there are similarities between these rules and machine learning algorithms. Traditional qualitative data analysis methods include the KJ method and the Grounded Theory Approach (GTA). These methods follow common procedures:

1. Classify fragments of collected information by similarity,
2. Extract common characteristics within each group.
3. Explain the relationships between each element (group, feature).

Even when researchers do not limit themselves to specific methods such as the KJ method or GTA method, reports are essentially created following these procedures. In particular, the KJ method involves arranging pieces of paper with fragments of information written on them on a two-dimensional board, which is somewhat reminiscent of self-organising maps (SOM).

In the research industry, there are traditionally qualitative research (document research and interview research) and quantitative research (questionnaires, etc.). Qualitative research is flexible, and although researchers conduct research according to a set plan, they can try out various perspectives as the research progresses. On the other hand, it is difficult to ensure the comprehensiveness and representativeness of the data. Quantitative surveys collect data in accordance with a comprehensive plan, and the collected data can be analysed using mathematical methods such as multivariate analysis. On the other hand, inflexible and often observable data can sometimes miss important perspectives.

From the perspective of multivariate analysis, 1. above corresponds to cluster analysis, 2. to profile analysis (multiple comparison testing), and 3. to graphical modelling. Needless to say, however, in the framework of conventional surveys, qualitative and quantitative data are completely incompatible, and it is impossible to apply quantitative analysis methods to qualitative surveys, and vice versa.

However, now that large language models (LLMs) can be easily used, the barriers that separated qualitative and quantitative data have been removed. In other words, datasets combining text chunks obtained from qualitative research with conventional quantitative data (numerical and categorical attributes) can be analysed using SOM and neural gas.

This is not only revolutionary for the analysis of survey data. SOM and neural gas are not only used as a type of multivariate analysis method, but also have emerged as a potentially useful interface between humans and AI. In the analysis, LLM can be used to interpret dimensions and clusters, but it can also infer concepts corresponding to coordinates in the semantic space by specifying the coordinates in the semantic space. The significance of this is considered to be very important.

AI has made remarkable progress and is now said to have the capabilities of a graduate student. Furthermore, there is a vision that in the future, AI will be able to conduct research and development autonomously, make strategic decisions for companies, and even perform all of an organisation’s work. If that happens, we will be faced with the philosophical question of what humans should do.

The human brain has between 100 billion and 150 billion nerve cells, each of which has approximately 10,000 synapses. In other words, humans perform even more multidimensional information processing than AI, but unfortunately, we cannot understand or explain this process with our conscious minds. Even about AI that operates using artificial neural networks, it is not possible to explain all of its internal processes.

SOM and neural gas may serve as tools for accessing the hyperdimensional space in which AI operates from the low-dimensional space that is comprehensible to human consciousness.