JST CREST — 2026–2032
Multiscale Spatiotemporal Virtualization
of Cultivation Environments for Twin Agriculture
Cultivation is an interaction between plants and people. We virtualize the entire cultivation environment to build the foundations of Twin Agriculture — growing crops in real and virtual space at once.
Every task carried out in the field is continuously recorded by drones, robots and farmers' own viewpoints, and mirrored in real time onto a virtual farm — the Twin Agriculture Platform. There, detailed plant structure and growth forecasts, the state of the whole field, and the know-how of farmers are fused, and the next optimal action is fed back to people and robots.
"Smart agriculture" has so far delivered harvesting, spraying, and relatively simple disease detection. Supporting the skilled, everyday work of farming — pruning, training, growth forecasting and yield maximization — requires virtualizing every leaf and branch, and even the intent of experienced farmers. This project takes on that bottleneck directly.
Real space
Virtual space
Plants are among the most challenging subjects for computer vision. Finding fruit is not enough to support skilled tasks such as pruning and training.
Photo: Agne27 / CC BY-SA 3.0Large, densely planted fields require separating individual plants and continuously updating crop and environmental information.
Farm work is not just physical motion; it is an intellectual act grounded in understanding the plant and years of experience.
For example, "which plant in the field, which branch, and with what intention is this person pruning?" — answering this requires virtualizing all three scales and linking them together.
Digital plants, branch by branch
Reconstructing the 3D structure of individual plants down to each branch, leaf and fruit, and extending it to temporal correspondence and growth prediction. Combining procedural modeling, large language models and physics simulation, we turn reconstructions into "virtual plants" that predict growth, mechanics and the outcome of farm work.
Led by Okura group (The University of Osaka)
4D digital twins of entire fields
Integrating RGB, multispectral and LiDAR data from drones, ground robots and environmental sensors to map crop distribution, growth status, environment and their changes into a 4D virtual space — a field-scale digital twin that is updated continuously and responsively.
Led by Guo group (The University of Tokyo)
How farmers perceive, decide and act
We study how farmers perceive their surroundings and act on them, drawing on video and a wide range of other information. By capturing the relationship between the experience and knowledge of skilled farmers and the state of plants and fields, we aim to develop AI technologies that support the next generation of agriculture.
Led by Sato & Shinoda group (The University of Tokyo)
Connecting the three scales
An integrated platform that stores and visualizes data from all three scales, aligned by shared spatio-temporal coordinates and plant IDs. It describes "which farmer did what, to which plant, in which field" in a unified way — the foundation for feeding insight back into the real world.
3D and temporal reconstruction of field-grown plants, 4D mapping of the research farm, and an egocentric dataset of key farm tasks — presented together as a cross-scale virtualization prototype.
Advanced virtualization robust to diverse farms, testbeds extended to commercial farms, and demonstrated effectiveness in real agricultural work.
↗
PRINCIPAL INVESTIGATOR
Associate Professor, SANKEN (Institute of Scientific and Industrial Research), The University of Osaka
Project lead / Plant-level / Platform Website ↗| Yuta Nakashima | Professor, SANKEN, The University of Osaka |
| Weng Ian Chan | Specially Appointed Researcher, SANKEN, The University of Osaka |
| Yosuke Toda | CEO, Phytometrics Inc. |
| Siyao Chen | Assistant Professor, Graduate School of Agricultural and Life Sciences, The University of Tokyo |
| Mashiro Okada | Project Researcher, Graduate School of Agricultural and Life Sciences, The University of Tokyo |
| Risa Shinoda | Assistant Professor, Institute of Industrial Science, The University of Tokyo |
| Masashi Hatano | Project Researcher, Institute of Industrial Science, The University of Tokyo |
