Introducing Shape Tokens, a continuous, compact, and easy-to-integrate 3D representation in machine learning models. Shape Tokens act as shape vectors, representing shape information within a 3D flow simulation model. This flow simulation model is trained to estimate probability density functions corresponding to delta functions embedded in 3D shape space. By combining Shape Tokens with various machine learning models, we can generate new shapes, convert images to 3D, match 3D shapes to text and graphics, and assign shapes directly to flexible, user-defined decisions. Additionally, Shape Tokens enable systematic analysis of geometric properties, including normal, density, and transition fields. In all tasks and tests, the use of Shape Tokens shows strong performance compared to existing frameworks.
Figure 1: Our representation of Shape Tokens can be easily used as input/output to machine learning models in various applications, including one-to-3D image (left), neural rendering of standard maps (top right) and 3D-CLIP alignment (bottom right). The resulting models obtain robust performance compared to the baselines for each function.Video 1: The video shows one of our image in 3D point cloud effects. Images are objects that can be seen in the Objaverse test set. Each video first shows the input image, then the generated point cloud. [Credits]Video 2: From the same abstract image, we generate multiple point clouds independently. [Credits]
Figure 2: Overview of our architecture. (Left) We model the 3D shape as a probability density function centered on the surface, creating a delta function in 3D. (Right) Our tokenizer uses cross-sectional attention to integrate information about the point cloud derived from the shape in ST. Velocity estimation uses only cross-correlation and MLP to preserve independence between points.
Figure 3: Reconstruction, density, and normalization of invisible point clouds in the GSO dataset. For each row, we are given a point cloud containing 16,384 points (xyz only), we calculate the ST and the iid sample of the result p(x|s) to get 262,144 points. Different columns give input and sample point clouds from different viewpoints. Shown with a label in parentheses, we color the input points according to their xyz coordinates and the sample points according to their original uvw noise coordinates and their estimated normality (last two columns). Note that we do not generally provide such input to the shape token.Video 4: We calculate Shape Tokens from input point clouds (16,384 points) from Google Scanned Objects. Then we sample 16x more points (262,144 points). The video shows the uvw-to-xyz trajectory of the sample flow matching process, that is, the ODE trajectory. We color the points by their starting position in the noise space (uvw). [Credits]
Figure 4: The integration path of the ODE defines the map from xyz (data) to uvw (noise).Video 3: The video shows the latest one-to-3D image modes in Google Scanned Objects, which are not visible in all modes.
From left to right:
Installation image
Spatter-image (CVPR 2024): trained on Objaverse
Point-e (2022): trained on several million 3D identity meshes.
Make-a-shape (ICML 2024): trained on 18 datasets, including Objaverse
Ours: trained in Objaverse
Note that this video is not intended to compare individual methods — these models differ in their training data (e.g., Point-e was trained for the identity of 3D meshes) and methods (e.g., Splatter image is not a generative model, our method. takes a model of a known camera). We provide viewer reference results. [Credits]
Video 5: The page shows the results of neural rendering on invisible clouds. From the Shape Tokens, we use a neural network to independently estimate the intersection point of each ray and its normal point.
