GLM-Image: A Breakdown of Zhipu AI’s New Image Generation Model
This text details Zhipu AI’s GLM-Image, a novel approach to image generation that leverages semantic tokens to improve learning, reliability, and efficiency. Here’s a summary of the key aspects:
1. Semantic Tokens as a Core Concept:
* GLM-Image doesn’t directly generate pixels. Rather, it first generates semantic tokens – representations that encode meaning about the image content (text, faces, backgrounds, etc.) rather than just color information.
* Zhipu AI claims this approach leads to faster learning and more reliable results.
2.Two-Stage Pipeline:
* Autoregressive Module (Planning): This module plans the image by generating a sequence of semantic tokens. It creates a low-resolution preview (around 256 tokens) to establish the basic layout. The model struggles with direct high-resolution generation, making this initial planning crucial.
* Diffusion Decoder (Refinement): This module takes the semantic tokens (and optionally reference images) and refines them into a high-resolution image (1024-4096 tokens). It’s responsible for adding detail and visual quality.
3. Text Rendering Improvement:
* A Glyph-by-T5 module is used to process characters individually,substantially improving text clarity,especially for complex scripts like Chinese.
* As the semantic tokens already contain rich information, a separate large text encoder isn’t needed, reducing computational costs and memory usage.
4. Optimization through Reinforcement Learning:
* Separate Fine-tuning: Each module (autoregressive and diffusion) is fine-tuned independently using reinforcement learning.
* Autoregressive Module Feedback: Receives feedback on both aesthetic appeal and content accuracy (prompt alignment and text legibility).
* Diffusion Decoder feedback: Trained to maximize visual quality, focusing on texture correctness and accurate details (like hands, using a specialized evaluation model). This targeted approach avoids compromising different aspects of image quality.
5. Key Benefits Highlighted:
* Improved Text Clarity: Particularly for complex languages.
* Reduced Computational Demands: By leveraging semantic tokens and avoiding a large text encoder.
* Targeted Optimization: Separate fine-tuning allows for focused improvements in specific areas.
* Control over High-Resolution Generation: the two-stage process with initial planning helps manage the complexity of generating detailed images.
In essence, GLM-Image represents a shift from direct pixel generation to a more structured, meaning-based approach, aiming for higher quality, efficiency, and control in image creation.
