ComfyUI — an open-source, node-based graphical interface for working and constructing generative AI workflows for content material creation — printed main updates this previous month, together with as much as 40% efficiency enhancements for NVIDIA RTX GPUs, and help for brand new AI fashions together with Wan 2.2, Qwen-Picture, FLUX.1 Krea [dev] and Hunyuan3D 2.1.
NVIDIA additionally launched NVIDIA TensorRT-optimized variations of widespread diffusion fashions like Secure Diffusion 3.5 and FLUX.1 Kontext as NVIDIA NIM microservices, permitting customers to run these fashions in ComfyUI as much as 3x sooner and with 50% much less VRAM.
Plus, an replace to NVIDIA RTX Remix — a platform that lets modders remaster traditional video games — launched at present, including a complicated path-traced particle system that delivers gorgeous visuals to breathe new life into traditional titles.
ComfyUI v3.57 Will get a Efficiency Enhance With RTX
NVIDIA has collaborated with ComfyUI to spice up AI mannequin efficiency by as much as 40%. To place this in perspective, GPU era upgrades sometimes solely ship a 20-30% efficiency increase.
Builders interested by optimizing efficiency and effectivity of diffusion fashions on their apps can learn extra on how NVIDIA helps speed up these workloads within the developer discussion board.
State-of-the-Artwork AI Fashions, Accelerated by RTX
Unimaginable fashions for AI content material creation have been launched within the final weeks, all of which at the moment are out there in ComfyUI.
Wan 2.2 is a brand new video mannequin that gives unimaginable high quality and management for video era on PCs. It’s the most recent mannequin from Wan AI — a inventive AI platform providing a powerful lineup of AI fashions, together with Textual content to Picture, Textual content to Video, Picture to Video and Speech to Video. GeForce RTX and NVIDIA RTX PRO GPUs are the one GPUs able to working Wan 2.2 14B fashions in ComfyUI with out main delays in output. Try the instance beneath created with the one immediate: “A robotic is cracking an egg, however by chance hits it outdoors the bowl.”
Qwen-Picture is a brand new picture era basis mannequin from Alibaba that achieves important advances in advanced textual content rendering and exact picture enhancing. It excels at rendering advanced textual content, dealing with intricate enhancing and sustaining each semantic and visible accuracy in generated photos. The mannequin runs 7x sooner on a GeForce RTX 5090 vs. Apple M3 Extremely.
Black Forest Labs’ new FLUX.1 Krea [dev] AI mannequin is the open-weight model of Krea 1, providing sturdy efficiency and educated to generate extra sensible, numerous photos that don’t comprise oversaturated textures. Black Forest Labs calls the mannequin “opinionated,” because it gives all kinds of numerous, visually fascinating photos. This mannequin runs 8x sooner on a GeForce RTX 5090 vs. Apple M3 Extremely.
Hunyuan3D 2.1 is a totally open-source, production-ready 3D generative system that transforms enter photos or textual content into high-fidelity 3D belongings enriched with bodily primarily based rendering supplies. Core elements embody a 3.3-billion-parameter mannequin for form era and a 2-billion-parameter mannequin for texture evaluation to rapidly generate extra sensible supplies. All of it runs sooner on Blackwell RTX GPUs.
Get Began With Superior Visible Generative Methods
Visible generative AI is a robust software, however getting began could be tough even for technical consultants and studying to make use of extra superior strategies sometimes takes months.
ComfyUI makes it straightforward to get began with superior workflows by offering templates or preset nodes that obtain a particular process, like holding a personality fixed all through totally different generations, adjusting the sunshine of a picture or loading a fine-tuning. This enables even non-technical artists to simply use superior AI workflows.
These are 10 key strategies to get began with generative AI:
- Information video era by defining the beginning body and finish body: Add a begin and finish body, in addition to how a video clip ought to start and finish. Wan 2.2 can then generate a clean, animated transition, filling the in-between frames to create a coherent animation. It’s very best for making animations, scene shifts or defining poses.
- Edit photos with pure language: Use FLUX.1 [dev] KONTEXT to edit particular textual content sections of a picture.
- Upscale photos or movies: Take picture or video at a decrease decision and improve its decision and element high quality by including sensible, high-frequency particulars.
- Management area composition: Assume extra granular management over picture era by controlling the association and structure of visible parts inside particular areas of a picture.
- Restyle photos: Use FLUX Redux to create totally different variations of photos whereas preserving core visible parts and particulars.
- Faucet image-to-multiview-to-3D fashions: Use a number of photos of an object captured from totally different angles to create a high-fidelity, textured 3D mannequin.
- Remodel sound to video: Create video clips or animations immediately from audio inputs reminiscent of speech, music or environmental sounds.
- Management video trajectory: Mechanically information the movement of objects, digicam or scenes inside movies.
- Edit photos with inpainting: Fill in or alter lacking or undesirable components of a digital picture in a visually seamless, contextually constant approach with surrounding areas.
- Develop the canvas with outpainting: Generate new picture content material to increase the boundaries of an current picture or video footage, add element to cropped sections or full a scene.
Observe ComfyUI on X for updates to inventive templates and workflows.
Develop Your ComfyUI Zone
ComfyUI plug-ins allow customers so as to add generative AI workflows into their current functions. The ComfyUI neighborhood has began constructing plug-ins for a number of the high widespread inventive functions.
The Adobe Photoshop plug-in enhances Photoshop’s native Firefly fashions by permitting customers to run their very own flows and choose specialised fashions for particular duties. Native inference additionally permits limitless generative fill with low latency.
The Blender plug-in — featured within the NVIDIA AI Blueprint for 3D-guided generative AI — permits customers to attach 2D and 3D workflows. Artists can use 3D scenes to regulate picture generations or create textures in ComfyUI and apply them to separate 3D belongings.
The Foundry Nuke plug-in — just like the Blender — permits a connection between 2D and 3D workflows so customers don’t should alt-tab and swap between functions.
The Unreal Engine plug-in permits ComfyUI nodes immediately within the Unreal Engine person interface to rapidly create and refine textures for scenes utilizing generative diffusion fashions. See the instance beneath.
Run Hyper-Optimized Fashions for NVIDIA RTX GPUs in ComfyUI
The easiest way to make use of NVIDIA RTX GPUs is with the TensorRT library — a high-performance deep studying inference engine designed to squeeze most velocity out of the Tensor Cores in NVIDIA RTX GPUs.
NVIDIA has collaborated with the highest AI labs to combine TensorRT of their fashions, reminiscent of Black Forest Labs’ fashions and Stability AI’s fashions. These fashions are additionally out there quantized — a compressed model of the community that makes use of 50-70% much less VRAM and gives as much as 2x sooner inference whereas sustaining related high quality.
TensorRT-optimized fashions could be run immediately in ComfyUI by means of the TensorRT node, which at the moment helps SDXL, SD3 and SD3.5, in addition to FLUX.1-dev and FLUX.1-schnell fashions. The node converts the AI mannequin right into a TensorRT-optimized mannequin, after which generates a TensorRT-optimized engine for the person’s GPU — a map of find out how to execute that mannequin with optimum effectivity for his or her specific {hardware} — offering important speedups.
Quantizing fashions, nonetheless, takes a bit extra work. For customers interested by working quantized and TensorRT-optimized fashions, NVIDIA gives preconfigured recordsdata in a easy container known as a NIM microservice. Customers can use the NIM node in ComfyUI to load these containers and use quantized variations of fashions like FLUX.1-dev, FLUX.1-schnell, FLUX.1 Kontext, SD3.5 Giant and Microsoft TRELLIS.
Remix Replace Provides a Path-Traced Particle System
A brand new RTX Remix replace launched at present by means of the NVIDIA app, including a complicated particle system that permits modders to reinforce conventional hearth and smoke results, in addition to extra fantastical results, like these within the online game Portal.
With RTX Remix, legacy particles from traditional video games could possibly be interpreted as path-traced, enabling them to forged sensible mild to reinforce the looks of many scenes. However in the end, these particles had been nonetheless over 20 years outdated, missing element, aptitude and fluid animations.
RTX Remix’s new particles have bodily correct properties and may work together with a recreation’s lighting and different results. This allows particles to collide, precisely transfer in response to wind and different forces, be mirrored on surfaces, forged shadows and assign their very own shadows.
For a whole breakdown of the brand new particle system, learn the GeForce article.
Every week, the RTX AI Storage weblog collection options community-driven AI improvements and content material for these trying to be taught extra about NVIDIA NIM microservices and AI Blueprints, in addition to constructing AI brokers, inventive workflows, productiveness apps and extra on AI PCs and workstations.
Plug in to NVIDIA AI PC on Fb, Instagram, TikTok and X — and keep knowledgeable by subscribing to the RTX AI PC publication. Be a part of NVIDIA’s Discord server to attach with neighborhood builders and AI fanatics for discussions on what’s potential with RTX AI.
Observe NVIDIA Workstation on LinkedIn and X.
See discover concerning software program product info.
