PaddlePaddle/PaddleMIX v2.1.0

v2.1.0

Repository: PaddlePaddle/PaddleMIX

Tag: v2.1.0

Published: 2024-10-22T12:42:55Z

Prerelease: no

Release notes:

更新内容

• 发布自研多模数据能力标签模型PP-InsCapTagger；可用于数据的分析和过滤，试验案例表明在保持模型效果的条件下可减少50%的数据量，大幅提高训练效率。

• 新增Qwen2-VL、InternVL2、Stable Diffusion 3 (SD3)等前沿模型。

• 多模态大模型InternVL2、LLaVA、SD3、SDXL适配昇腾910B，提供国产计算芯片上的训推能力。

What's Changed

• 【pir 】modify dy2static Sd and 3. Grounding DINO model by @xiaoguoguo626807 in https://github.com/PaddlePaddle/PaddleMIX/pull/689
• fix llava pretrain config by @pkhk-1 in https://github.com/PaddlePaddle/PaddleMIX/pull/685
• Re-network the DIT, fix some parameters, and simplify the model networking code by @chang-wenbin in https://github.com/PaddlePaddle/PaddleMIX/pull/632
• update DIT doc by @chang-wenbin in https://github.com/PaddlePaddle/PaddleMIX/pull/693
• [NPU] Add llava npu doc by @Birdylx in https://github.com/PaddlePaddle/PaddleMIX/pull/694
• sd3推理优化——避免同步 by @chang-wenbin in https://github.com/PaddlePaddle/PaddleMIX/pull/695
• 减少重复拷贝，修复BUG by @chang-wenbin in https://github.com/PaddlePaddle/PaddleMIX/pull/699
• Add Qwen2-VL infer codes by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/698
• [doc] Update requirements by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/703
• Llava bug by @LokeZhou in https://github.com/PaddlePaddle/PaddleMIX/pull/704
• Fix is inference mode by @zhoutianzi666 in https://github.com/PaddlePaddle/PaddleMIX/pull/711
• update readme by @lyuwenyu in https://github.com/PaddlePaddle/PaddleMIX/pull/705
• update opensora video save method by @westfish in https://github.com/PaddlePaddle/PaddleMIX/pull/712
• Limit the installed version of paddlenlp and fix bugs of llava-next. by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/716
• SD3 transformer部分的优化 by @zhoutianzi666 in https://github.com/PaddlePaddle/PaddleMIX/pull/713
• [wip] add mix scheme by @lyuwenyu in https://github.com/PaddlePaddle/PaddleMIX/pull/664
• [NPU] InternVL2 supports npu training by @Birdylx in https://github.com/PaddlePaddle/PaddleMIX/pull/714
• Add SD3 DreamBooth by @westfish in https://github.com/PaddlePaddle/PaddleMIX/pull/686
• remove phi3 in internvl2 and refine format by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/715
• add flash_atten for qw2vl by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/723
• [NPU] sdxl support NPU training by @wangna11BD in https://github.com/PaddlePaddle/PaddleMIX/pull/719
• [NPU] sdxl lora support NPU training by @warrentdrew in https://github.com/PaddlePaddle/PaddleMIX/pull/718
• Adapt fa for npu by @LielinJiang in https://github.com/PaddlePaddle/PaddleMIX/pull/706
• [NPU] fix readme doc for SDXL LoRA training by @warrentdrew in https://github.com/PaddlePaddle/PaddleMIX/pull/724
• [npu]sd3 dreambooth adapt for npu by @LielinJiang in https://github.com/PaddlePaddle/PaddleMIX/pull/726
• add pp-inscaptagger by @pkhk-1 in https://github.com/PaddlePaddle/PaddleMIX/pull/727
• ADD SD3 batch_parallel by @chang-wenbin in https://github.com/PaddlePaddle/PaddleMIX/pull/731
• support auto parallel in dit and largedit by @jeff41404 in https://github.com/PaddlePaddle/PaddleMIX/pull/551
• add env_run.sh and correct packages version by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/733
• [NPU] Fix typo by @Birdylx in https://github.com/PaddlePaddle/PaddleMIX/pull/696
• paddlemix v2.1 readme by @lyuwenyu in https://github.com/PaddlePaddle/PaddleMIX/pull/734
• 修复paddlenlp develop版本适配错误_10-11 by @Xiaobin-Lu in https://github.com/PaddlePaddle/PaddleMIX/pull/735
• 修复qwen2vl视频图像预处理 by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/737
• [wip] update v2.1 readme by @lyuwenyu in https://github.com/PaddlePaddle/PaddleMIX/pull/736
• fix internvl2 minimonkey dataset docs by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/741
• fix tests of evaclip and internvl2 by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/746
• image2text_generation rm use_fast by @LokeZhou in https://github.com/PaddlePaddle/PaddleMIX/pull/744
• fix readme for llava_next_interleave by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/748
• support Qwen2-VL sft training by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/739
• fix dit training by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/752
• fix tests by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/753
• remove use_fast in AutoTokenizer by @warrentdrew in https://github.com/PaddlePaddle/PaddleMIX/pull/747
• fix dit weights convert to ppdiffusers by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/759
• [PPDiffusers]fix bugs and release 0.29.0 by @westfish in https://github.com/PaddlePaddle/PaddleMIX/pull/742
• autolabel fix nltk download by @LokeZhou in https://github.com/PaddlePaddle/PaddleMIX/pull/763
• [NPU] fix npu llava infer by @Birdylx in https://github.com/PaddlePaddle/PaddleMIX/pull/757
• Add npu model list by @nepeplwu in https://github.com/PaddlePaddle/PaddleMIX/pull/758
• Fix docs of by @nemonameless in https://github.com/PaddlePaddle/PaddleMIX/pull/767
• merge upstream readme by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/766
• correct huggingface_hub version by @luyao-cv in https://github.com/PaddlePaddle/PaddleMIX/pull/771
• [NPU] Refine doc by @Birdylx in https://github.com/PaddlePaddle/PaddleMIX/pull/774

New Contributors

• @xiaoguoguo626807 made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/689
• @chang-wenbin made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/632
• @wangna11BD made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/719
• @LielinJiang made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/706
• @jeff41404 made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/551
• @Xiaobin-Lu made their first contribution in https://github.com/PaddlePaddle/PaddleMIX/pull/735
• @nepeplwu made their first contribution in…